findepi commented on issue #12720: URL: https://github.com/apache/datafusion/issues/12720#issuecomment-2662322265
The issue example is about plain, dictionary and REE encoded data (also covered by https://github.com/apache/datafusion/discussions/7421). However, we could do more. - For example, for `decimal(1,0)` type we currently use 128 bits per value, where 8 would suffice. Plenty of waste. https://github.com/apache/arrow-rs/issues/6661 Will help a bit, but more can be done. - For a column with `decimal(38,0)` type, we may still use 128 bit per value even once Decimal32 type is added. But what if the column actually contains only numbers 0..9? The runtime representation could be smaller size integer type. - judging on how Snowflake returns data over their Arrow interface, this is likely what they do internally -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
