adriangb commented on issue #19337: URL: https://github.com/apache/datafusion/issues/19337#issuecomment-4951997189
Pretty much, with a couple important twists: 1. If we are writing stats / zone maps into the parquet footer *we are already tracking the min/max for each column*. 2. We never cast the data / change how we write the data. We only ever change *the arrow type we write into the footer*. In other words: the only change in practice is a small decision at the end of writing that changes the metadata written. The point is that if we have to read the column and filter it we can read it as a smaller more compact type in memory because we determined at write time that the values are in that range. Alternatively / in addition to this: we could do this at read time. This may be the safer option. Using file stats at read time we could demote a column to a narrower physical type. One catch to be aware of: we need to make sure this is *correct*. Be careful with nulls / NaNs and float ordering. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
