saadtajwar commented on issue #19337: URL: https://github.com/apache/datafusion/issues/19337#issuecomment-4949696070
Hey @adriangb ! Getting back to this now, read through the Discord conversation and the issue description - my current understanding is that: - the existing `PhysicalExprAdapter` - if a DF table schema type and Parquet file physical schema differ - will rewrite pushed-down filters/casts to match the file at read-time - we want, at write time to Parquet file, to advertise narrower types than the table schema if possible (like in your example above) Based on the above, I'm thinking: - Before ArrowWriter starts, compute min/max over the batches to write for each integer column - If a column fits a narrower type, cast it and write with that Arrow schema (Parquet types / embedded Arrow metadata follow as usual) - Keep table schema wide, and let `PhysicalExprAdapter` handle filter pushdown like it already does - Add tests for wide table + narrow file and make sure pushed-down filter runs on the narrow type correctly - Add config field for this on the writer Please let me know your thoughts on that or if there's anything you think I might be missing/misunderstanding! I'm also only thinking about integer types right now, but if there are others that we'd want to do narrowing on too that I might've missed please lmk! Thanks again! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
