Rich-T-kid commented on PR #24227: URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5285122363
ideally we surface columns that are physically RLE_DICTIONARY-encoded in the parquet file as Arrow `Dictionary(Int32, Utf8)` arrays rather than decoding them back to plain `Utf8`. To know whether a specific column is RLE_DICTIONARY-encoded you need to read the parquet file footer. For the `infer_schema` path this happens at table registration, but for explicit schemas (`CREATE EXTERNAL TABLE (col VARCHAR)`) and direct `ParquetSource` construction no footer is ever read during planning, so per-column encoding information isn't available for all paths. Downstream physical operators (`FilterExec`, `AggregateExec`) are compiled against the scan's declared output schema during physical planning, before any files are opened. If the scan declares `Utf8` but produces `Dictionary(Int32, Utf8)` at execution time that's a type mismatch. So when the flag is enabled we promote all string/binary columns to dict at planning time, not just the ones that are actually RLE-encoded, because that's the only way to guarantee schema consistency across all parquet scan paths without introducing file I/O into the planning stage. I feel like i'm missing something here. if we could take a peak at the parquets metadata before physical planning and change the schema for all operators from the point forward that would be perfect. Im not sure this is currently possible -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
