Rich-T-kid opened a new pull request, #26977: URL: https://github.com/apache/datafusion/pull/26977
This is stacked on #26973; the final two commits are the change in this PR. When the table schema uses `Dictionary(Int32, Utf8)` but the physical Parquet schema is `LargeBinary`, the reader currently emits plain values and `DecoderProjection` dictionary-encodes every row. Preserve the physical value type in the reader schema instead, so arrow-rs decodes dictionary keys directly and the projection casts only the distinct dictionary values. The same rule handles Utf8/LargeUtf8 offset-width mismatches. The Q23 profile no longer contains `pack_byte_to_dictionary` or `GenericByteDictionaryBuilder::append_array`; it now enters parquet-rs `DictionaryDecoder` directly. A controlled ClickBench run is pending on #26964. Validated with `cargo test -p datafusion-datasource-parquet --lib`, package clippy with all targets/features, and the repository pre-push checks. Closes #26976. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
