xudong963 opened a new pull request, #24211: URL: https://github.com/apache/datafusion/pull/24211
## Which issue does this PR close? - Closes #24210. ## Rationale for this change When Hive partition columns are not kept in Parquet files, `FileSinkConfig::output_schema()` still describes the sink input while `get_writer_schema(&conf)` removes those partition columns. `ParquetFormat::create_writer_physical_plan` currently converts the input ordering directly to Parquet `sorting_columns`, so its column indices can reference removed columns or the wrong positions in the written schema. Reading that metadata can then panic in `SchemaDescriptor::column` with an out-of-bounds index. The execution ordering and the Parquet footer ordering have different schema domains: `DataSinkExec` must retain the original input ordering, while `ParquetSink` metadata must use indices from the actual writer schema. ## What changes are included in this PR? - Derive Parquet `sorting_columns` from both the input schema and `get_writer_schema(&conf)`. - Omit ordering keys that are removed from the written file, such as Hive partition columns. - Remap retained ordering keys to their writer-schema indices while preserving sort direction and null ordering. - Keep the original `order_requirements` unchanged for `DataSinkExec`. - Avoid writing an empty `sorting_columns` list when every ordering key is removed. - Add unit and end-to-end regression coverage for partitioned Parquet writes. ## Are these changes tested? Yes. The new tests cover the writer-schema remapping directly and verify the resulting Parquet footer after a partitioned write. The following checks were run locally: - `cargo fmt --all --check` - `CARGO_INCREMENTAL=0 cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-datasource-parquet test_lex_ordering_to_sorting_columns_uses_writer_schema` - `cargo test -p datafusion --test parquet_integration test_partitioned_table_remaps_sorting_columns` ## Are there any user-facing changes? There are no public API changes. Newly written partitioned Parquet files now contain valid `sorting_columns` metadata based on their physical file schema. Existing files with invalid metadata are unchanged and must be rewritten separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
