Anurag-S1ngh opened a new pull request, #4449: URL: https://github.com/apache/iceberg-python/pull/4449
<!-- Closes #3848 --> # Rationale for this change Arrow table writes ignored the table sort order and always wrote `sort_order_id=None`, so manifests disagreed with the table metadata and readers could not prune on sort order. Sort materialized Arrow tables when every sort field uses an identity transform, the column exists at the top level, and all fields agree on null placement. Unpartitioned tables are sorted before bin packing and each partition is sorted on its own. The table sort order id is written on each data file. Unsupported cases (non-identity transforms, nested or missing columns, mixed null placement, streaming `RecordBatchReader` writes) keep writing unsorted files and emit a warning. ## Are these changes tested? Yes. New tests in `tests/io/test_pyarrow.py` cover the four direction and null-placement combinations, the unsupported cases, sorted files per partition, and the streaming warning. `tests/table` and the local write tests pass. ## Are there any user-facing changes? Yes. Tables with an identity sort order now write physically sorted data files that carry the sort order id. Previously written files are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
