Anurag-S1ngh opened a new pull request, #4449:
URL: https://github.com/apache/iceberg-python/pull/4449

   <!-- Closes #3848 -->
   # Rationale for this change
   
   Arrow table writes ignored the table sort order and always wrote 
`sort_order_id=None`, so manifests disagreed with the table metadata and 
readers could not prune on sort order.
   
   Sort materialized Arrow tables when every sort field uses an identity 
transform, the column exists at the top level, and all fields agree on null 
placement. Unpartitioned tables are sorted before bin packing and each 
partition is sorted on its own. The table sort order id is written on each data 
file.
   
   Unsupported cases (non-identity transforms, nested or missing columns, mixed 
null placement, streaming `RecordBatchReader` writes) keep writing unsorted 
files and emit a warning.
   
   ## Are these changes tested?
   
   Yes. New tests in `tests/io/test_pyarrow.py` cover the four direction and 
null-placement combinations, the unsupported cases, sorted files per partition, 
and the streaming warning. `tests/table` and the local write tests pass.
   
   ## Are there any user-facing changes?
   
   Yes. Tables with an identity sort order now write physically sorted data 
files that carry the sort order id. Previously written files are unchanged.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to