rahulsmahadev opened a new pull request, #4423:
URL: https://github.com/apache/iceberg-python/pull/4423

   # Rationale for this change
   
   Follow-up to #4397. PyIceberg can register a partition statistics file but 
can't compute one, so users have to write the file themselves. This PR adds 
`table.maintenance.compute_partition_statistics()`. It computes the stats from 
the snapshot's manifests, writes the Parquet file under `metadata/` and 
registers it, like Java's `PartitionStatsHandler` plus Spark's 
`compute_partition_stats` procedure.
   
   ```python
   stats_file = table.maintenance.compute_partition_statistics()
   table.metadata.partition_statistics  # [stats_file]
   ```
   
   ebyhr proposed something similar in #3471 (closed).
   
   Same as Java:
   - Deleted entries only update `last_updated_at` and 
`last_updated_snapshot_id`.
   - NaN values form one partition, -0.0 and 0.0 stay separate, and rows are 
sorted with NULL first and NaN last.
   - A removed v1 partition field keeps its transform's output type.
   - If the snapshot already has a stats file, that file is returned.
   
   Not supported yet:
   - Incremental compute. Java reuses an earlier stats file when it can.
   - v3 tables (`NotImplementedError`) and tables whose `write.format.default` 
is not Parquet (`ValueError`).
   - UUID partition fields (`ArrowNotImplementedError`, same as 
`inspect.partitions()`, see #4358).
   - Partitions that only differ in a dropped source column end up in one row. 
Java keeps them apart.
   
   I put this under `table.maintenance` next to `expire_snapshots()`, since 
that's where Java's docs put it. Happy to move it to `Table`, or make it 
compute only, if you prefer.
   
   This also fixes the `Table.update_partition_statistics()` docstring example 
from #4397.
   
   ## Are these changes tested?
   
   Yes, added tests in `tests/table/test_statistics.py` that run on the memory, 
sql and sql_without_rowcount catalogs. They cover appends, overwrites and 
deletes, multiple partition specs, partition value types including float NaN 
and -0.0, delete files, expired snapshots, the file layout and the error cases. 
They also pass on PyArrow 18.0.0.
   
   ## Are there any user-facing changes?
   
   Yes, new API 
`MaintenanceTable.compute_partition_statistics(snapshot_id=None)`, documented 
under "Table Maintenance" in the API docs.
   
   Please add the `changelog` label.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to