rahulsmahadev opened a new pull request, #4423: URL: https://github.com/apache/iceberg-python/pull/4423
# Rationale for this change Follow-up to #4397. PyIceberg can register a partition statistics file but can't compute one, so users have to write the file themselves. This PR adds `table.maintenance.compute_partition_statistics()`. It computes the stats from the snapshot's manifests, writes the Parquet file under `metadata/` and registers it, like Java's `PartitionStatsHandler` plus Spark's `compute_partition_stats` procedure. ```python stats_file = table.maintenance.compute_partition_statistics() table.metadata.partition_statistics # [stats_file] ``` ebyhr proposed something similar in #3471 (closed). Same as Java: - Deleted entries only update `last_updated_at` and `last_updated_snapshot_id`. - NaN values form one partition, -0.0 and 0.0 stay separate, and rows are sorted with NULL first and NaN last. - A removed v1 partition field keeps its transform's output type. - If the snapshot already has a stats file, that file is returned. Not supported yet: - Incremental compute. Java reuses an earlier stats file when it can. - v3 tables (`NotImplementedError`) and tables whose `write.format.default` is not Parquet (`ValueError`). - UUID partition fields (`ArrowNotImplementedError`, same as `inspect.partitions()`, see #4358). - Partitions that only differ in a dropped source column end up in one row. Java keeps them apart. I put this under `table.maintenance` next to `expire_snapshots()`, since that's where Java's docs put it. Happy to move it to `Table`, or make it compute only, if you prefer. This also fixes the `Table.update_partition_statistics()` docstring example from #4397. ## Are these changes tested? Yes, added tests in `tests/table/test_statistics.py` that run on the memory, sql and sql_without_rowcount catalogs. They cover appends, overwrites and deletes, multiple partition specs, partition value types including float NaN and -0.0, delete files, expired snapshots, the file layout and the error cases. They also pass on PyArrow 18.0.0. ## Are there any user-facing changes? Yes, new API `MaintenanceTable.compute_partition_statistics(snapshot_id=None)`, documented under "Table Maintenance" in the API docs. Please add the `changelog` label. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
