ShivamSharma43 opened a new pull request, #4364:
URL: https://github.com/apache/iceberg-python/pull/4364

   # Rationale for this change
   
   When a snapshot is removed (`RemoveSnapshotsUpdate`, which 
`expire_snapshots` uses), pyiceberg removes the snapshot's statistics files but 
keeps its partition statistics files. After expiring a snapshot, the table 
metadata still has `partition-statistics` entries pointing at a snapshot that 
is gone.
   
   Iceberg Java removes both in 
`TableMetadata.Builder.rewriteSnapshotsInternal`:
   
   ```java
   removeStatistics(snapshotId);
   removePartitionStatistics(snapshotId);
   ```
   
   This adds the same removal for partition statistics, next to the existing 
one for statistics. A REST catalog server applies the update itself, so you 
mostly see this with catalogs where pyiceberg writes the metadata (SQL, Glue, 
Hive, DynamoDB). The test for #4201 works around the same gap by patching 
`commit_table` to drop these entries.
   
   I didn't find an existing issue for this.
   
   ## Are these changes tested?
   
   Yes. I added `test_update_remove_snapshots_with_partition_statistics` to 
`tests/table/test_expire_snapshots.py`, next to the existing statistics test. 
It fails on `main` (the partition statistics entry for the removed snapshot is 
still there) and passes with this change. `prek run` passes on the changed 
files.
   
   ## Are there any user-facing changes?
   
   After `expire_snapshots`, table metadata no longer lists partition 
statistics for snapshots that were removed.
   
   ## AI disclosure
   
   I used Claude Code (AI) to find this gap by comparing pyiceberg with Iceberg 
Java, and to write the fix and the test. I reviewed the change. One thing I'm 
not sure about: I kept the removal in the `RemoveSnapshotsUpdate` handler, 
applied as internal `RemovePartitionStatisticsUpdate`s the same way the 
statistics removal already works. I did not add a matching cleanup of the 
partition statistics files on storage. That belongs with the file-cleanup work 
in #4201.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_012XPuJeHi7A5CeHSKHjHmwx
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to