max-parke-scale opened a new issue, #4165: URL: https://github.com/apache/iceberg-python/issues/4165
### Apache Iceberg version 0.11.1. `main` has the same logic. ### Please describe the bug 🐞 When an overwrite removes files explicitly (`update_snapshot().overwrite()` with `delete_data_file`), the new snapshot keeps every earlier manifest that holds only DELETED entries. Each overwrite also writes a manifest recording its own deletions, so the manifest list grows by one per commit and only one manifest ever holds live files. Each later commit and scan plan opens all of them. In `_OverwriteFiles._existing_manifests`, each parent manifest is read with `fetch_manifest_entry(discard_deleted=True)`. A manifest whose entries are all DELETED comes back empty, so `found_deleted_entries` is empty and the manifest takes the "intercept is empty" branch, where it is kept unchanged. Java drops these. [`MergingSnapshotProducer.apply`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java) keeps a manifest only if `hasAddedFiles() || hasExistingFiles() || snapshotId() == snapshotId()`. The equivalent check in pyiceberg would skip parent manifests with no added or existing files before reading them. `Table.overwrite()` is not affected. It deletes through `_DeleteFiles` before appending. **Repro** (local SQL catalog): ```python import tempfile import pyarrow as pa from pyiceberg.catalog.sql import SqlCatalog from pyiceberg.io.pyarrow import _dataframe_to_data_files wh = tempfile.mkdtemp() catalog = SqlCatalog("repro", uri=f"sqlite:///{wh}/catalog.db", warehouse=f"file://{wh}") catalog.create_namespace("ns") table = catalog.create_table("ns.t", schema=pa.schema([("id", pa.int64())])) table.append(pa.table({"id": [0]})) for i in range(1, 11): table = catalog.load_table("ns.t") with table.transaction() as tx: new_files = list(_dataframe_to_data_files(tx.table_metadata, pa.table({"id": [i]}), table.io)) with tx.update_snapshot().overwrite() as producer: for task in table.scan().plan_files(): producer.delete_data_file(task.file) for f in new_files: producer.append_data_file(f) table = catalog.load_table("ns.t") manifests = table.current_snapshot().manifests(table.io) live = sum(1 for m in manifests if m.added_files_count or m.existing_files_count) print(f"overwrite {i}: {len(manifests)} manifests, {live} with live files") ``` Output: `overwrite 1: 2 manifests, 1 with live files` up to `overwrite 10: 11 manifests, 1 with live files`. In production, a table overwritten a few hundred times reached 388 manifests, 387 of them deleted-only. Each overwrite commit went from about 3 s to about 24 s on S3. ### Willingness to contribute - [x] I can contribute a fix for this bug independently 🧑💻🤖 — posted via [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
