debugmiller opened a new issue, #1393: URL: https://github.com/apache/iceberg-go/issues/1393
### Apache Iceberg version main (development) ### Please describe the bug 🐞 `overwriteFiles.existingManifests` iterates each parent snapshot manifest with `discardDeleted=true`, which skips entries with `status=DELETED`. For a manifest where every entry is `DELETED`, the iterator yields nothing (since `foundDeletedCount=0` amd `len(notDeleted)=0`) which the code interprets as "manifest was unaffected by this operation" and retains instead of dropping it at https://github.com/apache/iceberg-go/blob/a7c491261d495dcb055033583db34695eabe81c2/table/snapshot_producers.go#L190-L194 I think the fix is small, at [L190](https://github.com/apache/iceberg-go/blob/a7c491261d49/table/snapshot_producers.go#L190), only retain the manifest if the iterator actually found live entries: ```diff if foundDeletedCount == 0 { - existingFiles = append(existingFiles, m) + if len(notDeleted) > 0 { + existingFiles = append(existingFiles, m) + } + // len(notDeleted)==0: all entries are DELETED and were skipped by + // discardDeleted=true. The manifest is empty — drop it. continue } ``` We observed this when using overwrite on small tables, which produces one DELETE manifest via `deletedFilesManifests`. Since these are never pruned, they accumulate: after N overwrites the current snapshot's manifest list contains N all-DELETED manifests alongside the live data manifests, causing a growing set of unnecessary snapshots. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
