kevinjqliu commented on issue #3758: URL: https://github.com/apache/iceberg-python/issues/3758#issuecomment-5269570953
Hey folks, thanks for looking into this. I want to try to summarize the issue (as I understand it): The regression comes from manifest pruning added to the write path in #3011. During an overwrite, PyIceberg uses partition summaries to skip manifests that cannot contain any data files being replaced. This avoids opening every manifest before performing the exact `DataFile` membership check. The code mixes two predicate domains: - Row predicates contain source values and must be projected through the partition transform. - Predicates built from `DataFile.partition` contain already-transformed values and should be evaluated directly against the partition schema. For example, with `day(ts)`: ```text ts = 2026-01-06 12:00:00 DataFile.partition = 20459 ``` The row-predicate path applies the partition transform before manifest evaluation. Reusing that path for a predicate built from `DataFile.partition` applies `day()` to the already-transformed value `20459`, producing approximately `partition_day == 0`. This does not match the manifest’s actual value, `partition_day == 20459`, so the manifest is incorrectly skipped. Because the manifest is skipped, it is carried forward unchanged and the exact file-removal check never runs. The replacement files are still added, leaving both the old and new data in the table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
