karubian opened a new pull request, #19036: URL: https://github.com/apache/iceberg/pull/19036
Fixes #19035. `rewrite_position_delete_files` keeps only the position deletes whose data file appears in a `data_files` scan filtered to the file group's partition. That scan skips manifests by their partition summaries, and Iceberg before 1.11.0 could write wrong bounds for binary partition fields, #15128, so a live data file can look gone and its deletes are dropped from the current snapshot. The runner now leaves binary partition fields out of that filter; fields of other types keep it. On a table partitioned only by binary fields, every file group reads all data manifests and joins against all live data files, which can turn the broadcast join into a shuffle. Spark 3.5, 4.0 and 4.1 follow in a backport PR. Test plan: on a v2 table whose binary partition summary excludes its own data file, the rewrite keeps the delete of that live file and drops the delete of a removed file in the same group. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: [unknown - human to fill in] - Prompt Summary: Investigate a position delete that `rewrite_position_delete_files` dropped in the author's correctness test suite, then write a regression test and a fix in `SparkRewritePositionDeleteRunner`. The runner change, the test and this description are generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
