dramaticlly opened a new pull request, #19044: URL: https://github.com/apache/iceberg/pull/19044
With `startVersion` set, RewriteTablePath lists the statistics and partition statistics files of every version file it stages. Later versions still reference the start version's statistics files, so an incremental copy lists them again. The earlier copy already moved them, and the docs say an incremental rewrite returns only files added in the range. This change removes the start version's statistics and partition statistics files from the copy plan. Statistics files added after the start version are still listed, including new ones for snapshots that already existed. Full copies are unchanged. This targets Spark 4.2; backports to 3.5, 4.0 and 4.1 can follow. Test plan: - `incrementalRewriteSkipsStatisticsFilesOfStartVersion` fails without the fix (one statistics file listed, zero expected). Dropping only the partition statistics exclusion fails it on the partition statistics count. - `incrementalRewriteIncludesNewStatisticsFilesOfExistingSnapshot` guards against over-exclusion: it fails if incremental copies drop every statistics file. - `./gradlew -DallModules build -x test -x javadoc -x integrationTest` and `:iceberg-spark:iceberg-spark-4.2_2.13:check -DtestParallelism=8` pass. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: [unknown - human to fill in] - Prompt Summary: Port a fix for statistics files re-listed by incremental RewriteTablePath from a downstream fork: confirm the bug upstream, write the fix and tests, self-review. The fix and both tests are generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
