dramaticlly opened a new pull request, #19044:
URL: https://github.com/apache/iceberg/pull/19044

   With `startVersion` set, RewriteTablePath lists the statistics and partition 
statistics files of every version file it stages. Later versions still 
reference the start version's statistics files, so an incremental copy lists 
them again. The earlier copy already moved them, and the docs say an 
incremental rewrite returns only files added in the range. This change removes 
the start version's statistics and partition statistics files from the copy 
plan. Statistics files added after the start version are still listed, 
including new ones for snapshots that already existed. Full copies are 
unchanged. This targets Spark 4.2; backports to 3.5, 4.0 and 4.1 can follow.
   
   Test plan:
   - `incrementalRewriteSkipsStatisticsFilesOfStartVersion` fails without the 
fix (one statistics file listed, zero expected). Dropping only the partition 
statistics exclusion fails it on the partition statistics count.
   - `incrementalRewriteIncludesNewStatisticsFilesOfExistingSnapshot` guards 
against over-exclusion: it fails if incremental copies drop every statistics 
file.
   - `./gradlew -DallModules build -x test -x javadoc -x integrationTest` and 
`:iceberg-spark:iceberg-spark-4.2_2.13:check -DtestParallelism=8` pass.
   
   ---
   **AI Disclosure**
   - Model: Claude Opus 5.5
   - Platform/Tool: Claude Code
   - Human Oversight: [unknown - human to fill in]
   - Prompt Summary: Port a fix for statistics files re-listed by incremental 
RewriteTablePath from a downstream fork: confirm the bug upstream, write the 
fix and tests, self-review. The fix and both tests are generated.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to