Hi everyone, We support each Spark minor by copying the entire spark/ tree — spark/v3.5, spark/v4.0, spark/v4.1, soon spark/v4.2. Adding a version is a several-hundred-file, 100k+-line PR #14984 <https://github.com/apache/iceberg/pull/14984> With SPARK-54633 <https://docs.google.com/document/d/1gBoZ4KH5zQUWpgK3M7zAN7p6Glz4S_e9bO3PvQA9sQs/edit?tab=t.0#heading=h.vj8hviw7ebqz> proposing quarterly Spark minors, we'd be doing that roughly four times a year.
@Sebastian Baunsgaard <[email protected]> proposed in the last community sync <https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?tab=t.0#heading=h.r977qio1wsv2> a shared source tree with small per-version shim directories ,the approach Delta Lake uses, starting at Spark 4.3 and forward. Since 4.3 doesn't exist yet to test against, I built a proof of concept using 4.1/4.2 (from #14984 <https://github.com/apache/iceberg/pull/14984>'s branch) purely as a stand-in to prove the mechanism itself works. With Claude's help, I was able to make changes and run tests locally. There were 555 shared files vs. 73–74 per-version shim files (88% shared) with this experiment. [image: Screenshot 2026-07-31 at 2.48.06 PM.png] Some issues arose during the process: 1. Gradle's antlr plugin can't read a grammar outside the project directory — fixed by staging it into the build dir first with a Copy task. Better idiom out there? 2. The extensions module can't clear java.srcDirs like the core module does, since antlr needs that dir for its generated parser — fixed by just leaving it alone, but it fails silently and confusingly if you don't. 3. Spotless targets are project-relative and silently skip anything outside a version's own directory, I fixed by registering it from the :iceberg-spark container project instead. Open to better options. *Advantages* - Adding a version becomes a small, reviewable PR. - No backports needed for shared Spark changes. (could also be a risk :) ) - No user-facing changes *Risks* - This is a major refactor of the code structure. - A Spark release that changes a lot at once could still need its own full private tree. - Build tooling (Spotless, Checkstyle, CI matrix) must be deliberately pointed at the shared/shim dirs or it will silently skip them. - Bugs in shared sources can manifest in all versions. Have we thought about this idea before and what do you all think? If the community thinks this is useful, I can take it forward. Thanks, Anurag
