Hi all, Following up on the question of whether shared source is better overall than per-version copies, from a maintenance and dependency perspective, I did a draft POC ( #19025 <https://github.com/apache/iceberg/pull/19025> (do not merge)).
What it does: - 545 common source files in 4.1 and 4.2 move to spark/common. - Every file that differs stays as a full copy in its version directory. - There are no shims, which addresses Xin's governance concern. - Each Spark minor still compiles its own jars against its own Spark, and the 4.1 and 4.2 runtime jars match main's. - It also adds 4.3, with the same adaptations as #18749 <https://github.com/apache/iceberg/pull/18749>. What I observed since 4.2 landed (Sep 4): - 23 PRs changed only 4.2 and touched files this layout would share. - 13 of them were later backported to 4.1 by hand. With shared source, we would have avoided the backports. - 5 PRs used 4.2-only Spark APIs. These are the main 4.2 features. With shared source, each would have had to fork the shared files it touched first, or wait until 4.1 is dropped. This is the dependency cost Russell raised in the sync. Shared code can only use APIs that every member version has. The gains with this approach are the fewer needed backports, but the big cost is waiting for the last incompatible version to be dropped before adding a new API. Also, with AI, copying directories and backporting to older versions is easier than before. I think we should not pursue this. Let me know if you think otherwise. Thanks, Anurag On Mon, Aug 3, 2026 at 10:33 PM Anurag Mantripragada < [email protected]> wrote: > Hi Xin, > > Thanks for the feedback and for sharing your experience with Delta; this > is very useful input, and I agree with all the points you raised. > > To address a few of your suggestions based on the proof of concept (POC): > > - Scoping shims to genuine API adaptation: The POC already reflects > this. Of the 67 files that diverged between the 4.1 and 4.2 versions I > tested, only one was pure drift (which I moved to the shared folder). The > remaining 66 were due to actual Spark API changes. > - Stable Iceberg-owned contract: I agree that this will be the most > challenging part. I am happy to dig into what a concrete implementation > would look like if the community is interested. > - Shared business logic and test suites: The POC is already structured > this way. The full test suite of around 17,800 tests is shared, and only > version-specific expectations are shimmed. > - CI duplication reporting: I am thinking we could introduce a CI job > that periodically diffs adjacent-version shim directories to flag file > pairs above a certain similarity threshold as promotion candidates. It > could also flag any shim file that exceeds a specific size threshold for > additional review scrutiny. > > > Thanks, > Anurag > > > On Fri, Jul 31, 2026 at 4:03 PM Xin Huang via dev <[email protected]> > wrote: > >> Hi Anurag, >> >> Thank you and Sebastian for proposing this idea, and thank you for >> building and testing the proof of concept. I wanted to share a few thoughts >> based on my personal experience and feelings. >> >> I think a shared source tree is a promising direction. It should make >> Spark version upgrades smaller and easier to review. However, Delta’s >> experience suggests that shim governance should be considered from the >> beginning. Delta initially accumulated mirrored per-version shims and later >> needed #7201 <https://github.com/delta-io/delta/pull/7201> and #7202 >> <https://github.com/delta-io/delta/pull/7202> to remove roughly 2.8K >> lines of duplication and introduce additional cross-version shim >> directories. >> >> A few ideas on top of my mind that might help keep the shim layer >> manageable over time: >> >> - It may be useful to keep shims focused on Spark API adaptation. For >> example, the Spark 4.1 GeometryVal/GeographyVal APIs changed to >> BinaryView in Spark 4.2. A small geospatial adapter could handle that >> difference while allowing the surrounding Iceberg reader and writer logic >> to remain shared. >> - Where practical, shared code could depend on a stable, >> version-neutral contract. For example, shared view-handling logic could >> operate on an Iceberg-owned view metadata representation, while small >> version-specific adapters translate between it and Spark 4.1’s >> ViewInfo API or Spark 4.2’s new View API. This might be a main >> challenging point >> - Keeping business logic and complete test suites shared where >> possible may help avoid recreating some of the same maintenance burden >> inside the shim directories. >> - It may also help to derive source roots from one central definition >> and add CI reporting for duplicated or growing shim code. >> >> These are just some initial thoughts. Look forward to hearing what you >> think and would be happy to discuss further. >> Thanks >> Xin >> >>
