Shims are annoying but IMO much less error-prone than duplicating everything. As I opined in the community sync call this morning, Spark source compatibility has been fairly stable in practice, so IMO "large common, thin shims" is a useful bias.
Of course, when and if any particular shim starts to feel especially cursèd, some natural questions can always be raised: - what lives on either side of the common/shim split for any given version - whether any particular file should be duplicated or abstracted - "meta-shims" for specific compatibility "eras" (e.g. one for 3.5.x with minimal-to-no patch-specific content, one that covers 4.0 and 4.1 but perhaps not 4.2...) Thanks! -0xe1a On Wed, Nov 18, 2026 at 1:56 AM Anurag Mantripragada via dev < [email protected]> wrote: > Hi all, > > Following up on the question of whether shared source is better overall > than per-version copies, from a maintenance and dependency perspective, I > did a draft POC ( #19025 <https://github.com/apache/iceberg/pull/19025> > (do not merge)). > > What it does: > > - 545 common source files in 4.1 and 4.2 move to spark/common. > - Every file that differs stays as a full copy in its version > directory. > - There are no shims, which addresses Xin's governance concern. > - Each Spark minor still compiles its own jars against its own Spark, > and the 4.1 and 4.2 runtime jars match main's. > - It also adds 4.3, with the same adaptations as #18749 > <https://github.com/apache/iceberg/pull/18749>. > > > What I observed since 4.2 landed (Sep 4): > > - 23 PRs changed only 4.2 and touched files this layout would share. > - 13 of them were later backported to 4.1 by hand. With shared source, > we would have avoided the backports. > - 5 PRs used 4.2-only Spark APIs. These are the main 4.2 features. > With shared source, each would have had to fork the shared files it touched > first, or wait until 4.1 is dropped. This is the dependency cost Russell > raised in the sync. Shared code can only use APIs that every member version > has. > > The gains with this approach are the fewer needed backports, but the big > cost is waiting for the last incompatible version to be dropped before > adding a new API. > > Also, with AI, copying directories and backporting to older versions is > easier than before. I think we should not pursue this. Let me know if you > think otherwise. > > Thanks, > Anurag > > On Mon, Aug 3, 2026 at 10:33 PM Anurag Mantripragada < > [email protected]> wrote: > >> Hi Xin, >> >> Thanks for the feedback and for sharing your experience with Delta; this >> is very useful input, and I agree with all the points you raised. >> >> To address a few of your suggestions based on the proof of concept (POC): >> >> - Scoping shims to genuine API adaptation: The POC already reflects >> this. Of the 67 files that diverged between the 4.1 and 4.2 versions I >> tested, only one was pure drift (which I moved to the shared folder). The >> remaining 66 were due to actual Spark API changes. >> - Stable Iceberg-owned contract: I agree that this will be the most >> challenging part. I am happy to dig into what a concrete implementation >> would look like if the community is interested. >> - Shared business logic and test suites: The POC is already >> structured this way. The full test suite of around 17,800 tests is shared, >> and only version-specific expectations are shimmed. >> - CI duplication reporting: I am thinking we could introduce a CI job >> that periodically diffs adjacent-version shim directories to flag file >> pairs above a certain similarity threshold as promotion candidates. It >> could also flag any shim file that exceeds a specific size threshold for >> additional review scrutiny. >> >> >> Thanks, >> Anurag >> >> >> On Fri, Jul 31, 2026 at 4:03 PM Xin Huang via dev <[email protected]> >> wrote: >> >>> Hi Anurag, >>> >>> Thank you and Sebastian for proposing this idea, and thank you for >>> building and testing the proof of concept. I wanted to share a few thoughts >>> based on my personal experience and feelings. >>> >>> I think a shared source tree is a promising direction. It should make >>> Spark version upgrades smaller and easier to review. However, Delta’s >>> experience suggests that shim governance should be considered from the >>> beginning. Delta initially accumulated mirrored per-version shims and later >>> needed #7201 <https://github.com/delta-io/delta/pull/7201> and #7202 >>> <https://github.com/delta-io/delta/pull/7202> to remove roughly 2.8K >>> lines of duplication and introduce additional cross-version shim >>> directories. >>> >>> A few ideas on top of my mind that might help keep the shim layer >>> manageable over time: >>> >>> - It may be useful to keep shims focused on Spark API adaptation. >>> For example, the Spark 4.1 GeometryVal/GeographyVal APIs changed to >>> BinaryView in Spark 4.2. A small geospatial adapter could handle >>> that difference while allowing the surrounding Iceberg reader and writer >>> logic to remain shared. >>> - Where practical, shared code could depend on a stable, >>> version-neutral contract. For example, shared view-handling logic could >>> operate on an Iceberg-owned view metadata representation, while small >>> version-specific adapters translate between it and Spark 4.1’s >>> ViewInfo API or Spark 4.2’s new View API. This might be a main >>> challenging point >>> - Keeping business logic and complete test suites shared where >>> possible may help avoid recreating some of the same maintenance burden >>> inside the shim directories. >>> - It may also help to derive source roots from one central >>> definition and add CI reporting for duplicated or growing shim code. >>> >>> These are just some initial thoughts. Look forward to hearing what you >>> think and would be happy to discuss further. >>> Thanks >>> Xin >>> >>>
