Hi Alex, The draft I shared in #19025 <https://github.com/apache/iceberg/pull/19025> was the no-shim variant, Shims were discussed earlier as well. The concern there was that the adapter layer needs governance and tends to grow; Xin described Delta's experience on this thread. Spark also breaks APIs we depend on between minor releases. For example, 4.2 changed connector.catalog.View from an interface to a class.
At today's sync we decided to keep the copy-per-version layout for now. If someone wants to review the no-shim design I will leave #19025 <https://github.com/apache/iceberg/pull/19025> open. We can always revisit this if adding spark versions becomes a severe review burden. Thanks, Anurag On Wed, Nov 18, 2026 at 11:05 AM Alex Cruise <[email protected]> wrote: > Shims are annoying but IMO much less error-prone than duplicating > everything. As I opined in the community sync call this morning, Spark > source compatibility has been fairly stable in practice, so IMO "large > common, thin shims" is a useful bias. > > Of course, when and if any particular shim starts to feel especially > cursèd, some natural questions can always be raised: > - what lives on either side of the common/shim split for any given version > - whether any particular file should be duplicated or abstracted > - "meta-shims" for specific compatibility "eras" (e.g. one for 3.5.x with > minimal-to-no patch-specific content, one that covers 4.0 and 4.1 but > perhaps not 4.2...) > > Thanks! > > -0xe1a > > On Wed, Nov 18, 2026 at 1:56 AM Anurag Mantripragada via dev < > [email protected]> wrote: > >> Hi all, >> >> Following up on the question of whether shared source is better overall >> than per-version copies, from a maintenance and dependency perspective, I >> did a draft POC ( #19025 <https://github.com/apache/iceberg/pull/19025> >> (do not merge)). >> >> What it does: >> >> - 545 common source files in 4.1 and 4.2 move to spark/common. >> - Every file that differs stays as a full copy in its version >> directory. >> - There are no shims, which addresses Xin's governance concern. >> - Each Spark minor still compiles its own jars against its own Spark, >> and the 4.1 and 4.2 runtime jars match main's. >> - It also adds 4.3, with the same adaptations as #18749 >> <https://github.com/apache/iceberg/pull/18749>. >> >> >> What I observed since 4.2 landed (Sep 4): >> >> - 23 PRs changed only 4.2 and touched files this layout would share. >> - 13 of them were later backported to 4.1 by hand. With shared >> source, we would have avoided the backports. >> - 5 PRs used 4.2-only Spark APIs. These are the main 4.2 features. >> With shared source, each would have had to fork the shared files it >> touched >> first, or wait until 4.1 is dropped. This is the dependency cost Russell >> raised in the sync. Shared code can only use APIs that every member >> version >> has. >> >> The gains with this approach are the fewer needed backports, but the big >> cost is waiting for the last incompatible version to be dropped before >> adding a new API. >> >> Also, with AI, copying directories and backporting to older versions is >> easier than before. I think we should not pursue this. Let me know if you >> think otherwise. >> >> Thanks, >> Anurag >> >> On Mon, Aug 3, 2026 at 10:33 PM Anurag Mantripragada < >> [email protected]> wrote: >> >>> Hi Xin, >>> >>> Thanks for the feedback and for sharing your experience with Delta; this >>> is very useful input, and I agree with all the points you raised. >>> >>> To address a few of your suggestions based on the proof of concept (POC): >>> >>> - Scoping shims to genuine API adaptation: The POC already reflects >>> this. Of the 67 files that diverged between the 4.1 and 4.2 versions I >>> tested, only one was pure drift (which I moved to the shared folder). The >>> remaining 66 were due to actual Spark API changes. >>> - Stable Iceberg-owned contract: I agree that this will be the most >>> challenging part. I am happy to dig into what a concrete implementation >>> would look like if the community is interested. >>> - Shared business logic and test suites: The POC is already >>> structured this way. The full test suite of around 17,800 tests is >>> shared, >>> and only version-specific expectations are shimmed. >>> - CI duplication reporting: I am thinking we could introduce a CI >>> job that periodically diffs adjacent-version shim directories to flag >>> file >>> pairs above a certain similarity threshold as promotion candidates. It >>> could also flag any shim file that exceeds a specific size threshold for >>> additional review scrutiny. >>> >>> >>> Thanks, >>> Anurag >>> >>> >>> On Fri, Jul 31, 2026 at 4:03 PM Xin Huang via dev < >>> [email protected]> wrote: >>> >>>> Hi Anurag, >>>> >>>> Thank you and Sebastian for proposing this idea, and thank you for >>>> building and testing the proof of concept. I wanted to share a few thoughts >>>> based on my personal experience and feelings. >>>> >>>> I think a shared source tree is a promising direction. It should make >>>> Spark version upgrades smaller and easier to review. However, Delta’s >>>> experience suggests that shim governance should be considered from the >>>> beginning. Delta initially accumulated mirrored per-version shims and later >>>> needed #7201 <https://github.com/delta-io/delta/pull/7201> and #7202 >>>> <https://github.com/delta-io/delta/pull/7202> to remove roughly 2.8K >>>> lines of duplication and introduce additional cross-version shim >>>> directories. >>>> >>>> A few ideas on top of my mind that might help keep the shim layer >>>> manageable over time: >>>> >>>> - It may be useful to keep shims focused on Spark API adaptation. >>>> For example, the Spark 4.1 GeometryVal/GeographyVal APIs changed to >>>> BinaryView in Spark 4.2. A small geospatial adapter could handle >>>> that difference while allowing the surrounding Iceberg reader and writer >>>> logic to remain shared. >>>> - Where practical, shared code could depend on a stable, >>>> version-neutral contract. For example, shared view-handling logic could >>>> operate on an Iceberg-owned view metadata representation, while small >>>> version-specific adapters translate between it and Spark 4.1’s >>>> ViewInfo API or Spark 4.2’s new View API. This might be a main >>>> challenging point >>>> - Keeping business logic and complete test suites shared where >>>> possible may help avoid recreating some of the same maintenance burden >>>> inside the shim directories. >>>> - It may also help to derive source roots from one central >>>> definition and add CI reporting for duplicated or growing shim code. >>>> >>>> These are just some initial thoughts. Look forward to hearing what you >>>> think and would be happy to discuss further. >>>> Thanks >>>> Xin >>>> >>>>
