Hi Alex,

The draft I shared in #19025 <https://github.com/apache/iceberg/pull/19025> was
the no-shim variant, Shims were discussed earlier as well. The concern
there was that the adapter layer needs governance and tends to grow; Xin
described Delta's experience on this thread. Spark also breaks APIs we
depend on between minor releases. For example, 4.2 changed
connector.catalog.View from an interface to a class.

At today's sync we decided to keep the copy-per-version layout for now. If
someone wants to review the no-shim design I will leave #19025
<https://github.com/apache/iceberg/pull/19025> open. We can always revisit
this if adding spark versions becomes a severe review burden.

Thanks,
Anurag

On Wed, Nov 18, 2026 at 11:05 AM Alex Cruise <[email protected]> wrote:

> Shims are annoying but IMO much less error-prone than duplicating
> everything. As I opined in the community sync call this morning, Spark
> source compatibility has been fairly stable in practice, so IMO "large
> common, thin shims" is a useful bias.
>
> Of course, when and if any particular shim starts to feel especially
> cursèd, some natural questions can always be raised:
> - what lives on either side of the common/shim split for any given version
> - whether any particular file should be duplicated or abstracted
> - "meta-shims" for specific compatibility "eras" (e.g. one for 3.5.x with
> minimal-to-no patch-specific content, one that covers 4.0 and 4.1 but
> perhaps not 4.2...)
>
> Thanks!
>
> -0xe1a
>
> On Wed, Nov 18, 2026 at 1:56 AM Anurag Mantripragada via dev <
> [email protected]> wrote:
>
>> Hi all,
>>
>> Following up on the question of whether shared source is better overall
>> than per-version copies, from a maintenance and dependency perspective, I
>> did a draft POC ( #19025 <https://github.com/apache/iceberg/pull/19025>
>> (do not merge)).
>>
>> What it does:
>>
>>    - 545 common source files in 4.1 and 4.2 move to spark/common.
>>    - Every file that differs stays as a full copy in its version
>>    directory.
>>    - There are no shims, which addresses Xin's governance concern.
>>    - Each Spark minor still compiles its own jars against its own Spark,
>>    and the 4.1 and 4.2 runtime jars match main's.
>>    - It also adds 4.3, with the same adaptations as #18749
>>    <https://github.com/apache/iceberg/pull/18749>.
>>
>>
>> What I observed since 4.2 landed (Sep 4):
>>
>>    - 23 PRs changed only 4.2 and touched files this layout would share.
>>    - 13 of them were later backported to 4.1 by hand. With shared
>>    source, we would have avoided the backports.
>>    - 5 PRs used 4.2-only Spark APIs. These are the main 4.2 features.
>>    With shared source, each would have had to fork the shared files it 
>> touched
>>    first, or wait until 4.1 is dropped. This is the dependency cost Russell
>>    raised in the sync. Shared code can only use APIs that every member 
>> version
>>    has.
>>
>> The gains with this approach are the fewer needed backports, but the big
>> cost is waiting for the last incompatible version to be dropped before
>> adding a new API.
>>
>> Also, with AI, copying directories and backporting to older versions is
>> easier than before. I think we should not pursue this. Let me know if you
>> think otherwise.
>>
>> Thanks,
>> Anurag
>>
>> On Mon, Aug 3, 2026 at 10:33 PM Anurag Mantripragada <
>> [email protected]> wrote:
>>
>>> Hi Xin,
>>>
>>> Thanks for the feedback and for sharing your experience with Delta; this
>>> is very useful input, and I agree with all the points you raised.
>>>
>>> To address a few of your suggestions based on the proof of concept (POC):
>>>
>>>    - Scoping shims to genuine API adaptation: The POC already reflects
>>>    this. Of the 67 files that diverged between the 4.1 and 4.2 versions I
>>>    tested, only one was pure drift (which I moved to the shared folder). The
>>>    remaining 66 were due to actual Spark API changes.
>>>    - Stable Iceberg-owned contract: I agree that this will be the most
>>>    challenging part. I am happy to dig into what a concrete implementation
>>>    would look like if the community is interested.
>>>    - Shared business logic and test suites: The POC is already
>>>    structured this way. The full test suite of around 17,800 tests is 
>>> shared,
>>>    and only version-specific expectations are shimmed.
>>>    - CI duplication reporting: I am thinking we could introduce a CI
>>>    job that periodically diffs adjacent-version shim directories to flag 
>>> file
>>>    pairs above a certain similarity threshold as promotion candidates. It
>>>    could also flag any shim file that exceeds a specific size threshold for
>>>    additional review scrutiny.
>>>
>>>
>>> Thanks,
>>> Anurag
>>>
>>>
>>> On Fri, Jul 31, 2026 at 4:03 PM Xin Huang via dev <
>>> [email protected]> wrote:
>>>
>>>> Hi Anurag,
>>>>
>>>> Thank you and Sebastian for proposing this idea, and thank you for
>>>> building and testing the proof of concept. I wanted to share a few thoughts
>>>> based on my personal experience and feelings.
>>>>
>>>> I think a shared source tree is a promising direction. It should make
>>>> Spark version upgrades smaller and easier to review. However, Delta’s
>>>> experience suggests that shim governance should be considered from the
>>>> beginning. Delta initially accumulated mirrored per-version shims and later
>>>> needed #7201 <https://github.com/delta-io/delta/pull/7201> and #7202
>>>> <https://github.com/delta-io/delta/pull/7202> to remove roughly 2.8K
>>>> lines of duplication and introduce additional cross-version shim
>>>> directories.
>>>>
>>>> A few ideas on top of my mind that might help keep the shim layer
>>>> manageable over time:
>>>>
>>>>    - It may be useful to keep shims focused on Spark API adaptation.
>>>>    For example, the Spark 4.1 GeometryVal/GeographyVal APIs changed to
>>>>    BinaryView in Spark 4.2. A small geospatial adapter could handle
>>>>    that difference while allowing the surrounding Iceberg reader and writer
>>>>    logic to remain shared.
>>>>    - Where practical, shared code could depend on a stable,
>>>>    version-neutral contract. For example, shared view-handling logic could
>>>>    operate on an Iceberg-owned view metadata representation, while small
>>>>    version-specific adapters translate between it and Spark 4.1’s
>>>>    ViewInfo API or Spark 4.2’s new View API. This might be a main
>>>>    challenging point
>>>>    - Keeping business logic and complete test suites shared where
>>>>    possible may help avoid recreating some of the same maintenance burden
>>>>    inside the shim directories.
>>>>    - It may also help to derive source roots from one central
>>>>    definition and add CI reporting for duplicated or growing shim code.
>>>>
>>>> These are just some initial thoughts. Look forward to hearing what you
>>>> think and would be happy to discuss further.
>>>> Thanks
>>>> Xin
>>>>
>>>>

Reply via email to