Shims are annoying but IMO much less error-prone than duplicating
everything. As I opined in the community sync call this morning, Spark
source compatibility has been fairly stable in practice, so IMO "large
common, thin shims" is a useful bias.

Of course, when and if any particular shim starts to feel especially
cursèd, some natural questions can always be raised:
- what lives on either side of the common/shim split for any given version
- whether any particular file should be duplicated or abstracted
- "meta-shims" for specific compatibility "eras" (e.g. one for 3.5.x with
minimal-to-no patch-specific content, one that covers 4.0 and 4.1 but
perhaps not 4.2...)

Thanks!

-0xe1a

On Wed, Nov 18, 2026 at 1:56 AM Anurag Mantripragada via dev <
[email protected]> wrote:

> Hi all,
>
> Following up on the question of whether shared source is better overall
> than per-version copies, from a maintenance and dependency perspective, I
> did a draft POC ( #19025 <https://github.com/apache/iceberg/pull/19025>
> (do not merge)).
>
> What it does:
>
>    - 545 common source files in 4.1 and 4.2 move to spark/common.
>    - Every file that differs stays as a full copy in its version
>    directory.
>    - There are no shims, which addresses Xin's governance concern.
>    - Each Spark minor still compiles its own jars against its own Spark,
>    and the 4.1 and 4.2 runtime jars match main's.
>    - It also adds 4.3, with the same adaptations as #18749
>    <https://github.com/apache/iceberg/pull/18749>.
>
>
> What I observed since 4.2 landed (Sep 4):
>
>    - 23 PRs changed only 4.2 and touched files this layout would share.
>    - 13 of them were later backported to 4.1 by hand. With shared source,
>    we would have avoided the backports.
>    - 5 PRs used 4.2-only Spark APIs. These are the main 4.2 features.
>    With shared source, each would have had to fork the shared files it touched
>    first, or wait until 4.1 is dropped. This is the dependency cost Russell
>    raised in the sync. Shared code can only use APIs that every member version
>    has.
>
> The gains with this approach are the fewer needed backports, but the big
> cost is waiting for the last incompatible version to be dropped before
> adding a new API.
>
> Also, with AI, copying directories and backporting to older versions is
> easier than before. I think we should not pursue this. Let me know if you
> think otherwise.
>
> Thanks,
> Anurag
>
> On Mon, Aug 3, 2026 at 10:33 PM Anurag Mantripragada <
> [email protected]> wrote:
>
>> Hi Xin,
>>
>> Thanks for the feedback and for sharing your experience with Delta; this
>> is very useful input, and I agree with all the points you raised.
>>
>> To address a few of your suggestions based on the proof of concept (POC):
>>
>>    - Scoping shims to genuine API adaptation: The POC already reflects
>>    this. Of the 67 files that diverged between the 4.1 and 4.2 versions I
>>    tested, only one was pure drift (which I moved to the shared folder). The
>>    remaining 66 were due to actual Spark API changes.
>>    - Stable Iceberg-owned contract: I agree that this will be the most
>>    challenging part. I am happy to dig into what a concrete implementation
>>    would look like if the community is interested.
>>    - Shared business logic and test suites: The POC is already
>>    structured this way. The full test suite of around 17,800 tests is shared,
>>    and only version-specific expectations are shimmed.
>>    - CI duplication reporting: I am thinking we could introduce a CI job
>>    that periodically diffs adjacent-version shim directories to flag file
>>    pairs above a certain similarity threshold as promotion candidates. It
>>    could also flag any shim file that exceeds a specific size threshold for
>>    additional review scrutiny.
>>
>>
>> Thanks,
>> Anurag
>>
>>
>> On Fri, Jul 31, 2026 at 4:03 PM Xin Huang via dev <[email protected]>
>> wrote:
>>
>>> Hi Anurag,
>>>
>>> Thank you and Sebastian for proposing this idea, and thank you for
>>> building and testing the proof of concept. I wanted to share a few thoughts
>>> based on my personal experience and feelings.
>>>
>>> I think a shared source tree is a promising direction. It should make
>>> Spark version upgrades smaller and easier to review. However, Delta’s
>>> experience suggests that shim governance should be considered from the
>>> beginning. Delta initially accumulated mirrored per-version shims and later
>>> needed #7201 <https://github.com/delta-io/delta/pull/7201> and #7202
>>> <https://github.com/delta-io/delta/pull/7202> to remove roughly 2.8K
>>> lines of duplication and introduce additional cross-version shim
>>> directories.
>>>
>>> A few ideas on top of my mind that might help keep the shim layer
>>> manageable over time:
>>>
>>>    - It may be useful to keep shims focused on Spark API adaptation.
>>>    For example, the Spark 4.1 GeometryVal/GeographyVal APIs changed to
>>>    BinaryView in Spark 4.2. A small geospatial adapter could handle
>>>    that difference while allowing the surrounding Iceberg reader and writer
>>>    logic to remain shared.
>>>    - Where practical, shared code could depend on a stable,
>>>    version-neutral contract. For example, shared view-handling logic could
>>>    operate on an Iceberg-owned view metadata representation, while small
>>>    version-specific adapters translate between it and Spark 4.1’s
>>>    ViewInfo API or Spark 4.2’s new View API. This might be a main
>>>    challenging point
>>>    - Keeping business logic and complete test suites shared where
>>>    possible may help avoid recreating some of the same maintenance burden
>>>    inside the shim directories.
>>>    - It may also help to derive source roots from one central
>>>    definition and add CI reporting for duplicated or growing shim code.
>>>
>>> These are just some initial thoughts. Look forward to hearing what you
>>> think and would be happy to discuss further.
>>> Thanks
>>> Xin
>>>
>>>

Reply via email to