Hi Alex,

Each Iceberg runtime jar is built for one Spark minor, and running it on a
different minor fails at runtime (See this thread
<https://github.com/apache/iceberg/issues/17959#issuecomment-6121779365>
for example). Until we publish 4.2 jars, production Spark 4.2 users would
need to build their own from source.

~ Anurag

On Wed, Nov 18, 2026 at 10:33 AM Alex Cruise <[email protected]> wrote:

> On Wed, Nov 18, 2026 at 11:11 AM Russell Spitzer <
> [email protected]> wrote:
>
>> We don't ship binaries for 4.2 so there is no package target for them. A
>> user could just build their own lib and use it.
>>
>
> OK, but could a Spark 4.2 user use the 4.1 binaries unmodified? Would they
> see a dependency error at build time, and/or a principled refusal to
> operate at runtime, and/or some kind of LinkageError at runtime?
>
> Thanks!
>
> -0xe1a
>
>
>>
>> On Wed, Nov 18, 2026 at 12:06 PM Alex Cruise <[email protected]> wrote:
>>
>>> I apologize if this should be obvious, but does the lack of official
>>> Iceberg support for Spark 4.2 mean that Spark 4.2 users can't use Iceberg
>>> at all, or merely that they can't benefit from enhancements available in
>>> Spark 4.2?
>>>
>>> Thanks!
>>>
>>> -0xe1a
>>>
>>> On Wed, Nov 18, 2026 at 2:44 AM Cheng Pan <[email protected]> wrote:
>>>
>>>> Thanks for reviving this discussion and sharing your thoughts. But I
>>>> think we had reached a consensus on the Spark version support policy when
>>>> adding Spark 4.2 support: the latest LTS + 2 latest minors, and the
>>>> situation still hold - Spark community runs with the planned version policy
>>>> and has no change since we made decision.
>>>>
>>>> And for "We have not released Spark 4.2 binaries yet.”, this was raised
>>>> by Anton, I believe it should be up to him to assess whether the current
>>>> Spark 4.2 connector meets his expectations for a binary release, and to
>>>> determine whether it should be included in the next 1.12.x patch release or
>>>> in 1.13.0.
>>>>
>>>> Thanks,
>>>> Cheng Pan
>>>>
>>>>
>>>>
>>>> On Nov 18, 2026, at 16:41, Anurag Mantripragada via dev <
>>>> [email protected]> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> Reviving this thread because Spark 4.3.0 is out and Cheng opened #18749
>>>> <https://github.com/apache/iceberg/pull/18749> to add it. We need to
>>>> decide whether to add 4.3 or skip this version entirely.
>>>>
>>>> Where we are:
>>>>
>>>>    - We have not released Spark 4.2 binaries yet.
>>>>    - Spark 4.3 is a non-LTS minor with a 6-month maintenance window. #18749
>>>>    <https://github.com/apache/iceberg/pull/18749>shows the diff is
>>>>    small.
>>>>    - Spark 4.4 is expected around January (branch is cut
>>>>    <https://lists.apache.org/thread.html/8l3sv3ywtyk9j86r92gfmc3cj5xx1dyy>
>>>>    in Spark already).
>>>>
>>>> I lean towards skipping 4.3 entirely and focusing on completing the pending
>>>> items in Spark 4.2 <https://github.com/apache/iceberg/issues/17959>.
>>>> Depending on the timeline, we can look into adding Spark 4.4 after the 1.13
>>>> release.
>>>>
>>>> What do you all think?
>>>>
>>>> On Wed, Jun 17, 2026 at 3:56 PM Anton Okolnychyi <[email protected]>
>>>> wrote:
>>>>
>>>>> Given that we still rely on internals of Spark in some places, I think
>>>>> we will have to continue to have a module per Spark version. LTS + rolling
>>>>> window of 2 minors seems reasonable to me too.
>>>>>
>>>>> +1 on getting 4.2 support in to unlock development and dropping 4.0
>>>>> after that.
>>>>>
>>>>> - Anton
>>>>>
>>>>> пн, 15 черв. 2026 р. о 07:30 Anurag Mantripragada <
>>>>> [email protected]> пише:
>>>>>
>>>>>> And I think whether to dropping Spark 4.0 support does not block
>>>>>>> adding Spark 4.2 support (with current RC state)
>>>>>>
>>>>>>
>>>>>> +1, we can review and merge Spark 4.2 support first. I will start a
>>>>>> vote to remove Spark 4.0 after the community sync tomorrow.
>>>>>>
>>>>>> Thanks,
>>>>>> Anurag
>>>>>>
>>>>>> On Mon, Jun 15, 2026 at 1:08 AM Cheng Pan <[email protected]> wrote:
>>>>>>
>>>>>>> Keep support for Spark last LTS + 2 latest minors sounds reasonable,
>>>>>>> if Iceberg could have a roughly 3 months release periods, which is 
>>>>>>> similar
>>>>>>> to Spark and Flink, otherwise, it requires a wider version range to 
>>>>>>> avoid
>>>>>>> version jumping when upgrading Iceberg.
>>>>>>>
>>>>>>> And I think whether to dropping Spark 4.0 support does not block
>>>>>>> adding Spark 4.2 support (with current RC state), I see at least two
>>>>>>> committer expressed the willingness on merging the spark 4.2 connector 
>>>>>>> code
>>>>>>> as-is to unblock subsequent developing, and no objections, so, merge 
>>>>>>> Spark
>>>>>>> 4.2 support first? And then start a vote thread for deleting Spark 4.0
>>>>>>> support, if necessary?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Cheng Pan
>>>>>>>
>>>>>>>
>>>>

Reply via email to