Hi all,

We discussed this topic in today's community (18-Nov-2026)  sync and here
is the summary:

   1. It is still unclear how users will adapt to the new Spark rapid
   release schedule so skipping a version may mean users of that minor version
   cannot use Iceberg runtime jars. Hence, we will keep adding every Spark
   minor rather than skipping versions, so please review the Spark 4.3 PR (
   #18749 <https://github.com/apache/iceberg/pull/18749>). It is not yet
   decided which Iceberg release will ship 4.3.
   2. Removing a Spark version should be decoupled from adding one. Each
   removal will be decided individually, based on factors such as how many
   users still depend on that version.
   3. We will keep the copy-per-version layout for now. I'll keep the
    shared-source draft PR (#19025
   <https://github.com/apache/iceberg/pull/19025>) up if you want to review
   it but let's discuss that in it's own thread
   <https://lists.apache.org/thread.html/tr572xz4fm6xl7btw8m87wjtg1x01fk5>.

If you have any suggestions or comments, please let me know.

Thanks,
Anurag

On Wed, Nov 18, 2026 at 12:12 PM Anurag Mantripragada <
[email protected]> wrote:

> Hi Alex,
>
> Each Iceberg runtime jar is built for one Spark minor, and running it on a
> different minor fails at runtime (See this thread
> <https://github.com/apache/iceberg/issues/17959#issuecomment-6121779365>
> for example). Until we publish 4.2 jars, production Spark 4.2 users would
> need to build their own from source.
>
> ~ Anurag
>
> On Wed, Nov 18, 2026 at 10:33 AM Alex Cruise <[email protected]> wrote:
>
>> On Wed, Nov 18, 2026 at 11:11 AM Russell Spitzer <
>> [email protected]> wrote:
>>
>>> We don't ship binaries for 4.2 so there is no package target for them. A
>>> user could just build their own lib and use it.
>>>
>>
>> OK, but could a Spark 4.2 user use the 4.1 binaries unmodified? Would
>> they see a dependency error at build time, and/or a principled refusal to
>> operate at runtime, and/or some kind of LinkageError at runtime?
>>
>> Thanks!
>>
>> -0xe1a
>>
>>
>>>
>>> On Wed, Nov 18, 2026 at 12:06 PM Alex Cruise <[email protected]> wrote:
>>>
>>>> I apologize if this should be obvious, but does the lack of official
>>>> Iceberg support for Spark 4.2 mean that Spark 4.2 users can't use Iceberg
>>>> at all, or merely that they can't benefit from enhancements available in
>>>> Spark 4.2?
>>>>
>>>> Thanks!
>>>>
>>>> -0xe1a
>>>>
>>>> On Wed, Nov 18, 2026 at 2:44 AM Cheng Pan <[email protected]> wrote:
>>>>
>>>>> Thanks for reviving this discussion and sharing your thoughts. But I
>>>>> think we had reached a consensus on the Spark version support policy when
>>>>> adding Spark 4.2 support: the latest LTS + 2 latest minors, and the
>>>>> situation still hold - Spark community runs with the planned version 
>>>>> policy
>>>>> and has no change since we made decision.
>>>>>
>>>>> And for "We have not released Spark 4.2 binaries yet.”, this was
>>>>> raised by Anton, I believe it should be up to him to assess whether the
>>>>> current Spark 4.2 connector meets his expectations for a binary release,
>>>>> and to determine whether it should be included in the next 1.12.x patch
>>>>> release or in 1.13.0.
>>>>>
>>>>> Thanks,
>>>>> Cheng Pan
>>>>>
>>>>>
>>>>>
>>>>> On Nov 18, 2026, at 16:41, Anurag Mantripragada via dev <
>>>>> [email protected]> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> Reviving this thread because Spark 4.3.0 is out and Cheng opened
>>>>> #18749 <https://github.com/apache/iceberg/pull/18749> to add it. We
>>>>> need to decide whether to add 4.3 or skip this version entirely.
>>>>>
>>>>> Where we are:
>>>>>
>>>>>    - We have not released Spark 4.2 binaries yet.
>>>>>    - Spark 4.3 is a non-LTS minor with a 6-month maintenance window. 
>>>>> #18749
>>>>>    <https://github.com/apache/iceberg/pull/18749>shows the diff is
>>>>>    small.
>>>>>    - Spark 4.4 is expected around January (branch is cut
>>>>>    <https://lists.apache.org/thread.html/8l3sv3ywtyk9j86r92gfmc3cj5xx1dyy>
>>>>>    in Spark already).
>>>>>
>>>>> I lean towards skipping 4.3 entirely and focusing on completing the 
>>>>> pending
>>>>> items in Spark 4.2 <https://github.com/apache/iceberg/issues/17959>.
>>>>> Depending on the timeline, we can look into adding Spark 4.4 after the 
>>>>> 1.13
>>>>> release.
>>>>>
>>>>> What do you all think?
>>>>>
>>>>> On Wed, Jun 17, 2026 at 3:56 PM Anton Okolnychyi <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Given that we still rely on internals of Spark in some places, I
>>>>>> think we will have to continue to have a module per Spark version. LTS +
>>>>>> rolling window of 2 minors seems reasonable to me too.
>>>>>>
>>>>>> +1 on getting 4.2 support in to unlock development and dropping 4.0
>>>>>> after that.
>>>>>>
>>>>>> - Anton
>>>>>>
>>>>>> пн, 15 черв. 2026 р. о 07:30 Anurag Mantripragada <
>>>>>> [email protected]> пише:
>>>>>>
>>>>>>> And I think whether to dropping Spark 4.0 support does not block
>>>>>>>> adding Spark 4.2 support (with current RC state)
>>>>>>>
>>>>>>>
>>>>>>> +1, we can review and merge Spark 4.2 support first. I will start a
>>>>>>> vote to remove Spark 4.0 after the community sync tomorrow.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Anurag
>>>>>>>
>>>>>>> On Mon, Jun 15, 2026 at 1:08 AM Cheng Pan <[email protected]> wrote:
>>>>>>>
>>>>>>>> Keep support for Spark last LTS + 2 latest minors sounds
>>>>>>>> reasonable, if Iceberg could have a roughly 3 months release periods, 
>>>>>>>> which
>>>>>>>> is similar to Spark and Flink, otherwise, it requires a wider version 
>>>>>>>> range
>>>>>>>> to avoid version jumping when upgrading Iceberg.
>>>>>>>>
>>>>>>>> And I think whether to dropping Spark 4.0 support does not block
>>>>>>>> adding Spark 4.2 support (with current RC state), I see at least two
>>>>>>>> committer expressed the willingness on merging the spark 4.2 connector 
>>>>>>>> code
>>>>>>>> as-is to unblock subsequent developing, and no objections, so, merge 
>>>>>>>> Spark
>>>>>>>> 4.2 support first? And then start a vote thread for deleting Spark 4.0
>>>>>>>> support, if necessary?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Cheng Pan
>>>>>>>>
>>>>>>>>
>>>>>

Reply via email to