Re: [Discuss] Spark versioning strategy with accelerated Spark releases.

[Anton Okolnychyi]

Given that we still rely on internals of Spark in some places, I think we
will have to continue to have a module per Spark version. LTS + rolling
window of 2 minors seems reasonable to me too.

+1 on getting 4.2 support in to unlock development and dropping 4.0 after
that.

- Anton

пн, 15 черв. 2026 р. о 07:30 Anurag Mantripragada <
anuragmantriprag...@apache.org> пише:

> And I think whether to dropping Spark 4.0 support does not block adding
>> Spark 4.2 support (with current RC state)
>
>
> +1, we can review and merge Spark 4.2 support first. I will start a vote
> to remove Spark 4.0 after the community sync tomorrow.
>
> Thanks,
> Anurag
>
> On Mon, Jun 15, 2026 at 1:08 AM Cheng Pan  wrote:
>
>> Keep support for Spark last LTS + 2 latest minors sounds reasonable, if
>> Iceberg could have a roughly 3 months release periods, which is similar to
>> Spark and Flink, otherwise, it requires a wider version range to avoid
>> version jumping when upgrading Iceberg.
>>
>> And I think whether to dropping Spark 4.0 support does not block adding
>> Spark 4.2 support (with current RC state), I see at least two committer
>> expressed the willingness on merging the spark 4.2 connector code as-is to
>> unblock subsequent developing, and no objections, so, merge Spark 4.2
>> support first? And then start a vote thread for deleting Spark 4.0 support,
>> if necessary?
>>
>> Thanks,
>> Cheng Pan
>>
>>

Re: [Discuss] Spark versioning strategy with accelerated Spark releases.

[Anurag Mantripragada]

>
> And I think whether to dropping Spark 4.0 support does not block adding
> Spark 4.2 support (with current RC state)


+1, we can review and merge Spark 4.2 support first. I will start a vote to
remove Spark 4.0 after the community sync tomorrow.

Thanks,
Anurag

On Mon, Jun 15, 2026 at 1:08 AM Cheng Pan  wrote:

> Keep support for Spark last LTS + 2 latest minors sounds reasonable, if
> Iceberg could have a roughly 3 months release periods, which is similar to
> Spark and Flink, otherwise, it requires a wider version range to avoid
> version jumping when upgrading Iceberg.
>
> And I think whether to dropping Spark 4.0 support does not block adding
> Spark 4.2 support (with current RC state), I see at least two committer
> expressed the willingness on merging the spark 4.2 connector code as-is to
> unblock subsequent developing, and no objections, so, merge Spark 4.2
> support first? And then start a vote thread for deleting Spark 4.0 support,
> if necessary?
>
> Thanks,
> Cheng Pan
>
>

Re: [Discuss] Spark versioning strategy with accelerated Spark releases.

[Cheng Pan]

Keep support for Spark last LTS + 2 latest minors sounds reasonable, if Iceberg 
could have a roughly 3 months release periods, which is similar to Spark and 
Flink, otherwise, it requires a wider version range to avoid version jumping 
when upgrading Iceberg.

And I think whether to dropping Spark 4.0 support does not block adding Spark 
4.2 support (with current RC state), I see at least two committer expressed the 
willingness on merging the spark 4.2 connector code as-is to unblock subsequent 
developing, and no objections, so, merge Spark 4.2 support first? And then 
start a vote thread for deleting Spark 4.0 support, if necessary?

Thanks,
Cheng Pan

Re: [Discuss] Spark versioning strategy with accelerated Spark releases.

[Szehon Ho]

Yea , we discussed a little bit on
https://github.com/apache/iceberg/pull/14984.  We are trying via DSV2 to
minimize internal Spark class use.  But, given the Ci challenges and lack
of confidence, it sounds like Option 1: last LTS + 2 latest minors, is more
likely.  (Ie, we can start thinking about removing Spark 4.0 when adopting
4.2).  Love to hear thoughts if anyone thinks otherwise.

Thanks
Szehon

On Mon, Jun 15, 2026 at 12:12 AM Cheng Pan  wrote:

> > would an Iceberg-Spark runtime jar built for 4.2 also work on Spark 4.5?
>
> I am not optimistic about this unless Iceberg adds a cross-version
> verification workflow - based on the Iceberg Runtime Jar compiled for Spark
> 4.2, running smoke tests on higher versions of Spark distributions. This
> would consume a lot of CI resources, clearly defeating the purpose.
>
> Note, Iceberg uses a lot of Spark internal catalyst API, which has no
> compatibility guarantee, the SPIP only ensures Public API and User Facing
> behavior has no breaking changes across minor releases.
>
> Thanks,
> Cheng Pan
>
>
>
> On Jun 3, 2026, at 14:04, Anurag Mantripragada <
> anuragmantriprag...@apache.org> wrote:
>
> Hi Kurtis, Szehon, and Jingyi,
>
> Kurtis, to your point, we do need to support two major versions (3.5 and
> 4.x) during this transition period. I hope Szehon’s explanation clarified
> your questions regarding API changes and the frequency of Spark minor
> version upgrades moving forward.
>
> Szehon and Jingyi, thank you for providing more context from the Spark
> side. Since CI is becoming a bottleneck, we should restrict support to one
> LTS and two minor versions.
>
> Regarding compatibility: if we have versions 4.1, 4.2, and 4.5 (the LTS
> and final version in the 4.x line), would an Iceberg-Spark runtime jar
> built for 4.2 also work on Spark 4.5?
>
> ~ Anurag
>
>
> On Mon, Jun 1, 2026 at 7:25 PM Szehon Ho  wrote:
>
>> Yes , the policy is that the same Spark major version should now avoid
>> breaking change: https://spark.apache.org/versioning-policy.html.   One
>> additional detail is that LTS is going to be the last minor of every major
>> version release.
>>
>> But there is still the issue.  If Iceberg is eager to consume new Spark
>> API's, then we still need multiple Spark-Iceberg versions.  Ie, if we
>> consume new DSV2 features in Spark 4.3, Iceberg-Spark 4.3 won't run with
>> Spark 4.2.
>>
>> There is some advantage of Spark major version backward compatibility.
>> One Iceberg-Spark version will in theory work with future Spark versions of
>> the same major.  For example, we can have one Iceberg-Spark jar for
>> Iceberg-Spark 4.0 (or any 'selected minor' as Anurag said), and in
>> theory it will work with all future Spark versions in the same major (ie,
>> 4.1, 4.2).  So something like 1. last Spark LTS, 2. first Spark version of
>> current major, 3. last Spark version of current major does increase the
>> coverage, but becomes a bit more complex for users and community (more
>> cross-version testing needed -- needing more CI, and users need to
>> understand it).
>>
>> Thanks,
>> Szehon
>>
>>
>> On Mon, Jun 1, 2026 at 4:52 PM  wrote:
>>
>>> Hi Anurag,
>>>
>>> Thank you for calling this out, TIL about Spark quarterly updates!
>>>
>>> A few naive question, do we need to support more than 2 major Spark
>>> versions in CI?
>>> Is it correct to assume API  interface changes should only happen across
>>> major version updates?
>>> Is the Spark community doing this with the built in assumption that
>>> minor version upgrades will be relatively easy going forward?
>>>
>>> Thank you,
>>> Kurtis C. Wright
>>>
>>> On Jun 1, 2026, at 15:25, Anurag Mantripragada <
>>> anuragmantriprag...@apache.org> wrote:
>>>
>>> 
>>> Hi all,
>>>
>>>
>>>
>>> With Spark 3.4 now removed  after the 1.11 release, and Spark
>>> community proposing
>>> 
>>> quarterly minor releases, I'd like to start a discussion on how Iceberg
>>> should adapt its Spark version support strategy going forward.
>>>
>>> *Where we are today   *
>>>
>>>
>>> On main we support three Spark versions: 3.5, 4.0, and 4.1. Our CI
>>> matrix runs 16 jobs across these which is already becoming a bottleneck
>>> .
>>>
>>>
>>>
>>> Historically, we have deprecated and removed Spark versions in an ad-hoc
>>> fashion. This worked with ~2 Spark minors per year, but with the new
>>> quarterly releases of spark it may not scale.
>>>
>>> As per the Spark SPIP we have this coming next
>>>
>>> Date
>>> Release
>>> Maintenance
>>> Notes
>>> April 2026
>>> 4.2
>>> 6 months
>>> Non-LTS (Past)
>>> July 2026
>>> 4.3
>>> 6 months
>>> Non-LTS
>>> October 2026
>>> 4.4
>>> 6 months
>>> Non-LTS
>>> January 2027
>>> 4.5
>>> 18 months
>>> LTS
>>> April 2027
>>> 5.0
>>> —
>>> Major
>>>
>>> This means 4 Spark minors per year, each with only a 6-month maintenance
>>> window, and 

Re: [Discuss] Spark versioning strategy with accelerated Spark releases.

[Cheng Pan]

> would an Iceberg-Spark runtime jar built for 4.2 also work on Spark 4.5?

I am not optimistic about this unless Iceberg adds a cross-version verification 
workflow - based on the Iceberg Runtime Jar compiled for Spark 4.2, running 
smoke tests on higher versions of Spark distributions. This would consume a lot 
of CI resources, clearly defeating the purpose.

Note, Iceberg uses a lot of Spark internal catalyst API, which has no 
compatibility guarantee, the SPIP only ensures Public API and User Facing 
behavior has no breaking changes across minor releases.

Thanks,
Cheng Pan



> On Jun 3, 2026, at 14:04, Anurag Mantripragada 
>  wrote:
> 
> Hi Kurtis, Szehon, and Jingyi,
> 
> Kurtis, to your point, we do need to support two major versions (3.5 and 4.x) 
> during this transition period. I hope Szehon’s explanation clarified your 
> questions regarding API changes and the frequency of Spark minor version 
> upgrades moving forward.
> 
> Szehon and Jingyi, thank you for providing more context from the Spark side. 
> Since CI is becoming a bottleneck, we should restrict support to one LTS and 
> two minor versions.
> 
> Regarding compatibility: if we have versions 4.1, 4.2, and 4.5 (the LTS and 
> final version in the 4.x line), would an Iceberg-Spark runtime jar built for 
> 4.2 also work on Spark 4.5?
> 
> ~ Anurag
> 
> 
> On Mon, Jun 1, 2026 at 7:25 PM Szehon Ho  > wrote:
>> Yes , the policy is that the same Spark major version should now avoid 
>> breaking change: https://spark.apache.org/versioning-policy.html.   One 
>> additional detail is that LTS is going to be the last minor of every major 
>> version release.
>> 
>> But there is still the issue.  If Iceberg is eager to consume new Spark 
>> API's, then we still need multiple Spark-Iceberg versions.  Ie, if we 
>> consume new DSV2 features in Spark 4.3, Iceberg-Spark 4.3 won't run with 
>> Spark 4.2.
>> 
>> There is some advantage of Spark major version backward compatibility.  One 
>> Iceberg-Spark version will in theory work with future Spark versions of the 
>> same major.  For example, we can have one Iceberg-Spark jar for 
>> Iceberg-Spark 4.0 (or any 'selected minor' as Anurag said), and in theory it 
>> will work with all future Spark versions in the same major (ie, 4.1, 4.2).  
>> So something like 1. last Spark LTS, 2. first Spark version of current 
>> major, 3. last Spark version of current major does increase the coverage, 
>> but becomes a bit more complex for users and community (more cross-version 
>> testing needed -- needing more CI, and users need to understand it).
>> 
>> Thanks,
>> Szehon
>> 
>> 
>> On Mon, Jun 1, 2026 at 4:52 PM > > wrote:
>>> Hi Anurag,
>>> 
>>> Thank you for calling this out, TIL about Spark quarterly updates!
>>> 
>>> A few naive question, do we need to support more than 2 major Spark 
>>> versions in CI? 
>>> Is it correct to assume API  interface changes should only happen across 
>>> major version updates? 
>>> Is the Spark community doing this with the built in assumption that minor 
>>> version upgrades will be relatively easy going forward?
>>> 
>>> Thank you,
>>> Kurtis C. Wright
>>> 
 On Jun 1, 2026, at 15:25, Anurag Mantripragada 
 mailto:anuragmantriprag...@apache.org>> 
 wrote:
 
 
 Hi all,  


 
 With Spark 3.4 now removed  after the 1.11 release, and Spark 
 community proposing 
 
  quarterly minor releases, I'd like to start a discussion on how Iceberg 
 should adapt its Spark version support strategy going forward.
 
 Where we are today 

   
 On main we support three Spark versions: 3.5, 4.0, and 4.1. Our CI matrix 
 runs 16 jobs across these which is already becoming a bottleneck 
 . 


 
 Historically, we have deprecated and removed Spark versions in an ad-hoc 
 fashion. This worked with ~2 Spark minors per year, but with the new 
 quarterly releases of spark it may not scale.
 
 As per the Spark SPIP we have this coming next 
 
 Date
 Release
 Maintenance
 Notes
 April 2026
 4.2
 6 months
 Non-LTS (Past)
 July 2026
 4.3
 6 months
 Non-LTS
 October 2026
 4.4
 6 months
 Non-LTS
 January 2027
 4.5
 18 mont

Re: [Discuss] Spark versioning strategy with accelerated Spark releases.

[Szehon Ho]

Add back Jingyi (author of the SPIP), who is not on this list :).

Yes, exactly, Iceberg-Spark 4.x would work in theory with anything for y >
x.

I think this corresponds to Option 2 in your list.  But we would be
bottlenecked by CI to verify it.  In terms of CI, it may not be better than
just increasing the number of supported Spark versions.  As now with AI,
keeping code in sync is no longer really a bottleneck as much as CI.

Thanks
Szehon

On Tue, Jun 2, 2026 at 11:05 PM Anurag Mantripragada <
anuragmantriprag...@apache.org> wrote:

> Hi Kurtis, Szehon, and Jingyi,
>
> Kurtis, to your point, we do need to support two major versions (3.5 and
> 4.x) during this transition period. I hope Szehon’s explanation clarified
> your questions regarding API changes and the frequency of Spark minor
> version upgrades moving forward.
>
> Szehon and Jingyi, thank you for providing more context from the Spark
> side. Since CI is becoming a bottleneck, we should restrict support to one
> LTS and two minor versions.
>
> Regarding compatibility: if we have versions 4.1, 4.2, and 4.5 (the LTS
> and final version in the 4.x line), would an Iceberg-Spark runtime jar
> built for 4.2 also work on Spark 4.5?
>
> ~ Anurag
>
>
> On Mon, Jun 1, 2026 at 7:25 PM Szehon Ho  wrote:
>
>> Yes , the policy is that the same Spark major version should now avoid
>> breaking change: https://spark.apache.org/versioning-policy.html.   One
>> additional detail is that LTS is going to be the last minor of every major
>> version release.
>>
>> But there is still the issue.  If Iceberg is eager to consume new Spark
>> API's, then we still need multiple Spark-Iceberg versions.  Ie, if we
>> consume new DSV2 features in Spark 4.3, Iceberg-Spark 4.3 won't run with
>> Spark 4.2.
>>
>> There is some advantage of Spark major version backward compatibility.
>> One Iceberg-Spark version will in theory work with future Spark versions of
>> the same major.  For example, we can have one Iceberg-Spark jar for
>> Iceberg-Spark 4.0 (or any 'selected minor' as Anurag said), and in
>> theory it will work with all future Spark versions in the same major (ie,
>> 4.1, 4.2).  So something like 1. last Spark LTS, 2. first Spark version of
>> current major, 3. last Spark version of current major does increase the
>> coverage, but becomes a bit more complex for users and community (more
>> cross-version testing needed -- needing more CI, and users need to
>> understand it).
>>
>> Thanks,
>> Szehon
>>
>>
>> On Mon, Jun 1, 2026 at 4:52 PM  wrote:
>>
>>> Hi Anurag,
>>>
>>> Thank you for calling this out, TIL about Spark quarterly updates!
>>>
>>> A few naive question, do we need to support more than 2 major Spark
>>> versions in CI?
>>> Is it correct to assume API  interface changes should only happen across
>>> major version updates?
>>> Is the Spark community doing this with the built in assumption that
>>> minor version upgrades will be relatively easy going forward?
>>>
>>> Thank you,
>>> Kurtis C. Wright
>>>
>>> On Jun 1, 2026, at 15:25, Anurag Mantripragada <
>>> anuragmantriprag...@apache.org> wrote:
>>>
>>> 
>>> Hi all,
>>>
>>>
>>>
>>> With Spark 3.4 now removed  after the 1.11 release, and Spark
>>> community proposing
>>> 
>>> quarterly minor releases, I'd like to start a discussion on how Iceberg
>>> should adapt its Spark version support strategy going forward.
>>>
>>> *Where we are today   *
>>>
>>>
>>> On main we support three Spark versions: 3.5, 4.0, and 4.1. Our CI
>>> matrix runs 16 jobs across these which is already becoming a bottleneck
>>> .
>>>
>>>
>>>
>>> Historically, we have deprecated and removed Spark versions in an ad-hoc
>>> fashion. This worked with ~2 Spark minors per year, but with the new
>>> quarterly releases of spark it may not scale.
>>>
>>> As per the Spark SPIP we have this coming next
>>>
>>> Date
>>>
>>> Release
>>>
>>> Maintenance
>>>
>>> Notes
>>>
>>> April 2026
>>>
>>> 4.2
>>>
>>> 6 months
>>>
>>> Non-LTS (Past)
>>>
>>> July 2026
>>>
>>> 4.3
>>>
>>> 6 months
>>>
>>> Non-LTS
>>>
>>> October 2026
>>>
>>> 4.4
>>>
>>> 6 months
>>>
>>> Non-LTS
>>>
>>> January 2027
>>>
>>> 4.5
>>>
>>> 18 months
>>>
>>> LTS
>>>
>>> April 2027
>>>
>>> 5.0
>>>
>>> —
>>>
>>> Major
>>>
>>> This means 4 Spark minors per year, each with only a 6-month maintenance
>>> window, and an LTS roughly once a year.
>>>
>>> I propose we adopt a policy instead of making ad-hoc decisions. Some
>>> options I see:
>>>
>>>
>>>1. *LTS + rolling window of 2 minors*: Support the current Spark LTS
>>>and the 2 most recent minors. For example, when 4.2 GA ships, add it and
>>>deprecate 4.0 and when 4.3 ships, add it and deprecate 4.1. This provides
>>>predictable cadence but also means a version add/drop every quarter.
>>>
>>>2. *LTS + selective minors*:  Support the Spark LTS and choose
>>>

Re: [Discuss] Spark versioning strategy with accelerated Spark releases.

[Anurag Mantripragada]

Hi Kurtis, Szehon, and Jingyi,

Kurtis, to your point, we do need to support two major versions (3.5 and
4.x) during this transition period. I hope Szehon’s explanation clarified
your questions regarding API changes and the frequency of Spark minor
version upgrades moving forward.

Szehon and Jingyi, thank you for providing more context from the Spark
side. Since CI is becoming a bottleneck, we should restrict support to one
LTS and two minor versions.

Regarding compatibility: if we have versions 4.1, 4.2, and 4.5 (the LTS and
final version in the 4.x line), would an Iceberg-Spark runtime jar built
for 4.2 also work on Spark 4.5?

~ Anurag


On Mon, Jun 1, 2026 at 7:25 PM Szehon Ho  wrote:

> Yes , the policy is that the same Spark major version should now avoid
> breaking change: https://spark.apache.org/versioning-policy.html.   One
> additional detail is that LTS is going to be the last minor of every major
> version release.
>
> But there is still the issue.  If Iceberg is eager to consume new Spark
> API's, then we still need multiple Spark-Iceberg versions.  Ie, if we
> consume new DSV2 features in Spark 4.3, Iceberg-Spark 4.3 won't run with
> Spark 4.2.
>
> There is some advantage of Spark major version backward compatibility.
> One Iceberg-Spark version will in theory work with future Spark versions of
> the same major.  For example, we can have one Iceberg-Spark jar for
> Iceberg-Spark 4.0 (or any 'selected minor' as Anurag said), and in
> theory it will work with all future Spark versions in the same major (ie,
> 4.1, 4.2).  So something like 1. last Spark LTS, 2. first Spark version of
> current major, 3. last Spark version of current major does increase the
> coverage, but becomes a bit more complex for users and community (more
> cross-version testing needed -- needing more CI, and users need to
> understand it).
>
> Thanks,
> Szehon
>
>
> On Mon, Jun 1, 2026 at 4:52 PM  wrote:
>
>> Hi Anurag,
>>
>> Thank you for calling this out, TIL about Spark quarterly updates!
>>
>> A few naive question, do we need to support more than 2 major Spark
>> versions in CI?
>> Is it correct to assume API  interface changes should only happen across
>> major version updates?
>> Is the Spark community doing this with the built in assumption that minor
>> version upgrades will be relatively easy going forward?
>>
>> Thank you,
>> Kurtis C. Wright
>>
>> On Jun 1, 2026, at 15:25, Anurag Mantripragada <
>> anuragmantriprag...@apache.org> wrote:
>>
>> 
>> Hi all,
>>
>>
>>
>> With Spark 3.4 now removed  after the 1.11 release, and Spark
>> community proposing
>> 
>> quarterly minor releases, I'd like to start a discussion on how Iceberg
>> should adapt its Spark version support strategy going forward.
>>
>> *Where we are today   *
>>
>>
>> On main we support three Spark versions: 3.5, 4.0, and 4.1. Our CI matrix
>> runs 16 jobs across these which is already becoming a bottleneck
>> .
>>
>>
>>
>> Historically, we have deprecated and removed Spark versions in an ad-hoc
>> fashion. This worked with ~2 Spark minors per year, but with the new
>> quarterly releases of spark it may not scale.
>>
>> As per the Spark SPIP we have this coming next
>>
>> Date
>>
>> Release
>>
>> Maintenance
>>
>> Notes
>>
>> April 2026
>>
>> 4.2
>>
>> 6 months
>>
>> Non-LTS (Past)
>>
>> July 2026
>>
>> 4.3
>>
>> 6 months
>>
>> Non-LTS
>>
>> October 2026
>>
>> 4.4
>>
>> 6 months
>>
>> Non-LTS
>>
>> January 2027
>>
>> 4.5
>>
>> 18 months
>>
>> LTS
>>
>> April 2027
>>
>> 5.0
>>
>> —
>>
>> Major
>>
>> This means 4 Spark minors per year, each with only a 6-month maintenance
>> window, and an LTS roughly once a year.
>>
>> I propose we adopt a policy instead of making ad-hoc decisions. Some
>> options I see:
>>
>>
>>1. *LTS + rolling window of 2 minors*: Support the current Spark LTS
>>and the 2 most recent minors. For example, when 4.2 GA ships, add it and
>>deprecate 4.0 and when 4.3 ships, add it and deprecate 4.1. This provides
>>predictable cadence but also means a version add/drop every quarter.
>>
>>2. *LTS + selective minors*:  Support the Spark LTS and choose minors
>>that have meaningful DSv2 API changes, skipping versions that are
>>incremental. More flexible but less predictable for users. (This is the
>>current strategy)
>>
>>
>> Any strategy must account for CI infra ceiling too. Recent improvements
>>  have helped, but I
>> think we should support at most 3 versions to keep this under control.
>>
>>
>>

Re: [Discuss] Spark versioning strategy with accelerated Spark releases.

[Szehon Ho]

I am cc'ing my teammate Jingyi who drove the Spark proposal, if we need any
other help from Spark side.

>From Spark side, I believe the idea is that most users will stay on LTS
versions.  Advanced users will try newer Spark major version (ie Spark 4.x)
for new features, but will probably be ok to upgrade aggressively until
that major graduates to an LTS.  Especially as Spark will try no breaking
user-side changes within the major version.  (Jingyi can correct me if I am
wrong).

Is max 3 Spark versions a hard limit due to CI restriction?  If so,
then probably Option 1: LTS + 2 rolling minor is easier.  Option 2: LTS +
selected minor will be good for increasing coverage due to Spark API
backward-compatibility, as I mentioned above, but is definitely a bit more
confusing for users/community.

Thanks,
Szehon

On Mon, Jun 1, 2026 at 7:25 PM Szehon Ho  wrote:

> Yes , the policy is that the same Spark major version should now avoid
> breaking change: https://spark.apache.org/versioning-policy.html.   One
> additional detail is that LTS is going to be the last minor of every major
> version release.
>
> But there is still the issue.  If Iceberg is eager to consume new Spark
> API's, then we still need multiple Spark-Iceberg versions.  Ie, if we
> consume new DSV2 features in Spark 4.3, Iceberg-Spark 4.3 won't run with
> Spark 4.2.
>
> There is some advantage of Spark major version backward compatibility.
> One Iceberg-Spark version will in theory work with future Spark versions of
> the same major.  For example, we can have one Iceberg-Spark jar for
> Iceberg-Spark 4.0 (or any 'selected minor' as Anurag said), and in
> theory it will work with all future Spark versions in the same major (ie,
> 4.1, 4.2).  So something like 1. last Spark LTS, 2. first Spark version of
> current major, 3. last Spark version of current major does increase the
> coverage, but becomes a bit more complex for users and community (more
> cross-version testing needed -- needing more CI, and users need to
> understand it).
>
> Thanks,
> Szehon
>
>
> On Mon, Jun 1, 2026 at 4:52 PM  wrote:
>
>> Hi Anurag,
>>
>> Thank you for calling this out, TIL about Spark quarterly updates!
>>
>> A few naive question, do we need to support more than 2 major Spark
>> versions in CI?
>> Is it correct to assume API  interface changes should only happen across
>> major version updates?
>> Is the Spark community doing this with the built in assumption that minor
>> version upgrades will be relatively easy going forward?
>>
>> Thank you,
>> Kurtis C. Wright
>>
>> On Jun 1, 2026, at 15:25, Anurag Mantripragada <
>> anuragmantriprag...@apache.org> wrote:
>>
>> 
>> Hi all,
>>
>>
>>
>> With Spark 3.4 now removed  after the 1.11 release, and Spark
>> community proposing
>> 
>> quarterly minor releases, I'd like to start a discussion on how Iceberg
>> should adapt its Spark version support strategy going forward.
>>
>> *Where we are today   *
>>
>>
>> On main we support three Spark versions: 3.5, 4.0, and 4.1. Our CI matrix
>> runs 16 jobs across these which is already becoming a bottleneck
>> .
>>
>>
>>
>> Historically, we have deprecated and removed Spark versions in an ad-hoc
>> fashion. This worked with ~2 Spark minors per year, but with the new
>> quarterly releases of spark it may not scale.
>>
>> As per the Spark SPIP we have this coming next
>>
>> Date
>>
>> Release
>>
>> Maintenance
>>
>> Notes
>>
>> April 2026
>>
>> 4.2
>>
>> 6 months
>>
>> Non-LTS (Past)
>>
>> July 2026
>>
>> 4.3
>>
>> 6 months
>>
>> Non-LTS
>>
>> October 2026
>>
>> 4.4
>>
>> 6 months
>>
>> Non-LTS
>>
>> January 2027
>>
>> 4.5
>>
>> 18 months
>>
>> LTS
>>
>> April 2027
>>
>> 5.0
>>
>> —
>>
>> Major
>>
>> This means 4 Spark minors per year, each with only a 6-month maintenance
>> window, and an LTS roughly once a year.
>>
>> I propose we adopt a policy instead of making ad-hoc decisions. Some
>> options I see:
>>
>>
>>1. *LTS + rolling window of 2 minors*: Support the current Spark LTS
>>and the 2 most recent minors. For example, when 4.2 GA ships, add it and
>>deprecate 4.0 and when 4.3 ships, add it and deprecate 4.1. This provides
>>predictable cadence but also means a version add/drop every quarter.
>>
>>2. *LTS + selective minors*:  Support the Spark LTS and choose minors
>>that have meaningful DSv2 API changes, skipping versions that are
>>incremental. More flexible but less predictable for users. (This is the
>>current strategy)
>>
>>
>> Any strategy must account for CI infra ceiling too. Recent improvements
>>  have helped, but I
>> think we should support at most 3 versions to keep this under control.
>>
>>
>>

Re: [Discuss] Spark versioning strategy with accelerated Spark releases.

[Szehon Ho]

Yes , the policy is that the same Spark major version should now avoid
breaking change: https://spark.apache.org/versioning-policy.html.   One
additional detail is that LTS is going to be the last minor of every major
version release.

But there is still the issue.  If Iceberg is eager to consume new Spark
API's, then we still need multiple Spark-Iceberg versions.  Ie, if we
consume new DSV2 features in Spark 4.3, Iceberg-Spark 4.3 won't run with
Spark 4.2.

There is some advantage of Spark major version backward compatibility.  One
Iceberg-Spark version will in theory work with future Spark versions of the
same major.  For example, we can have one Iceberg-Spark jar for
Iceberg-Spark 4.0 (or any 'selected minor' as Anurag said), and in
theory it will work with all future Spark versions in the same major (ie,
4.1, 4.2).  So something like 1. last Spark LTS, 2. first Spark version of
current major, 3. last Spark version of current major does increase the
coverage, but becomes a bit more complex for users and community (more
cross-version testing needed -- needing more CI, and users need to
understand it).

Thanks,
Szehon


On Mon, Jun 1, 2026 at 4:52 PM  wrote:

> Hi Anurag,
>
> Thank you for calling this out, TIL about Spark quarterly updates!
>
> A few naive question, do we need to support more than 2 major Spark
> versions in CI?
> Is it correct to assume API  interface changes should only happen across
> major version updates?
> Is the Spark community doing this with the built in assumption that minor
> version upgrades will be relatively easy going forward?
>
> Thank you,
> Kurtis C. Wright
>
> On Jun 1, 2026, at 15:25, Anurag Mantripragada <
> anuragmantriprag...@apache.org> wrote:
>
> 
> Hi all,
>
>
>
> With Spark 3.4 now removed  after the 1.11 release, and Spark
> community proposing
> 
> quarterly minor releases, I'd like to start a discussion on how Iceberg
> should adapt its Spark version support strategy going forward.
>
> *Where we are today   *
>
>
> On main we support three Spark versions: 3.5, 4.0, and 4.1. Our CI matrix
> runs 16 jobs across these which is already becoming a bottleneck
> .
>
>
>
> Historically, we have deprecated and removed Spark versions in an ad-hoc
> fashion. This worked with ~2 Spark minors per year, but with the new
> quarterly releases of spark it may not scale.
>
> As per the Spark SPIP we have this coming next
>
> Date
>
> Release
>
> Maintenance
>
> Notes
>
> April 2026
>
> 4.2
>
> 6 months
>
> Non-LTS (Past)
>
> July 2026
>
> 4.3
>
> 6 months
>
> Non-LTS
>
> October 2026
>
> 4.4
>
> 6 months
>
> Non-LTS
>
> January 2027
>
> 4.5
>
> 18 months
>
> LTS
>
> April 2027
>
> 5.0
>
> —
>
> Major
>
> This means 4 Spark minors per year, each with only a 6-month maintenance
> window, and an LTS roughly once a year.
>
> I propose we adopt a policy instead of making ad-hoc decisions. Some
> options I see:
>
>
>1. *LTS + rolling window of 2 minors*: Support the current Spark LTS
>and the 2 most recent minors. For example, when 4.2 GA ships, add it and
>deprecate 4.0 and when 4.3 ships, add it and deprecate 4.1. This provides
>predictable cadence but also means a version add/drop every quarter.
>
>2. *LTS + selective minors*:  Support the Spark LTS and choose minors
>that have meaningful DSv2 API changes, skipping versions that are
>incremental. More flexible but less predictable for users. (This is the
>current strategy)
>
>
> Any strategy must account for CI infra ceiling too. Recent improvements
>  have helped, but I think
> we should support at most 3 versions to keep this under control.
>
>
>

Re: [Discuss] Spark versioning strategy with accelerated Spark releases.

[kurtiscwright]

Hi Anurag,Thank you for calling this out, TIL about Spark quarterly updates!A few naive question, do we need to support more than 2 major Spark versions in CI? Is it correct to assume API  interface changes should only happen across major version updates? Is the Spark community doing this with the built in assumption that minor version upgrades will be relatively easy going forward?Thank you,Kurtis C. WrightOn Jun 1, 2026, at 15:25, Anurag Mantripragada  wrote:Hi all,                                                                                                                                                                                            With Spark 3.4 now removed after the 1.11 release, and Spark community proposing quarterly minor releases, I'd like to start a discussion on how Iceberg should adapt its Spark version support strategy going forward.Where we are today                                                                                                                                      On main we support three Spark versions: 3.5, 4.0, and 4.1. Our CI matrix runs 16 jobs across these which is already becoming a bottleneck.                                                                                                                                                           Historically, we have deprecated and removed Spark versions in an ad-hoc fashion. This worked with ~2 Spark minors per year, but with the new quarterly releases of spark it may not scale.As per the Spark SPIP we have this coming next DateReleaseMaintenanceNotesApril 20264.26 monthsNon-LTS (Past)July 20264.36 monthsNon-LTSOctober 20264.46 monthsNon-LTSJanuary 20274.518 monthsLTSApril 20275.0—Major   This means 4 Spark minors per year, each with only a 6-month maintenance window, and an LTS roughly once a year.I propose we adopt a policy instead of making ad-hoc decisions. Some options I see:                                                                                  LTS + rolling window of 2 minors: Support the current Spark LTS and the 2 most recent minors. For example, when 4.2 GA ships, add it and deprecate 4.0 and when 4.3 ships, add it and deprecate 4.1. This provides predictable cadence but also means a version add/drop every quarter.                                                        LTS + selective minors:  Support the Spark LTS and choose minors that have meaningful DSv2 API changes, skipping versions that are incremental. More flexible but less predictable for users. (This is the current strategy)                                                                                                    Any strategy must account for CI infra ceiling too. Recent improvements have helped, but I think we should support at most 3 versions to keep this under control.