The Spark 3.4 removal PR has been merged! ( https://github.com/apache/iceberg/pull/14122) Spark 3.4 is now considered "End of Life", https://iceberg.apache.org/multi-engine-support/#apache-spark And the last Iceberg support is version 1.11.0
Thanks everyone! On Thu, Jan 8, 2026 at 6:49 PM Manu Zhang <[email protected]> wrote: > Totally agreed. I'd propose we formalize this process in the doc. > > 1. Mark Spark X / Flink Y as deprecated for Iceberg Z in multi-engine > support doc[1]. > 2. Release Z with Spark X / Flink Y. > 3. Remove Spark X / Flink Y support as part of post action of release Z. > 4. Update multi-engine support doc accordingly. > > Regards, > Manu > > On Fri, Jan 9, 2026 at 3:19 AM Kevin Liu <[email protected]> wrote: > >> Hey folks, >> >> Picking this thread up again. I've gathered feedback from the community, >> and here's what I think is the best course of action. >> >> >> *Current State*We've already marked Spark 3.4 as "Deprecated" at >> https://iceberg.apache.org/multi-engine-support/#apache-spark. This >> means Spark 3.4 is no longer actively maintained as of 1.10.x. Backports >> can still be added, but the *community will not invest effort in >> achieving feature parity*. We recommend users move to a newer version: >> Spark 3.5 or Spark 4.x. >> >> Since the 1.10 release, there have been 34 backport PRs [1] currently on >> the main branch. Since Spark 3.4 is already marked as "Deprecated," *there's >> no obligation for the community to ensure all changes are backported to >> Spark 3.4.* I propose we release 1.11 with the available changes and >> fully remove Spark 3.4 support after the release. If further changes are >> needed for Spark 3.4 afterward, they can go into the 1.11.x branch as patch >> releases. >> >> >> *Proposed Actions*1. Allow Spark 3.4 backports with no obligation to >> ensure all features are backported >> 2. Release 1.11 with Spark 3.4 support >> 3. Fully remove the `spark/v3.4` folder immediately after 1.11 >> 4. Mark Spark 3.4 as "End-of-life" with 1.11 as the last supported version >> >> Please share your thoughts. I'll also link this to the vote thread >> on removing Spark 3.4. >> >> Best, >> Kevin Liu >> >> >> [1] git log --pretty=oneline apache-iceberg-1.10.1..main -- spark/v3.4/ | >> nl >> >> On Mon, Oct 13, 2025 at 10:03 PM huaxin gao <[email protected]> >> wrote: >> >>> Instead of removing Spark 3.4 in 1.11, could we keep it for a couple >>> more versions? >>> >>> Thanks, >>> Huaxin >>> >>> On Mon, Oct 13, 2025 at 9:57 AM Parth Chandra <[email protected]> wrote: >>> >>>> I'm wondering if it is too late to request just one more release of >>>> Iceberg with Spark 3.4 support? I have a PR open >>>> https://github.com/apache/iceberg/pull/13786 that addresses a whole >>>> host of issues with using Comet and Iceberg. This completes the work >>>> started by Huaxin in https://github.com/apache/iceberg/pull/13378. >>>> At the moment Comet still supports Spark 3.4 and the PR accordingly >>>> updates the Spark 3.4 module as well. If we deprecate Spark 3.4, users of >>>> Spark 3.4 who want to use Comet will have to build their own. It's not the >>>> end of the world, Comet already provides a patch and instructions on how to >>>> do this. It is, however, less than ideal. >>>> I know that voting has already started (and is practically over) on >>>> this issue, but I figured it doesn't hurt to try. >>>> >>>> Parth >>>> >>>> On Tue, Oct 7, 2025 at 7:49 AM Kevin Liu <[email protected]> wrote: >>>> >>>>> The PR has been merged (https://github.com/apache/iceberg/pull/14099), >>>>> and I can see the change reflected on the website, >>>>> https://iceberg.apache.org/multi-engine-support/#apache-spark >>>>> Thanks everyone for the review. >>>>> >>>>> I'll start a separate thread for removing spark 3.4 from the repo. >>>>> >>>>> Best, >>>>> Kevin Liu >>>>> >>>>> >>>>> >>>>> On Mon, Oct 6, 2025 at 3:03 PM Kevin Liu <[email protected]> >>>>> wrote: >>>>> >>>>>> Hey folks, >>>>>> >>>>>> All (except 1) recent contributors to spark/v3.4 have responded on >>>>>> the slack thread above and support removing spark 3.4. The last >>>>>> contributor >>>>>> has not responded yet. >>>>>> >>>>>> If there are no objections, I plan to merge this PR to change the >>>>>> status for Spark 3.4 from "Maintained" -> "Deprecated" . >>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>> Thanks Eduard for the review. >>>>>> >>>>>> Best, >>>>>> Kevin Liu >>>>>> >>>>>> On Mon, Sep 29, 2025 at 9:52 AM Kevin Liu <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Started a slack thread, >>>>>>> https://apache-iceberg.slack.com/archives/C03LG1D563F/p1759163920874249 >>>>>>> Based on the recently commit activities, I messaged (on slack and >>>>>>> pr) 4 contributors who are actively backporting Spark 3.4. >>>>>>> >>>>>>> Best, >>>>>>> Kevin Liu >>>>>>> >>>>>>> On Mon, Sep 29, 2025 at 9:32 AM Kevin Liu <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Bumping this thread. >>>>>>>> Since there's consensus to mark Spark 3.4 as deprecated, let's >>>>>>>> merge this PR to change the status on the docs, >>>>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>>>> I'll start a new thread for the decision to mark Spark 3.4 as >>>>>>>> end-of-life and for removal from the codebase. >>>>>>>> >>>>>>>> Best, >>>>>>>> Kevin Liu >>>>>>>> >>>>>>>> On Mon, Sep 22, 2025 at 11:13 AM Steve Loughran >>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>>> making hadoop 3.4.1 the base hadoop version has sort of made >>>>>>>>> iceberg incompatible with spark 3.4, it just doesn't surface very >>>>>>>>> much. >>>>>>>>> >>>>>>>>> My PR, https://github.com/apache/iceberg/pull/10233 , which uses >>>>>>>>> a hadoop 3.4.1 BulkDelete call not in the spark 3.4 classpath flags >>>>>>>>> this: >>>>>>>>> the spark 3.4 tests fail with classpath errors. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, 19 Sept 2025 at 22:02, Kevin Liu <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks Anton and Eduard. I'm ok with being more aggressive with >>>>>>>>>> the deprecation schedule. >>>>>>>>>> Looking at the git history for `spark/v3.4/` >>>>>>>>>> <https://github.com/apache/iceberg/commits/main/spark/v3.4>, >>>>>>>>>> there are 5 new commits since the 1.10 release. Only 1 commit ( >>>>>>>>>> 3bbdee9 >>>>>>>>>> <https://github.com/apache/iceberg/commit/3bbdee97b0f33794df091c3292de8d54a05dcc1e>) >>>>>>>>>> is to backport specifically for Spark 3.4. >>>>>>>>>> >>>>>>>>>> From Engine Version Lifecycle >>>>>>>>>> <https://iceberg.apache.org/multi-engine-support/#engine-version-lifecycle> >>>>>>>>>> , >>>>>>>>>> > Deprecated: an engine version is no longer actively >>>>>>>>>> maintained. People who are still interested in the version can >>>>>>>>>> backport any >>>>>>>>>> necessary feature or bug fix from newer versions, but the community >>>>>>>>>> will >>>>>>>>>> not spend effort in achieving feature parity. Iceberg recommends >>>>>>>>>> users to >>>>>>>>>> move towards a newer version. Contributions to a deprecated version >>>>>>>>>> is >>>>>>>>>> expected to diminish over time, so that eventually no change is >>>>>>>>>> added to a >>>>>>>>>> deprecated version. >>>>>>>>>> and >>>>>>>>>> > End-of-life: a vote can be initiated in the community to fully >>>>>>>>>> remove a deprecated version out of the Iceberg repository to mark as >>>>>>>>>> its >>>>>>>>>> end of life. >>>>>>>>>> >>>>>>>>>> Let's change the status for Spark 3.4 to "Deprecated". I have >>>>>>>>>> prepared a PR already >>>>>>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>>>>>> >>>>>>>>>> I can start another vote thread for removal of Spark 3.4 in the >>>>>>>>>> upcoming 1.11 release and mark it as "End-of-life". I already >>>>>>>>>> prepared a PR >>>>>>>>>> to fully remove Spark 3.4 from the codebase, >>>>>>>>>> https://github.com/apache/iceberg/pull/14122 >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Kevin Liu >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Sep 19, 2025 at 11:29 AM Eduard Tudenhöfner < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> I agree with Anton and I would be in favor of just removing it >>>>>>>>>>> in the next release. By updating the docs now we can already signal >>>>>>>>>>> immediately that Spark 3.4 is deprecated and people can always use >>>>>>>>>>> Iceberg >>>>>>>>>>> 1.10 when needing Spark 3.4 support. >>>>>>>>>>> >>>>>>>>>>> On Fri, Sep 19, 2025 at 7:06 PM Anton Okolnychyi < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> I know we followed this rule of deprecating a Spark version in >>>>>>>>>>>> one release and then removing it in the next one. Shall we ask >>>>>>>>>>>> ourselves >>>>>>>>>>>> whether it is still the model we want to follow? >>>>>>>>>>>> >>>>>>>>>>>> My problem like before is that we release a new Iceberg jar >>>>>>>>>>>> that is supposed to contain the latest and greatest features but >>>>>>>>>>>> the >>>>>>>>>>>> functionality for older Spark versions is severely lagging. >>>>>>>>>>>> >>>>>>>>>>>> We initially kept older Spark modules in main to give folks in >>>>>>>>>>>> the community a place to maintain these older integrations and >>>>>>>>>>>> collaborate. >>>>>>>>>>>> I don’t see a lot of interest in that, if I am being honest. >>>>>>>>>>>> Instead, it >>>>>>>>>>>> became a liability for devs as all major format features now have >>>>>>>>>>>> to work >>>>>>>>>>>> with those old Spark integrations. It is hurting the velocity of >>>>>>>>>>>> the >>>>>>>>>>>> project and recent row ID work is an example of that. >>>>>>>>>>>> >>>>>>>>>>>> - Anton >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Sep 19, 2025 at 6:33 PM Kevin Liu < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> > why not just remove Spark 3.4 for the next 1.11 release? Or >>>>>>>>>>>>> do we usually wait for one more release and remove it in the 1.12 >>>>>>>>>>>>> release >>>>>>>>>>>>> after marking 3.4 as deprecated in the engine status doc page? >>>>>>>>>>>>> >>>>>>>>>>>>> My preference is to mark as deprecated for one release and >>>>>>>>>>>>> remove in the following. >>>>>>>>>>>>> >>>>>>>>>>>>> To quote JB: >>>>>>>>>>>>> "announce" the deprecation in 1.11 and remove 1.12, it gives >>>>>>>>>>>>> time for users to "adapt". >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> Kevin Liu >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Sep 19, 2025 at 9:26 AM Steven Wu < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Following up on Manu's question, why not just remove Spark >>>>>>>>>>>>>> 3.4 for the next 1.11 release? Or do we usually wait for one >>>>>>>>>>>>>> more release >>>>>>>>>>>>>> and remove it in the 1.12 release after marking 3.4 as >>>>>>>>>>>>>> deprecated in the >>>>>>>>>>>>>> engine status doc page? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 9:12 AM Kevin Liu < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Given the many +1's here, I've moved the PR to deprecate 3.4 >>>>>>>>>>>>>>> to "ready for review", >>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > Does it mean we will stop back-porting PRs to Spark 3.4 >>>>>>>>>>>>>>> for 1.11? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Not necessarily. There's a lot of Spark 3.4 backports >>>>>>>>>>>>>>> already, >>>>>>>>>>>>>>> https://github.com/apache/iceberg/commits/main/spark/v3.4 >>>>>>>>>>>>>>> I suggest we continue to backport for consistency and then >>>>>>>>>>>>>>> stop right after the 1.11 release. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> Kevin Liu >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 6:18 AM Amogh Jahagirdar < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 2:03 AM Péter Váry < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Eduard Tudenhöfner <[email protected]> ezt írta >>>>>>>>>>>>>>>>> (időpont: 2025. szept. 19., P, 8:56): >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> +1 on deprecating Spark 3.4 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, Sep 18, 2025 at 8:36 AM Steve < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Wed, Sep 17, 2025 at 22:52 Jean-Baptiste Onofré < >>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I agree about the plan to "announce" the deprecation in >>>>>>>>>>>>>>>>>>>> 1.11 and >>>>>>>>>>>>>>>>>>>> remove 1.12, it gives time for users to "adapt". >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>>>> JB >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Wed, Sep 17, 2025 at 10:31 PM Kevin Liu < >>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Hi everyone, >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > I’d like to bring up the topic of deprecating Spark >>>>>>>>>>>>>>>>>>>> 3.4 in an upcoming release. Anton initially suggested this >>>>>>>>>>>>>>>>>>>> during our >>>>>>>>>>>>>>>>>>>> previous dev list discussion about maintaining feature >>>>>>>>>>>>>>>>>>>> parity across the >>>>>>>>>>>>>>>>>>>> Spark versions we support for 1.10. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Currently, we support two different Spark 3.x >>>>>>>>>>>>>>>>>>>> versions, 3.4 and 3.5. Spark 3.4’s last maintenance >>>>>>>>>>>>>>>>>>>> release was in October >>>>>>>>>>>>>>>>>>>> 2024, and it is now considered end-of-life. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > What are your thoughts on marking Spark 3.4 as >>>>>>>>>>>>>>>>>>>> deprecated in 1.11 and removing it in 1.12? >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > For reference, here's the previous discussion thread >>>>>>>>>>>>>>>>>>>> on deprecating Spark 3.3. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Best, >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Kevin Liu >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>
