Good question about backports. Walaa and Edgar, are you backporting fixes to 3.1? It makes sense to have a place to collaborate, but only if people are actively keeping them updated.
On Fri, Apr 21, 2023 at 3:54 PM Steven Wu <stevenz...@gmail.com> wrote: > For the 3.1 activities that Ryan linked, 3.1 are updated probably for the > requirement of backporting (keeping 3.1, 3.2, 3.3 in sync). It is the > adopted policy. Not sure if it is an indication that people are actively > collaborating on 3.1. > > As Anton was saying, backporting/syncing 4 versions (3.1, 3.2, 3.3, 3.4) > is a pretty high budden. > > On Fri, Apr 21, 2023 at 2:29 PM Anton Okolnychyi > <aokolnyc...@apple.com.invalid> wrote: > >> If it is being used by folks in the community, let’s keep it for now. >> That said, let’s come up with a strategy on when to eventually drop it as >> the list cannot grow indefinitely. Our initial agreement was to keep last 3 >> (except Spark LTS versions), which worked well for 18 months of support >> promised by the Spark community. At this point, Spark will not release any >> bug fixes for 3.1, even critical. >> >> Walaa, Edgar, can you tell us a little bit about the Spark 3.1 >> integration you depend on? Do you have your own Iceberg/Spark forks? Is an >> updated Iceberg core module the primary thing you are looking for? How >> do you deal with Spark bugs? >> >> My biggest worry is that our Spark 3.1 integration randomly gets some >> updates from time to time. By releasing those jars with each Iceberg >> version, we send a message that it is being actively maintained and worked >> on. That’s actually not true, we cherry-pick only some changes. Also, it is >> still part of our release cycle, so it must be checked and tested (our next >> release will have 3.1, 3.2, 3.3 and 3.4 integrations to test). >> >> I am going to close the PR for now but it would be great to find a good >> way to handle this in the future. At least, we have to document what kind >> of expectations our users should have. Do we promise that all bug fixes >> discovered in newer Spark versions will be cherry-picked to all older Spark >> versions? I am not sure that’s true at this point. >> >> - Anton >> >> >> On Apr 21, 2023, at 10:29 AM, Ryan Blue <b...@tabular.io> wrote: >> >> According to Spark docs, a minor release will be supported for 18 months >> and 3.1 was released 2021-03-02, more than 2 years ago. I don't think we >> should expect any further updates from the Spark community for the 3.1 line. >> >> I'm also not sure that there is a problem continuing to release Iceberg's >> module for 3.1. It is still being updated >> <https://github.com/apache/iceberg/commits/master/spark/v3.1> and I >> don't think it is preventing us from continuing work on the later versions. >> Makes sense to me to keep it if people are collaborating there. We should >> evaluate this again soon though. >> >> On Fri, Apr 21, 2023 at 8:27 AM Edgar Rodriguez < >> edgar.rodrig...@airbnb.com.invalid> wrote: >> >>> Airbnb is also still on Spark 3.1 and I echo some of Walaa's comments. >>> >>> Cheers, >>> >>> On Thu, Apr 20, 2023 at 8:14 PM Walaa Eldin Moustafa < >>> wa.moust...@gmail.com> wrote: >>> >>>> LinkedIn is still on Spark 3.1. I am guessing a number of other >>>> companies could be in the same boat. I feel the argument for Spark 2.4 is >>>> different from that of Spark 3.1 and it would be great if we can continue >>>> to support 3.1 for some time. >>>> >>>> On Wed, Apr 19, 2023 at 11:06 AM Ryan Blue <b...@tabular.io> wrote: >>>> >>>>> +1 >>>>> >>>>> As we said in the 2.4 discussion, the format itself should provide >>>>> forward compatibility with tables and it is more clear that we aren't >>>>> adding new features if you have to use older versions for Spark 3.1. >>>>> >>>>> On Wed, Apr 19, 2023 at 10:08 AM Anton Okolnychyi < >>>>> aokolnyc...@apple.com.invalid> wrote: >>>>> >>>>>> Hey folks, >>>>>> >>>>>> What does everybody think about Spark 3.1 support after we add Spark >>>>>> 3.4 support? Our initial plan was to release jars for the last 3 >>>>>> versions. >>>>>> Are there any blockers for dropping 3.1? >>>>>> >>>>>> - Anton >>>>> >>>>> >>>>> >>>>> -- >>>>> Ryan Blue >>>>> Tabular >>>>> >>>> >>> >>> -- >>> Edgar R >>> >> >> >> -- >> Ryan Blue >> Tabular >> >> >> -- Ryan Blue Tabular
