Re: [DISCUSS] Deprecate Spark 3.2 support?
Exactly ! Thanks for sharing ! I didn't find this page. Thanks, so we have it :) Regards JB On Thu, Sep 21, 2023 at 6:42 PM Ryan Blue wrote: > > Do you mean like this page? > https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status > > On Thu, Sep 21, 2023 at 8:52 AM Jean-Baptiste Onofré > wrote: >> >> Hi Ryan, >> >> Yes it makes sense. The way we discuss and decide the Spark versions >> is totally fine. >> >> My proposal was more to clearly announce the Spark/Flink/Java/Python >> versions supported by Iceberg releases. I know that it's obvious on >> the artifacts name (containing the spark/flink versions) as we share >> on https://iceberg.apache.org/releases/. >> The idea is just to anticipate a bit to inform our users/community, >> for instance having a clear table about the supported layers (a bit >> like on https://karaf.apache.org/download.html or >> https://kafka.apache.org/downloads). >> >> Thanks ! >> Regards >> JB >> >> On Thu, Sep 21, 2023 at 5:40 PM Ryan Blue wrote: >> > >> > JB, I don't think that we need a policy on which Spark versions we intend >> > to keep. Having discussions like this are more effective. Just look at the >> > support for Spark 2.4, which we kept for a lot longer to help people >> > transition. Policy is a way of making decisions by algorithm and I don't >> > think we want to do that here. >> > >> > On Thu, Sep 21, 2023 at 1:48 AM Jean-Baptiste Onofré >> > wrote: >> >> >> >> Just to elaborate a bit :) >> >> >> >> - As Iceberg 1.4.0 is new "major" release, it's good time to >> >> deprecate/remove old version support (of Spark and other things) >> >> - Spark 3.2 users can still use previous Iceberg version >> >> - I will start a discussion about LTS policy with a clear "target" >> >> support for our users (something like the table you can see here >> >> https://karaf.apache.org/download.html), we can list supported Java, >> >> Python, Spark, Flink, support >> >> >> >> Regards >> >> JB >> >> >> >> On Thu, Sep 21, 2023 at 12:01 AM Anton Okolnychyi >> >> wrote: >> >> > >> >> > Shall we consider deprecating our Spark 3.2 support? That Spark version >> >> > is no longer being maintained by the Spark community and is not under >> >> > active development in Iceberg. It was released in October, 2021 and >> >> > passed the 18 month maintenance mark in Spark. >> >> > >> >> > - Anton >> > >> > >> > >> > -- >> > Ryan Blue >> > Tabular > > > > -- > Ryan Blue > Tabular
Re: [DISCUSS] Deprecate Spark 3.2 support?
Do you mean like this page? https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status On Thu, Sep 21, 2023 at 8:52 AM Jean-Baptiste Onofré wrote: > Hi Ryan, > > Yes it makes sense. The way we discuss and decide the Spark versions > is totally fine. > > My proposal was more to clearly announce the Spark/Flink/Java/Python > versions supported by Iceberg releases. I know that it's obvious on > the artifacts name (containing the spark/flink versions) as we share > on https://iceberg.apache.org/releases/. > The idea is just to anticipate a bit to inform our users/community, > for instance having a clear table about the supported layers (a bit > like on https://karaf.apache.org/download.html or > https://kafka.apache.org/downloads). > > Thanks ! > Regards > JB > > On Thu, Sep 21, 2023 at 5:40 PM Ryan Blue wrote: > > > > JB, I don't think that we need a policy on which Spark versions we > intend to keep. Having discussions like this are more effective. Just look > at the support for Spark 2.4, which we kept for a lot longer to help people > transition. Policy is a way of making decisions by algorithm and I don't > think we want to do that here. > > > > On Thu, Sep 21, 2023 at 1:48 AM Jean-Baptiste Onofré > wrote: > >> > >> Just to elaborate a bit :) > >> > >> - As Iceberg 1.4.0 is new "major" release, it's good time to > >> deprecate/remove old version support (of Spark and other things) > >> - Spark 3.2 users can still use previous Iceberg version > >> - I will start a discussion about LTS policy with a clear "target" > >> support for our users (something like the table you can see here > >> https://karaf.apache.org/download.html), we can list supported Java, > >> Python, Spark, Flink, support > >> > >> Regards > >> JB > >> > >> On Thu, Sep 21, 2023 at 12:01 AM Anton Okolnychyi > >> wrote: > >> > > >> > Shall we consider deprecating our Spark 3.2 support? That Spark > version is no longer being maintained by the Spark community and is not > under active development in Iceberg. It was released in October, 2021 and > passed the 18 month maintenance mark in Spark. > >> > > >> > - Anton > > > > > > > > -- > > Ryan Blue > > Tabular > -- Ryan Blue Tabular
Re: [DISCUSS] Deprecate Spark 3.2 support?
Hi Ryan, Yes it makes sense. The way we discuss and decide the Spark versions is totally fine. My proposal was more to clearly announce the Spark/Flink/Java/Python versions supported by Iceberg releases. I know that it's obvious on the artifacts name (containing the spark/flink versions) as we share on https://iceberg.apache.org/releases/. The idea is just to anticipate a bit to inform our users/community, for instance having a clear table about the supported layers (a bit like on https://karaf.apache.org/download.html or https://kafka.apache.org/downloads). Thanks ! Regards JB On Thu, Sep 21, 2023 at 5:40 PM Ryan Blue wrote: > > JB, I don't think that we need a policy on which Spark versions we intend to > keep. Having discussions like this are more effective. Just look at the > support for Spark 2.4, which we kept for a lot longer to help people > transition. Policy is a way of making decisions by algorithm and I don't > think we want to do that here. > > On Thu, Sep 21, 2023 at 1:48 AM Jean-Baptiste Onofré > wrote: >> >> Just to elaborate a bit :) >> >> - As Iceberg 1.4.0 is new "major" release, it's good time to >> deprecate/remove old version support (of Spark and other things) >> - Spark 3.2 users can still use previous Iceberg version >> - I will start a discussion about LTS policy with a clear "target" >> support for our users (something like the table you can see here >> https://karaf.apache.org/download.html), we can list supported Java, >> Python, Spark, Flink, support >> >> Regards >> JB >> >> On Thu, Sep 21, 2023 at 12:01 AM Anton Okolnychyi >> wrote: >> > >> > Shall we consider deprecating our Spark 3.2 support? That Spark version is >> > no longer being maintained by the Spark community and is not under active >> > development in Iceberg. It was released in October, 2021 and passed the 18 >> > month maintenance mark in Spark. >> > >> > - Anton > > > > -- > Ryan Blue > Tabular
Re: [DISCUSS] Deprecate Spark 3.2 support?
JB, I don't think that we need a policy on which Spark versions we intend to keep. Having discussions like this are more effective. Just look at the support for Spark 2.4, which we kept for a lot longer to help people transition. Policy is a way of making decisions by algorithm and I don't think we want to do that here. On Thu, Sep 21, 2023 at 1:48 AM Jean-Baptiste Onofré wrote: > Just to elaborate a bit :) > > - As Iceberg 1.4.0 is new "major" release, it's good time to > deprecate/remove old version support (of Spark and other things) > - Spark 3.2 users can still use previous Iceberg version > - I will start a discussion about LTS policy with a clear "target" > support for our users (something like the table you can see here > https://karaf.apache.org/download.html), we can list supported Java, > Python, Spark, Flink, support > > Regards > JB > > On Thu, Sep 21, 2023 at 12:01 AM Anton Okolnychyi > wrote: > > > > Shall we consider deprecating our Spark 3.2 support? That Spark version > is no longer being maintained by the Spark community and is not under > active development in Iceberg. It was released in October, 2021 and passed > the 18 month maintenance mark in Spark. > > > > - Anton > -- Ryan Blue Tabular
Re: [DISCUSS] Deprecate Spark 3.2 support?
+1 on deprecating Spark 3.2 On Thu, Sep 21, 2023 at 10:48 AM Jean-Baptiste Onofré wrote: > Just to elaborate a bit :) > > - As Iceberg 1.4.0 is new "major" release, it's good time to > deprecate/remove old version support (of Spark and other things) > - Spark 3.2 users can still use previous Iceberg version > - I will start a discussion about LTS policy with a clear "target" > support for our users (something like the table you can see here > https://karaf.apache.org/download.html), we can list supported Java, > Python, Spark, Flink, support > > Regards > JB > > On Thu, Sep 21, 2023 at 12:01 AM Anton Okolnychyi > wrote: > > > > Shall we consider deprecating our Spark 3.2 support? That Spark version > is no longer being maintained by the Spark community and is not under > active development in Iceberg. It was released in October, 2021 and passed > the 18 month maintenance mark in Spark. > > > > - Anton >
Re: [DISCUSS] Deprecate Spark 3.2 support?
Just to elaborate a bit :) - As Iceberg 1.4.0 is new "major" release, it's good time to deprecate/remove old version support (of Spark and other things) - Spark 3.2 users can still use previous Iceberg version - I will start a discussion about LTS policy with a clear "target" support for our users (something like the table you can see here https://karaf.apache.org/download.html), we can list supported Java, Python, Spark, Flink, support Regards JB On Thu, Sep 21, 2023 at 12:01 AM Anton Okolnychyi wrote: > > Shall we consider deprecating our Spark 3.2 support? That Spark version is no > longer being maintained by the Spark community and is not under active > development in Iceberg. It was released in October, 2021 and passed the 18 > month maintenance mark in Spark. > > - Anton
Re: [DISCUSS] Deprecate Spark 3.2 support?
+1 Regards JB On Thu, Sep 21, 2023 at 12:01 AM Anton Okolnychyi wrote: > > Shall we consider deprecating our Spark 3.2 support? That Spark version is no > longer being maintained by the Spark community and is not under active > development in Iceberg. It was released in October, 2021 and passed the 18 > month maintenance mark in Spark. > > - Anton
Re: [DISCUSS] Deprecate Spark 3.2 support?
Correct, the proposal is to initially deprecate but still release whatever the community cherry-picks to that branch. Releasing without the deprecation signals that the dependency is actively maintained while it is not. The 3.2 integration significantly lags behind and does not represent the latest and most capable integration with Spark. I've raised a separate question about 3.1. On 2023/09/21 00:06:35 Pucheng Yang wrote: > Thanks! > > On Wed, Sep 20, 2023 at 5:03 PM Ryan Blue wrote: > > > Ah, right. I actually was talking about if we removed 3.2 from the source > > tree. But you're right. If we just deprecated it, you'd still be able to > > backport things to it until we actually do remove it. > > > > On Wed, Sep 20, 2023 at 5:02 PM Pucheng Yang > > wrote: > > > >> Got it, so "deprecate spark 3.2 support" does not mean removing the spark > >> 3.2 module in the Iceberg project right? > >> > >> And it also means maybe some new changes will only be available on Spark > >> 3.3 +, and such change will likely not get backported. to <3.3 ? But if we > >> are interested, we can do that ourselves, right? > >> > >> Thanks > >> > >> On Wed, Sep 20, 2023 at 4:57 PM Ryan Blue wrote: > >> > >>> Pucheng, you can continue using older releases of Iceberg or you may > >>> already have your own fork. The older versions will continue to work and > >>> we > >>> can still do patch releases for the point versions if there are serious > >>> issues to fix. > >>> > >>> On Wed, Sep 20, 2023 at 4:35 PM Pucheng Yang > >>> wrote: > >>> > Like Linkedin (mentioned in another thread), Pinterest is on Spark 3.2 > and there is no immediate plan to upgrade to the new Spark version as the > migration cost is very high and the process is slow. > > What will be the implications of not having Spark-3.2 module in Iceberg > any more? Based on your past experience, how do companies move forward to > maintain Iceberg with removed Spark versions? Do those companies own > their > enhancements or fixes internally? > > Thanks! > > On Wed, Sep 20, 2023 at 4:00 PM Ryan Blue wrote: > > > +1 > > > > On Wed, Sep 20, 2023 at 3:02 PM Anton Okolnychyi > > wrote: > > > >> Shall we consider deprecating our Spark 3.2 support? That Spark > >> version is no longer being maintained by the Spark community and is not > >> under active development in Iceberg. It was released in October, 2021 > >> and > >> passed the 18 month maintenance mark in Spark. > >> > >> - Anton > > > > > > > > -- > > Ryan Blue > > Tabular > > > > >>> > >>> -- > >>> Ryan Blue > >>> Tabular > >>> > >> > > > > -- > > Ryan Blue > > Tabular > > >
Re: [DISCUSS] Deprecate Spark 3.2 support?
Thanks! On Wed, Sep 20, 2023 at 5:03 PM Ryan Blue wrote: > Ah, right. I actually was talking about if we removed 3.2 from the source > tree. But you're right. If we just deprecated it, you'd still be able to > backport things to it until we actually do remove it. > > On Wed, Sep 20, 2023 at 5:02 PM Pucheng Yang > wrote: > >> Got it, so "deprecate spark 3.2 support" does not mean removing the spark >> 3.2 module in the Iceberg project right? >> >> And it also means maybe some new changes will only be available on Spark >> 3.3 +, and such change will likely not get backported. to <3.3 ? But if we >> are interested, we can do that ourselves, right? >> >> Thanks >> >> On Wed, Sep 20, 2023 at 4:57 PM Ryan Blue wrote: >> >>> Pucheng, you can continue using older releases of Iceberg or you may >>> already have your own fork. The older versions will continue to work and we >>> can still do patch releases for the point versions if there are serious >>> issues to fix. >>> >>> On Wed, Sep 20, 2023 at 4:35 PM Pucheng Yang >>> wrote: >>> Like Linkedin (mentioned in another thread), Pinterest is on Spark 3.2 and there is no immediate plan to upgrade to the new Spark version as the migration cost is very high and the process is slow. What will be the implications of not having Spark-3.2 module in Iceberg any more? Based on your past experience, how do companies move forward to maintain Iceberg with removed Spark versions? Do those companies own their enhancements or fixes internally? Thanks! On Wed, Sep 20, 2023 at 4:00 PM Ryan Blue wrote: > +1 > > On Wed, Sep 20, 2023 at 3:02 PM Anton Okolnychyi > wrote: > >> Shall we consider deprecating our Spark 3.2 support? That Spark >> version is no longer being maintained by the Spark community and is not >> under active development in Iceberg. It was released in October, 2021 and >> passed the 18 month maintenance mark in Spark. >> >> - Anton > > > > -- > Ryan Blue > Tabular > >>> >>> -- >>> Ryan Blue >>> Tabular >>> >> > > -- > Ryan Blue > Tabular >
Re: [DISCUSS] Deprecate Spark 3.2 support?
Ah, right. I actually was talking about if we removed 3.2 from the source tree. But you're right. If we just deprecated it, you'd still be able to backport things to it until we actually do remove it. On Wed, Sep 20, 2023 at 5:02 PM Pucheng Yang wrote: > Got it, so "deprecate spark 3.2 support" does not mean removing the spark > 3.2 module in the Iceberg project right? > > And it also means maybe some new changes will only be available on Spark > 3.3 +, and such change will likely not get backported. to <3.3 ? But if we > are interested, we can do that ourselves, right? > > Thanks > > On Wed, Sep 20, 2023 at 4:57 PM Ryan Blue wrote: > >> Pucheng, you can continue using older releases of Iceberg or you may >> already have your own fork. The older versions will continue to work and we >> can still do patch releases for the point versions if there are serious >> issues to fix. >> >> On Wed, Sep 20, 2023 at 4:35 PM Pucheng Yang >> wrote: >> >>> Like Linkedin (mentioned in another thread), Pinterest is on Spark 3.2 >>> and there is no immediate plan to upgrade to the new Spark version as the >>> migration cost is very high and the process is slow. >>> >>> What will be the implications of not having Spark-3.2 module in Iceberg >>> any more? Based on your past experience, how do companies move forward to >>> maintain Iceberg with removed Spark versions? Do those companies own their >>> enhancements or fixes internally? >>> >>> Thanks! >>> >>> On Wed, Sep 20, 2023 at 4:00 PM Ryan Blue wrote: >>> +1 On Wed, Sep 20, 2023 at 3:02 PM Anton Okolnychyi wrote: > Shall we consider deprecating our Spark 3.2 support? That Spark > version is no longer being maintained by the Spark community and is not > under active development in Iceberg. It was released in October, 2021 and > passed the 18 month maintenance mark in Spark. > > - Anton -- Ryan Blue Tabular >>> >> >> -- >> Ryan Blue >> Tabular >> > -- Ryan Blue Tabular
Re: [DISCUSS] Deprecate Spark 3.2 support?
Got it, so "deprecate spark 3.2 support" does not mean removing the spark 3.2 module in the Iceberg project right? And it also means maybe some new changes will only be available on Spark 3.3 +, and such change will likely not get backported. to <3.3 ? But if we are interested, we can do that ourselves, right? Thanks On Wed, Sep 20, 2023 at 4:57 PM Ryan Blue wrote: > Pucheng, you can continue using older releases of Iceberg or you may > already have your own fork. The older versions will continue to work and we > can still do patch releases for the point versions if there are serious > issues to fix. > > On Wed, Sep 20, 2023 at 4:35 PM Pucheng Yang > wrote: > >> Like Linkedin (mentioned in another thread), Pinterest is on Spark 3.2 >> and there is no immediate plan to upgrade to the new Spark version as the >> migration cost is very high and the process is slow. >> >> What will be the implications of not having Spark-3.2 module in Iceberg >> any more? Based on your past experience, how do companies move forward to >> maintain Iceberg with removed Spark versions? Do those companies own their >> enhancements or fixes internally? >> >> Thanks! >> >> On Wed, Sep 20, 2023 at 4:00 PM Ryan Blue wrote: >> >>> +1 >>> >>> On Wed, Sep 20, 2023 at 3:02 PM Anton Okolnychyi >>> wrote: >>> Shall we consider deprecating our Spark 3.2 support? That Spark version is no longer being maintained by the Spark community and is not under active development in Iceberg. It was released in October, 2021 and passed the 18 month maintenance mark in Spark. - Anton >>> >>> >>> >>> -- >>> Ryan Blue >>> Tabular >>> >> > > -- > Ryan Blue > Tabular >
Re: [DISCUSS] Deprecate Spark 3.2 support?
Pucheng, you can continue using older releases of Iceberg or you may already have your own fork. The older versions will continue to work and we can still do patch releases for the point versions if there are serious issues to fix. On Wed, Sep 20, 2023 at 4:35 PM Pucheng Yang wrote: > Like Linkedin (mentioned in another thread), Pinterest is on Spark 3.2 and > there is no immediate plan to upgrade to the new Spark version as the > migration cost is very high and the process is slow. > > What will be the implications of not having Spark-3.2 module in Iceberg > any more? Based on your past experience, how do companies move forward to > maintain Iceberg with removed Spark versions? Do those companies own their > enhancements or fixes internally? > > Thanks! > > On Wed, Sep 20, 2023 at 4:00 PM Ryan Blue wrote: > >> +1 >> >> On Wed, Sep 20, 2023 at 3:02 PM Anton Okolnychyi >> wrote: >> >>> Shall we consider deprecating our Spark 3.2 support? That Spark version >>> is no longer being maintained by the Spark community and is not under >>> active development in Iceberg. It was released in October, 2021 and passed >>> the 18 month maintenance mark in Spark. >>> >>> - Anton >> >> >> >> -- >> Ryan Blue >> Tabular >> > -- Ryan Blue Tabular
Re: [DISCUSS] Deprecate Spark 3.2 support?
Like Linkedin (mentioned in another thread), Pinterest is on Spark 3.2 and there is no immediate plan to upgrade to the new Spark version as the migration cost is very high and the process is slow. What will be the implications of not having Spark-3.2 module in Iceberg any more? Based on your past experience, how do companies move forward to maintain Iceberg with removed Spark versions? Do those companies own their enhancements or fixes internally? Thanks! On Wed, Sep 20, 2023 at 4:00 PM Ryan Blue wrote: > +1 > > On Wed, Sep 20, 2023 at 3:02 PM Anton Okolnychyi > wrote: > >> Shall we consider deprecating our Spark 3.2 support? That Spark version >> is no longer being maintained by the Spark community and is not under >> active development in Iceberg. It was released in October, 2021 and passed >> the 18 month maintenance mark in Spark. >> >> - Anton > > > > -- > Ryan Blue > Tabular >
Re: [DISCUSS] Deprecate Spark 3.2 support?
+1 On Wed, Sep 20, 2023 at 3:02 PM Anton Okolnychyi wrote: > Shall we consider deprecating our Spark 3.2 support? That Spark version is > no longer being maintained by the Spark community and is not under active > development in Iceberg. It was released in October, 2021 and passed the 18 > month maintenance mark in Spark. > > - Anton -- Ryan Blue Tabular
