Re: JDK version support policy?
Thanks all for the discussion here. Based on this I think we'll stick with Java 8 for now and then upgrade to Java 11 around or after Spark 4. On Thu, Jun 8, 2023, at 07:17, Sean Owen wrote: > Noted, but for that you'd simply run your app on Java 17. If Spark works, and > your app's dependencies work on Java 17 because you compile it for 17 (and > jakarta.* classes for example) then there's no issue. > > On Thu, Jun 8, 2023 at 3:13 AM Martin Andersson > wrote: >> There are some reasons to drop Java 11 as well. Java 17 included a large >> change, breaking backwards compatibility with their transition from Java EE >> to Jakarta EE >> <https://blogs.oracle.com/javamagazine/post/transition-from-java-ee-to-jakarta-ee>. >> This means that any users using Spark 4.0 together with Spring 6.x or any >> recent version of servlet containers such as Tomcat or Jetty will experience >> issues. (For security reasons it's beneficial to float your dependencies to >> the latest version of these libraries/frameworks) >> >> I'm not explicitly saying Java 11 should be dropped in Spark 4, just thought >> I'd bring this issue to your attention. >> >> Best Regards, Martin >> >> >> *From:* Jungtaek Lim >> *Sent:* Wednesday, June 7, 2023 23:19 >> *To:* Sean Owen >> *Cc:* Dongjoon Hyun ; Holden Karau >> ; dev >> *Subject:* Re: JDK version support policy? >> >> >> >> EXTERNAL SENDER. Do not click links or open attachments unless you recognize >> the sender and know the content is safe. DO NOT provide your username or >> password. >> >> >> >> +1 to drop Java 8 but +1 to set the lowest support version to Java 11. >> >> Considering the phase for only security updates, 11 LTS would not be EOLed >> in very long time. Unless that’s coupled with other deps which require >> bumping JDK version (hope someone can bring up lists), it doesn’t seem to >> buy much. And given the strong backward compatibility JDK provides, that’s >> less likely. >> >> Purely from the project’s source code view, does anyone know how much >> benefits we can leverage for picking up 17 rather than 11? I lost the track, >> but some of their proposals are more likely catching up with other >> languages, which don’t make us be happy since Scala provides them for years. >> >> 2023년 6월 8일 (목) 오전 2:35, Sean Owen 님이 작성: >>> I also generally perceive that, after Java 9, there is much less breaking >>> change. So working on Java 11 probably means it works on 20, or can be >>> easily made to without pain. Like I think the tweaks for Java 17 were quite >>> small. >>> >>> Targeting Java >11 excludes Java 11 users and probably wouldn't buy much. >>> Keeping the support probably doesn't interfere with working on much newer >>> JVMs either. >>> >>> On Wed, Jun 7, 2023, 12:29 PM Holden Karau wrote: >>>> So JDK 11 is still supported in open JDK until 2026, I'm not sure if we're >>>> going to see enough folks moving to JRE17 by the Spark 4 release unless we >>>> have a strong benefit from dropping 11 support I'd be inclined to keep it. >>>> >>>> On Tue, Jun 6, 2023 at 9:08 PM Dongjoon Hyun wrote: >>>>> I'm also +1 on dropping both Java 8 and 11 in Apache Spark 4.0, too. >>>>> >>>>> Dongjoon. >>>>> >>>>> On 2023/06/07 02:42:19 yangjie01 wrote: >>>>> > +1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only >>>>> > support Java 17 and the upcoming Java 21. >>>>> > >>>>> > 发件人: Denny Lee >>>>> > 日期: 2023年6月7日 星期三 07:10 >>>>> > 收件人: Sean Owen >>>>> > 抄送: David Li , "dev@spark.apache.org" >>>>> > >>>>> > 主题: Re: JDK version support policy? >>>>> > >>>>> > +1 on dropping Java 8 in Spark 4.0, saying this as a fan of the >>>>> > fast-paced (positive) updates to Arrow, eh?! >>>>> > >>>>> > On Tue, Jun 6, 2023 at 4:02 PM Sean Owen >>>>> > mailto:sro...@gmail.com>> wrote: >>>>> > I haven't followed this discussion closely, but I think we could/should >>>>> > drop Java 8 in Spark 4.0, which is up next after 3.5? >>>>> > >>>>> > On Tue, Jun 6, 2023 at 2:44 PM David Li >>>>> > mailto:lidav...@apache.org>> wrote: >>>>
Re: JDK version support policy?
Noted, but for that you'd simply run your app on Java 17. If Spark works, and your app's dependencies work on Java 17 because you compile it for 17 (and jakarta.* classes for example) then there's no issue. On Thu, Jun 8, 2023 at 3:13 AM Martin Andersson wrote: > There are some reasons to drop Java 11 as well. Java 17 included a large > change, breaking backwards compatibility with their transition from Java > EE to Jakarta EE > <https://blogs.oracle.com/javamagazine/post/transition-from-java-ee-to-jakarta-ee>. > This means that any users using Spark 4.0 together with Spring 6.x or any > recent version of servlet containers such as Tomcat or Jetty will > experience issues. (For security reasons it's beneficial to float your > dependencies to the latest version of these libraries/frameworks) > > I'm not explicitly saying Java 11 should be dropped in Spark 4, just > thought I'd bring this issue to your attention. > > Best Regards, Martin > -- > *From:* Jungtaek Lim > *Sent:* Wednesday, June 7, 2023 23:19 > *To:* Sean Owen > *Cc:* Dongjoon Hyun ; Holden Karau < > hol...@pigscanfly.ca>; dev > *Subject:* Re: JDK version support policy? > > > EXTERNAL SENDER. Do not click links or open attachments unless you > recognize the sender and know the content is safe. DO NOT provide your > username or password. > > +1 to drop Java 8 but +1 to set the lowest support version to Java 11. > > Considering the phase for only security updates, 11 LTS would not be EOLed > in very long time. Unless that’s coupled with other deps which require > bumping JDK version (hope someone can bring up lists), it doesn’t seem to > buy much. And given the strong backward compatibility JDK provides, that’s > less likely. > > Purely from the project’s source code view, does anyone know how much > benefits we can leverage for picking up 17 rather than 11? I lost the > track, but some of their proposals are more likely catching up with other > languages, which don’t make us be happy since Scala provides them for years. > > 2023년 6월 8일 (목) 오전 2:35, Sean Owen 님이 작성: > > I also generally perceive that, after Java 9, there is much less breaking > change. So working on Java 11 probably means it works on 20, or can be > easily made to without pain. Like I think the tweaks for Java 17 were quite > small. > > Targeting Java >11 excludes Java 11 users and probably wouldn't buy much. > Keeping the support probably doesn't interfere with working on much newer > JVMs either. > > On Wed, Jun 7, 2023, 12:29 PM Holden Karau wrote: > > So JDK 11 is still supported in open JDK until 2026, I'm not sure if we're > going to see enough folks moving to JRE17 by the Spark 4 release unless we > have a strong benefit from dropping 11 support I'd be inclined to keep it. > > On Tue, Jun 6, 2023 at 9:08 PM Dongjoon Hyun wrote: > > I'm also +1 on dropping both Java 8 and 11 in Apache Spark 4.0, too. > > Dongjoon. > > On 2023/06/07 02:42:19 yangjie01 wrote: > > +1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only > support Java 17 and the upcoming Java 21. > > > > 发件人: Denny Lee > > 日期: 2023年6月7日 星期三 07:10 > > 收件人: Sean Owen > > 抄送: David Li , "dev@spark.apache.org" < > dev@spark.apache.org> > > 主题: Re: JDK version support policy? > > > > +1 on dropping Java 8 in Spark 4.0, saying this as a fan of the > fast-paced (positive) updates to Arrow, eh?! > > > > On Tue, Jun 6, 2023 at 4:02 PM Sean Owen sro...@gmail.com>> wrote: > > I haven't followed this discussion closely, but I think we could/should > drop Java 8 in Spark 4.0, which is up next after 3.5? > > > > On Tue, Jun 6, 2023 at 2:44 PM David Li lidav...@apache.org>> wrote: > > Hello Spark developers, > > > > I'm from the Apache Arrow project. We've discussed Java version support > [1], and crucially, whether to continue supporting Java 8 or not. As Spark > is a big user of Arrow in Java, I was curious what Spark's policy here was. > > > > If Spark intends to stay on Java 8, for instance, we may also want to > stay on Java 8 or otherwise provide some supported version of Arrow for > Java 8. > > > > We've seen dependencies dropping or planning to drop support. gRPC may > drop Java 8 at any time [2], possibly this September [3], which may affect > Spark (due to Spark Connect). And today we saw that Arrow had issues > running tests with Mockito on Java 20, but we couldn't update Mockito since > it had dropped Java 8 support. (We pinned the JDK version in that CI > pipeline for now.) > > > > So at least, I am curious if Arrow could start the long proces
Re: JDK version support policy?
There are some reasons to drop Java 11 as well. Java 17 included a large change, breaking backwards compatibility with their transition from Java EE to Jakarta EE<https://blogs.oracle.com/javamagazine/post/transition-from-java-ee-to-jakarta-ee>. This means that any users using Spark 4.0 together with Spring 6.x or any recent version of servlet containers such as Tomcat or Jetty will experience issues. (For security reasons it's beneficial to float your dependencies to the latest version of these libraries/frameworks) I'm not explicitly saying Java 11 should be dropped in Spark 4, just thought I'd bring this issue to your attention. Best Regards, Martin From: Jungtaek Lim Sent: Wednesday, June 7, 2023 23:19 To: Sean Owen Cc: Dongjoon Hyun ; Holden Karau ; dev Subject: Re: JDK version support policy? EXTERNAL SENDER. Do not click links or open attachments unless you recognize the sender and know the content is safe. DO NOT provide your username or password. +1 to drop Java 8 but +1 to set the lowest support version to Java 11. Considering the phase for only security updates, 11 LTS would not be EOLed in very long time. Unless that’s coupled with other deps which require bumping JDK version (hope someone can bring up lists), it doesn’t seem to buy much. And given the strong backward compatibility JDK provides, that’s less likely. Purely from the project’s source code view, does anyone know how much benefits we can leverage for picking up 17 rather than 11? I lost the track, but some of their proposals are more likely catching up with other languages, which don’t make us be happy since Scala provides them for years. 2023년 6월 8일 (목) 오전 2:35, Sean Owen mailto:sro...@gmail.com>>님이 작성: I also generally perceive that, after Java 9, there is much less breaking change. So working on Java 11 probably means it works on 20, or can be easily made to without pain. Like I think the tweaks for Java 17 were quite small. Targeting Java >11 excludes Java 11 users and probably wouldn't buy much. Keeping the support probably doesn't interfere with working on much newer JVMs either. On Wed, Jun 7, 2023, 12:29 PM Holden Karau mailto:hol...@pigscanfly.ca>> wrote: So JDK 11 is still supported in open JDK until 2026, I'm not sure if we're going to see enough folks moving to JRE17 by the Spark 4 release unless we have a strong benefit from dropping 11 support I'd be inclined to keep it. On Tue, Jun 6, 2023 at 9:08 PM Dongjoon Hyun mailto:dongj...@apache.org>> wrote: I'm also +1 on dropping both Java 8 and 11 in Apache Spark 4.0, too. Dongjoon. On 2023/06/07 02:42:19 yangjie01 wrote: > +1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only > support Java 17 and the upcoming Java 21. > > 发件人: Denny Lee mailto:denny.g@gmail.com>> > 日期: 2023年6月7日 星期三 07:10 > 收件人: Sean Owen mailto:sro...@gmail.com>> > 抄送: David Li mailto:lidav...@apache.org>>, > "dev@spark.apache.org<mailto:dev@spark.apache.org>" > mailto:dev@spark.apache.org>> > 主题: Re: JDK version support policy? > > +1 on dropping Java 8 in Spark 4.0, saying this as a fan of the fast-paced > (positive) updates to Arrow, eh?! > > On Tue, Jun 6, 2023 at 4:02 PM Sean Owen > mailto:sro...@gmail.com><mailto:sro...@gmail.com<mailto:sro...@gmail.com>>> > wrote: > I haven't followed this discussion closely, but I think we could/should drop > Java 8 in Spark 4.0, which is up next after 3.5? > > On Tue, Jun 6, 2023 at 2:44 PM David Li > mailto:lidav...@apache.org><mailto:lidav...@apache.org<mailto:lidav...@apache.org>>> > wrote: > Hello Spark developers, > > I'm from the Apache Arrow project. We've discussed Java version support [1], > and crucially, whether to continue supporting Java 8 or not. As Spark is a > big user of Arrow in Java, I was curious what Spark's policy here was. > > If Spark intends to stay on Java 8, for instance, we may also want to stay on > Java 8 or otherwise provide some supported version of Arrow for Java 8. > > We've seen dependencies dropping or planning to drop support. gRPC may drop > Java 8 at any time [2], possibly this September [3], which may affect Spark > (due to Spark Connect). And today we saw that Arrow had issues running tests > with Mockito on Java 20, but we couldn't update Mockito since it had dropped > Java 8 support. (We pinned the JDK version in that CI pipeline for now.) > > So at least, I am curious if Arrow could start the long process of migrating > Java versions without impacting Spark, or if we should continue to cooperate. > Arrow Java doesn't see quite so much activity these days, so it's not quite > critical, but it's possible that these dependency issues will start to affect > us more soon. And l
Re: JDK version support policy?
+1 to drop Java 8 but +1 to set the lowest support version to Java 11. Considering the phase for only security updates, 11 LTS would not be EOLed in very long time. Unless that’s coupled with other deps which require bumping JDK version (hope someone can bring up lists), it doesn’t seem to buy much. And given the strong backward compatibility JDK provides, that’s less likely. Purely from the project’s source code view, does anyone know how much benefits we can leverage for picking up 17 rather than 11? I lost the track, but some of their proposals are more likely catching up with other languages, which don’t make us be happy since Scala provides them for years. 2023년 6월 8일 (목) 오전 2:35, Sean Owen 님이 작성: > I also generally perceive that, after Java 9, there is much less breaking > change. So working on Java 11 probably means it works on 20, or can be > easily made to without pain. Like I think the tweaks for Java 17 were quite > small. > > Targeting Java >11 excludes Java 11 users and probably wouldn't buy much. > Keeping the support probably doesn't interfere with working on much newer > JVMs either. > > On Wed, Jun 7, 2023, 12:29 PM Holden Karau wrote: > >> So JDK 11 is still supported in open JDK until 2026, I'm not sure if >> we're going to see enough folks moving to JRE17 by the Spark 4 release >> unless we have a strong benefit from dropping 11 support I'd be inclined to >> keep it. >> >> On Tue, Jun 6, 2023 at 9:08 PM Dongjoon Hyun wrote: >> >>> I'm also +1 on dropping both Java 8 and 11 in Apache Spark 4.0, too. >>> >>> Dongjoon. >>> >>> On 2023/06/07 02:42:19 yangjie01 wrote: >>> > +1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only >>> support Java 17 and the upcoming Java 21. >>> > >>> > 发件人: Denny Lee >>> > 日期: 2023年6月7日 星期三 07:10 >>> > 收件人: Sean Owen >>> > 抄送: David Li , "dev@spark.apache.org" < >>> dev@spark.apache.org> >>> > 主题: Re: JDK version support policy? >>> > >>> > +1 on dropping Java 8 in Spark 4.0, saying this as a fan of the >>> fast-paced (positive) updates to Arrow, eh?! >>> > >>> > On Tue, Jun 6, 2023 at 4:02 PM Sean Owen >> sro...@gmail.com>> wrote: >>> > I haven't followed this discussion closely, but I think we >>> could/should drop Java 8 in Spark 4.0, which is up next after 3.5? >>> > >>> > On Tue, Jun 6, 2023 at 2:44 PM David Li >> lidav...@apache.org>> wrote: >>> > Hello Spark developers, >>> > >>> > I'm from the Apache Arrow project. We've discussed Java version >>> support [1], and crucially, whether to continue supporting Java 8 or not. >>> As Spark is a big user of Arrow in Java, I was curious what Spark's policy >>> here was. >>> > >>> > If Spark intends to stay on Java 8, for instance, we may also want to >>> stay on Java 8 or otherwise provide some supported version of Arrow for >>> Java 8. >>> > >>> > We've seen dependencies dropping or planning to drop support. gRPC may >>> drop Java 8 at any time [2], possibly this September [3], which may affect >>> Spark (due to Spark Connect). And today we saw that Arrow had issues >>> running tests with Mockito on Java 20, but we couldn't update Mockito since >>> it had dropped Java 8 support. (We pinned the JDK version in that CI >>> pipeline for now.) >>> > >>> > So at least, I am curious if Arrow could start the long process of >>> migrating Java versions without impacting Spark, or if we should continue >>> to cooperate. Arrow Java doesn't see quite so much activity these days, so >>> it's not quite critical, but it's possible that these dependency issues >>> will start to affect us more soon. And looking forward, Java is working on >>> APIs that should also allow us to ditch the --add-opens flag requirement >>> too. >>> > >>> > [1]: https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj< >>> https://mailshield.baidu.com/check?q=Nz%2bGj2hdKguk92URjA7sg0PfbSN%2fXUIMgrHTmW45gOOKEr3Shre45B7TRzhEpb%2baVsnyuRL%2fl%2f0cu7IVGHunSGDVnxM%3d >>> > >>> > [2]: >>> https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md< >>> https://mailshield.baidu.com/check?q=s89S3eo8GCJkV7Mpx7aG1SXId7uCRYGjQMA6DeLuX9duS86LhIODZMJfeFdGMWdFzJ8S7minyHoC7mCrzHagbJXCXYTBH%2fpZBpfTbw%3d%3d >>> > >>> > [3]: https://github.com/grpc/grpc-java/issues/9386< >>> https://mailshield.baidu.com/check?q=R0HtWZIkY5eIxpz8jtqHLzd0ugNbcaXIKW2LbUUxpIn0t9Y9yAhuHPuZ4buryfNwRnnJTA%3d%3d >>> > >>> > >>> >>> - >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >>> >> >> -- >> Twitter: https://twitter.com/holdenkarau >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> >
Re: JDK version support policy?
I also generally perceive that, after Java 9, there is much less breaking change. So working on Java 11 probably means it works on 20, or can be easily made to without pain. Like I think the tweaks for Java 17 were quite small. Targeting Java >11 excludes Java 11 users and probably wouldn't buy much. Keeping the support probably doesn't interfere with working on much newer JVMs either. On Wed, Jun 7, 2023, 12:29 PM Holden Karau wrote: > So JDK 11 is still supported in open JDK until 2026, I'm not sure if we're > going to see enough folks moving to JRE17 by the Spark 4 release unless we > have a strong benefit from dropping 11 support I'd be inclined to keep it. > > On Tue, Jun 6, 2023 at 9:08 PM Dongjoon Hyun wrote: > >> I'm also +1 on dropping both Java 8 and 11 in Apache Spark 4.0, too. >> >> Dongjoon. >> >> On 2023/06/07 02:42:19 yangjie01 wrote: >> > +1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only >> support Java 17 and the upcoming Java 21. >> > >> > 发件人: Denny Lee >> > 日期: 2023年6月7日 星期三 07:10 >> > 收件人: Sean Owen >> > 抄送: David Li , "dev@spark.apache.org" < >> dev@spark.apache.org> >> > 主题: Re: JDK version support policy? >> > >> > +1 on dropping Java 8 in Spark 4.0, saying this as a fan of the >> fast-paced (positive) updates to Arrow, eh?! >> > >> > On Tue, Jun 6, 2023 at 4:02 PM Sean Owen > sro...@gmail.com>> wrote: >> > I haven't followed this discussion closely, but I think we could/should >> drop Java 8 in Spark 4.0, which is up next after 3.5? >> > >> > On Tue, Jun 6, 2023 at 2:44 PM David Li > lidav...@apache.org>> wrote: >> > Hello Spark developers, >> > >> > I'm from the Apache Arrow project. We've discussed Java version support >> [1], and crucially, whether to continue supporting Java 8 or not. As Spark >> is a big user of Arrow in Java, I was curious what Spark's policy here was. >> > >> > If Spark intends to stay on Java 8, for instance, we may also want to >> stay on Java 8 or otherwise provide some supported version of Arrow for >> Java 8. >> > >> > We've seen dependencies dropping or planning to drop support. gRPC may >> drop Java 8 at any time [2], possibly this September [3], which may affect >> Spark (due to Spark Connect). And today we saw that Arrow had issues >> running tests with Mockito on Java 20, but we couldn't update Mockito since >> it had dropped Java 8 support. (We pinned the JDK version in that CI >> pipeline for now.) >> > >> > So at least, I am curious if Arrow could start the long process of >> migrating Java versions without impacting Spark, or if we should continue >> to cooperate. Arrow Java doesn't see quite so much activity these days, so >> it's not quite critical, but it's possible that these dependency issues >> will start to affect us more soon. And looking forward, Java is working on >> APIs that should also allow us to ditch the --add-opens flag requirement >> too. >> > >> > [1]: https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj< >> https://mailshield.baidu.com/check?q=Nz%2bGj2hdKguk92URjA7sg0PfbSN%2fXUIMgrHTmW45gOOKEr3Shre45B7TRzhEpb%2baVsnyuRL%2fl%2f0cu7IVGHunSGDVnxM%3d >> > >> > [2]: >> https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md< >> https://mailshield.baidu.com/check?q=s89S3eo8GCJkV7Mpx7aG1SXId7uCRYGjQMA6DeLuX9duS86LhIODZMJfeFdGMWdFzJ8S7minyHoC7mCrzHagbJXCXYTBH%2fpZBpfTbw%3d%3d >> > >> > [3]: https://github.com/grpc/grpc-java/issues/9386< >> https://mailshield.baidu.com/check?q=R0HtWZIkY5eIxpz8jtqHLzd0ugNbcaXIKW2LbUUxpIn0t9Y9yAhuHPuZ4buryfNwRnnJTA%3d%3d >> > >> > >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > > -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >
Re: JDK version support policy?
So JDK 11 is still supported in open JDK until 2026, I'm not sure if we're going to see enough folks moving to JRE17 by the Spark 4 release unless we have a strong benefit from dropping 11 support I'd be inclined to keep it. On Tue, Jun 6, 2023 at 9:08 PM Dongjoon Hyun wrote: > I'm also +1 on dropping both Java 8 and 11 in Apache Spark 4.0, too. > > Dongjoon. > > On 2023/06/07 02:42:19 yangjie01 wrote: > > +1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only > support Java 17 and the upcoming Java 21. > > > > 发件人: Denny Lee > > 日期: 2023年6月7日 星期三 07:10 > > 收件人: Sean Owen > > 抄送: David Li , "dev@spark.apache.org" < > dev@spark.apache.org> > > 主题: Re: JDK version support policy? > > > > +1 on dropping Java 8 in Spark 4.0, saying this as a fan of the > fast-paced (positive) updates to Arrow, eh?! > > > > On Tue, Jun 6, 2023 at 4:02 PM Sean Owen sro...@gmail.com>> wrote: > > I haven't followed this discussion closely, but I think we could/should > drop Java 8 in Spark 4.0, which is up next after 3.5? > > > > On Tue, Jun 6, 2023 at 2:44 PM David Li lidav...@apache.org>> wrote: > > Hello Spark developers, > > > > I'm from the Apache Arrow project. We've discussed Java version support > [1], and crucially, whether to continue supporting Java 8 or not. As Spark > is a big user of Arrow in Java, I was curious what Spark's policy here was. > > > > If Spark intends to stay on Java 8, for instance, we may also want to > stay on Java 8 or otherwise provide some supported version of Arrow for > Java 8. > > > > We've seen dependencies dropping or planning to drop support. gRPC may > drop Java 8 at any time [2], possibly this September [3], which may affect > Spark (due to Spark Connect). And today we saw that Arrow had issues > running tests with Mockito on Java 20, but we couldn't update Mockito since > it had dropped Java 8 support. (We pinned the JDK version in that CI > pipeline for now.) > > > > So at least, I am curious if Arrow could start the long process of > migrating Java versions without impacting Spark, or if we should continue > to cooperate. Arrow Java doesn't see quite so much activity these days, so > it's not quite critical, but it's possible that these dependency issues > will start to affect us more soon. And looking forward, Java is working on > APIs that should also allow us to ditch the --add-opens flag requirement > too. > > > > [1]: https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj< > https://mailshield.baidu.com/check?q=Nz%2bGj2hdKguk92URjA7sg0PfbSN%2fXUIMgrHTmW45gOOKEr3Shre45B7TRzhEpb%2baVsnyuRL%2fl%2f0cu7IVGHunSGDVnxM%3d > > > > [2]: > https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md< > https://mailshield.baidu.com/check?q=s89S3eo8GCJkV7Mpx7aG1SXId7uCRYGjQMA6DeLuX9duS86LhIODZMJfeFdGMWdFzJ8S7minyHoC7mCrzHagbJXCXYTBH%2fpZBpfTbw%3d%3d > > > > [3]: https://github.com/grpc/grpc-java/issues/9386< > https://mailshield.baidu.com/check?q=R0HtWZIkY5eIxpz8jtqHLzd0ugNbcaXIKW2LbUUxpIn0t9Y9yAhuHPuZ4buryfNwRnnJTA%3d%3d > > > > > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Re: JDK version support policy?
I'm also +1 on dropping both Java 8 and 11 in Apache Spark 4.0, too. Dongjoon. On 2023/06/07 02:42:19 yangjie01 wrote: > +1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only > support Java 17 and the upcoming Java 21. > > 发件人: Denny Lee > 日期: 2023年6月7日 星期三 07:10 > 收件人: Sean Owen > 抄送: David Li , "dev@spark.apache.org" > > 主题: Re: JDK version support policy? > > +1 on dropping Java 8 in Spark 4.0, saying this as a fan of the fast-paced > (positive) updates to Arrow, eh?! > > On Tue, Jun 6, 2023 at 4:02 PM Sean Owen > mailto:sro...@gmail.com>> wrote: > I haven't followed this discussion closely, but I think we could/should drop > Java 8 in Spark 4.0, which is up next after 3.5? > > On Tue, Jun 6, 2023 at 2:44 PM David Li > mailto:lidav...@apache.org>> wrote: > Hello Spark developers, > > I'm from the Apache Arrow project. We've discussed Java version support [1], > and crucially, whether to continue supporting Java 8 or not. As Spark is a > big user of Arrow in Java, I was curious what Spark's policy here was. > > If Spark intends to stay on Java 8, for instance, we may also want to stay on > Java 8 or otherwise provide some supported version of Arrow for Java 8. > > We've seen dependencies dropping or planning to drop support. gRPC may drop > Java 8 at any time [2], possibly this September [3], which may affect Spark > (due to Spark Connect). And today we saw that Arrow had issues running tests > with Mockito on Java 20, but we couldn't update Mockito since it had dropped > Java 8 support. (We pinned the JDK version in that CI pipeline for now.) > > So at least, I am curious if Arrow could start the long process of migrating > Java versions without impacting Spark, or if we should continue to cooperate. > Arrow Java doesn't see quite so much activity these days, so it's not quite > critical, but it's possible that these dependency issues will start to affect > us more soon. And looking forward, Java is working on APIs that should also > allow us to ditch the --add-opens flag requirement too. > > [1]: > https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj<https://mailshield.baidu.com/check?q=Nz%2bGj2hdKguk92URjA7sg0PfbSN%2fXUIMgrHTmW45gOOKEr3Shre45B7TRzhEpb%2baVsnyuRL%2fl%2f0cu7IVGHunSGDVnxM%3d> > [2]: > https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md<https://mailshield.baidu.com/check?q=s89S3eo8GCJkV7Mpx7aG1SXId7uCRYGjQMA6DeLuX9duS86LhIODZMJfeFdGMWdFzJ8S7minyHoC7mCrzHagbJXCXYTBH%2fpZBpfTbw%3d%3d> > [3]: > https://github.com/grpc/grpc-java/issues/9386<https://mailshield.baidu.com/check?q=R0HtWZIkY5eIxpz8jtqHLzd0ugNbcaXIKW2LbUUxpIn0t9Y9yAhuHPuZ4buryfNwRnnJTA%3d%3d> > - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: JDK version support policy?
+1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only support Java 17 and the upcoming Java 21. 发件人: Denny Lee 日期: 2023年6月7日 星期三 07:10 收件人: Sean Owen 抄送: David Li , "dev@spark.apache.org" 主题: Re: JDK version support policy? +1 on dropping Java 8 in Spark 4.0, saying this as a fan of the fast-paced (positive) updates to Arrow, eh?! On Tue, Jun 6, 2023 at 4:02 PM Sean Owen mailto:sro...@gmail.com>> wrote: I haven't followed this discussion closely, but I think we could/should drop Java 8 in Spark 4.0, which is up next after 3.5? On Tue, Jun 6, 2023 at 2:44 PM David Li mailto:lidav...@apache.org>> wrote: Hello Spark developers, I'm from the Apache Arrow project. We've discussed Java version support [1], and crucially, whether to continue supporting Java 8 or not. As Spark is a big user of Arrow in Java, I was curious what Spark's policy here was. If Spark intends to stay on Java 8, for instance, we may also want to stay on Java 8 or otherwise provide some supported version of Arrow for Java 8. We've seen dependencies dropping or planning to drop support. gRPC may drop Java 8 at any time [2], possibly this September [3], which may affect Spark (due to Spark Connect). And today we saw that Arrow had issues running tests with Mockito on Java 20, but we couldn't update Mockito since it had dropped Java 8 support. (We pinned the JDK version in that CI pipeline for now.) So at least, I am curious if Arrow could start the long process of migrating Java versions without impacting Spark, or if we should continue to cooperate. Arrow Java doesn't see quite so much activity these days, so it's not quite critical, but it's possible that these dependency issues will start to affect us more soon. And looking forward, Java is working on APIs that should also allow us to ditch the --add-opens flag requirement too. [1]: https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj<https://mailshield.baidu.com/check?q=Nz%2bGj2hdKguk92URjA7sg0PfbSN%2fXUIMgrHTmW45gOOKEr3Shre45B7TRzhEpb%2baVsnyuRL%2fl%2f0cu7IVGHunSGDVnxM%3d> [2]: https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md<https://mailshield.baidu.com/check?q=s89S3eo8GCJkV7Mpx7aG1SXId7uCRYGjQMA6DeLuX9duS86LhIODZMJfeFdGMWdFzJ8S7minyHoC7mCrzHagbJXCXYTBH%2fpZBpfTbw%3d%3d> [3]: https://github.com/grpc/grpc-java/issues/9386<https://mailshield.baidu.com/check?q=R0HtWZIkY5eIxpz8jtqHLzd0ugNbcaXIKW2LbUUxpIn0t9Y9yAhuHPuZ4buryfNwRnnJTA%3d%3d>
Re: JDK version support policy?
+1 on dropping Java 8 in Spark 4.0, saying this as a fan of the fast-paced (positive) updates to Arrow, eh?! On Tue, Jun 6, 2023 at 4:02 PM Sean Owen wrote: > I haven't followed this discussion closely, but I think we could/should > drop Java 8 in Spark 4.0, which is up next after 3.5? > > On Tue, Jun 6, 2023 at 2:44 PM David Li wrote: > >> Hello Spark developers, >> >> I'm from the Apache Arrow project. We've discussed Java version support >> [1], and crucially, whether to continue supporting Java 8 or not. As Spark >> is a big user of Arrow in Java, I was curious what Spark's policy here was. >> >> If Spark intends to stay on Java 8, for instance, we may also want to >> stay on Java 8 or otherwise provide some supported version of Arrow for >> Java 8. >> >> We've seen dependencies dropping or planning to drop support. gRPC may >> drop Java 8 at any time [2], possibly this September [3], which may affect >> Spark (due to Spark Connect). And today we saw that Arrow had issues >> running tests with Mockito on Java 20, but we couldn't update Mockito since >> it had dropped Java 8 support. (We pinned the JDK version in that CI >> pipeline for now.) >> >> So at least, I am curious if Arrow could start the long process of >> migrating Java versions without impacting Spark, or if we should continue >> to cooperate. Arrow Java doesn't see quite so much activity these days, so >> it's not quite critical, but it's possible that these dependency issues >> will start to affect us more soon. And looking forward, Java is working on >> APIs that should also allow us to ditch the --add-opens flag requirement >> too. >> >> [1]: https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj >> [2]: >> https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md >> [3]: https://github.com/grpc/grpc-java/issues/9386 >> >
Re: JDK version support policy?
I haven't followed this discussion closely, but I think we could/should drop Java 8 in Spark 4.0, which is up next after 3.5? On Tue, Jun 6, 2023 at 2:44 PM David Li wrote: > Hello Spark developers, > > I'm from the Apache Arrow project. We've discussed Java version support > [1], and crucially, whether to continue supporting Java 8 or not. As Spark > is a big user of Arrow in Java, I was curious what Spark's policy here was. > > If Spark intends to stay on Java 8, for instance, we may also want to stay > on Java 8 or otherwise provide some supported version of Arrow for Java 8. > > We've seen dependencies dropping or planning to drop support. gRPC may > drop Java 8 at any time [2], possibly this September [3], which may affect > Spark (due to Spark Connect). And today we saw that Arrow had issues > running tests with Mockito on Java 20, but we couldn't update Mockito since > it had dropped Java 8 support. (We pinned the JDK version in that CI > pipeline for now.) > > So at least, I am curious if Arrow could start the long process of > migrating Java versions without impacting Spark, or if we should continue > to cooperate. Arrow Java doesn't see quite so much activity these days, so > it's not quite critical, but it's possible that these dependency issues > will start to affect us more soon. And looking forward, Java is working on > APIs that should also allow us to ditch the --add-opens flag requirement > too. > > [1]: https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj > [2]: > https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md > [3]: https://github.com/grpc/grpc-java/issues/9386 >
JDK version support policy?
Hello Spark developers, I'm from the Apache Arrow project. We've discussed Java version support [1], and crucially, whether to continue supporting Java 8 or not. As Spark is a big user of Arrow in Java, I was curious what Spark's policy here was. If Spark intends to stay on Java 8, for instance, we may also want to stay on Java 8 or otherwise provide some supported version of Arrow for Java 8. We've seen dependencies dropping or planning to drop support. gRPC may drop Java 8 at any time [2], possibly this September [3], which may affect Spark (due to Spark Connect). And today we saw that Arrow had issues running tests with Mockito on Java 20, but we couldn't update Mockito since it had dropped Java 8 support. (We pinned the JDK version in that CI pipeline for now.) So at least, I am curious if Arrow could start the long process of migrating Java versions without impacting Spark, or if we should continue to cooperate. Arrow Java doesn't see quite so much activity these days, so it's not quite critical, but it's possible that these dependency issues will start to affect us more soon. And looking forward, Java is working on APIs that should also allow us to ditch the --add-opens flag requirement too. [1]: https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj [2]: https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md [3]: https://github.com/grpc/grpc-java/issues/9386
