Fokko, thank you for these datapoints! It's great to see how other low level Java OSS projects are approaching this.
JB, I believe yes we have formal consensus to drop Java 8 in Arrow. There was no contention in current discussions across [GitHub issues | Arrow Mailing List | Community Syncs]. We can save Java 11 deprecation for a future discussion. For users on Java 11, I do anticipate this discussion to come shortly after Java 8 deprecation is released. On Fri, May 17, 2024 at 10:02 AM Fokko Driesprong <fo...@apache.org> wrote: > I was traveling the last few weeks, so just a follow-up from my end. > > Fokko, can you elaborate on the discussions held in other OSS projects to >> drop Java <17? How did they weigh the benefits/drawbacks for dropping both >> Java 8 and 11 LTS versions? I'd also be curious if other projects plan to >> support older branches with security patches. > > > So, the ones that I'm involved with (including a TLDR): > > - Avro: > - (April 2024: Consensus on moving to 11+, +1 for moving to 17+) > https://lists.apache.org/thread/6vbd3w5qk7mpb5lyrfyf2s0z1cymjt5w > - (Jan 2024: Consensus on dropping 8) > https://lists.apache.org/thread/bd39zhk655pgzfctq763vp3z4xrjpx58 > - Iceberg: > - (Jan 2023: Concerns about Hive): > https://lists.apache.org/thread/hr7rdxvddw3fklfyg3dfbqbsy81hzhyk > - (Feb 2024: Concensus to drop Hadoop 2.x, and move to JDK11+, > also +1's for moving to 17+): > https://lists.apache.org/thread/ntrk2thvsg9tdccwd4flsdz9gg743368 > > I think the most noteworthy (slow-moving in general): > > - Spark 4 supports JDK 17+ > - Hive 4 is still on Java 8 > <https://github.com/apache/hive?tab=readme-ov-file#java> > > It looks like most of the projects are looking at each other. Keep in > mind, that projects that still support older versions of Java, can still > use older versions of Arrow. > > [image: spiderman-pointing-at-spiderman.jpeg] > (in case the image doesn't come through, that's Spiderman pointing at > Spiderman) > > Concerning the Java 11 support, some data: > > - Oracle 11: support until January 2032 (extended fee has been waived) > - Cornetto 11: September 2027 > - Adoptium 11: At least Oct 2027 > - Zulu 11: Jan 2032 > - OpenJDK11: October 2024 > > I think it is fair to support 11 for the time being, but at some point, we > also have to move on and start exploiting the new features and make sure > that we keep up to date. For example, Java 8 also has extended support > until 2030. Dependabot on the Iceberg project > <https://github.com/apache/iceberg/pulls?q=is%3Aopen+is%3Apr+label%3Adependencies> > nicely shows which projects are already at JDK11+ :) > > Thanks Dane for driving this! > > Kind regards, > Fokko > > > > > > Op vr 17 mei 2024 om 07:44 schreef Jean-Baptiste Onofré <j...@nanthrax.net>: > >> Hi Dane >> >> Do we have a formal consensus about Java version in regards of arrow >> version ? >> I agree with the plan but just wondering if it’s ok from everyone with the >> community. >> >> Regards >> JB >> >> Le jeu. 16 mai 2024 à 18:05, Dane Pitkin <dpit...@apache.org> a écrit : >> >> > To wrap up this thread on Java 8 deprecation, here is my current plan of >> > action: >> > >> > 1) Arrow v17 will be the last version supporting Java 8 and the release >> > notes will warn of its impending deprecation. >> > 2) Arrow v18 will be the first release supporting min version Java 11. >> > >> > I have updated the GH issue[1] to reflect this. >> > >> > [1]https://github.com/apache/arrow/issues/38051 >> > >> > On Wed, May 8, 2024 at 5:46 PM Dane Pitkin <d...@voltrondata.com.invalid >> > >> > wrote: >> > >> > > Thank you all for your valuable input. The consensus from my >> > understanding >> > > is that dropping Java 8 is not contentious, so we will move forward >> here. >> > > >> > > We won't drop Java 11 yet, but there's a chance it will happen sooner >> > than >> > > later. I brought up Java 8 & 11 deprecation in the community sync >> again >> > > today. The summary is that the ASF could be enforcing stricter >> security >> > > practices in the near future. Arrow Java may be forced to drop Java >> 11 if >> > > any of its dependencies no longer support Java 11. This is something >> > we'll >> > > have to investigate and monitor. When the time is right, we should >> start >> > a >> > > new thread on the mailing list to discuss. >> > > >> > > Thanks, >> > > Dane >> > > >> > > On Sat, May 4, 2024 at 2:51 AM <martin.trave...@icloud.com.invalid> >> > wrote: >> > > >> > > > Hi, >> > > > >> > > > We were originally expecting to keep Java 11 to the 2026 EOL date >> for >> > > > extended support, but now that date is moved to 2032 which feels >> like >> > > more >> > > > time than we need. The issue for us is that getting technology >> approved >> > > for >> > > > use in an enterprise can have ridiculously long lead times, so >> having a >> > > > minimum supported version that is only 2 years old, while probably >> ok >> > in >> > > > most case, would be a bit aggressive. We use optional dependencies >> > where >> > > we >> > > > can, so e.g. the Java 17 dependency for Spark 4 would only affect >> > clients >> > > > using Spark 4, and they could wait to upgrade. But we chose to use >> > Arrow >> > > in >> > > > the core of our product, it is the internal format everything else >> goes >> > > > through. On the compliance side we have to keep current with >> security >> > > > updates, so there is no option to stick on an old version. >> > > > >> > > > If we were to drop Java 11 after the next LTS comes out, i.e. 2025 / >> > > 2026, >> > > > then the three latest LTS versions would be supported and the >> minimum >> > > > version would have been available for 4 - 5 years. I think it would >> be >> > > very >> > > > hard to argue 17 can’t be made available at that point. If Arrow >> forces >> > > our >> > > > hand then obviously we’ll have to go sooner, but it wouldn’t be >> ideal >> > for >> > > > us. >> > > > >> > > > Lastly just on language capabilities, the only things we’re really >> > > > interested in are performance related, probably virtual threads and >> > > foreign >> > > > memory would be the main ones. Both of the those could be optional >> > > > dependencies, in the case of FFM we’d rely on either yourselves or >> > Netty >> > > > anyway to provide an allocator. So in fact there is very little >> benefit >> > > for >> > > > us to drop Java 11 early, all it costs us is one extra CI job. >> > > > >> > > > Hope some of this is helpful - apologies for the high latency, busy >> as >> > > > always!! >> > > > >> > > > Martin. >> > > > >> > > > >> > > > > On 1 May 2024, at 22:38, Dane Pitkin <d...@voltrondata.com.INVALID >> > >> > > > wrote: >> > > > > >> > > > > Thanks, Martin. It's great to hear of real-world use cases. Do you >> > > > > anticipate any timeline for dropping Java 11 for your product? If >> > Arrow >> > > > did >> > > > > drop Java 11, then it sounds like pinning Arrow Java to an older >> > > version >> > > > > wouldn't be an ideal option if security patches are not >> backported. >> > > > > >> > > > > Fokko, can you elaborate on the discussions held in other OSS >> > projects >> > > to >> > > > > drop Java <17? How did they weigh the benefits/drawbacks for >> dropping >> > > > both >> > > > > Java 8 and 11 LTS versions? I'd also be curious if other projects >> > plan >> > > to >> > > > > support older branches with security patches. >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > On Tue, Apr 30, 2024 at 4:14 PM <martin.trave...@icloud.com >> .invalid> >> > > > wrote: >> > > > > >> > > > >> Speaking for my own product we would like to see Java 11 >> support, we >> > > > rely >> > > > >> heavily on Arrow and have Java 11 as our minimum supported >> version. >> > > We’d >> > > > >> like to keep doing that if possible. Our clients are big >> enterprises >> > > > with >> > > > >> notoriously sluggish update cycles, so we want to offer maximum >> > > > >> compatibility. Once security patches are no longer available on >> the >> > > > regular >> > > > >> public channels then there is a compliance issue, so we generally >> > > follow >> > > > >> the EOL schedule of our dependencies. >> > > > >> >> > > > >> Corretto, Adoptium and Zulu all have recent public builds of >> both 8 >> > > and >> > > > 11 >> > > > >> and look set to support them with public builds for many years to >> > > come. >> > > > >> Several organisations I have worked with switched away from >> Oracle >> > > when >> > > > >> they made their licensing blunder with Java 8 and although that >> is >> > > > >> rectified now, the change seems to have stuck in quite a few >> places >> > > (at >> > > > >> least in my anecdotal experience). >> > > > >> >> > > > >> A major practical difference to me in Java 17 is the strong >> > > > encapsulation >> > > > >> of internals. Since that affects the majority of serious Java >> > > > applications >> > > > >> then perhaps most people have figured out by now to add the JVM >> > params >> > > > that >> > > > >> let Java continue working. Still, it could be a consideration, if >> > > > Java17 >> > > > >> is the baseline supported version. >> > > > >> >> > > > >> Regards, >> > > > >> Martin. >> > > > >> >> > > > >> - In case anyone is curious why we don’t support Java 8 per our >> own >> > > > >> policy, it’s because of the “var” keyword - seriously, why did >> Java >> > > > take so >> > > > >> long with that, even C++ got there sooner! >> > > > >> >> > > > >>> On 30 Apr 2024, at 16:20, Jacob Wujciak <assignu...@apache.org> >> > > wrote: >> > > > >>> >> > > > >>> Hello everyone! >> > > > >>> Great to see this move forward! >> > > > >>> +1 on dropping both 8 and 11 unless there is very good reason to >> > keep >> > > > 11 >> > > > >>> around. >> > > > >>> Otherwise people will just move to 11 and then have the pain of >> > > > migration >> > > > >>> again when we drop that (which will happen soon regardless imo). >> > > > >>> >> > > > >>> Am Di., 30. Apr. 2024 um 16:18 Uhr schrieb Dane Pitkin >> > > > >>> <d...@voltrondata.com.invalid>: >> > > > >>> >> > > > >>>> Thanks, JB. Are we aware of any downstream dependencies that >> would >> > > > >> benefit >> > > > >>>> from maintaining Java 11 support? Apache Spark jumped straight >> to >> > > Java >> > > > >> 17. >> > > > >>>> It seems other projects are dropping both 8 and 11 at the same >> > time >> > > as >> > > > >>>> mentioned by Fokko. From a maintenance perspective, it would be >> > nice >> > > > to >> > > > >>>> drop both. >> > > > >>>> >> > > > >>>> On Mon, Apr 29, 2024 at 11:20 AM Jean-Baptiste Onofré < >> > > > j...@nanthrax.net> >> > > > >>>> wrote: >> > > > >>>> >> > > > >>>>> Hi >> > > > >>>>> >> > > > >>>>> I think it's time to drop JDK8 support. I would say that we >> > should >> > > > >>>>> keep Java11 (jumping directly to Java17 would be problematic >> > > > >>>>> potentially for some users I guess). >> > > > >>>>> >> > > > >>>>> Regards >> > > > >>>>> JB >> > > > >>>>> >> > > > >>>>> On Thu, Apr 25, 2024 at 10:21 PM James Duong >> > > > >>>>> <james.du...@improving.com.invalid> wrote: >> > > > >>>>>> >> > > > >>>>>> If we dropped JDK 8, we could use the JDK to compile >> > > > module-info.java >> > > > >>>>> files. Then we could remove the custom maven plugin we’re >> using >> > for >> > > > >>>>> compiling module-info.java files for JPMS support and get >> better >> > > IDE >> > > > >>>>> integration (as what we’re doing currently somewhat shoe-horns >> > > module >> > > > >>>>> information alongside JDK8 bytecode). >> > > > >>>>>> >> > > > >>>>>> From: Dane Pitkin <d...@voltrondata.com.INVALID> >> > > > >>>>>> Date: Thursday, April 25, 2024 at 1:02 PM >> > > > >>>>>> To: dev@arrow.apache.org <dev@arrow.apache.org> >> > > > >>>>>> Subject: [DISCUSS] Drop Java 8 support >> > > > >>>>>> Hi all, >> > > > >>>>>> >> > > > >>>>>> I would like to revisit the discussion of dropping Java 8 >> (and >> > > maybe >> > > > >>>> 11) >> > > > >>>>>> from Arrow's Java implementation. See GH issue[1] below. This >> > was >> > > > also >> > > > >>>>>> discussed in the last Arrow community sync meeting on >> > 2024-04-24. >> > > > >>>>>> >> > > > >>>>>> For context, this was discussed[2] last year on this mailing >> > list. >> > > > We >> > > > >>>>>> decided to revisit the discussion around the June 2024 >> release >> > > > (Arrow >> > > > >>>>> v17). >> > > > >>>>>> The timing coincides with the initial release of Apache Spark >> > > 4.0.0, >> > > > >>>>> which >> > > > >>>>>> drops both Java 8 and 11 support. >> > > > >>>>>> >> > > > >>>>>> For background, we chose not to drop Java 8 support last year >> > > > because >> > > > >>>>> Arrow >> > > > >>>>>> is seen as a low level library that should support as many >> > > > >> environments >> > > > >>>>> as >> > > > >>>>>> possible. Nowadays, we see more enthusiasm for dropping Java >> 8 >> > > (and >> > > > >> 11) >> > > > >>>>> as >> > > > >>>>>> exemplified by Apache Spark as well as Apache Iceberg[3]. >> > > > >>>>>> >> > > > >>>>>> Is it time to consider dropping Java 8? Should we drop Java >> 11 >> > and >> > > > >> skip >> > > > >>>>>> straight to Java 17 as our minimum version? What >> implications do >> > > we >> > > > >>>> need >> > > > >>>>> to >> > > > >>>>>> be aware of? >> > > > >>>>>> >> > > > >>>>>> Thanks, >> > > > >>>>>> Dane >> > > > >>>>>> >> > > > >>>>>> [1]https://github.com/apache/arrow/issues/38051 >> > > > >>>>>> [2] >> > > https://lists.apache.org/thread/s07jx58yw4mkl54t3bkggnyg0sftcrr8 >> > > > >>>>>> [3] >> > > https://lists.apache.org/thread/ntrk2thvsg9tdccwd4flsdz9gg743368 >> > > > >>>>> >> > > > >>>> >> > > > >> >> > > > >> >> > > > >> > > > >> > > >> > >> >
