Fantastic progress! Looking forward to the new platform! Wooohooo! On Fri, Sep 15, 2023, 7:17 PM Joe Witt <joe.w...@gmail.com> wrote:
> Caveat to that - the build works right now as the Java release version in > the pom is set to 17. So we're building/running Java 21 but the form it is > being built in is enforcing Java 17. This is ensuring the Groovy compiler > can work. There is not current obvious path/option to build Groovy with > Java 21 so we likely should focus on removal of any groovy things requiring > compilation in our build. Groovy runs on Java 21 just fine but the eclipse > compiler as yet does not. Either way we're on track to get those tests and > groovy based source out. > > Still good progress on getting to NiFi 2.0 and Java 21 (which officially > releases Sep 19th). > > Thanks > > On Fri, Sep 15, 2023 at 4:03 PM Joe Witt <joe.w...@gmail.com> wrote: > > > Team > > > > Long story short we are now able to build and run NiFi main/2.x line on > > Java 21! > > > > We've made a lot of progress this week on top of all the great work > > already done for getting to NiFi 2.0 but specifically to this thread of > > making NiFi 2.0 be based on Java 21. > > > > There are a couple PRs outstanding but once they land I'll put up a PR > for > > this commit [1] and we will be building with Java 21-ea on the main line. > > The full clean build with all tests and all profiles I could find is now > > working locally and is also now running in my fork before I put up the PR > > [2], [3]. NiFi also runs on Java 21. We did have to make a bunch of > > updates to all things Groovy and we're reducing/eliminating a lot of > pieces > > that are poorly maintained or need fundamentally different > implementations > > in Java 21. The toolkit is likely to be removed it seems and we can > later > > introduce back specific pieces if/as needed but designed better/more > easily > > maintained. > > > > What would be ideal is we land a couple more key pieces like ensuring > > every deprecated component is actually removed and ensuring the > flow.xml.gz > > is entirely gone. Then we kick out a NiFi 2.0 M1 release for people to > > work with. > > > > [1] > > > https://github.com/joewitt/nifi/commit/c3d16d949f8153441c64a1f98fd641cf80178f43 > > [2] https://github.com/joewitt/nifi/actions/runs/6203527626 > > [3] https://github.com/joewitt/nifi/actions/runs/6203527625 > > > > Thanks > > Joe > > > > On Fri, Sep 15, 2023 at 1:06 PM Pierre Villard < > > pierre.villard...@gmail.com> wrote: > > > >> Templates are already out. Flow.xml being removed should be reviewed > soon > >> (PR was rebased today). I'm working on the removal of variables. I hope > to > >> get a PR for this in the next few days. > >> > >> Le ven. 15 sept. 2023, 19:22, Joe Witt <joe.w...@gmail.com> a écrit : > >> > >> > Timeline - we remain in full blitz mode to get things ready for 2.0. > No > >> > clear ETA but we need to be getting it out soon. At least a milestone > >> > release of it for people to work with. There is a big change needed > to > >> get > >> > rid of the flow.xml.gz in favor of the json form and that is in > >> progress. > >> > I am not sure offhand whether templates got the boot yet. > >> > > >> > Latest fun is wrestling our rather messy situation with Groovy in the > >> build > >> > as that seems not ready for Java 21 generally. > >> > > >> > > >> > > >> > On Fri, Sep 15, 2023 at 10:19 AM Ryan Hendrickson < > >> > ryan.andrew.hendrick...@gmail.com> wrote: > >> > > >> > > I think NiFi 2.x going to Java 21 for all the reasons outlined > makes a > >> > lot > >> > > of sense. > >> > > > >> > > Is there a timeline for 2.x? > >> > > > >> > > On Mon, Sep 11, 2023 at 5:00 AM Pierre Villard < > >> > > pierre.villard...@gmail.com> > >> > > wrote: > >> > > > >> > > > Thanks Joe, it makes total sense and I agree that the ones that > >> would > >> > > > likely be slow at adopting Java 21 would not go to NiFi 2.0 super > >> > quickly > >> > > > anyway. Being able to bring the latest and greatest in NiFi is > great > >> > and > >> > > > given all of the features announced in Java 21, I imagine a lot of > >> > > projects > >> > > > we depend on will be doing the same. > >> > > > > >> > > > Le jeu. 7 sept. 2023 à 19:36, Joe Witt <joe.w...@gmail.com> a > >> écrit : > >> > > > > >> > > > > Pierre > >> > > > > > >> > > > > A few concerns you raised so want to address my view on each: > >> > > > > > >> > > > > Will users be able to adopt Java 21 fast enough? > >> > > > > I share Brandon's view on that in terms of their adoption > >> timeline. > >> > It > >> > > > > will likely align well with NiFi 2.0 itself in my estimation. > >> > > > > > >> > > > > Will this delay NiFi 2.0? > >> > > > > If it would then I'd not be supportive. I don't think we need > to > >> > > bother > >> > > > > with adopting any of the features now. What I would like us to > >> have > >> > is > >> > > > the > >> > > > > option to adopt them as we progress. We should get 2.0 done > asap > >> and > >> > > if > >> > > > > this added delay then I'd be way less interested in this idea. > >> > > > > > >> > > > > Feature benefits of 21 and what will that bring? > >> > > > > Mark spoke well to the key one that stood out to me which was > the > >> new > >> > > > > threading model available. It would be awfully nice to leverage > >> that > >> > > for > >> > > > > the efficiency it represents and especially if it can reduce > some > >> of > >> > > our > >> > > > > heap usage which is valuable in cloud/shared compute contexts. > >> > > > > > >> > > > > Performance benefits of Java 21? > >> > > > > It appears from some analysis found with googling that Java 21 > >> brings > >> > > out > >> > > > > of the box 4-5% performance increases generally. Not amazing > but > >> > > useful. > >> > > > > > >> > > > > User experience otherwise with Java 21? > >> > > > > I believe it would be consistent with Java 17 for their point of > >> view > >> > > in > >> > > > > terms of install/config/etc.. > >> > > > > > >> > > > > My motivation for this is fairly pure honestly. Since we're > >> setting > >> > > our > >> > > > > new minimum bar that lives for as long as the 2.x release line > >> lives > >> > > I'd > >> > > > > like to set it at the current LTS available when we ship that > >> line as > >> > > > well. > >> > > > > > >> > > > > Thanks > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Thu, Sep 7, 2023 at 4:22 AM Brandon DeVries < > >> > > > brandon.devr...@gmail.com> > >> > > > > wrote: > >> > > > > > >> > > > > > +1 to requiring java 21. Starting off as "up to date" as > >> possibly > >> > > > makes a > >> > > > > > lot of sense, and some of the new features seem especially > >> relevant > >> > > to > >> > > > > NiFi. > >> > > > > > > >> > > > > > I definitely understand the concerns about organizations being > >> > > willing > >> > > > / > >> > > > > > able to approve Java 21... But those same organizations might > >> also > >> > be > >> > > > > > hesitant to move to NiFi 2.0. We will continue to support java > >> 17 & > >> > > > NiFi > >> > > > > > 1.x for some time, so hopefully those groups will have the > time > >> > they > >> > > > need > >> > > > > > to get approvals, do evaluations, and upgrade. > >> > > > > > > >> > > > > > Brandon > >> > > > > > ________________________________ > >> > > > > > From: Pierre Villard <pierre.villard...@gmail.com> > >> > > > > > Sent: Thursday, September 7, 2023 6:15:58 AM > >> > > > > > To: dev@nifi.apache.org <dev@nifi.apache.org> > >> > > > > > Subject: Re: [discuss] nifi 2.0 and Java 21… > >> > > > > > > >> > > > > > Hi all, > >> > > > > > > >> > > > > > I share the concerns raised by Chris regarding how quickly > >> users of > >> > > > NiFi > >> > > > > > will be able to adopt Java 21. > >> > > > > > > >> > > > > > While I'm definitely in favor of using the latest and > greatest, > >> > > > > especially > >> > > > > > when it brings to the table such significant features, we need > >> to > >> > be > >> > > > > > careful as it may significantly delay the adoption of NiFi 2.0 > >> in > >> > big > >> > > > > > companies where deploying Java 21 will take time. I > acknowledge > >> > that > >> > > > > going > >> > > > > > from Java 8 to Java 17 is certainly the same effort as going > >> from > >> > > Java > >> > > > 8 > >> > > > > to > >> > > > > > Java 21 but how quickly security-sensitive environments will > >> adopt > >> > a > >> > > > new > >> > > > > > release of Java that is completely new? > >> > > > > > > >> > > > > > In addition to that, it sounds like we would add a significant > >> > rework > >> > > > of > >> > > > > > the framework in NiFi 2.0 assuming we adopt Java 21 as the > >> minimum > >> > > > > version. > >> > > > > > Do we think this is going to significantly delay the first > >> release > >> > of > >> > > > > NiFi > >> > > > > > 2.0? We still have work to do but adding this on top could > delay > >> > the > >> > > > > > release, no? > >> > > > > > > >> > > > > > Finally, the features that Java 21 are bringing sound super > >> > > interesting > >> > > > > in > >> > > > > > the context of NiFi but do we already have an idea of what it > >> will > >> > > > > improve? > >> > > > > > is it the user experience, and if so, how will it change? is > it > >> the > >> > > > > > performance, and if so, do we have an idea of how things will > >> > > improve? > >> > > > > > > >> > > > > > Thanks, > >> > > > > > Pierre > >> > > > > > > >> > > > > > Le mer. 6 sept. 2023 à 23:07, Chris Sampson > >> > > > > > <chris.samp...@naimuri.com.invalid> a écrit : > >> > > > > > > >> > > > > > > Yeah, I understand the need to move to 21 as a minimum to > take > >> > > > > advantage > >> > > > > > of > >> > > > > > > its features. Hopefully the wider java ecosystem won't be an > >> > issue > >> > > in > >> > > > > the > >> > > > > > > short term. > >> > > > > > > > >> > > > > > > I just wanted the discussion to be clear about this being a > >> > change > >> > > to > >> > > > > the > >> > > > > > > Java baseline/minimum for NiFi 2.0. > >> > > > > > > > >> > > > > > > It's a +1 from me. > >> > > > > > > > >> > > > > > > On Wed, 6 Sept 2023, 19:01 Joe Witt, <joe.w...@gmail.com> > >> wrote: > >> > > > > > > > >> > > > > > > > Chris > >> > > > > > > > > >> > > > > > > > My suggestion is rooted in making Java 21 the minimum of > the > >> > NiFi > >> > > > 2.0 > >> > > > > > > > line. It would not work on Java 17. The reason for this > >> is so > >> > > > that > >> > > > > we > >> > > > > > > can > >> > > > > > > > leverage the longest duration of a given LTS line while > also > >> > > > > benefiting > >> > > > > > > > from the language improvements that affords. Maintaining > >> > > > > compatibility > >> > > > > > > > with future versions we generally have to do. But > whatever > >> the > >> > > > > minimum > >> > > > > > > > version we accept dictates which language features we can > >> > > leverage. > >> > > > > So > >> > > > > > > if > >> > > > > > > > it is 17 then we can't leverage anything from the 21 line > >> for > >> > > > > example. > >> > > > > > > > > >> > > > > > > > GIven the nature and timelines of LTS I don't really think > >> > there > >> > > is > >> > > > > the > >> > > > > > > > same burn in logic that we'd have all known in the past > >> before > >> > > the > >> > > > > > > > LTS/STS/etc.. release constructs existed. > >> > > > > > > > > >> > > > > > > > Thanks > >> > > > > > > > > >> > > > > > > > On Wed, Sep 6, 2023 at 10:53 AM Chris Sampson > >> > > > > > > > <chris.samp...@naimuri.com.invalid> wrote: > >> > > > > > > > > >> > > > > > > > > To be clear, is the discussion one of making Java 21 the > >> > > minimum > >> > > > > > > > > requirement for NiFi 2.0.0, or rather NiFi 2.x be > >> compatible > >> > > with > >> > > > > > Java > >> > > > > > > > 21, > >> > > > > > > > > while retaining Java 17 as a minimum? > >> > > > > > > > > > >> > > > > > > > > If we moved straight to a Java 21 requirement, will we > run > >> > into > >> > > > > > > > > compatibility issues that delay initial NiFi 2 release? > >> Will > >> > > the > >> > > > > move > >> > > > > > > to > >> > > > > > > > > Java 21 mean some organisations delay their migration to > >> > NiFi 2 > >> > > > > > through > >> > > > > > > > not > >> > > > > > > > > wanting to move to the latest Java LTS version until > it's > >> > had a > >> > > > > time > >> > > > > > > for > >> > > > > > > > > "settling" through security/bug patches, etc.? And are > >> either > >> > > of > >> > > > > > these > >> > > > > > > > > sufficient concern to pause Java 21 becoming the > >> requirement, > >> > > as > >> > > > we > >> > > > > > may > >> > > > > > > > > then need to extend NiFi 1.x maintenance for longer into > >> the > >> > > > > future? > >> > > > > > > > > > >> > > > > > > > > Generally, I'm in favour of moving to "latest and > >> greatest", > >> > > > > > > particularly > >> > > > > > > > > for LTS versions of technologies, but the rate of Java > >> > version > >> > > > > > adoption > >> > > > > > > > > across the community gives me pause. > >> > > > > > > > > > >> > > > > > > > > I certainly see the advantage of new Java features for > >> NiFi > >> > in > >> > > > Java > >> > > > > > 21, > >> > > > > > > > > such as the already mentioned virtual threads. > >> > > > > > > > > > >> > > > > > > > > On Wed, 6 Sept 2023, 18:34 Mike Thomsen, < > >> > > mikerthom...@gmail.com > >> > > > > > >> > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > +1 100% > >> > > > > > > > > > > >> > > > > > > > > > On Wed, Sep 6, 2023 at 11:48 AM Adam Taft < > >> > a...@adamtaft.com > >> > > > > >> > > > > > wrote: > >> > > > > > > > > > > >> > > > > > > > > > > Yes, please. +1 Exactly what Mark said. Virtual > >> threads > >> > > have > >> > > > > > > > potential > >> > > > > > > > > to > >> > > > > > > > > > > be extremely impactful to applications like NiFi. > >> > > > > > > > > > > > >> > > > > > > > > > > /Adam > >> > > > > > > > > > > > >> > > > > > > > > > > On Wed, Sep 6, 2023 at 7:26 AM Mark Payne < > >> > > > > marka...@hotmail.com> > >> > > > > > > > > wrote: > >> > > > > > > > > > > > >> > > > > > > > > > > > Thanks for bringing his up, Joe. > >> > > > > > > > > > > > > >> > > > > > > > > > > > I would definitely be a +1. I think the new > virtual > >> > > thread > >> > > > > > > concept > >> > > > > > > > > > would > >> > > > > > > > > > > > have great impact on us. > >> > > > > > > > > > > > It would allow us to significantly simplify our > >> > > scheduling > >> > > > > > logic, > >> > > > > > > > > which > >> > > > > > > > > > > > would help with code maintainability > >> > > > > > > > > > > > but would also make configuration simpler. This is > >> one > >> > of > >> > > > the > >> > > > > > > most > >> > > > > > > > > > > > difficult things for users to configure, > >> > > > > > > > > > > > and I would very much welcome the ability to > >> simplify > >> > > this. > >> > > > > It > >> > > > > > > > would > >> > > > > > > > > > > > likely also yield better off-heap memory > >> > > > > > > > > > > > utilization by reducing the number of native > threads > >> > > > > necessary. > >> > > > > > > > > > > > > >> > > > > > > > > > > > Thanks > >> > > > > > > > > > > > -Mark > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Sep 6, 2023, at 10:20 AM, Joe Witt < > >> > > > joe.w...@gmail.com> > >> > > > > > > > wrote: > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Team > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Thought it might be worth relighting this thread > >> with > >> > > > Java > >> > > > > 21 > >> > > > > > > GA > >> > > > > > > > > > > > imminent. > >> > > > > > > > > > > > > Given the timing we should give consideration to > >> > having > >> > > > > Java > >> > > > > > 21 > >> > > > > > > > as > >> > > > > > > > > > the > >> > > > > > > > > > > > > basis for nifi 2.x to buy maximum time with LTS > >> > > > alignment. > >> > > > > > > There > >> > > > > > > > > are > >> > > > > > > > > > > > also > >> > > > > > > > > > > > > a couple interesting language features we can > >> likely > >> > > take > >> > > > > > > > advantage > >> > > > > > > > > > of. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > What do you think? > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks > >> > > > > > > > > > > > > Joe > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > On Wed, Jun 21, 2023 at 6:21 AM David > Handermann < > >> > > > > > > > > > > > > exceptionfact...@apache.org> wrote: > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >> Hi Dirk, > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> Thanks for summarizing your findings in the > >> > referenced > >> > > > > Jira > >> > > > > > > > > issues. > >> > > > > > > > > > It > >> > > > > > > > > > > > >> sounds like subsequent discussion of Nashorn > >> support > >> > > may > >> > > > > be > >> > > > > > > > better > >> > > > > > > > > > on > >> > > > > > > > > > > a > >> > > > > > > > > > > > new > >> > > > > > > > > > > > >> thread. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> The Spring 6 and Jetty 11 upgrades are going to > >> > > require > >> > > > > > > > > significant > >> > > > > > > > > > > > work. > >> > > > > > > > > > > > >> One incremental step in that direction was > making > >> > Java > >> > > > 17 > >> > > > > > the > >> > > > > > > > > > minimum > >> > > > > > > > > > > > >> version, and upgrading to Jetty 10 should also > >> help > >> > > move > >> > > > > > > things > >> > > > > > > > > > > forward. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> Although compiling NiFi modules with a > reference > >> to > >> > > the > >> > > > > > > > standalone > >> > > > > > > > > > > > Nashorn > >> > > > > > > > > > > > >> library may introduce issues, there should be > >> other > >> > > > > options > >> > > > > > > for > >> > > > > > > > > > > > referencing > >> > > > > > > > > > > > >> the library at runtime. That requires custom > >> class > >> > > > > loading, > >> > > > > > > > which > >> > > > > > > > > > some > >> > > > > > > > > > > > >> Processors support, so that seems like the > >> general > >> > > > > direction > >> > > > > > > to > >> > > > > > > > > go. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> If you have additional findings, feel free to > >> start > >> > a > >> > > > new > >> > > > > > > > > developer > >> > > > > > > > > > > list > >> > > > > > > > > > > > >> thread and that may gather additional feedback. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> Regards, > >> > > > > > > > > > > > >> David Handermann > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> On Wed, Jun 21, 2023 at 12:17 AM Dirk Arends < > >> > > > > > > > > > > dirk.are...@fontis.com.au > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >> wrote: > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >>> Since initially raising concerns about the > move > >> to > >> > > Java > >> > > > > 17 > >> > > > > > > > losing > >> > > > > > > > > > > > >> Nashorn, > >> > > > > > > > > > > > >>> I have been investigating the suggestion to > use > >> > > Nashorn > >> > > > > as > >> > > > > > a > >> > > > > > > > > > > standalone > >> > > > > > > > > > > > >>> package as potential easier alternative to > >> GraalVM. > >> > > [1] > >> > > > > > > > > > > > >>> > >> > > > > > > > > > > > >>> While making some progress, a number of issues > >> have > >> > > > been > >> > > > > > > > > > encountered > >> > > > > > > > > > > > >> which > >> > > > > > > > > > > > >>> I haven't been able to resolve as yet. More > >> details > >> > > are > >> > > > > > > > included > >> > > > > > > > > in > >> > > > > > > > > > > > >>> relevant JIRA tickets, but summarising: > >> > > > > > > > > > > > >>> > >> > > > > > > > > > > > >>> - Building NiFi with a recent Nashorn > dependency > >> > > leads > >> > > > to > >> > > > > > > > errors > >> > > > > > > > > > > > >>> "Unsupported class file major version 61" [2] > >> > > > > > > > > > > > >>> - Building NiFi using Java 17 highlights > issues > >> > with > >> > > > the > >> > > > > > > > current > >> > > > > > > > > > > Jetty > >> > > > > > > > > > > > >>> version, which I believe would require an > >> upgrade > >> > > from > >> > > > > > 9.4.51 > >> > > > > > > > to > >> > > > > > > > > > > > 11.0.15 > >> > > > > > > > > > > > >>> [3] > >> > > > > > > > > > > > >>> - Jetty 11 then requires an upgrade of the > >> Spring > >> > > > > Framework > >> > > > > > > > > > version 5 > >> > > > > > > > > > > > to > >> > > > > > > > > > > > >> 6. > >> > > > > > > > > > > > >>> [4] > >> > > > > > > > > > > > >>> > >> > > > > > > > > > > > >>> The current steps to remove references to > >> > > "Javascript" > >> > > > > as a > >> > > > > > > > > > > > preinstalled > >> > > > > > > > > > > > >>> scripting language [5] are understandable, but > >> it > >> > > does > >> > > > > seem > >> > > > > > > > there > >> > > > > > > > > > is > >> > > > > > > > > > > > >> still > >> > > > > > > > > > > > >>> quite a bit to do before Nashorn or another > >> > external > >> > > > > > > scripting > >> > > > > > > > > > engine > >> > > > > > > > > > > > >> could > >> > > > > > > > > > > > >>> be used. > >> > > > > > > > > > > > >>> > >> > > > > > > > > > > > >>> [1] > >> > https://issues.apache.org/jira/browse/NIFI-11700 > >> > > : > >> > > > > Java > >> > > > > > > 17 > >> > > > > > > > > > > Nashorn > >> > > > > > > > > > > > >>> standalone support > >> > > > > > > > > > > > >>> [2] > >> > https://issues.apache.org/jira/browse/NIFI-11701 > >> > > : > >> > > > > > > Support > >> > > > > > > > > > > building > >> > > > > > > > > > > > >>> with > >> > > > > > > > > > > > >>> version 61 class files > >> > > > > > > > > > > > >>> [3] > >> > https://issues.apache.org/jira/browse/NIFI-11702 > >> > > : > >> > > > > > > Upgrade > >> > > > > > > > > > Jetty > >> > > > > > > > > > > to > >> > > > > > > > > > > > >>> version 11 > >> > > > > > > > > > > > >>> [4] > >> > https://issues.apache.org/jira/browse/NIFI-11703 > >> > > : > >> > > > > > > Upgrade > >> > > > > > > > > > Spring > >> > > > > > > > > > > > >>> Framework to version 6 > >> > > > > > > > > > > > >>> [5] > >> > https://issues.apache.org/jira/browse/NIFI-11713 > >> > > : > >> > > > > > Remove > >> > > > > > > > > > > > Deprecated > >> > > > > > > > > > > > >>> ECMAScript Support > >> > > > > > > > > > > > >>> > >> > > > > > > > > > > > >>> Regards, > >> > > > > > > > > > > > >>> Dirk Arends > >> > > > > > > > > > > > >>> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >
