Feeling sympathetic to Pierre's point of view here. Picking specifically on the repository encryption issue itself, we have deprecation of a fundamental and well-known capability of NiFi. Repository encryption has been a key highlight and feature for those using NiFi in highly secure enclaves that maintain the strictest of auditing oversight. This change will no doubt cause reassessment efforts and scrutiny over what will be perceived as a loss of information security.
The justification for the change is probably warranted. Maybe repository encryption should never have been a problem NiFi attempted to solve. But regardless, the change was made without reasonable discussion, at minimum, here on the mailing lists where an Apache project is supposed to have such discussions. - The PR to remove this capability was created 5 days ago. [1] - The github PR was accepted and merged 3 days ago, on July 4th, a US national holiday no less. [2] - The deprecated components page was updated to reflect this change *retroactively* 1 day ago. [3] Granted, I might not have seen additional mailing list discussions on this topic. I grepped my inbox, but maybe I missed it. But still, for a security feature to be removed in the timeline outlined above, even if it was brought to the mailing list, would have been aggressive. So yes, there's a smell here. Clearly there needs to be a balance of doing small/normal changes without a lot (or any) needed discussion on the mailing lists, using Jira as the main project driver. But likewise, there has to be a sense of when to promote a discussion (or even awareness) to the mailing list level. A removed feature that has any sort of touchpoint to information security should most definitely be brought forward here. I'm supportive of the removal, to be clear. But I'm not supportive of how it was handled. And saying we should need "*specific examples where something looks like it was handled poorly*" is a good way to close off conversation. That's a lot of work looking back at a ship that has already sailed. It's better to trust that an impacting feature will be raised to the larger audience than to hunt out and find surprises after the fact. Community trust is an essential aspect of an Apache OSS project. [1] https://issues.apache.org/jira/browse/NIFI-13494 [2] https://github.com/apache/nifi/pull/9039/commits [3] https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=235834208 On Sun, Jul 7, 2024 at 1:34 PM Joe Witt <joe.w...@gmail.com> wrote: > Pierre, > > We are seeing active maintenance on the 1.x line and it is by the same > people active on the 2.x line. Is there a specific concern to highlight > and discuss? > > From the previous email the reference to things being removed quickly and > without discussion is something we need to ensure is being handled well. > If there are specific examples where something looks like it was handled > poorly then let's surface them and address them. > > The items you noted in your latest email specifically having concerns with: > > Kafka 2.6 parity concern - Can you share more on what is missing? > Obviously supporting Kafka users within NiFi is quite important but I think > we're making the right steps here. Whatever gap we have now let's identify > and see who can knock it out. > > Kudu: Releases look very infrequent for Kudu. We're on the latest. We > have vulnerable library hits and they've been there for a very long time. > This is an ideal candidate for a repository for components not likely to > see active changes. > > Repository Encryption: Such a component is both highly security sensitive > but also a tremendous code/maintenance issue. Considering how key > management for it worked to this point we should collaborate with users to > understand how to better achieve the desired outcome. > > Legacy Kafka and Kudu are good candidates for an extensions repo for things > which will likely not see active maintenance. Establishing such a > repository and the build process and so on for it is a considerable effort. > Just needs a volunteer to build that infra/process out. > > Thanks > > > > On Sun, Jul 7, 2024 at 11:04 AM Pierre Villard < > pierre.villard...@gmail.com> > wrote: > > > I'm not discussing the motivations behind the removal of what has been > > removed this week. It makes sense. But we can't expect to have users > > quickly move to 2.x when the amount of things users will have to take > care > > of when moving to 2.x keeps growing and is very significant. > > > > For the extensions, I'm a bit surprised by the removal of some extensions > > such as Kudu, that we know is used a lot by NiFi users, and has been > > actively maintained over the last few years. The one thing that concerns > me > > the most is the removal of the Kafka 2.6 components knowing that we don't > > have feature parity, yet, with the new approach. I understand that we > have > > to do the cut at some point and there is no easy solution but again it > > means that we should likely actively maintain 1.x for a longer period of > > time. > > > > On the framework side, I'm a bit concerned by the removal of repository > > encryption which is also something many users are using. Not saying this > is > > perfect and should not be removed but we can't just say "switch to OS > level > > encryption / cloud provider encryption". Again, this will take a lot of > > time for users using all of those things to adjust. > > > > We are putting more and more expectations on users to move to 2.x when > they > > have been using NiFi for many many years and have systems running > smoothly. > > I'm not saying we should not clean things up, but by taking this > approach, > > I think we owe our community an active maintenance of the 1.x branch as > > many users will need time to upgrade. > > > > Le dim. 7 juil. 2024 à 15:35, Joe Witt <joe.w...@gmail.com> a écrit : > > > > > Pierre > > > > > > Which things that have been removed are problematic and which bits got > > > removed too fast? There is no intent to make migration harder but > there > > is > > > to make it more sustainable. > > > > > > The focus is getting things with less known usage but highest tech debt > > > removed. These are things with problematic socket code, limited > security > > > features, capabilities we made new versions for and ultimately which > have > > > minimal maintenance. > > > > > > If there is something controversial getting removed lets discuss it. I > > > dont think keeping any of this in the main codebase is workable but it > > can > > > prompt having an extensions repository for lesser maintained > components. > > > This would help communicate these components have limited maintenance > but > > > may still be useful. > > > > > > If the bar to clean things up is higher than the bar to add new we > have a > > > known quality problem. We either have a similar bar or we need other > > > solutions. > > > > > > If there is a desire to see some (all?) decomissioned components kept > and > > > placed in an extensions repo to ease migration or just end user needs > is > > > there anybody available and interested to do that work? > > > > > > Thanks > > > > > > On Sun, Jul 7, 2024 at 5:42 AM Pierre Villard < > > pierre.villard...@gmail.com > > > > > > > wrote: > > > > > > > Given all the things that are being removed in 2.x without even > having > > a > > > > single discussion about it (PR opened and merged within 48 hours), we > > > > should not expect users (especially very large ones) to move from 1.x > > to > > > > 2.x any time soon. Things that have been removed over the last few > days > > > are > > > > going to be extremely impactful and will require users to do a > > > significant > > > > amount of work / re-architecturing to move to 2.x. It's a bit sad to > > see > > > so > > > > many things being done without any discussion in the community. > > > > > > > > Le ven. 5 juil. 2024 à 20:11, Michael Moser <moser...@gmail.com> a > > > écrit : > > > > > > > > > My opinion is that the NiFi 1.x line is already in maintenance and > > > > doesn't > > > > > need new features or APIs, but bug fixes are still important to get > > in, > > > > and > > > > > security fixes are still critical. > > > > > > > > > > That said, this is an open source community-driven project, so if > you > > > can > > > > > find contributors and committers that want to put in extra work to > > > > support > > > > > 1.x, then why not? I'm sure many of us have seen something new in > > NiFi > > > > and > > > > > thought "I wouldn't have done that". But others in the community > > > thought > > > > > it was a good idea, so why not? > > > > > > > > > > I don't really agree with the argument that putting too much work > > into > > > > 1.x > > > > > will delay people's transition to 2.0. People will transition to > 2.0 > > > on > > > > > their own schedule, with myriad reasons for that schedule. As a > > > > community, > > > > > we can simply make recommendations and provide solutions that help > > > with a > > > > > preferred direction. > > > > > > > > > > Kind regards, > > > > > -- Mike > > > > > > > > > > > > > > > On Wed, Jul 3, 2024 at 1:27 PM Joe Witt <joe.w...@gmail.com> > wrote: > > > > > > > > > > > Judith, > > > > > > > > > > > > The Apache NiFi 2.x line is built on and requires Java 21 and > will > > > not > > > > > > support any versions prior. > > > > > > > > > > > > NiFi 1.x line is built on Java 8 and requires either Java 8, 11, > or > > > 17. > > > > > > > > > > > > The dates you mention which extend to 2030/etc.. for a particular > > > > > > distributor of JVMs is a function of what sort of support that > > > > commercial > > > > > > vendor is making available to customers and it is different from > > what > > > > > that > > > > > > means for the public generally. > > > > > > > > > > > > Notably though it is also a very different concern from critical > > > > > > dependencies we have and how they evolve. For example we rely on > > > things > > > > > > like Spring and Jetty and so on. And these all choose to adopt > > > > language > > > > > > features or discontinue key lines which tie to certain Java > > versions, > > > > > > etc.. We can't control whether they choose to move to new major > > > lines > > > > > when > > > > > > they fix vulnerabilities or whether they backport, etc.. We held > > the > > > > > line > > > > > > on Java 8 support far longer than I imagined we could but the > > current > > > > > > confluence of events/scenarios means we made the jump all the way > > to > > > > Java > > > > > > 21 (an LTS) release with the hopes that we can set a good clock > for > > > > > people > > > > > > to work with for some time. > > > > > > > > > > > > The Java 1.x line will continue to be available for users to > > download > > > > and > > > > > > will still see general maintenance to the extent there are > > > contributors > > > > > > available to do that. You also have access to vendor supported > > paths > > > > to > > > > > > consider there which is similar to the stated 'Extended Support > > > > > > Availability' of Java. But they too will have to navigate the > > > > complexity > > > > > > of the reported vulnerabilities. > > > > > > > > > > > > My experience in the enterprise customer space is that tolerance > > for > > > > > > components which contain reported vulnerabilities is far far > lower > > > than > > > > > > ever before. This in turn means the focus shifts to giving > people > > > > strong > > > > > > upgrade paths to consider. > > > > > > > > > > > > Thanks > > > > > > Joe > > > > > > > > > > > > On Wed, Jul 3, 2024 at 10:10 AM Maucieri, Judith < > copos...@ctc.com > > > > > > > > wrote: > > > > > > > > > > > > > If I may: One large obstacle to "our" shift to v2.x is the > > absence > > > > of > > > > > > > Java 8 support (unless I overlooked updates to the plan stated > in > > > > > > November > > > > > > > 2022 during release of version 1.19.0, Java 8 only remains in > > NiFi > > > > 1.x > > > > > > > releases, which you hinted at remaining accurate). > > > > > > > > > > > > > > I make this statement in support of multiple scenarios where we > > > > employ > > > > > > > Apache NiFi. CVEs that are not addressed in v1.x would be a > > major > > > > > > > concern. In our case, per requirements levied upon us, we must > > > > > maintain > > > > > > > compatibility with all active Long Term Support (LTS) versions > of > > > > Java. > > > > > > I > > > > > > > feel this is reinforced by LTS of RHEL 7, which just began and > > > > extends > > > > > > into > > > > > > > 2028 and employs (by default) Java 8. > > > > > > > > > > > > > > Note the LTS dates currently indicated (reference > > > > > > > https://en.wikipedia.org/wiki/Java_version_history): > > > > > > > - Java 8 December 2030 > > > > > > > - Java 11 January 2032 > > > > > > > - Java 17 September 2029 > > > > > > > - Java 21 September 2031 > > > > > > > > > > > > > > Thank you, > > > > > > > JM > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Matt Burgess <mattyb...@apache.org> > > > > > > > Sent: Tuesday, July 2, 2024 4:05 PM > > > > > > > To: dev@nifi.apache.org > > > > > > > Subject: [DISCUSS] 1.x Maintenance > > > > > > > > > > > > > > > > > !-------------------------------------------------------------------| > > > > > > > This Message Is From an External Sender > > > > > > > Please use caution when clicking links or opening > > > > > > > attachments. > > > > > > > > > > |-------------------------------------------------------------------! > > > > > > > > > > > > > > There have been some ongoing discussions [1,2] about what to > > bring > > > > back > > > > > > > for PRs to 1.x vs trying to push forward with 2.x. There are of > > > > course > > > > > > > great points from everyone. On the 2.x front, namely that 2.x > has > > > > many > > > > > > > improvements not just to components but the framework and UI as > > > well, > > > > > and > > > > > > > that we've seen less contributions to the maintenance on the > 1.x > > > > line. > > > > > On > > > > > > > the 1.x front that community members need to support 1.x for > > their > > > > > users > > > > > > > for the time being. > > > > > > > > > > > > > > I'm opening this discussion thread so we can collect everyone's > > > > > thoughts > > > > > > > so the PMC can make a sensible recommendation/decision moving > > > > forward. > > > > > > For > > > > > > > myself, I think all bug PRs should be backported where > > > > > prudent/possible, > > > > > > > and even new features (although that can be a difficult > backport > > > due > > > > to > > > > > > > moving the extensions in [3], but I recommend a separate PR > > against > > > > > 1.x, > > > > > > > hopefully a simple copy-paste if there are no Java 21-specific > > code > > > > > > issues, > > > > > > > but please run contrib-check against Java 8 first). > > > > > > > > > > > > > > I disagree with the "atrophy" sentiment from [2], a mature > > release > > > > line > > > > > > > normally experiences less contributions because it is mostly > > stable > > > > in > > > > > > its > > > > > > > own right. Guiding users to 2.x IMO is the right call but many > of > > > us > > > > > have > > > > > > > to deal with the fact that it doesn't usually happen overnight, > > > > > > especially > > > > > > > without a GA release. > > > > > > > > > > > > > > I opened this discussion with my opinions but if I may humbly > > speak > > > > for > > > > > > > the PMC, we need your voice, and we look forward to all of your > > > > > thoughts, > > > > > > > opinions, and questions. > > > > > > > > > > > > > > Thank you, > > > > > > > Matty B > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://issues.apache.org/jira/browse/NIFI-13196__;!!HkW_WPl1peM!oK4J-OlEsDT4Hl0lVpbgtfPr1qvI-MyLLzjIjPmf0O8cfZFlSIaP7B8Ei8u6ou8_hDUuBxGEIQDtUKArfA$ > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://github.com/apache/nifi/pull/9018__;!!HkW_WPl1peM!oK4J-OlEsDT4Hl0lVpbgtfPr1qvI-MyLLzjIjPmf0O8cfZFlSIaP7B8Ei8u6ou8_hDUuBxGEIQD2THE6MA$ > > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://issues.apache.org/jira/browse/NIFI-12998__;!!HkW_WPl1peM!oK4J-OlEsDT4Hl0lVpbgtfPr1qvI-MyLLzjIjPmf0O8cfZFlSIaP7B8Ei8u6ou8_hDUuBxGEIQD3D-iGiA$ > > > > > > > > > > > > > > > ----------------------------------------------------------------- > > > > > > > This message and any files transmitted within are intended > > > > > > > solely for the addressee or its representative and may contain > > > > > > > company proprietary information. If you are not the intended > > > > > > > recipient, notify the sender immediately and delete this > > > > > > > message. Publication, reproduction, forwarding, or content > > > > > > > disclosure is prohibited without the consent of the original > > > > > > > sender and may be unlawful. > > > > > > > > > > > > > > Concurrent Technologies Corporation and its Affiliates. > > > > > > > www.ctc.com 1-800-282-4392 > > > > > > > > ----------------------------------------------------------------- > > > > > > > > > > > > > > > > > > > > > > > > > > > >
