Matt, Although it is technically possible to have encrypted repository implementations in a separate NAR, the original design did not take that approach. Improvements to the property configuration process also abstracted away the need to specify individual repository implementation classes.
The removal pull request [1] provides a good reference point for the classes and supporting modules that were necessary for repository encryption. This presents an opportunity for anyone to create a separate project based on the historical capabilities. With that being said, the original reasons for deprecating and removing repository encryption still stand. Refactoring the design to isolated modules would require a notable amount of work, as many of the implementation classes extend from the standard framework classes. Moving support to an optional profile does not solve the fundamental problems related to continued maintenance of a problematic implementation. Regards, David Handermann [1] https://github.com/apache/nifi/pull/9039 On Thu, Jul 18, 2024 at 1:49 PM Matt Burgess <mattyb...@gmail.com> wrote: > > Couldn't we restore the encrypted repository implementation as a NAR and > possibly make adding it a non-active profile by default? It doesn't solve > the security issue but it does remove it from the convenience binary yet > leaves it to be added by whoever wants to build NiFi their own way. > Arguably all things with public API interfaces (extensions and framework) > should be extensible so it probably should've been in its own NAR anyway, > like the Provenance Repository implementations. > > Regards, > Matt > > On Mon, Jul 8, 2024 at 12:43 AM Joe Witt <joe.w...@gmail.com> wrote: > > > Adam, > > > > We are all sympathetic to Pierre's view. As one of the people who > > participates in contributing code, code reviews, discussions, conducting > > releases, and more, his concerns and actions carry considerable weight for > > those of us that do so as well and the broader community. > > > > For repository encryption we should have raised a discussion thread. > > Fortunately every code change is reversible. Nothing is lost. If there is > > anything needing undone let's discuss them. There are no ships sailing. > > Just contributors contributing. > > > > If you keep the rest of the sentence in the quote you snipped it reads > > "let's surface them and address them" which does the opposite of close off > > conversation. > > > > Thanks > > Joe > > > > > > On Mon, Jul 8, 2024 at 12:07 AM Adam Taft <a...@adamtaft.com> wrote: > > > > > Feeling sympathetic to Pierre's point of view here. > > > > > > Picking specifically on the repository encryption issue itself, we have > > > deprecation of a fundamental and well-known capability of NiFi. > > Repository > > > encryption has been a key highlight and feature for those using NiFi in > > > highly secure enclaves that maintain the strictest of auditing oversight. > > > This change will no doubt cause reassessment efforts and scrutiny over > > what > > > will be perceived as a loss of information security. > > > > > > The justification for the change is probably warranted. Maybe repository > > > encryption should never have been a problem NiFi attempted to solve. But > > > regardless, the change was made without reasonable discussion, at > > minimum, > > > here on the mailing lists where an Apache project is supposed to have > > such > > > discussions. > > > > > > - The PR to remove this capability was created 5 days ago. [1] > > > - The github PR was accepted and merged 3 days ago, on July 4th, a US > > > national holiday no less. [2] > > > - The deprecated components page was updated to reflect this change > > > *retroactively* 1 day ago. [3] > > > > > > Granted, I might not have seen additional mailing list discussions on > > this > > > topic. I grepped my inbox, but maybe I missed it. But still, for a > > security > > > feature to be removed in the timeline outlined above, even if it was > > > brought to the mailing list, would have been aggressive. > > > > > > So yes, there's a smell here. Clearly there needs to be a balance of > > doing > > > small/normal changes without a lot (or any) needed discussion on the > > > mailing lists, using Jira as the main project driver. But likewise, there > > > has to be a sense of when to promote a discussion (or even awareness) to > > > the mailing list level. A removed feature that has any sort of touchpoint > > > to information security should most definitely be brought forward here. > > > > > > I'm supportive of the removal, to be clear. But I'm not supportive of how > > > it was handled. And saying we should need "*specific examples where > > > something looks like it was handled poorly*" is a good way to close off > > > conversation. That's a lot of work looking back at a ship that has > > already > > > sailed. It's better to trust that an impacting feature will be raised to > > > the larger audience than to hunt out and find surprises after the fact. > > > Community trust is an essential aspect of an Apache OSS project. > > > > > > [1] https://issues.apache.org/jira/browse/NIFI-13494 > > > [2] https://github.com/apache/nifi/pull/9039/commits > > > [3] > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=235834208 > > > > > > On Sun, Jul 7, 2024 at 1:34 PM Joe Witt <joe.w...@gmail.com> wrote: > > > > > > > Pierre, > > > > > > > > We are seeing active maintenance on the 1.x line and it is by the same > > > > people active on the 2.x line. Is there a specific concern to > > highlight > > > > and discuss? > > > > > > > > From the previous email the reference to things being removed quickly > > and > > > > without discussion is something we need to ensure is being handled > > well. > > > > If there are specific examples where something looks like it was > > handled > > > > poorly then let's surface them and address them. > > > > > > > > The items you noted in your latest email specifically having concerns > > > with: > > > > > > > > Kafka 2.6 parity concern - Can you share more on what is missing? > > > > Obviously supporting Kafka users within NiFi is quite important but I > > > think > > > > we're making the right steps here. Whatever gap we have now let's > > > identify > > > > and see who can knock it out. > > > > > > > > Kudu: Releases look very infrequent for Kudu. We're on the latest. > > We > > > > have vulnerable library hits and they've been there for a very long > > time. > > > > This is an ideal candidate for a repository for components not likely > > to > > > > see active changes. > > > > > > > > Repository Encryption: Such a component is both highly security > > sensitive > > > > but also a tremendous code/maintenance issue. Considering how key > > > > management for it worked to this point we should collaborate with users > > > to > > > > understand how to better achieve the desired outcome. > > > > > > > > Legacy Kafka and Kudu are good candidates for an extensions repo for > > > things > > > > which will likely not see active maintenance. Establishing such a > > > > repository and the build process and so on for it is a considerable > > > effort. > > > > Just needs a volunteer to build that infra/process out. > > > > > > > > Thanks > > > > > > > > > > > > > > > > On Sun, Jul 7, 2024 at 11:04 AM Pierre Villard < > > > > pierre.villard...@gmail.com> > > > > wrote: > > > > > > > > > I'm not discussing the motivations behind the removal of what has > > been > > > > > removed this week. It makes sense. But we can't expect to have users > > > > > quickly move to 2.x when the amount of things users will have to take > > > > care > > > > > of when moving to 2.x keeps growing and is very significant. > > > > > > > > > > For the extensions, I'm a bit surprised by the removal of some > > > extensions > > > > > such as Kudu, that we know is used a lot by NiFi users, and has been > > > > > actively maintained over the last few years. The one thing that > > > concerns > > > > me > > > > > the most is the removal of the Kafka 2.6 components knowing that we > > > don't > > > > > have feature parity, yet, with the new approach. I understand that we > > > > have > > > > > to do the cut at some point and there is no easy solution but again > > it > > > > > means that we should likely actively maintain 1.x for a longer period > > > of > > > > > time. > > > > > > > > > > On the framework side, I'm a bit concerned by the removal of > > repository > > > > > encryption which is also something many users are using. Not saying > > > this > > > > is > > > > > perfect and should not be removed but we can't just say "switch to OS > > > > level > > > > > encryption / cloud provider encryption". Again, this will take a lot > > of > > > > > time for users using all of those things to adjust. > > > > > > > > > > We are putting more and more expectations on users to move to 2.x > > when > > > > they > > > > > have been using NiFi for many many years and have systems running > > > > smoothly. > > > > > I'm not saying we should not clean things up, but by taking this > > > > approach, > > > > > I think we owe our community an active maintenance of the 1.x branch > > as > > > > > many users will need time to upgrade. > > > > > > > > > > Le dim. 7 juil. 2024 à 15:35, Joe Witt <joe.w...@gmail.com> a écrit > > : > > > > > > > > > > > Pierre > > > > > > > > > > > > Which things that have been removed are problematic and which bits > > > got > > > > > > removed too fast? There is no intent to make migration harder but > > > > there > > > > > is > > > > > > to make it more sustainable. > > > > > > > > > > > > The focus is getting things with less known usage but highest tech > > > debt > > > > > > removed. These are things with problematic socket code, limited > > > > security > > > > > > features, capabilities we made new versions for and ultimately > > which > > > > have > > > > > > minimal maintenance. > > > > > > > > > > > > If there is something controversial getting removed lets discuss > > > it. I > > > > > > dont think keeping any of this in the main codebase is workable but > > > it > > > > > can > > > > > > prompt having an extensions repository for lesser maintained > > > > components. > > > > > > This would help communicate these components have limited > > maintenance > > > > but > > > > > > may still be useful. > > > > > > > > > > > > If the bar to clean things up is higher than the bar to add new we > > > > have a > > > > > > known quality problem. We either have a similar bar or we need > > other > > > > > > solutions. > > > > > > > > > > > > If there is a desire to see some (all?) decomissioned components > > kept > > > > and > > > > > > placed in an extensions repo to ease migration or just end user > > needs > > > > is > > > > > > there anybody available and interested to do that work? > > > > > > > > > > > > Thanks > > > > > > > > > > > > On Sun, Jul 7, 2024 at 5:42 AM Pierre Villard < > > > > > pierre.villard...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Given all the things that are being removed in 2.x without even > > > > having > > > > > a > > > > > > > single discussion about it (PR opened and merged within 48 > > hours), > > > we > > > > > > > should not expect users (especially very large ones) to move from > > > 1.x > > > > > to > > > > > > > 2.x any time soon. Things that have been removed over the last > > few > > > > days > > > > > > are > > > > > > > going to be extremely impactful and will require users to do a > > > > > > significant > > > > > > > amount of work / re-architecturing to move to 2.x. It's a bit sad > > > to > > > > > see > > > > > > so > > > > > > > many things being done without any discussion in the community. > > > > > > > > > > > > > > Le ven. 5 juil. 2024 à 20:11, Michael Moser <moser...@gmail.com> > > a > > > > > > écrit : > > > > > > > > > > > > > > > My opinion is that the NiFi 1.x line is already in maintenance > > > and > > > > > > > doesn't > > > > > > > > need new features or APIs, but bug fixes are still important to > > > get > > > > > in, > > > > > > > and > > > > > > > > security fixes are still critical. > > > > > > > > > > > > > > > > That said, this is an open source community-driven project, so > > if > > > > you > > > > > > can > > > > > > > > find contributors and committers that want to put in extra work > > > to > > > > > > > support > > > > > > > > 1.x, then why not? I'm sure many of us have seen something new > > > in > > > > > NiFi > > > > > > > and > > > > > > > > thought "I wouldn't have done that". But others in the > > community > > > > > > thought > > > > > > > > it was a good idea, so why not? > > > > > > > > > > > > > > > > I don't really agree with the argument that putting too much > > work > > > > > into > > > > > > > 1.x > > > > > > > > will delay people's transition to 2.0. People will transition > > to > > > > 2.0 > > > > > > on > > > > > > > > their own schedule, with myriad reasons for that schedule. As > > a > > > > > > > community, > > > > > > > > we can simply make recommendations and provide solutions that > > > help > > > > > > with a > > > > > > > > preferred direction. > > > > > > > > > > > > > > > > Kind regards, > > > > > > > > -- Mike > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 3, 2024 at 1:27 PM Joe Witt <joe.w...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > Judith, > > > > > > > > > > > > > > > > > > The Apache NiFi 2.x line is built on and requires Java 21 and > > > > will > > > > > > not > > > > > > > > > support any versions prior. > > > > > > > > > > > > > > > > > > NiFi 1.x line is built on Java 8 and requires either Java 8, > > > 11, > > > > or > > > > > > 17. > > > > > > > > > > > > > > > > > > The dates you mention which extend to 2030/etc.. for a > > > particular > > > > > > > > > distributor of JVMs is a function of what sort of support > > that > > > > > > > commercial > > > > > > > > > vendor is making available to customers and it is different > > > from > > > > > what > > > > > > > > that > > > > > > > > > means for the public generally. > > > > > > > > > > > > > > > > > > Notably though it is also a very different concern from > > > critical > > > > > > > > > dependencies we have and how they evolve. For example we rely > > > on > > > > > > things > > > > > > > > > like Spring and Jetty and so on. And these all choose to > > adopt > > > > > > > language > > > > > > > > > features or discontinue key lines which tie to certain Java > > > > > versions, > > > > > > > > > etc.. We can't control whether they choose to move to new > > > major > > > > > > lines > > > > > > > > when > > > > > > > > > they fix vulnerabilities or whether they backport, etc.. We > > > held > > > > > the > > > > > > > > line > > > > > > > > > on Java 8 support far longer than I imagined we could but the > > > > > current > > > > > > > > > confluence of events/scenarios means we made the jump all the > > > way > > > > > to > > > > > > > Java > > > > > > > > > 21 (an LTS) release with the hopes that we can set a good > > clock > > > > for > > > > > > > > people > > > > > > > > > to work with for some time. > > > > > > > > > > > > > > > > > > The Java 1.x line will continue to be available for users to > > > > > download > > > > > > > and > > > > > > > > > will still see general maintenance to the extent there are > > > > > > contributors > > > > > > > > > available to do that. You also have access to vendor > > supported > > > > > paths > > > > > > > to > > > > > > > > > consider there which is similar to the stated 'Extended > > Support > > > > > > > > > Availability' of Java. But they too will have to navigate > > the > > > > > > > complexity > > > > > > > > > of the reported vulnerabilities. > > > > > > > > > > > > > > > > > > My experience in the enterprise customer space is that > > > tolerance > > > > > for > > > > > > > > > components which contain reported vulnerabilities is far far > > > > lower > > > > > > than > > > > > > > > > ever before. This in turn means the focus shifts to giving > > > > people > > > > > > > strong > > > > > > > > > upgrade paths to consider. > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > Joe > > > > > > > > > > > > > > > > > > On Wed, Jul 3, 2024 at 10:10 AM Maucieri, Judith < > > > > copos...@ctc.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > If I may: One large obstacle to "our" shift to v2.x is the > > > > > absence > > > > > > > of > > > > > > > > > > Java 8 support (unless I overlooked updates to the plan > > > stated > > > > in > > > > > > > > > November > > > > > > > > > > 2022 during release of version 1.19.0, Java 8 only remains > > in > > > > > NiFi > > > > > > > 1.x > > > > > > > > > > releases, which you hinted at remaining accurate). > > > > > > > > > > > > > > > > > > > > I make this statement in support of multiple scenarios > > where > > > we > > > > > > > employ > > > > > > > > > > Apache NiFi. CVEs that are not addressed in v1.x would be > > a > > > > > major > > > > > > > > > > concern. In our case, per requirements levied upon us, we > > > must > > > > > > > > maintain > > > > > > > > > > compatibility with all active Long Term Support (LTS) > > > versions > > > > of > > > > > > > Java. > > > > > > > > > I > > > > > > > > > > feel this is reinforced by LTS of RHEL 7, which just began > > > and > > > > > > > extends > > > > > > > > > into > > > > > > > > > > 2028 and employs (by default) Java 8. > > > > > > > > > > > > > > > > > > > > Note the LTS dates currently indicated (reference > > > > > > > > > > https://en.wikipedia.org/wiki/Java_version_history): > > > > > > > > > > - Java 8 December 2030 > > > > > > > > > > - Java 11 January 2032 > > > > > > > > > > - Java 17 September 2029 > > > > > > > > > > - Java 21 September 2031 > > > > > > > > > > > > > > > > > > > > Thank you, > > > > > > > > > > JM > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: Matt Burgess <mattyb...@apache.org> > > > > > > > > > > Sent: Tuesday, July 2, 2024 4:05 PM > > > > > > > > > > To: dev@nifi.apache.org > > > > > > > > > > Subject: [DISCUSS] 1.x Maintenance > > > > > > > > > > > > > > > > > > > > > > > > > > > > !-------------------------------------------------------------------| > > > > > > > > > > This Message Is From an External Sender > > > > > > > > > > Please use caution when clicking links or opening > > > > > > > > > > attachments. > > > > > > > > > > > > > > > > > > |-------------------------------------------------------------------! > > > > > > > > > > > > > > > > > > > > There have been some ongoing discussions [1,2] about what > > to > > > > > bring > > > > > > > back > > > > > > > > > > for PRs to 1.x vs trying to push forward with 2.x. There > > are > > > of > > > > > > > course > > > > > > > > > > great points from everyone. On the 2.x front, namely that > > 2.x > > > > has > > > > > > > many > > > > > > > > > > improvements not just to components but the framework and > > UI > > > as > > > > > > well, > > > > > > > > and > > > > > > > > > > that we've seen less contributions to the maintenance on > > the > > > > 1.x > > > > > > > line. > > > > > > > > On > > > > > > > > > > the 1.x front that community members need to support 1.x > > for > > > > > their > > > > > > > > users > > > > > > > > > > for the time being. > > > > > > > > > > > > > > > > > > > > I'm opening this discussion thread so we can collect > > > everyone's > > > > > > > > thoughts > > > > > > > > > > so the PMC can make a sensible recommendation/decision > > moving > > > > > > > forward. > > > > > > > > > For > > > > > > > > > > myself, I think all bug PRs should be backported where > > > > > > > > prudent/possible, > > > > > > > > > > and even new features (although that can be a difficult > > > > backport > > > > > > due > > > > > > > to > > > > > > > > > > moving the extensions in [3], but I recommend a separate PR > > > > > against > > > > > > > > 1.x, > > > > > > > > > > hopefully a simple copy-paste if there are no Java > > > 21-specific > > > > > code > > > > > > > > > issues, > > > > > > > > > > but please run contrib-check against Java 8 first). > > > > > > > > > > > > > > > > > > > > I disagree with the "atrophy" sentiment from [2], a mature > > > > > release > > > > > > > line > > > > > > > > > > normally experiences less contributions because it is > > mostly > > > > > stable > > > > > > > in > > > > > > > > > its > > > > > > > > > > own right. Guiding users to 2.x IMO is the right call but > > > many > > > > of > > > > > > us > > > > > > > > have > > > > > > > > > > to deal with the fact that it doesn't usually happen > > > overnight, > > > > > > > > > especially > > > > > > > > > > without a GA release. > > > > > > > > > > > > > > > > > > > > I opened this discussion with my opinions but if I may > > humbly > > > > > speak > > > > > > > for > > > > > > > > > > the PMC, we need your voice, and we look forward to all of > > > your > > > > > > > > thoughts, > > > > > > > > > > opinions, and questions. > > > > > > > > > > > > > > > > > > > > Thank you, > > > > > > > > > > Matty B > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://issues.apache.org/jira/browse/NIFI-13196__;!!HkW_WPl1peM!oK4J-OlEsDT4Hl0lVpbgtfPr1qvI-MyLLzjIjPmf0O8cfZFlSIaP7B8Ei8u6ou8_hDUuBxGEIQDtUKArfA$ > > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://github.com/apache/nifi/pull/9018__;!!HkW_WPl1peM!oK4J-OlEsDT4Hl0lVpbgtfPr1qvI-MyLLzjIjPmf0O8cfZFlSIaP7B8Ei8u6ou8_hDUuBxGEIQD2THE6MA$ > > > > > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://issues.apache.org/jira/browse/NIFI-12998__;!!HkW_WPl1peM!oK4J-OlEsDT4Hl0lVpbgtfPr1qvI-MyLLzjIjPmf0O8cfZFlSIaP7B8Ei8u6ou8_hDUuBxGEIQD3D-iGiA$ > > > > > > > > > > > > > > > > > > > > > > > > ----------------------------------------------------------------- > > > > > > > > > > This message and any files transmitted within are intended > > > > > > > > > > solely for the addressee or its representative and may > > > contain > > > > > > > > > > company proprietary information. If you are not the > > intended > > > > > > > > > > recipient, notify the sender immediately and delete this > > > > > > > > > > message. Publication, reproduction, forwarding, or content > > > > > > > > > > disclosure is prohibited without the consent of the > > original > > > > > > > > > > sender and may be unlawful. > > > > > > > > > > > > > > > > > > > > Concurrent Technologies Corporation and its Affiliates. > > > > > > > > > > www.ctc.com 1-800-282-4392 > > > > > > > > > > > > > > ----------------------------------------------------------------- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
