> > Kafka 2.6 parity concern - Can you share more on what is missing? > Obviously supporting Kafka users within NiFi is quite important but I think > we're making the right steps here. Whatever gap we have now let's identify > and see who can knock it out.
I cited a few in the PR description [1]. > - The PR as is should support the following authentication strategies: > PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL. Support for additional > authentication strategies could be handled via follow on JIRAs. > - Support for the KafkaRecordSink form factor could be handled via > follow on JIRAs. > - Support for Kafka "exactly once" semantics could be handled via > follow on JIRAs. > - Migration documentation could be handled via follow on JIRAs. > > This should not be considered a complete list. I consider it likely that there are others that I did not identify. I did try to capture that thinking in the PR [2]. Of note, there is a follow on JIRA [3] to flag the authentication features that were backed in *Kafka_2_6 by processor properties. There likely should be other JIRAs. In my opinion, this upgrade is more involved than the upgrade from *Kafka*_2_0 to *Kafka*_2_6, from a design perspective, an implementation perspective, and from the perspective of representing usages in a flow definition. This has the potential to cause a lot of problems for the community. It is hard for me to weigh all of the factors that should go into the retain / remove decision for Kafka 2.6, and I understand that I am a guest at this party. It does complicate things that the community is trying to close the book on NiFi 2.0 with a minimum of technical debt. But I'm not comfortable saying that the known (and unknown) functionality gaps can be addressed in the timeline I've seen discussed elsewhere for 2.0 GA. And I don't think that 2.0 should be held up for this issue. There is so much good in the main line, when compared against 1.x. [1] https://github.com/apache/nifi/pull/8463 [2] https://github.com/apache/nifi/pull/8463#issuecomment-2090939791 [3] https://issues.apache.org/jira/browse/NIFI-12952 On Mon, Jul 8, 2024 at 8:24 AM David Handermann <exceptionfact...@apache.org> wrote: > All, > > The discussion thus far has highlighted some important aspects of > project maintenance at both general and specific levels. Clearly > everyone shares a common goal to provide a supported and maintainable > product. The concerns mentioned reflect the natural tension in any > kind of technical debt evaluation, so the challenge is to determine > how to navigate those concerns. > > As Joe said, nothing is lost, the source code and previous releases > remain available for both internal and external use. > > Regarding the timeline for handling the removal of particular > features, it is important to note that nothing has occurred outside of > standard procedures. Every change has had a Jira issue, a pull > request, feedback, and signed-off commits. Regarding updates to the > deprecated features page, that has always occurred after merging, not > before, because it reflects committed changes, rather than proposed > changes. I understand the concern Pierre raised regarding the pace of > some changes, so if contributing members of the PMC see a need to have > a standard waiting period for Jira issues or pull requests, we should > consider that on a separate thread. Not having that right now, it > remains important that both removals and additions provide some sort > of rationale as background. Sometimes that only requires a sentence or > two, sometimes more, but that also characterizes the majority of > changes. We have had past discussions about the need for feature > proposals, and if it would improve maintainability to consider review > timelines, active contributors should discuss the options. For now, > however, raising issues even after merging changes continues to be a > valid way to handle things that may have been missed. > > On repository encryption itself, the conversation thus far rightly > acknowledged the problems noted in the Jira issue [1] NIFI-13494. > Removal of repository encryption is one several security-focused > removals, along with support for historical encryption processors and > application property protection. Although these removals will > undoubtedly impact some users, this is an area where we cannot > maintain extended support at the community level. Encryption features > include an inherent promise of security. If certain encryption > features have fundamental flaws, it is better to remove those features > than to hold on to compatibility. Major version changes are the > standard way to introduce breaking changes, and that has been one of > the primary themes for NiFi 2.0. With that understanding in mind, > there is value in considering future alternatives. If there are any > unanswered questions about security-focused feature removals, I would > be glad to provide more background as one who has both introduced > features and removed them during this process. > > The project has enjoyed significant adoption and contribution over the > years. The process of preparing for NiFi 2.0 has highlighted a number > of things, particularly that it is far easier to introduce > capabilities than to remove them. I believe we are on a solid path > forward. When we focus on specific concerns, such as those Pierre > raised regarding Kafka or Kudu, we can incorporate solutions as we go > forward. > > Regards, > David Handermann > > [1] https://issues.apache.org/jira/browse/NIFI-13494 > > On Sun, Jul 7, 2024 at 11:44 PM Joe Witt <joe.w...@gmail.com> wrote: > > > > Adam, > > > > We are all sympathetic to Pierre's view. As one of the people who > > participates in contributing code, code reviews, discussions, conducting > > releases, and more, his concerns and actions carry considerable weight > for > > those of us that do so as well and the broader community. > > > > For repository encryption we should have raised a discussion thread. > > Fortunately every code change is reversible. Nothing is lost. If there > is > > anything needing undone let's discuss them. There are no ships sailing. > > Just contributors contributing. > > > > If you keep the rest of the sentence in the quote you snipped it reads > > "let's surface them and address them" which does the opposite of close > off > > conversation. > > > > Thanks > > Joe > > > > > > On Mon, Jul 8, 2024 at 12:07 AM Adam Taft <a...@adamtaft.com> wrote: > > > > > Feeling sympathetic to Pierre's point of view here. > > > > > > Picking specifically on the repository encryption issue itself, we have > > > deprecation of a fundamental and well-known capability of NiFi. > Repository > > > encryption has been a key highlight and feature for those using NiFi in > > > highly secure enclaves that maintain the strictest of auditing > oversight. > > > This change will no doubt cause reassessment efforts and scrutiny over > what > > > will be perceived as a loss of information security. > > > > > > The justification for the change is probably warranted. Maybe > repository > > > encryption should never have been a problem NiFi attempted to solve. > But > > > regardless, the change was made without reasonable discussion, at > minimum, > > > here on the mailing lists where an Apache project is supposed to have > such > > > discussions. > > > > > > - The PR to remove this capability was created 5 days ago. [1] > > > - The github PR was accepted and merged 3 days ago, on July 4th, a US > > > national holiday no less. [2] > > > - The deprecated components page was updated to reflect this change > > > *retroactively* 1 day ago. [3] > > > > > > Granted, I might not have seen additional mailing list discussions on > this > > > topic. I grepped my inbox, but maybe I missed it. But still, for a > security > > > feature to be removed in the timeline outlined above, even if it was > > > brought to the mailing list, would have been aggressive. > > > > > > So yes, there's a smell here. Clearly there needs to be a balance of > doing > > > small/normal changes without a lot (or any) needed discussion on the > > > mailing lists, using Jira as the main project driver. But likewise, > there > > > has to be a sense of when to promote a discussion (or even awareness) > to > > > the mailing list level. A removed feature that has any sort of > touchpoint > > > to information security should most definitely be brought forward here. > > > > > > I'm supportive of the removal, to be clear. But I'm not supportive of > how > > > it was handled. And saying we should need "*specific examples where > > > something looks like it was handled poorly*" is a good way to close off > > > conversation. That's a lot of work looking back at a ship that has > already > > > sailed. It's better to trust that an impacting feature will be raised > to > > > the larger audience than to hunt out and find surprises after the fact. > > > Community trust is an essential aspect of an Apache OSS project. > > > > > > [1] https://issues.apache.org/jira/browse/NIFI-13494 > > > [2] https://github.com/apache/nifi/pull/9039/commits > > > [3] > > > > > > > https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=235834208 > > > > > > On Sun, Jul 7, 2024 at 1:34 PM Joe Witt <joe.w...@gmail.com> wrote: > > > > > > > Pierre, > > > > > > > > We are seeing active maintenance on the 1.x line and it is by the > same > > > > people active on the 2.x line. Is there a specific concern to > highlight > > > > and discuss? > > > > > > > > From the previous email the reference to things being removed > quickly and > > > > without discussion is something we need to ensure is being handled > well. > > > > If there are specific examples where something looks like it was > handled > > > > poorly then let's surface them and address them. > > > > > > > > The items you noted in your latest email specifically having concerns > > > with: > > > > > > > > Kafka 2.6 parity concern - Can you share more on what is missing? > > > > Obviously supporting Kafka users within NiFi is quite important but I > > > think > > > > we're making the right steps here. Whatever gap we have now let's > > > identify > > > > and see who can knock it out. > > > > > > > > Kudu: Releases look very infrequent for Kudu. We're on the > latest. We > > > > have vulnerable library hits and they've been there for a very long > time. > > > > This is an ideal candidate for a repository for components not > likely to > > > > see active changes. > > > > > > > > Repository Encryption: Such a component is both highly security > sensitive > > > > but also a tremendous code/maintenance issue. Considering how key > > > > management for it worked to this point we should collaborate with > users > > > to > > > > understand how to better achieve the desired outcome. > > > > > > > > Legacy Kafka and Kudu are good candidates for an extensions repo for > > > things > > > > which will likely not see active maintenance. Establishing such a > > > > repository and the build process and so on for it is a considerable > > > effort. > > > > Just needs a volunteer to build that infra/process out. > > > > > > > > Thanks > > > > > > > > > > > > > > > > On Sun, Jul 7, 2024 at 11:04 AM Pierre Villard < > > > > pierre.villard...@gmail.com> > > > > wrote: > > > > > > > > > I'm not discussing the motivations behind the removal of what has > been > > > > > removed this week. It makes sense. But we can't expect to have > users > > > > > quickly move to 2.x when the amount of things users will have to > take > > > > care > > > > > of when moving to 2.x keeps growing and is very significant. > > > > > > > > > > For the extensions, I'm a bit surprised by the removal of some > > > extensions > > > > > such as Kudu, that we know is used a lot by NiFi users, and has > been > > > > > actively maintained over the last few years. The one thing that > > > concerns > > > > me > > > > > the most is the removal of the Kafka 2.6 components knowing that we > > > don't > > > > > have feature parity, yet, with the new approach. I understand that > we > > > > have > > > > > to do the cut at some point and there is no easy solution but > again it > > > > > means that we should likely actively maintain 1.x for a longer > period > > > of > > > > > time. > > > > > > > > > > On the framework side, I'm a bit concerned by the removal of > repository > > > > > encryption which is also something many users are using. Not saying > > > this > > > > is > > > > > perfect and should not be removed but we can't just say "switch to > OS > > > > level > > > > > encryption / cloud provider encryption". Again, this will take a > lot of > > > > > time for users using all of those things to adjust. > > > > > > > > > > We are putting more and more expectations on users to move to 2.x > when > > > > they > > > > > have been using NiFi for many many years and have systems running > > > > smoothly. > > > > > I'm not saying we should not clean things up, but by taking this > > > > approach, > > > > > I think we owe our community an active maintenance of the 1.x > branch as > > > > > many users will need time to upgrade. > > > > > > > > > > Le dim. 7 juil. 2024 à 15:35, Joe Witt <joe.w...@gmail.com> a > écrit : > > > > > > > > > > > Pierre > > > > > > > > > > > > Which things that have been removed are problematic and which > bits > > > got > > > > > > removed too fast? There is no intent to make migration harder > but > > > > there > > > > > is > > > > > > to make it more sustainable. > > > > > > > > > > > > The focus is getting things with less known usage but highest > tech > > > debt > > > > > > removed. These are things with problematic socket code, limited > > > > security > > > > > > features, capabilities we made new versions for and ultimately > which > > > > have > > > > > > minimal maintenance. > > > > > > > > > > > > If there is something controversial getting removed lets discuss > > > it. I > > > > > > dont think keeping any of this in the main codebase is workable > but > > > it > > > > > can > > > > > > prompt having an extensions repository for lesser maintained > > > > components. > > > > > > This would help communicate these components have limited > maintenance > > > > but > > > > > > may still be useful. > > > > > > > > > > > > If the bar to clean things up is higher than the bar to add new > we > > > > have a > > > > > > known quality problem. We either have a similar bar or we need > other > > > > > > solutions. > > > > > > > > > > > > If there is a desire to see some (all?) decomissioned components > kept > > > > and > > > > > > placed in an extensions repo to ease migration or just end user > needs > > > > is > > > > > > there anybody available and interested to do that work? > > > > > > > > > > > > Thanks > > > > > > > > > > > > On Sun, Jul 7, 2024 at 5:42 AM Pierre Villard < > > > > > pierre.villard...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Given all the things that are being removed in 2.x without even > > > > having > > > > > a > > > > > > > single discussion about it (PR opened and merged within 48 > hours), > > > we > > > > > > > should not expect users (especially very large ones) to move > from > > > 1.x > > > > > to > > > > > > > 2.x any time soon. Things that have been removed over the last > few > > > > days > > > > > > are > > > > > > > going to be extremely impactful and will require users to do a > > > > > > significant > > > > > > > amount of work / re-architecturing to move to 2.x. It's a bit > sad > > > to > > > > > see > > > > > > so > > > > > > > many things being done without any discussion in the community. > > > > > > > > > > > > > > Le ven. 5 juil. 2024 à 20:11, Michael Moser < > moser...@gmail.com> a > > > > > > écrit : > > > > > > > > > > > > > > > My opinion is that the NiFi 1.x line is already in > maintenance > > > and > > > > > > > doesn't > > > > > > > > need new features or APIs, but bug fixes are still important > to > > > get > > > > > in, > > > > > > > and > > > > > > > > security fixes are still critical. > > > > > > > > > > > > > > > > That said, this is an open source community-driven project, > so if > > > > you > > > > > > can > > > > > > > > find contributors and committers that want to put in extra > work > > > to > > > > > > > support > > > > > > > > 1.x, then why not? I'm sure many of us have seen something > new > > > in > > > > > NiFi > > > > > > > and > > > > > > > > thought "I wouldn't have done that". But others in the > community > > > > > > thought > > > > > > > > it was a good idea, so why not? > > > > > > > > > > > > > > > > I don't really agree with the argument that putting too much > work > > > > > into > > > > > > > 1.x > > > > > > > > will delay people's transition to 2.0. People will > transition to > > > > 2.0 > > > > > > on > > > > > > > > their own schedule, with myriad reasons for that schedule. > As a > > > > > > > community, > > > > > > > > we can simply make recommendations and provide solutions that > > > help > > > > > > with a > > > > > > > > preferred direction. > > > > > > > > > > > > > > > > Kind regards, > > > > > > > > -- Mike > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 3, 2024 at 1:27 PM Joe Witt <joe.w...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > Judith, > > > > > > > > > > > > > > > > > > The Apache NiFi 2.x line is built on and requires Java 21 > and > > > > will > > > > > > not > > > > > > > > > support any versions prior. > > > > > > > > > > > > > > > > > > NiFi 1.x line is built on Java 8 and requires either Java > 8, > > > 11, > > > > or > > > > > > 17. > > > > > > > > > > > > > > > > > > The dates you mention which extend to 2030/etc.. for a > > > particular > > > > > > > > > distributor of JVMs is a function of what sort of support > that > > > > > > > commercial > > > > > > > > > vendor is making available to customers and it is different > > > from > > > > > what > > > > > > > > that > > > > > > > > > means for the public generally. > > > > > > > > > > > > > > > > > > Notably though it is also a very different concern from > > > critical > > > > > > > > > dependencies we have and how they evolve. For example we > rely > > > on > > > > > > things > > > > > > > > > like Spring and Jetty and so on. And these all choose to > adopt > > > > > > > language > > > > > > > > > features or discontinue key lines which tie to certain Java > > > > > versions, > > > > > > > > > etc.. We can't control whether they choose to move to new > > > major > > > > > > lines > > > > > > > > when > > > > > > > > > they fix vulnerabilities or whether they backport, etc.. > We > > > held > > > > > the > > > > > > > > line > > > > > > > > > on Java 8 support far longer than I imagined we could but > the > > > > > current > > > > > > > > > confluence of events/scenarios means we made the jump all > the > > > way > > > > > to > > > > > > > Java > > > > > > > > > 21 (an LTS) release with the hopes that we can set a good > clock > > > > for > > > > > > > > people > > > > > > > > > to work with for some time. > > > > > > > > > > > > > > > > > > The Java 1.x line will continue to be available for users > to > > > > > download > > > > > > > and > > > > > > > > > will still see general maintenance to the extent there are > > > > > > contributors > > > > > > > > > available to do that. You also have access to vendor > supported > > > > > paths > > > > > > > to > > > > > > > > > consider there which is similar to the stated 'Extended > Support > > > > > > > > > Availability' of Java. But they too will have to navigate > the > > > > > > > complexity > > > > > > > > > of the reported vulnerabilities. > > > > > > > > > > > > > > > > > > My experience in the enterprise customer space is that > > > tolerance > > > > > for > > > > > > > > > components which contain reported vulnerabilities is far > far > > > > lower > > > > > > than > > > > > > > > > ever before. This in turn means the focus shifts to giving > > > > people > > > > > > > strong > > > > > > > > > upgrade paths to consider. > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > Joe > > > > > > > > > > > > > > > > > > On Wed, Jul 3, 2024 at 10:10 AM Maucieri, Judith < > > > > copos...@ctc.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > If I may: One large obstacle to "our" shift to v2.x is > the > > > > > absence > > > > > > > of > > > > > > > > > > Java 8 support (unless I overlooked updates to the plan > > > stated > > > > in > > > > > > > > > November > > > > > > > > > > 2022 during release of version 1.19.0, Java 8 only > remains in > > > > > NiFi > > > > > > > 1.x > > > > > > > > > > releases, which you hinted at remaining accurate). > > > > > > > > > > > > > > > > > > > > I make this statement in support of multiple scenarios > where > > > we > > > > > > > employ > > > > > > > > > > Apache NiFi. CVEs that are not addressed in v1.x would > be a > > > > > major > > > > > > > > > > concern. In our case, per requirements levied upon us, > we > > > must > > > > > > > > maintain > > > > > > > > > > compatibility with all active Long Term Support (LTS) > > > versions > > > > of > > > > > > > Java. > > > > > > > > > I > > > > > > > > > > feel this is reinforced by LTS of RHEL 7, which just > began > > > and > > > > > > > extends > > > > > > > > > into > > > > > > > > > > 2028 and employs (by default) Java 8. > > > > > > > > > > > > > > > > > > > > Note the LTS dates currently indicated (reference > > > > > > > > > > https://en.wikipedia.org/wiki/Java_version_history): > > > > > > > > > > - Java 8 December 2030 > > > > > > > > > > - Java 11 January 2032 > > > > > > > > > > - Java 17 September 2029 > > > > > > > > > > - Java 21 September 2031 > > > > > > > > > > > > > > > > > > > > Thank you, > > > > > > > > > > JM > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: Matt Burgess <mattyb...@apache.org> > > > > > > > > > > Sent: Tuesday, July 2, 2024 4:05 PM > > > > > > > > > > To: dev@nifi.apache.org > > > > > > > > > > Subject: [DISCUSS] 1.x Maintenance > > > > > > > > > > > > > > > > > > > > > > > > > > > !-------------------------------------------------------------------| > > > > > > > > > > This Message Is From an External Sender > > > > > > > > > > Please use caution when clicking links or opening > > > > > > > > > > attachments. > > > > > > > > > > > > > > > > > |-------------------------------------------------------------------! > > > > > > > > > > > > > > > > > > > > There have been some ongoing discussions [1,2] about > what to > > > > > bring > > > > > > > back > > > > > > > > > > for PRs to 1.x vs trying to push forward with 2.x. There > are > > > of > > > > > > > course > > > > > > > > > > great points from everyone. On the 2.x front, namely > that 2.x > > > > has > > > > > > > many > > > > > > > > > > improvements not just to components but the framework > and UI > > > as > > > > > > well, > > > > > > > > and > > > > > > > > > > that we've seen less contributions to the maintenance on > the > > > > 1.x > > > > > > > line. > > > > > > > > On > > > > > > > > > > the 1.x front that community members need to support 1.x > for > > > > > their > > > > > > > > users > > > > > > > > > > for the time being. > > > > > > > > > > > > > > > > > > > > I'm opening this discussion thread so we can collect > > > everyone's > > > > > > > > thoughts > > > > > > > > > > so the PMC can make a sensible recommendation/decision > moving > > > > > > > forward. > > > > > > > > > For > > > > > > > > > > myself, I think all bug PRs should be backported where > > > > > > > > prudent/possible, > > > > > > > > > > and even new features (although that can be a difficult > > > > backport > > > > > > due > > > > > > > to > > > > > > > > > > moving the extensions in [3], but I recommend a separate > PR > > > > > against > > > > > > > > 1.x, > > > > > > > > > > hopefully a simple copy-paste if there are no Java > > > 21-specific > > > > > code > > > > > > > > > issues, > > > > > > > > > > but please run contrib-check against Java 8 first). > > > > > > > > > > > > > > > > > > > > I disagree with the "atrophy" sentiment from [2], a > mature > > > > > release > > > > > > > line > > > > > > > > > > normally experiences less contributions because it is > mostly > > > > > stable > > > > > > > in > > > > > > > > > its > > > > > > > > > > own right. Guiding users to 2.x IMO is the right call but > > > many > > > > of > > > > > > us > > > > > > > > have > > > > > > > > > > to deal with the fact that it doesn't usually happen > > > overnight, > > > > > > > > > especially > > > > > > > > > > without a GA release. > > > > > > > > > > > > > > > > > > > > I opened this discussion with my opinions but if I may > humbly > > > > > speak > > > > > > > for > > > > > > > > > > the PMC, we need your voice, and we look forward to all > of > > > your > > > > > > > > thoughts, > > > > > > > > > > opinions, and questions. > > > > > > > > > > > > > > > > > > > > Thank you, > > > > > > > > > > Matty B > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://issues.apache.org/jira/browse/NIFI-13196__;!!HkW_WPl1peM!oK4J-OlEsDT4Hl0lVpbgtfPr1qvI-MyLLzjIjPmf0O8cfZFlSIaP7B8Ei8u6ou8_hDUuBxGEIQDtUKArfA$ > > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://github.com/apache/nifi/pull/9018__;!!HkW_WPl1peM!oK4J-OlEsDT4Hl0lVpbgtfPr1qvI-MyLLzjIjPmf0O8cfZFlSIaP7B8Ei8u6ou8_hDUuBxGEIQD2THE6MA$ > > > > > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://issues.apache.org/jira/browse/NIFI-12998__;!!HkW_WPl1peM!oK4J-OlEsDT4Hl0lVpbgtfPr1qvI-MyLLzjIjPmf0O8cfZFlSIaP7B8Ei8u6ou8_hDUuBxGEIQD3D-iGiA$ > > > > > > > > > > > > > > > > > > > > > > > > ----------------------------------------------------------------- > > > > > > > > > > This message and any files transmitted within are > intended > > > > > > > > > > solely for the addressee or its representative and may > > > contain > > > > > > > > > > company proprietary information. If you are not the > intended > > > > > > > > > > recipient, notify the sender immediately and delete this > > > > > > > > > > message. Publication, reproduction, forwarding, or > content > > > > > > > > > > disclosure is prohibited without the consent of the > original > > > > > > > > > > sender and may be unlawful. > > > > > > > > > > > > > > > > > > > > Concurrent Technologies Corporation and its Affiliates. > > > > > > > > > > www.ctc.com 1-800-282-4392 > > > > > > > > > > > > > > ----------------------------------------------------------------- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
