I'm strongly in favor of reducing tech debt, and moving deliberately to a 2.0 release. I have a concern with just one processor that is currently marked as deprecated: PostHTTP. (I have not evaluated specifically any other deprecated components; I cannot say if there are or are not similar issues elsewhere.) I understand the rationale for deprecating this processor in that it eliminates a processor whose functionality is available elsewhere, specifically in the more flexible InvokeHTTP. However, in my experience and testing, PostHTTP performs transfers with far greater throughput than InvokeHTTP. I would not be in favor of removing PostHTTP unless/until InvokeHTTP is refactored to increase its throughput capability.
Has anyone else continued to use PostHTTP over InvokeHTTP for similar reasons? Or, is there a performance-related configuration for InvokeHTTP I may have missed? Also, in order to help facilitate a smooth transition to 2.0 from a user perspective, would it be advisable to add some sort of visual indicator in the UI for components that are currently annotated as @Deprecated? This might help users proactively modify their flow prior to a release that would otherwise break it. On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <bbe...@gmail.com> wrote: > With the merging of MiNiFi and registry into the NiFi repo, we've > actually gone more towards a mono-repo where everything is released > together. Eventually it would still be nice to have a smaller base > distribution containing just the framework and standard NARs, but it > is hard to tackle that until we provide a way for users to easily > obtain the NARs in some other way. > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <edward.ar...@gmail.com> > wrote: > > > > Given the major version number shift and the spliting up of processors > into > > multiple NAR's. I'd like to suggest that we start individually versioning > > individual NARs/ bundles. > > > > I can see this bringing a large number of benifits including making Nifi > > more deployable with things RPM, but also potentially allowing those that > > have to deploy managed Nifi instances easier to upgrade. > > > > Edward > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <ottobackwa...@gmail.com> wrote: > > > > > The issue with updating the aws sdk is if it breaks any one of the > > > processors. > > > the Web Gateway API invoke processor for example is not using a high > level > > > purpose build client and may break. > > > > > > If we change the aws version, we need to coordinate in such a way that > they > > > all > > > can come along reasonably. > > > IE: what happens if 1 or 2 break but the rest or OK? > > > > > > > > > > > > From: David Handermann <exceptionfact...@apache.org> > > > <exceptionfact...@apache.org> > > > Reply: dev@nifi.apache.org <dev@nifi.apache.org> <dev@nifi.apache.org> > > > Date: July 26, 2021 at 09:33:42 > > > To: dev@nifi.apache.org <dev@nifi.apache.org> <dev@nifi.apache.org> > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > Chris, > > > > > > Thanks for the reply and recommendations. It seems like some of the > work to > > > reorganize the module structure could be done outside of a major > release, > > > but it would be great to target any breaking changes for 2.0. Perhaps a > > > separate feature proposal on module restructuring, with the goal of > > > supporting optimized builds, would be a helpful way to move that part > of > > > the discussion forward. > > > > > > Regarding updating AWS SDK to version 2, it seems like that might be > > > possible now. I haven't taken a close look at the referencing > components, > > > so I'm not sure about the level of effort involved. Minor NiFi version > > > updates have incorporated major new versions of dependencies. For > example, > > > NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one > > > hand, including the AWS SDK update as part of a major release seems > > > helpful, but unless there are changes that break existing component > > > properties, upgrading the AWS SDK could be worked independently. > Others may > > > have more insight into particular usage of that library. > > > > > > Regards, > > > David Handermann > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson > > > <chris.samp...@naimuri.com.invalid> wrote: > > > > > > > Might be worth considering refactoring the build as part of this work > > > too, > > > > e.g. only building the bits of the repo affected by a commit, etc. - > > > > discussed briefly in previous threads but don't think any changes > made > > > yet. > > > > If NARs/components are likely to be split up and refactored then such > > > work > > > > around the build probably makes sense to consider. > > > > > > > > I've a couple of PRs open that include updates to Elasticsearch > versions > > > > already, although I stopped at 7.10.2 (after which Elastic changed > > > licence) > > > > in case there were licence concerns. But more work can be done to > tidy up > > > > the processors, absolutely. > > > > > > > > AWS libraries to v2 would seem a sensible move and a refactor of > those > > > > processors as well. > > > > > > > > > > > > Cheers, > > > > > > > > Chris Sampson > > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, < > > > exceptionfact...@apache.org> > > > > > > > wrote: > > > > > > > > > Thanks for pointing out the standard NAR bundles, Mark. There are a > > > > number > > > > > of components in the standard NAR bundles with particular > dependencies > > > > that > > > > > would make more sense in separate NARs. Reorganizing the standard > NAR > > > to > > > > > components with limited dependencies and wide applicability would > > > > > definitely help with future maintenance. > > > > > > > > > > Regards, > > > > > David Handermann > > > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <marka...@hotmail.com> > > > > wrote: > > > > > > > > > > > There’s also some code that exists in order to maintain backward > > > > > > compatibility in the repositories. I would very much like the > > > > > repositories > > > > > > to contain no unnecessary code. And swap file format supports > really > > > > old > > > > > > formats. And the old impls of the repositories themselves, like > > > > > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of > stuff > > > > > there > > > > > > that can be removed. And some methods in ProcessSession that are > > > never > > > > > used > > > > > > by any processor in the codebase but exists in the public API so > > > can’t > > > > be > > > > > > removed till 2.0. > > > > > > > > > > > > I think his is also a great time to clean up the “standard nar.” > At > > > > this > > > > > > point, it’s something like 70 MB. And many of the components > there > > > are > > > > > not > > > > > > really “standard” - things like connecting to FTP & SFTP > servers, XML > > > > > > processing, Jolt transform, etc. could potentially be moved into > > > other > > > > > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is > 6.9 MB > > > is > > > > > not > > > > > > necessary for stateless or minifi java. Lots of things probably > to > > > > > > reconsider within the standard nar. > > > > > > > > > > > > I definitely think this is a reasonable approach, to allow for a > 2.0 > > > > that > > > > > > is not a huge feature release but allows the project to be > simpler > > > and > > > > > more > > > > > > nimble. > > > > > > > > > > > > Thanks > > > > > > -Mark > > > > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen < > mikerthom...@gmail.com > > > > > <mailto: > > > > > > mikerthom...@gmail.com>> wrote: > > > > > > > > > > > > Russell, > > > > > > > > > > > > AFAICT from looking at Elastic's repos, the low level REST > client is > > > > > > still fine. > > > > > > > > > > > > > > > > > > > > > > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java > > > > > > > > > > > > Our Elasticsearch support is spread over two NARs at present. One > > > uses > > > > > > OkHttp and the other uses that low level Elastic REST client. > > > > > > Therefore, I think we're fine on licensing for the moment. > > > > > > > > > > > > Mike > > > > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman < > > > r...@windofkeltia.com > > > > > > <mailto:r...@windofkeltia.com>> wrote: > > > > > > > > > > > > Bringing up Elastic also reminds me that the Elastic framework > has > > > just > > > > > > recently transitioned out of Open Source, so to acknowledge that, > > > maybe > > > > > > some effort toward OpenSearch--I say this not understanding > exactly > > > how > > > > > > this sort of thing is considered in a large-scale, world-class > > > software > > > > > > project like Apache NiFi. (I'm not a contributor, just a grateful > > > > > > consumer.) > > > > > > > > > > > > Russ > > > > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote: > > > > > > Along with the itemized list for ancient components we should > look at > > > > > > updating versions of drivers, SDKs, etc. for external systems > such as > > > > > > Elasticsearch, Cassandra, etc. There may be breaking changes but > 2.0 > > > > > > is probably the right time to get things up to date to make them > more > > > > > > useful to more people. > > > > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough < > thena...@gmail.com > > > > > <mailto: > > > > > > thena...@gmail.com>> wrote: > > > > > > I'm a +1 for removing pretty much all of this stuff. There are > > > security > > > > > > implications to keeping old dependencies around, so the more old > code > > > > we > > > > > > can remove the better. I agree that eventually we need to move to > > > > > > supporting only Java 11+, and as our next release will probably > be > > > > about > > > > > 4 > > > > > > - 6 months from now that doesn't seem too soon. We could > potentially > > > > > break > > > > > > this in two and remove the deprecated processors and leave 1.x on > > > Java > > > > 8, > > > > > > and finally start on 2.x which would support only Java 11. I'm > unsure > > > > of > > > > > > what implications changing the date and time handling would have > - > > > for > > > > > > running systems that use long term historical logs, unexpected > > > impacts > > > > to > > > > > > time logging could be a problem. > > > > > > > > > > > > As Joe says I think feature work will have to be dedicated to > 2.x and > > > > we > > > > > > could support 1.x for security fixes for some period of time. 2.x > > > seems > > > > > > like a gargantuan task but it's probably time to get started. Not > > > sure > > > > > how > > > > > > we handle all open PRs and the transition between 1.x and 2.x. > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <joe.w...@gmail.com > > > <mailto: > > > > > > joe.w...@gmail.com>> wrote: > > > > > > > > > > > > Jon > > > > > > > > > > > > You're right we have to be careful and you're right there are > still > > > > > > significant Java 8 users out there. But we also have to be > careful > > > > > > about security and sustainability of the codebase. If we had > talked > > > > > > about this last year when that article came out I'd have agreed > it is > > > > > > too early. Interestingly that link seems to get updated and I > tried > > > > > > [1] and found more recent data (not sure how recent). Anyway it > > > > > > suggests Java 8 is still the top dog but we see good growth on > 11. In > > > > > > my $dayjob this aligns to what I'm seeing too. Customers didn't > seem > > > > > > to care about Java 11 until later half last year and now > suddenly it > > > > > > is all over the place. > > > > > > > > > > > > I think once we put out a NiFi 2.0 release we'd see rapid > decrease in > > > > > > work on the 1.x line just being blunt. We did this many years ago > > > > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a > year or > > > > > > so) but it was purely bug fix/security related bits. We would > need to > > > > > > do something similar. But feature work would almost certainly go > to > > > > > > the 2.x line. Maybe there are other workable models but my > instinct > > > > > > suggests this is likely to follow a similar path. > > > > > > > > > > > > ...anyway I agree it isn't that easy of a call to dump Java 8. We > > > > > > need to make the call in both the interests of the user base and > the > > > > > > contributor base of the community. > > > > > > > > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/ > > > > > > > > > > > > > > > > > > Thanks > > > > > > Joe > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <joe.w...@gmail.com > <mailto: > > > > > > joe.w...@gmail.com>> wrote: > > > > > > Russ > > > > > > > > > > > > Yeah the flow registry is a key part of it. But also now you can > > > > > > download the flow definition in JSON (upload i think is there now > > > > > > too). Templates offered a series of challenges such as we store > them > > > > > > in the flow definition which has made flows massive in an > unintended > > > > > > way which isn't fun for cluster behavior. > > > > > > > > > > > > We have a couple cases where we headed down a particular concept > and > > > > > > came up with better approaches later. We need to reconcile these > with > > > > > > the benefit of hindsight, and while being careful to be not > overly > > > > > > disruptive to existing users, to reduce the codebase/maintenance > > > > > > burden and allow continued evolution of the project. > > > > > > > > > > > > Thanks > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman < > > > r...@windofkeltia.com > > > > > > <mailto:r...@windofkeltia.com>> > > > > > > wrote: > > > > > > Joe, > > > > > > > > > > > > I apologize for the off-topic intrusion, but what replaces > templates? > > > > > > The Registry? Templates rocked and we have used them since 0.5.x. > > > > > > > > > > > > Russ > > > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote: > > > > > > David, > > > > > > > > > > > > I think this is a highly reasonable approach and such a focus > will > > > > > > greatly help make a 2.0 release far more approachable to knock > out. > > > > > > Not only that but tech debt reduction would help make work > towards > > > > > > major features we'd think about in a 'major release' sense more > > > > > > approachable. > > > > > > > > > > > > We should remove all deprecated things (as well as verify we > have the > > > > > > right list). We should remove/consider removal of deprecated > > > > > > concepts > > > > > > like templates. We should consider whether we can resolve the > > > > > > various > > > > > > ways we've handled what are now parameters down to one clean > > > > > > approach. > > > > > > We should remove options in the nifi.properties which turn out to > > > > > > never be used quite right (if there are). There is quite a bit we > > > > > > can > > > > > > do purely in the name of tech debt reduction. > > > > > > > > > > > > Lots to consider here but I think this is the right discussion. > > > > > > > > > > > > Than ks > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bbe...@gmail.com > > > <mailto: > > > > > > bbe...@gmail.com>> > > > > > > wrote: > > > > > > I'm a +1 for this... Not sure if this falls under "Removing > > > > > > Deprecated > > > > > > Components", but I think we should also look at anything that has > > > > > > been > > > > > > marked as deprecated throughout the code base as a candidate for > > > > > > removal. There are quite a few classes, methods, properties, etc > > > > > > that > > > > > > have been waiting for a chance to be removed. > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann > > > > > > <exceptionfact...@apache.org<mailto:exceptionfact...@apache.org > >> > > > > wrote: > > > > > > Team, > > > > > > > > > > > > With all of the excellent work that many have contributed to NiFi > > > > > > over the > > > > > > years, the code base has also accumulated some amount of > technical > > > > > > debt. A > > > > > > handful of components have been marked as deprecated, and some > > > > > > components > > > > > > remain in the code base to support integration with old versions > > > > > > of various > > > > > > products. Following the principles of semantic versioning, > > > > > > introducing a > > > > > > major release would provide the opportunity to remove these > > > > > > deprecated and > > > > > > unsupported components. > > > > > > > > > > > > Rather than focusing the next major release on new features, what > > > > > > do you > > > > > > think about focusing on technical debt removal? This approach > > > > > > would not > > > > > > make for the most interesting release, but it provides the > > > > > > opportunity to > > > > > > clean up elements that involve breaking changes. > > > > > > > > > > > > Focusing on technical debt, at least three primary goals come to > > > > > > mind for > > > > > > the next major release: > > > > > > > > > > > > 1. Removal of deprecated and unmaintained components > > > > > > 2. Require Java 11 as the minimum supported version > > > > > > 3. Transition internal date and time handling to JSR 310 > java.time > > > > > > components > > > > > > > > > > > > *Removing Deprecated Components* > > > > > > > > > > > > Removing support for older and deprecated components provides a > > > > > > great > > > > > > opportunity to improve the overall security posture when it comes > > > > > > to > > > > > > maintaining dependencies. The OWASP dependency plugin report > > > > > > currently > > > > > > generates 50 MB of HTML for questionable dependencies, many of > > > > > > which are > > > > > > related to old versions of various libraries. > > > > > > > > > > > > As a starting point, here are a handful of components and > > > > > > extension modules > > > > > > that could be targeted for removal in a major version: > > > > > > > > > > > > - PostHTTP and GetHTTP > > > > > > - ListenLumberjack and the entire nifi-lumberjack-bundle > > > > > > - ListenBeats and the entire nifi-beats-bundle > > > > > > - Elasticsearch 5 components > > > > > > - Hive 1 and 2 components > > > > > > > > > > > > *Requiring Java 11* > > > > > > > > > > > > Java 8 is now over seven years old, and NiFi has supported > general > > > > > > compatibility with Java 11 for several years. NiFi 1.14.0 > > > > > > incorporated > > > > > > internal improvements specifically related to TLS 1.3, which > > > > > > allowed > > > > > > closing out the long-running Java 11 compatibility epic > NIFI-5174. > > > > > > Making > > > > > > Java 11 the minimum required version provides the opportunity to > > > > > > address > > > > > > any lingering edge cases and put NiFi in a better position to > > > > > > support > > > > > > current Java versions. > > > > > > > > > > > > *JSR 310 for Date and Time Handling* > > > > > > > > > > > > Without making the scope too broad, transitioning internal date > > > > > > and time > > > > > > handling to use DateTimeFormatter instead of SimpleDateFormat > > > > > > would provide > > > > > > a number of advantages. The Java Time components provide much > > > > > > better > > > > > > clarity when it comes to handling localized date and time > > > > > > representations, > > > > > > and also avoid the inherent confusion of java.sql.Date extending > > > > > > java.util.Date. Many internal components, specifically > > > > > > Record-oriented > > > > > > processors and services, rely on date parsing, leading to > > > > > > confusion and > > > > > > various workarounds. The pattern formats of SimpleDateFormat and > > > > > > DateTimeFormatter are very similar, but there are a few subtle > > > > > > differences. > > > > > > Making this transition would provide a much better foundation > > > > > > going forward. > > > > > > *Conclusion* > > > > > > > > > > > > Thanks for giving this proposal some consideration. Many of you > > > > > > have been > > > > > > developing NiFi for years and I look forward to your feedback. I > > > > > > would be > > > > > > glad to put together a more formalized recommendation on > > > > > > Confluence and > > > > > > write up Jira epics if this general approach sounds agreeable to > > > > > > the > > > > > > community. > > > > > > > > > > > > Regards, > > > > > > David Handermann > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
