Mark, Thanks for the feedback. It may be better to start a separate thread on PostHTTP, but can you provide an example flow demonstrating the performance differences between PostHTTP and InvokeHTTP?
PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP uses OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for OkHttp, so it would be important to test with the most recent version. It is also worth noting that test classes for PostHTTP are now integration tests as opposed to unit tests, which means they are not executed as part of the automated builds. The NiFi documentation includes the contents of the DeprecationNotice for PostHTTP: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html Regards, David Handermann On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <[email protected]> wrote: > I'm strongly in favor of reducing tech debt, and moving deliberately to a > 2.0 release. I have a concern with just one processor that is currently > marked as deprecated: PostHTTP. (I have not evaluated specifically any > other deprecated components; I cannot say if there are or are not similar > issues elsewhere.) I understand the rationale for deprecating this > processor in that it eliminates a processor whose functionality is > available elsewhere, specifically in the more flexible InvokeHTTP. However, > in my experience and testing, PostHTTP performs transfers with far greater > throughput than InvokeHTTP. I would not be in favor of removing PostHTTP > unless/until InvokeHTTP is refactored to increase its throughput > capability. > > Has anyone else continued to use PostHTTP over InvokeHTTP for similar > reasons? Or, is there a performance-related configuration for InvokeHTTP I > may have missed? > > Also, in order to help facilitate a smooth transition to 2.0 from a user > perspective, would it be advisable to add some sort of visual indicator in > the UI for components that are currently annotated as @Deprecated? This > might help users proactively modify their flow prior to a release that > would otherwise break it. > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <[email protected]> wrote: > > > With the merging of MiNiFi and registry into the NiFi repo, we've > > actually gone more towards a mono-repo where everything is released > > together. Eventually it would still be nice to have a smaller base > > distribution containing just the framework and standard NARs, but it > > is hard to tackle that until we provide a way for users to easily > > obtain the NARs in some other way. > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <[email protected]> > > wrote: > > > > > > Given the major version number shift and the spliting up of processors > > into > > > multiple NAR's. I'd like to suggest that we start individually > versioning > > > individual NARs/ bundles. > > > > > > I can see this bringing a large number of benifits including making > Nifi > > > more deployable with things RPM, but also potentially allowing those > that > > > have to deploy managed Nifi instances easier to upgrade. > > > > > > Edward > > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <[email protected]> > wrote: > > > > > > > The issue with updating the aws sdk is if it breaks any one of the > > > > processors. > > > > the Web Gateway API invoke processor for example is not using a high > > level > > > > purpose build client and may break. > > > > > > > > If we change the aws version, we need to coordinate in such a way > that > > they > > > > all > > > > can come along reasonably. > > > > IE: what happens if 1 or 2 break but the rest or OK? > > > > > > > > > > > > > > > > From: David Handermann <[email protected]> > > > > <[email protected]> > > > > Reply: [email protected] <[email protected]> < > [email protected]> > > > > Date: July 26, 2021 at 09:33:42 > > > > To: [email protected] <[email protected]> <[email protected]> > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > > > Chris, > > > > > > > > Thanks for the reply and recommendations. It seems like some of the > > work to > > > > reorganize the module structure could be done outside of a major > > release, > > > > but it would be great to target any breaking changes for 2.0. > Perhaps a > > > > separate feature proposal on module restructuring, with the goal of > > > > supporting optimized builds, would be a helpful way to move that part > > of > > > > the discussion forward. > > > > > > > > Regarding updating AWS SDK to version 2, it seems like that might be > > > > possible now. I haven't taken a close look at the referencing > > components, > > > > so I'm not sure about the level of effort involved. Minor NiFi > version > > > > updates have incorporated major new versions of dependencies. For > > example, > > > > NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the > one > > > > hand, including the AWS SDK update as part of a major release seems > > > > helpful, but unless there are changes that break existing component > > > > properties, upgrading the AWS SDK could be worked independently. > > Others may > > > > have more insight into particular usage of that library. > > > > > > > > Regards, > > > > David Handermann > > > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson > > > > <[email protected]> wrote: > > > > > > > > > Might be worth considering refactoring the build as part of this > work > > > > too, > > > > > e.g. only building the bits of the repo affected by a commit, etc. > - > > > > > discussed briefly in previous threads but don't think any changes > > made > > > > yet. > > > > > If NARs/components are likely to be split up and refactored then > such > > > > work > > > > > around the build probably makes sense to consider. > > > > > > > > > > I've a couple of PRs open that include updates to Elasticsearch > > versions > > > > > already, although I stopped at 7.10.2 (after which Elastic changed > > > > licence) > > > > > in case there were licence concerns. But more work can be done to > > tidy up > > > > > the processors, absolutely. > > > > > > > > > > AWS libraries to v2 would seem a sensible move and a refactor of > > those > > > > > processors as well. > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Chris Sampson > > > > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, < > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > Thanks for pointing out the standard NAR bundles, Mark. There > are a > > > > > number > > > > > > of components in the standard NAR bundles with particular > > dependencies > > > > > that > > > > > > would make more sense in separate NARs. Reorganizing the standard > > NAR > > > > to > > > > > > components with limited dependencies and wide applicability would > > > > > > definitely help with future maintenance. > > > > > > > > > > > > Regards, > > > > > > David Handermann > > > > > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > There’s also some code that exists in order to maintain > backward > > > > > > > compatibility in the repositories. I would very much like the > > > > > > repositories > > > > > > > to contain no unnecessary code. And swap file format supports > > really > > > > > old > > > > > > > formats. And the old impls of the repositories themselves, like > > > > > > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of > > stuff > > > > > > there > > > > > > > that can be removed. And some methods in ProcessSession that > are > > > > never > > > > > > used > > > > > > > by any processor in the codebase but exists in the public API > so > > > > can’t > > > > > be > > > > > > > removed till 2.0. > > > > > > > > > > > > > > I think his is also a great time to clean up the “standard > nar.” > > At > > > > > this > > > > > > > point, it’s something like 70 MB. And many of the components > > there > > > > are > > > > > > not > > > > > > > really “standard” - things like connecting to FTP & SFTP > > servers, XML > > > > > > > processing, Jolt transform, etc. could potentially be moved > into > > > > other > > > > > > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is > > 6.9 MB > > > > is > > > > > > not > > > > > > > necessary for stateless or minifi java. Lots of things probably > > to > > > > > > > reconsider within the standard nar. > > > > > > > > > > > > > > I definitely think this is a reasonable approach, to allow for > a > > 2.0 > > > > > that > > > > > > > is not a huge feature release but allows the project to be > > simpler > > > > and > > > > > > more > > > > > > > nimble. > > > > > > > > > > > > > > Thanks > > > > > > > -Mark > > > > > > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen < > > [email protected] > > > > > > <mailto: > > > > > > > [email protected]>> wrote: > > > > > > > > > > > > > > Russell, > > > > > > > > > > > > > > AFAICT from looking at Elastic's repos, the low level REST > > client is > > > > > > > still fine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java > > > > > > > > > > > > > > Our Elasticsearch support is spread over two NARs at present. > One > > > > uses > > > > > > > OkHttp and the other uses that low level Elastic REST client. > > > > > > > Therefore, I think we're fine on licensing for the moment. > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman < > > > > [email protected] > > > > > > > <mailto:[email protected]>> wrote: > > > > > > > > > > > > > > Bringing up Elastic also reminds me that the Elastic framework > > has > > > > just > > > > > > > recently transitioned out of Open Source, so to acknowledge > that, > > > > maybe > > > > > > > some effort toward OpenSearch--I say this not understanding > > exactly > > > > how > > > > > > > this sort of thing is considered in a large-scale, world-class > > > > software > > > > > > > project like Apache NiFi. (I'm not a contributor, just a > grateful > > > > > > > consumer.) > > > > > > > > > > > > > > Russ > > > > > > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote: > > > > > > > Along with the itemized list for ancient components we should > > look at > > > > > > > updating versions of drivers, SDKs, etc. for external systems > > such as > > > > > > > Elasticsearch, Cassandra, etc. There may be breaking changes > but > > 2.0 > > > > > > > is probably the right time to get things up to date to make > them > > more > > > > > > > useful to more people. > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough < > > [email protected] > > > > > > <mailto: > > > > > > > [email protected]>> wrote: > > > > > > > I'm a +1 for removing pretty much all of this stuff. There are > > > > security > > > > > > > implications to keeping old dependencies around, so the more > old > > code > > > > > we > > > > > > > can remove the better. I agree that eventually we need to move > to > > > > > > > supporting only Java 11+, and as our next release will probably > > be > > > > > about > > > > > > 4 > > > > > > > - 6 months from now that doesn't seem too soon. We could > > potentially > > > > > > break > > > > > > > this in two and remove the deprecated processors and leave 1.x > on > > > > Java > > > > > 8, > > > > > > > and finally start on 2.x which would support only Java 11. I'm > > unsure > > > > > of > > > > > > > what implications changing the date and time handling would > have > > - > > > > for > > > > > > > running systems that use long term historical logs, unexpected > > > > impacts > > > > > to > > > > > > > time logging could be a problem. > > > > > > > > > > > > > > As Joe says I think feature work will have to be dedicated to > > 2.x and > > > > > we > > > > > > > could support 1.x for security fixes for some period of time. > 2.x > > > > seems > > > > > > > like a gargantuan task but it's probably time to get started. > Not > > > > sure > > > > > > how > > > > > > > we handle all open PRs and the transition between 1.x and 2.x. > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <[email protected] > > > > <mailto: > > > > > > > [email protected]>> wrote: > > > > > > > > > > > > > > Jon > > > > > > > > > > > > > > You're right we have to be careful and you're right there are > > still > > > > > > > significant Java 8 users out there. But we also have to be > > careful > > > > > > > about security and sustainability of the codebase. If we had > > talked > > > > > > > about this last year when that article came out I'd have agreed > > it is > > > > > > > too early. Interestingly that link seems to get updated and I > > tried > > > > > > > [1] and found more recent data (not sure how recent). Anyway it > > > > > > > suggests Java 8 is still the top dog but we see good growth on > > 11. In > > > > > > > my $dayjob this aligns to what I'm seeing too. Customers didn't > > seem > > > > > > > to care about Java 11 until later half last year and now > > suddenly it > > > > > > > is all over the place. > > > > > > > > > > > > > > I think once we put out a NiFi 2.0 release we'd see rapid > > decrease in > > > > > > > work on the 1.x line just being blunt. We did this many years > ago > > > > > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a > > year or > > > > > > > so) but it was purely bug fix/security related bits. We would > > need to > > > > > > > do something similar. But feature work would almost certainly > go > > to > > > > > > > the 2.x line. Maybe there are other workable models but my > > instinct > > > > > > > suggests this is likely to follow a similar path. > > > > > > > > > > > > > > ...anyway I agree it isn't that easy of a call to dump Java 8. > We > > > > > > > need to make the call in both the interests of the user base > and > > the > > > > > > > contributor base of the community. > > > > > > > > > > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/ > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > Joe > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <[email protected] > > <mailto: > > > > > > > [email protected]>> wrote: > > > > > > > Russ > > > > > > > > > > > > > > Yeah the flow registry is a key part of it. But also now you > can > > > > > > > download the flow definition in JSON (upload i think is there > now > > > > > > > too). Templates offered a series of challenges such as we store > > them > > > > > > > in the flow definition which has made flows massive in an > > unintended > > > > > > > way which isn't fun for cluster behavior. > > > > > > > > > > > > > > We have a couple cases where we headed down a particular > concept > > and > > > > > > > came up with better approaches later. We need to reconcile > these > > with > > > > > > > the benefit of hindsight, and while being careful to be not > > overly > > > > > > > disruptive to existing users, to reduce the > codebase/maintenance > > > > > > > burden and allow continued evolution of the project. > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman < > > > > [email protected] > > > > > > > <mailto:[email protected]>> > > > > > > > wrote: > > > > > > > Joe, > > > > > > > > > > > > > > I apologize for the off-topic intrusion, but what replaces > > templates? > > > > > > > The Registry? Templates rocked and we have used them since > 0.5.x. > > > > > > > > > > > > > > Russ > > > > > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote: > > > > > > > David, > > > > > > > > > > > > > > I think this is a highly reasonable approach and such a focus > > will > > > > > > > greatly help make a 2.0 release far more approachable to knock > > out. > > > > > > > Not only that but tech debt reduction would help make work > > towards > > > > > > > major features we'd think about in a 'major release' sense more > > > > > > > approachable. > > > > > > > > > > > > > > We should remove all deprecated things (as well as verify we > > have the > > > > > > > right list). We should remove/consider removal of deprecated > > > > > > > concepts > > > > > > > like templates. We should consider whether we can resolve the > > > > > > > various > > > > > > > ways we've handled what are now parameters down to one clean > > > > > > > approach. > > > > > > > We should remove options in the nifi.properties which turn out > to > > > > > > > never be used quite right (if there are). There is quite a bit > we > > > > > > > can > > > > > > > do purely in the name of tech debt reduction. > > > > > > > > > > > > > > Lots to consider here but I think this is the right discussion. > > > > > > > > > > > > > > Than ks > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <[email protected] > > > > <mailto: > > > > > > > [email protected]>> > > > > > > > wrote: > > > > > > > I'm a +1 for this... Not sure if this falls under "Removing > > > > > > > Deprecated > > > > > > > Components", but I think we should also look at anything that > has > > > > > > > been > > > > > > > marked as deprecated throughout the code base as a candidate > for > > > > > > > removal. There are quite a few classes, methods, properties, > etc > > > > > > > that > > > > > > > have been waiting for a chance to be removed. > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann > > > > > > > <[email protected]<mailto: > [email protected] > > >> > > > > > wrote: > > > > > > > Team, > > > > > > > > > > > > > > With all of the excellent work that many have contributed to > NiFi > > > > > > > over the > > > > > > > years, the code base has also accumulated some amount of > > technical > > > > > > > debt. A > > > > > > > handful of components have been marked as deprecated, and some > > > > > > > components > > > > > > > remain in the code base to support integration with old > versions > > > > > > > of various > > > > > > > products. Following the principles of semantic versioning, > > > > > > > introducing a > > > > > > > major release would provide the opportunity to remove these > > > > > > > deprecated and > > > > > > > unsupported components. > > > > > > > > > > > > > > Rather than focusing the next major release on new features, > what > > > > > > > do you > > > > > > > think about focusing on technical debt removal? This approach > > > > > > > would not > > > > > > > make for the most interesting release, but it provides the > > > > > > > opportunity to > > > > > > > clean up elements that involve breaking changes. > > > > > > > > > > > > > > Focusing on technical debt, at least three primary goals come > to > > > > > > > mind for > > > > > > > the next major release: > > > > > > > > > > > > > > 1. Removal of deprecated and unmaintained components > > > > > > > 2. Require Java 11 as the minimum supported version > > > > > > > 3. Transition internal date and time handling to JSR 310 > > java.time > > > > > > > components > > > > > > > > > > > > > > *Removing Deprecated Components* > > > > > > > > > > > > > > Removing support for older and deprecated components provides a > > > > > > > great > > > > > > > opportunity to improve the overall security posture when it > comes > > > > > > > to > > > > > > > maintaining dependencies. The OWASP dependency plugin report > > > > > > > currently > > > > > > > generates 50 MB of HTML for questionable dependencies, many of > > > > > > > which are > > > > > > > related to old versions of various libraries. > > > > > > > > > > > > > > As a starting point, here are a handful of components and > > > > > > > extension modules > > > > > > > that could be targeted for removal in a major version: > > > > > > > > > > > > > > - PostHTTP and GetHTTP > > > > > > > - ListenLumberjack and the entire nifi-lumberjack-bundle > > > > > > > - ListenBeats and the entire nifi-beats-bundle > > > > > > > - Elasticsearch 5 components > > > > > > > - Hive 1 and 2 components > > > > > > > > > > > > > > *Requiring Java 11* > > > > > > > > > > > > > > Java 8 is now over seven years old, and NiFi has supported > > general > > > > > > > compatibility with Java 11 for several years. NiFi 1.14.0 > > > > > > > incorporated > > > > > > > internal improvements specifically related to TLS 1.3, which > > > > > > > allowed > > > > > > > closing out the long-running Java 11 compatibility epic > > NIFI-5174. > > > > > > > Making > > > > > > > Java 11 the minimum required version provides the opportunity > to > > > > > > > address > > > > > > > any lingering edge cases and put NiFi in a better position to > > > > > > > support > > > > > > > current Java versions. > > > > > > > > > > > > > > *JSR 310 for Date and Time Handling* > > > > > > > > > > > > > > Without making the scope too broad, transitioning internal date > > > > > > > and time > > > > > > > handling to use DateTimeFormatter instead of SimpleDateFormat > > > > > > > would provide > > > > > > > a number of advantages. The Java Time components provide much > > > > > > > better > > > > > > > clarity when it comes to handling localized date and time > > > > > > > representations, > > > > > > > and also avoid the inherent confusion of java.sql.Date > extending > > > > > > > java.util.Date. Many internal components, specifically > > > > > > > Record-oriented > > > > > > > processors and services, rely on date parsing, leading to > > > > > > > confusion and > > > > > > > various workarounds. The pattern formats of SimpleDateFormat > and > > > > > > > DateTimeFormatter are very similar, but there are a few subtle > > > > > > > differences. > > > > > > > Making this transition would provide a much better foundation > > > > > > > going forward. > > > > > > > *Conclusion* > > > > > > > > > > > > > > Thanks for giving this proposal some consideration. Many of you > > > > > > > have been > > > > > > > developing NiFi for years and I look forward to your feedback. > I > > > > > > > would be > > > > > > > glad to put together a more formalized recommendation on > > > > > > > Confluence and > > > > > > > write up Jira epics if this general approach sounds agreeable > to > > > > > > > the > > > > > > > community. > > > > > > > > > > > > > > Regards, > > > > > > > David Handermann > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
