I'm not seeing the side thread that was going to discuss deprecation of PostHTTP. Has that thread started and I just don't see it?
One (significant?) concern with PostHTTP is the smooth integration of NiFi-to-NiFi communication that is very transparently enabled with the ListenHTTP and PostHTTP processors. There's some special logic in there for handling flowfiles that InvokeHTTP doesn't really (nor should really) have. I know of several (large) NiFi installations that rely on the PostHTTP / ListenHTTP combination. It has enabled NiFi to NiFi communication for folks reluctant or unable to enable site-to-site type configuration. Honestly, I don't know why we'd want to "deprecate" any processor, as opposed to just moving it to a new location. If these processors can be ported and maintained to whatever the 2.0 API looks like, there's possibly little harm keeping them around. And by the way, what happened to the "marketplace" concept? Is this being considered for 2.0 as well? Because relocating the deprecated processors to an external nar might be the best solution. Losing PostHTTP entirely I think would be a mistake, but I'd conceptually support relocating it. Thanks, /Adam On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <joe.w...@gmail.com> wrote: > Looks like we just need to knock out 5 JIRAs :) [1] > > I felt like we had a label folks were using at one point but quickly > looking revealed nothing exciting. After this confluence page > stabilizes a bit we can probably knock out some JIRAs and such. > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599 > > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > > I find myself wishing I had a list of all the jiras / issues that have > > been put off for a 2.0 release because they required some change or > another > > :( > > > > From: Joe Witt <joe.w...@gmail.com> <joe.w...@gmail.com> > > Reply: dev@nifi.apache.org <dev@nifi.apache.org> <dev@nifi.apache.org> > > Date: July 27, 2021 at 12:30:35 > > To: dev@nifi.apache.org <dev@nifi.apache.org> <dev@nifi.apache.org> > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > A few thoughts: > > > > 1. I would love to see deprecation notices show up in the UI in > > various ways to help motivate users to move off things to more > > supportable things. That is not a prerequisite for anything happening > > however. Just a good feature/nice thing to do for users when someone > > is able to tackle it. > > > > 2. The decision to deprecate something and to further remove it need > > not mean there is a superior solution available. If that thing itself > > isn't getting the love/attention it needs to be > > maintained/supported/healthy going forward that alone is enough to > > remove it. That might well be the case with PostHTTP [1] and for > > comparison you can see how much effort has gone into InvokeHTTP [2]. > > > > 3. When discussing a 2.0 release each thing we add as a 'must do' the > > further away from reality such a release will become. We'll have to > > get very specific about 'musts' vs 'wants'. > > > > [1] > > > https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java > > [2] > > > https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java > > > > On Tue, Jul 27, 2021 at 8:47 AM David Handermann > > <exceptionfact...@apache.org> wrote: > > > > > > Thanks Mark, providing a template or comparison statistics with Java > > > versions and component configuration details would be very helpful. If > it > > > is possible to run tests using a public API or deployable service, that > > > would also help confirm potential differences. > > > > > > Showing a deprecation notice in the UI could be helpful, perhaps as a > > > configurable option. NIFI-8650 describes a general Flow Analysis > > > capability, and it seems like that might be one possible way to surface > > > deprecation warnings. For something more specific to component > > deprecation, > > > it seems important to find a balance between making it obvious and > making > > > it something that ends up getting ignored. > > > > > > Regards, > > > David Handermann > > > > > > On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <mark.o.b...@gmail.com> > wrote: > > > > > > > I'll start a new thread for PostHTTP when I get a template and/or > > detailed > > > > stats. > > > > > > > > I know the deprecation is noted in the documentation. That's a > > necessary > > > > and minimum level of notification. I was suggesting it be more > obvious > > in > > > > the UI. I think it would be beneficial to somehow be aware of the > > > > deprecation status simply by looking at the flow (perhaps even on the > > > > summary pages too), and without having to open the documentation for > > every > > > > processor to confirm whether or not a component is marked as > > deprecated. > > > > > > > > Thanks, > > > > Mark > > > > > > > > > > > > On Tue, Jul 27, 2021 at 11:16 AM David Handermann < > > > > exceptionfact...@apache.org> wrote: > > > > > > > > > Mark, > > > > > > > > > > Thanks for the feedback. It may be better to start a separate > thread > > on > > > > > PostHTTP, but can you provide an example flow demonstrating the > > > > performance > > > > > differences between PostHTTP and InvokeHTTP? > > > > > > > > > > PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP > > uses > > > > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for > > OkHttp, > > > > > so it would be important to test with the most recent version. It > is > > also > > > > > worth noting that test classes for PostHTTP are now integration > tests > > as > > > > > opposed to unit tests, which means they are not executed as part of > > the > > > > > automated builds. > > > > > > > > > > The NiFi documentation includes the contents of the > DeprecationNotice > > for > > > > > PostHTTP: > > > > > > > > > > > > > > > > > > > > > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html > > > > > > > > > > Regards, > > > > > David Handermann > > > > > > > > > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <mark.o.b...@gmail.com> > > wrote: > > > > > > > > > > > I'm strongly in favor of reducing tech debt, and moving > > deliberately > > > > to a > > > > > > 2.0 release. I have a concern with just one processor that is > > currently > > > > > > marked as deprecated: PostHTTP. (I have not evaluated > specifically > > any > > > > > > other deprecated components; I cannot say if there are or are not > > > > similar > > > > > > issues elsewhere.) I understand the rationale for deprecating > this > > > > > > processor in that it eliminates a processor whose functionality > is > > > > > > available elsewhere, specifically in the more flexible > InvokeHTTP. > > > > > However, > > > > > > in my experience and testing, PostHTTP performs transfers with > far > > > > > greater > > > > > > throughput than InvokeHTTP. I would not be in favor of removing > > > > PostHTTP > > > > > > unless/until InvokeHTTP is refactored to increase its throughput > > > > > > capability. > > > > > > > > > > > > Has anyone else continued to use PostHTTP over InvokeHTTP for > > similar > > > > > > reasons? Or, is there a performance-related configuration for > > > > InvokeHTTP > > > > > I > > > > > > may have missed? > > > > > > > > > > > > Also, in order to help facilitate a smooth transition to 2.0 > from a > > > > user > > > > > > perspective, would it be advisable to add some sort of visual > > indicator > > > > > in > > > > > > the UI for components that are currently annotated as > @Deprecated? > > This > > > > > > might help users proactively modify their flow prior to a release > > that > > > > > > would otherwise break it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <bbe...@gmail.com> > > wrote: > > > > > > > > > > > > > With the merging of MiNiFi and registry into the NiFi repo, > we've > > > > > > > actually gone more towards a mono-repo where everything is > > released > > > > > > > together. Eventually it would still be nice to have a smaller > > base > > > > > > > distribution containing just the framework and standard NARs, > but > > it > > > > > > > is hard to tackle that until we provide a way for users to > easily > > > > > > > obtain the NARs in some other way. > > > > > > > > > > > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes < > > edward.ar...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > Given the major version number shift and the spliting up of > > > > > processors > > > > > > > into > > > > > > > > multiple NAR's. I'd like to suggest that we start > individually > > > > > > versioning > > > > > > > > individual NARs/ bundles. > > > > > > > > > > > > > > > > I can see this bringing a large number of benifits including > > making > > > > > > Nifi > > > > > > > > more deployable with things RPM, but also potentially > allowing > > > > those > > > > > > that > > > > > > > > have to deploy managed Nifi instances easier to upgrade. > > > > > > > > > > > > > > > > Edward > > > > > > > > > > > > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, < > ottobackwa...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > The issue with updating the aws sdk is if it breaks any one > > of > > > > the > > > > > > > > > processors. > > > > > > > > > the Web Gateway API invoke processor for example is not > using > > a > > > > > high > > > > > > > level > > > > > > > > > purpose build client and may break. > > > > > > > > > > > > > > > > > > If we change the aws version, we need to coordinate in > such a > > way > > > > > > that > > > > > > > they > > > > > > > > > all > > > > > > > > > can come along reasonably. > > > > > > > > > IE: what happens if 1 or 2 break but the rest or OK? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: David Handermann <exceptionfact...@apache.org> > > > > > > > > > <exceptionfact...@apache.org> > > > > > > > > > Reply: dev@nifi.apache.org <dev@nifi.apache.org> < > > > > > > dev@nifi.apache.org> > > > > > > > > > Date: July 26, 2021 at 09:33:42 > > > > > > > > > To: dev@nifi.apache.org <dev@nifi.apache.org> < > > > > dev@nifi.apache.org > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > > > > > > > > > > > > > Chris, > > > > > > > > > > > > > > > > > > Thanks for the reply and recommendations. It seems like > some > > of > > > > the > > > > > > > work to > > > > > > > > > reorganize the module structure could be done outside of a > > major > > > > > > > release, > > > > > > > > > but it would be great to target any breaking changes for > 2.0. > > > > > > Perhaps a > > > > > > > > > separate feature proposal on module restructuring, with the > > goal > > > > of > > > > > > > > > supporting optimized builds, would be a helpful way to move > > that > > > > > part > > > > > > > of > > > > > > > > > the discussion forward. > > > > > > > > > > > > > > > > > > Regarding updating AWS SDK to version 2, it seems like that > > might > > > > > be > > > > > > > > > possible now. I haven't taken a close look at the > referencing > > > > > > > components, > > > > > > > > > so I'm not sure about the level of effort involved. Minor > > NiFi > > > > > > version > > > > > > > > > updates have incorporated major new versions of > dependencies. > > For > > > > > > > example, > > > > > > > > > NiFi 1.14 included an upgrade from Spring Framework 4 to 5. > > On > > > > the > > > > > > one > > > > > > > > > hand, including the AWS SDK update as part of a major > release > > > > seems > > > > > > > > > helpful, but unless there are changes that break existing > > > > component > > > > > > > > > properties, upgrading the AWS SDK could be worked > > independently. > > > > > > > Others may > > > > > > > > > have more insight into particular usage of that library. > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > David Handermann > > > > > > > > > > > > > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson > > > > > > > > > <chris.samp...@naimuri.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > Might be worth considering refactoring the build as part > of > > > > this > > > > > > work > > > > > > > > > too, > > > > > > > > > > e.g. only building the bits of the repo affected by a > > commit, > > > > > etc. > > > > > > - > > > > > > > > > > discussed briefly in previous threads but don't think any > > > > changes > > > > > > > made > > > > > > > > > yet. > > > > > > > > > > If NARs/components are likely to be split up and > refactored > > > > then > > > > > > such > > > > > > > > > work > > > > > > > > > > around the build probably makes sense to consider. > > > > > > > > > > > > > > > > > > > > I've a couple of PRs open that include updates to > > Elasticsearch > > > > > > > versions > > > > > > > > > > already, although I stopped at 7.10.2 (after which > Elastic > > > > > changed > > > > > > > > > licence) > > > > > > > > > > in case there were licence concerns. But more work can be > > done > > > > to > > > > > > > tidy up > > > > > > > > > > the processors, absolutely. > > > > > > > > > > > > > > > > > > > > AWS libraries to v2 would seem a sensible move and a > > refactor > > > > of > > > > > > > those > > > > > > > > > > processors as well. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > Chris Sampson > > > > > > > > > > > > > > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, < > > > > > > > > > exceptionfact...@apache.org> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks for pointing out the standard NAR bundles, Mark. > > There > > > > > > are a > > > > > > > > > > number > > > > > > > > > > > of components in the standard NAR bundles with > particular > > > > > > > dependencies > > > > > > > > > > that > > > > > > > > > > > would make more sense in separate NARs. Reorganizing > the > > > > > standard > > > > > > > NAR > > > > > > > > > to > > > > > > > > > > > components with limited dependencies and wide > > applicability > > > > > would > > > > > > > > > > > definitely help with future maintenance. > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > David Handermann > > > > > > > > > > > > > > > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne < > > > > > > marka...@hotmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > There’s also some code that exists in order to > maintain > > > > > > backward > > > > > > > > > > > > compatibility in the repositories. I would very much > > like > > > > the > > > > > > > > > > > repositories > > > > > > > > > > > > to contain no unnecessary code. And swap file format > > > > supports > > > > > > > really > > > > > > > > > > old > > > > > > > > > > > > formats. And the old impls of the repositories > > themselves, > > > > > like > > > > > > > > > > > > PersistentProvRepo instead of WriteAheadProv Repo, > etc. > > > > Lots > > > > > of > > > > > > > stuff > > > > > > > > > > > there > > > > > > > > > > > > that can be removed. And some methods in > ProcessSession > > > > that > > > > > > are > > > > > > > > > never > > > > > > > > > > > used > > > > > > > > > > > > by any processor in the codebase but exists in the > > public > > > > API > > > > > > so > > > > > > > > > can’t > > > > > > > > > > be > > > > > > > > > > > > removed till 2.0. > > > > > > > > > > > > > > > > > > > > > > > > I think his is also a great time to clean up the > > “standard > > > > > > nar.” > > > > > > > At > > > > > > > > > > this > > > > > > > > > > > > point, it’s something like 70 MB. And many of the > > > > components > > > > > > > there > > > > > > > > > are > > > > > > > > > > > not > > > > > > > > > > > > really “standard” - things like connecting to FTP & > > SFTP > > > > > > > servers, XML > > > > > > > > > > > > processing, Jolt transform, etc. could potentially be > > moved > > > > > > into > > > > > > > > > other > > > > > > > > > > > > nars. The > > nifi-standard-content-viewer-1.15.0-SNAPSHOT.war > > > > is > > > > > > > 6.9 MB > > > > > > > > > is > > > > > > > > > > > not > > > > > > > > > > > > necessary for stateless or minifi java. Lots of > things > > > > > probably > > > > > > > to > > > > > > > > > > > > reconsider within the standard nar. > > > > > > > > > > > > > > > > > > > > > > > > I definitely think this is a reasonable approach, to > > allow > > > > > for > > > > > > a > > > > > > > 2.0 > > > > > > > > > > that > > > > > > > > > > > > is not a huge feature release but allows the project > to > > be > > > > > > > simpler > > > > > > > > > and > > > > > > > > > > > more > > > > > > > > > > > > nimble. > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > -Mark > > > > > > > > > > > > > > > > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen < > > > > > > > mikerthom...@gmail.com > > > > > > > > > > > <mailto: > > > > > > > > > > > > mikerthom...@gmail.com>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > Russell, > > > > > > > > > > > > > > > > > > > > > > > > AFAICT from looking at Elastic's repos, the low level > > REST > > > > > > > client is > > > > > > > > > > > > still fine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java > > > > > > > > > > > > > > > > > > > > > > > > Our Elasticsearch support is spread over two NARs at > > > > present. > > > > > > One > > > > > > > > > uses > > > > > > > > > > > > OkHttp and the other uses that low level Elastic REST > > > > client. > > > > > > > > > > > > Therefore, I think we're fine on licensing for the > > moment. > > > > > > > > > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman < > > > > > > > > > r...@windofkeltia.com > > > > > > > > > > > > <mailto:r...@windofkeltia.com>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > Bringing up Elastic also reminds me that the Elastic > > > > > framework > > > > > > > has > > > > > > > > > just > > > > > > > > > > > > recently transitioned out of Open Source, so to > > acknowledge > > > > > > that, > > > > > > > > > maybe > > > > > > > > > > > > some effort toward OpenSearch--I say this not > > understanding > > > > > > > exactly > > > > > > > > > how > > > > > > > > > > > > this sort of thing is considered in a large-scale, > > > > > world-class > > > > > > > > > software > > > > > > > > > > > > project like Apache NiFi. (I'm not a contributor, > just > > a > > > > > > grateful > > > > > > > > > > > > consumer.) > > > > > > > > > > > > > > > > > > > > > > > > Russ > > > > > > > > > > > > > > > > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote: > > > > > > > > > > > > Along with the itemized list for ancient components > we > > > > should > > > > > > > look at > > > > > > > > > > > > updating versions of drivers, SDKs, etc. for external > > > > systems > > > > > > > such as > > > > > > > > > > > > Elasticsearch, Cassandra, etc. There may be breaking > > > > changes > > > > > > but > > > > > > > 2.0 > > > > > > > > > > > > is probably the right time to get things up to date > to > > make > > > > > > them > > > > > > > more > > > > > > > > > > > > useful to more people. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough < > > > > > > > thena...@gmail.com > > > > > > > > > > > <mailto: > > > > > > > > > > > > thena...@gmail.com>> wrote: > > > > > > > > > > > > I'm a +1 for removing pretty much all of this stuff. > > There > > > > > are > > > > > > > > > security > > > > > > > > > > > > implications to keeping old dependencies around, so > the > > > > more > > > > > > old > > > > > > > code > > > > > > > > > > we > > > > > > > > > > > > can remove the better. I agree that eventually we > need > > to > > > > > move > > > > > > to > > > > > > > > > > > > supporting only Java 11+, and as our next release > will > > > > > probably > > > > > > > be > > > > > > > > > > about > > > > > > > > > > > 4 > > > > > > > > > > > > - 6 months from now that doesn't seem too soon. We > > could > > > > > > > potentially > > > > > > > > > > > break > > > > > > > > > > > > this in two and remove the deprecated processors and > > leave > > > > > 1.x > > > > > > on > > > > > > > > > Java > > > > > > > > > > 8, > > > > > > > > > > > > and finally start on 2.x which would support only > Java > > 11. > > > > > I'm > > > > > > > unsure > > > > > > > > > > of > > > > > > > > > > > > what implications changing the date and time handling > > would > > > > > > have > > > > > > > - > > > > > > > > > for > > > > > > > > > > > > running systems that use long term historical logs, > > > > > unexpected > > > > > > > > > impacts > > > > > > > > > > to > > > > > > > > > > > > time logging could be a problem. > > > > > > > > > > > > > > > > > > > > > > > > As Joe says I think feature work will have to be > > dedicated > > > > to > > > > > > > 2.x and > > > > > > > > > > we > > > > > > > > > > > > could support 1.x for security fixes for some period > of > > > > time. > > > > > > 2.x > > > > > > > > > seems > > > > > > > > > > > > like a gargantuan task but it's probably time to get > > > > started. > > > > > > Not > > > > > > > > > sure > > > > > > > > > > > how > > > > > > > > > > > > we handle all open PRs and the transition between 1.x > > and > > > > > 2.x. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt < > > > > > joe.w...@gmail.com > > > > > > > > > <mailto: > > > > > > > > > > > > joe.w...@gmail.com>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > Jon > > > > > > > > > > > > > > > > > > > > > > > > You're right we have to be careful and you're right > > there > > > > are > > > > > > > still > > > > > > > > > > > > significant Java 8 users out there. But we also have > to > > be > > > > > > > careful > > > > > > > > > > > > about security and sustainability of the codebase. If > > we > > > > had > > > > > > > talked > > > > > > > > > > > > about this last year when that article came out I'd > > have > > > > > agreed > > > > > > > it is > > > > > > > > > > > > too early. Interestingly that link seems to get > updated > > > > and I > > > > > > > tried > > > > > > > > > > > > [1] and found more recent data (not sure how recent). > > > > Anyway > > > > > it > > > > > > > > > > > > suggests Java 8 is still the top dog but we see good > > growth > > > > > on > > > > > > > 11. In > > > > > > > > > > > > my $dayjob this aligns to what I'm seeing too. > > Customers > > > > > didn't > > > > > > > seem > > > > > > > > > > > > to care about Java 11 until later half last year and > > now > > > > > > > suddenly it > > > > > > > > > > > > is all over the place. > > > > > > > > > > > > > > > > > > > > > > > > I think once we put out a NiFi 2.0 release we'd see > > rapid > > > > > > > decrease in > > > > > > > > > > > > work on the 1.x line just being blunt. We did this > many > > > > years > > > > > > ago > > > > > > > > > > > > with 0.x to 1.x and we stood behind 0.x for a while > > (maybe > > > > a > > > > > > > year or > > > > > > > > > > > > so) but it was purely bug fix/security related bits. > We > > > > would > > > > > > > need to > > > > > > > > > > > > do something similar. But feature work would almost > > > > certainly > > > > > > go > > > > > > > to > > > > > > > > > > > > the 2.x line. Maybe there are other workable models > but > > my > > > > > > > instinct > > > > > > > > > > > > suggests this is likely to follow a similar path. > > > > > > > > > > > > > > > > > > > > > > > > ...anyway I agree it isn't that easy of a call to > dump > > Java > > > > > 8. > > > > > > We > > > > > > > > > > > > need to make the call in both the interests of the > user > > > > base > > > > > > and > > > > > > > the > > > > > > > > > > > > contributor base of the community. > > > > > > > > > > > > > > > > > > > > > > > > [1] > https://www.jetbrains.com/lp/devecosystem-2021/java/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > Joe > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt < > > > > joe.w...@gmail.com > > > > > > > <mailto: > > > > > > > > > > > > joe.w...@gmail.com>> wrote: > > > > > > > > > > > > Russ > > > > > > > > > > > > > > > > > > > > > > > > Yeah the flow registry is a key part of it. But also > > now > > > > you > > > > > > can > > > > > > > > > > > > download the flow definition in JSON (upload i think > is > > > > there > > > > > > now > > > > > > > > > > > > too). Templates offered a series of challenges such > as > > we > > > > > store > > > > > > > them > > > > > > > > > > > > in the flow definition which has made flows massive > in > > an > > > > > > > unintended > > > > > > > > > > > > way which isn't fun for cluster behavior. > > > > > > > > > > > > > > > > > > > > > > > > We have a couple cases where we headed down a > > particular > > > > > > concept > > > > > > > and > > > > > > > > > > > > came up with better approaches later. We need to > > reconcile > > > > > > these > > > > > > > with > > > > > > > > > > > > the benefit of hindsight, and while being careful to > be > > not > > > > > > > overly > > > > > > > > > > > > disruptive to existing users, to reduce the > > > > > > codebase/maintenance > > > > > > > > > > > > burden and allow continued evolution of the project. > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman < > > > > > > > > > r...@windofkeltia.com > > > > > > > > > > > > <mailto:r...@windofkeltia.com>> > > > > > > > > > > > > wrote: > > > > > > > > > > > > Joe, > > > > > > > > > > > > > > > > > > > > > > > > I apologize for the off-topic intrusion, but what > > replaces > > > > > > > templates? > > > > > > > > > > > > The Registry? Templates rocked and we have used them > > since > > > > > > 0.5.x. > > > > > > > > > > > > > > > > > > > > > > > > Russ > > > > > > > > > > > > > > > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote: > > > > > > > > > > > > David, > > > > > > > > > > > > > > > > > > > > > > > > I think this is a highly reasonable approach and > such a > > > > focus > > > > > > > will > > > > > > > > > > > > greatly help make a 2.0 release far more approachable > > to > > > > > knock > > > > > > > out. > > > > > > > > > > > > Not only that but tech debt reduction would help make > > work > > > > > > > towards > > > > > > > > > > > > major features we'd think about in a 'major release' > > sense > > > > > more > > > > > > > > > > > > approachable. > > > > > > > > > > > > > > > > > > > > > > > > We should remove all deprecated things (as well as > > verify > > > > we > > > > > > > have the > > > > > > > > > > > > right list). We should remove/consider removal of > > > > deprecated > > > > > > > > > > > > concepts > > > > > > > > > > > > like templates. We should consider whether we can > > resolve > > > > the > > > > > > > > > > > > various > > > > > > > > > > > > ways we've handled what are now parameters down to > one > > > > clean > > > > > > > > > > > > approach. > > > > > > > > > > > > We should remove options in the nifi.properties which > > turn > > > > > out > > > > > > to > > > > > > > > > > > > never be used quite right (if there are). There is > > quite a > > > > > bit > > > > > > we > > > > > > > > > > > > can > > > > > > > > > > > > do purely in the name of tech debt reduction. > > > > > > > > > > > > > > > > > > > > > > > > Lots to consider here but I think this is the right > > > > > discussion. > > > > > > > > > > > > > > > > > > > > > > > > Than ks > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende < > > > > > bbe...@gmail.com > > > > > > > > > <mailto: > > > > > > > > > > > > bbe...@gmail.com>> > > > > > > > > > > > > wrote: > > > > > > > > > > > > I'm a +1 for this... Not sure if this falls under > > "Removing > > > > > > > > > > > > Deprecated > > > > > > > > > > > > Components", but I think we should also look at > > anything > > > > that > > > > > > has > > > > > > > > > > > > been > > > > > > > > > > > > marked as deprecated throughout the code base as a > > > > candidate > > > > > > for > > > > > > > > > > > > removal. There are quite a few classes, methods, > > > > properties, > > > > > > etc > > > > > > > > > > > > that > > > > > > > > > > > > have been waiting for a chance to be removed. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann > > > > > > > > > > > > <exceptionfact...@apache.org<mailto: > > > > > > exceptionfact...@apache.org > > > > > > > >> > > > > > > > > > > wrote: > > > > > > > > > > > > Team, > > > > > > > > > > > > > > > > > > > > > > > > With all of the excellent work that many have > > contributed > > > > to > > > > > > NiFi > > > > > > > > > > > > over the > > > > > > > > > > > > years, the code base has also accumulated some amount > > of > > > > > > > technical > > > > > > > > > > > > debt. A > > > > > > > > > > > > handful of components have been marked as deprecated, > > and > > > > > some > > > > > > > > > > > > components > > > > > > > > > > > > remain in the code base to support integration with > old > > > > > > versions > > > > > > > > > > > > of various > > > > > > > > > > > > products. Following the principles of semantic > > versioning, > > > > > > > > > > > > introducing a > > > > > > > > > > > > major release would provide the opportunity to remove > > these > > > > > > > > > > > > deprecated and > > > > > > > > > > > > unsupported components. > > > > > > > > > > > > > > > > > > > > > > > > Rather than focusing the next major release on new > > > > features, > > > > > > what > > > > > > > > > > > > do you > > > > > > > > > > > > think about focusing on technical debt removal? This > > > > approach > > > > > > > > > > > > would not > > > > > > > > > > > > make for the most interesting release, but it > provides > > the > > > > > > > > > > > > opportunity to > > > > > > > > > > > > clean up elements that involve breaking changes. > > > > > > > > > > > > > > > > > > > > > > > > Focusing on technical debt, at least three primary > > goals > > > > come > > > > > > to > > > > > > > > > > > > mind for > > > > > > > > > > > > the next major release: > > > > > > > > > > > > > > > > > > > > > > > > 1. Removal of deprecated and unmaintained components > > > > > > > > > > > > 2. Require Java 11 as the minimum supported version > > > > > > > > > > > > 3. Transition internal date and time handling to JSR > > 310 > > > > > > > java.time > > > > > > > > > > > > components > > > > > > > > > > > > > > > > > > > > > > > > *Removing Deprecated Components* > > > > > > > > > > > > > > > > > > > > > > > > Removing support for older and deprecated components > > > > > provides a > > > > > > > > > > > > great > > > > > > > > > > > > opportunity to improve the overall security posture > > when it > > > > > > comes > > > > > > > > > > > > to > > > > > > > > > > > > maintaining dependencies. The OWASP dependency plugin > > > > report > > > > > > > > > > > > currently > > > > > > > > > > > > generates 50 MB of HTML for questionable > dependencies, > > many > > > > > of > > > > > > > > > > > > which are > > > > > > > > > > > > related to old versions of various libraries. > > > > > > > > > > > > > > > > > > > > > > > > As a starting point, here are a handful of components > > and > > > > > > > > > > > > extension modules > > > > > > > > > > > > that could be targeted for removal in a major > version: > > > > > > > > > > > > > > > > > > > > > > > > - PostHTTP and GetHTTP > > > > > > > > > > > > - ListenLumberjack and the entire > > nifi-lumberjack-bundle > > > > > > > > > > > > - ListenBeats and the entire nifi-beats-bundle > > > > > > > > > > > > - Elasticsearch 5 components > > > > > > > > > > > > - Hive 1 and 2 components > > > > > > > > > > > > > > > > > > > > > > > > *Requiring Java 11* > > > > > > > > > > > > > > > > > > > > > > > > Java 8 is now over seven years old, and NiFi has > > supported > > > > > > > general > > > > > > > > > > > > compatibility with Java 11 for several years. NiFi > > 1.14.0 > > > > > > > > > > > > incorporated > > > > > > > > > > > > internal improvements specifically related to TLS > 1.3, > > > > which > > > > > > > > > > > > allowed > > > > > > > > > > > > closing out the long-running Java 11 compatibility > epic > > > > > > > NIFI-5174. > > > > > > > > > > > > Making > > > > > > > > > > > > Java 11 the minimum required version provides the > > > > opportunity > > > > > > to > > > > > > > > > > > > address > > > > > > > > > > > > any lingering edge cases and put NiFi in a better > > position > > > > to > > > > > > > > > > > > support > > > > > > > > > > > > current Java versions. > > > > > > > > > > > > > > > > > > > > > > > > *JSR 310 for Date and Time Handling* > > > > > > > > > > > > > > > > > > > > > > > > Without making the scope too broad, transitioning > > internal > > > > > date > > > > > > > > > > > > and time > > > > > > > > > > > > handling to use DateTimeFormatter instead of > > > > SimpleDateFormat > > > > > > > > > > > > would provide > > > > > > > > > > > > a number of advantages. The Java Time components > > provide > > > > much > > > > > > > > > > > > better > > > > > > > > > > > > clarity when it comes to handling localized date and > > time > > > > > > > > > > > > representations, > > > > > > > > > > > > and also avoid the inherent confusion of > java.sql.Date > > > > > > extending > > > > > > > > > > > > java.util.Date. Many internal components, > specifically > > > > > > > > > > > > Record-oriented > > > > > > > > > > > > processors and services, rely on date parsing, > leading > > to > > > > > > > > > > > > confusion and > > > > > > > > > > > > various workarounds. The pattern formats of > > > > SimpleDateFormat > > > > > > and > > > > > > > > > > > > DateTimeFormatter are very similar, but there are a > few > > > > > subtle > > > > > > > > > > > > differences. > > > > > > > > > > > > Making this transition would provide a much better > > > > foundation > > > > > > > > > > > > going forward. > > > > > > > > > > > > *Conclusion* > > > > > > > > > > > > > > > > > > > > > > > > Thanks for giving this proposal some consideration. > > Many of > > > > > you > > > > > > > > > > > > have been > > > > > > > > > > > > developing NiFi for years and I look forward to your > > > > > feedback. > > > > > > I > > > > > > > > > > > > would be > > > > > > > > > > > > glad to put together a more formalized recommendation > > on > > > > > > > > > > > > Confluence and > > > > > > > > > > > > write up Jira epics if this general approach sounds > > > > agreeable > > > > > > to > > > > > > > > > > > > the > > > > > > > > > > > > community. > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > David Handermann > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
