Re: [DISCUSS] NiFi 2.0 Release Goals
Thanks for the reply Joe. If the "marketplace" concept can be scoped to something like "not in core NiFi but can be downloaded here", then it could fit in the definition of achievable in 2.0. The concept of the marketplace being this magical dynamic processor lookup service always felt a little too vaporware to me. One thing that would help the NiFi build size in general is to really focus on what needs to be in "core" and what can just be external nars that the dataflow/systems manager can add to the 'extensions' directory. We've struggled with having really large builds, so maybe this is an opportunity to address that in 2.0. Radical brainstorm warning: Imagine a completely "thin" NiFi distribution. e.g. with no processors or other components at all. Then a configuration manager could source additional nar functionality from a different distribution source. We could still build processor bundles, but they would just be completely separate from the framework. Separate git repository, separate versioning, separate download, etc. This would be the baby steps towards the marketplace concept. Let the ecosystem of processors and other components thrive outside of the main framework distribution. In this way, things like PostHTTP could live on easily without burdening the framework. And groups of similar processor functionality can exist in separately managed nars. Thanks for entertaining the thought at least. /Adam On Fri, Jul 30, 2021 at 4:50 PM Joe Witt wrote: > Adam > > In cases of ‘what happened’ it just hasnt happened. The ideas are still > good. > > I do agree with the ‘why kill anything’ mentality. We can simply park > these somewhere. That said there is an ongoing maint cost we do have to > rationalize. Builds are longer, deps to maintain, etc.. > > Also we need to prob pumps the breaks a bit on thoughs of removing or > changing many things. If we do we may well get to 2.0 but nobody could > adopt it. We have honestly a pretty massive deployment base and we cannot > make it a pain for users. We got away with things at 0.x to 1.0 that we > cannot get away with on 2.0 > > Thanks > > On Fri, Jul 30, 2021 at 3:41 PM Adam Taft wrote: > > > I'm not seeing the side thread that was going to discuss deprecation of > > PostHTTP. Has that thread started and I just don't see it? > > > > One (significant?) concern with PostHTTP is the smooth integration of > > NiFi-to-NiFi communication that is very transparently enabled with the > > ListenHTTP and PostHTTP processors. There's some special logic in there > for > > handling flowfiles that InvokeHTTP doesn't really (nor should really) > have. > > > > I know of several (large) NiFi installations that rely on the PostHTTP / > > ListenHTTP combination. It has enabled NiFi to NiFi communication for > folks > > reluctant or unable to enable site-to-site type configuration. > > > > Honestly, I don't know why we'd want to "deprecate" any processor, as > > opposed to just moving it to a new location. If these processors can be > > ported and maintained to whatever the 2.0 API looks like, there's > possibly > > little harm keeping them around. > > > > And by the way, what happened to the "marketplace" concept? Is this being > > considered for 2.0 as well? Because relocating the deprecated processors > > to an external nar might be the best solution. Losing PostHTTP entirely I > > think would be a mistake, but I'd conceptually support relocating it. > > > > Thanks, > > > > /Adam > > > > On Tue, Jul 27, 2021 at 2:11 PM Joe Witt wrote: > > > > > Looks like we just need to knock out 5 JIRAs :) [1] > > > > > > I felt like we had a label folks were using at one point but quickly > > > looking revealed nothing exciting. After this confluence page > > > stabilizes a bit we can probably knock out some JIRAs and such. > > > > > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599 > > > > > > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler > > > wrote: > > > > > > > > I find myself wishing I had a list of all the jiras / issues that > have > > > > been put off for a 2.0 release because they required some change or > > > another > > > > :( > > > > > > > > From: Joe Witt > > > > Reply: dev@nifi.apache.org < > dev@nifi.apache.org> > > > > Date: July 27, 2021 at 12:30:35 > > > > To: dev@nifi.apache.org > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > > > A few thoughts: > > > > > > > > 1. I would love to see deprecation notices show up in the UI in > > > > various ways to help motivate users to move off things to more > > > > supportable things. That is not a prerequisite for anything happening > > > > however. Just a good feature/nice thing to do for users when someone > > > > is able to tackle it. > > > > > > > > 2. The decision to deprecate something and to further remove it need > > > > not mean there is a superior solution available. If that thing itself > > > > isn't getting the love/attention it needs to be > > > >
Re: [DISCUSS] NiFi 2.0 Release Goals
Adam In cases of ‘what happened’ it just hasnt happened. The ideas are still good. I do agree with the ‘why kill anything’ mentality. We can simply park these somewhere. That said there is an ongoing maint cost we do have to rationalize. Builds are longer, deps to maintain, etc.. Also we need to prob pumps the breaks a bit on thoughs of removing or changing many things. If we do we may well get to 2.0 but nobody could adopt it. We have honestly a pretty massive deployment base and we cannot make it a pain for users. We got away with things at 0.x to 1.0 that we cannot get away with on 2.0 Thanks On Fri, Jul 30, 2021 at 3:41 PM Adam Taft wrote: > I'm not seeing the side thread that was going to discuss deprecation of > PostHTTP. Has that thread started and I just don't see it? > > One (significant?) concern with PostHTTP is the smooth integration of > NiFi-to-NiFi communication that is very transparently enabled with the > ListenHTTP and PostHTTP processors. There's some special logic in there for > handling flowfiles that InvokeHTTP doesn't really (nor should really) have. > > I know of several (large) NiFi installations that rely on the PostHTTP / > ListenHTTP combination. It has enabled NiFi to NiFi communication for folks > reluctant or unable to enable site-to-site type configuration. > > Honestly, I don't know why we'd want to "deprecate" any processor, as > opposed to just moving it to a new location. If these processors can be > ported and maintained to whatever the 2.0 API looks like, there's possibly > little harm keeping them around. > > And by the way, what happened to the "marketplace" concept? Is this being > considered for 2.0 as well? Because relocating the deprecated processors > to an external nar might be the best solution. Losing PostHTTP entirely I > think would be a mistake, but I'd conceptually support relocating it. > > Thanks, > > /Adam > > On Tue, Jul 27, 2021 at 2:11 PM Joe Witt wrote: > > > Looks like we just need to knock out 5 JIRAs :) [1] > > > > I felt like we had a label folks were using at one point but quickly > > looking revealed nothing exciting. After this confluence page > > stabilizes a bit we can probably knock out some JIRAs and such. > > > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599 > > > > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler > > wrote: > > > > > > I find myself wishing I had a list of all the jiras / issues that have > > > been put off for a 2.0 release because they required some change or > > another > > > :( > > > > > > From: Joe Witt > > > Reply: dev@nifi.apache.org > > > Date: July 27, 2021 at 12:30:35 > > > To: dev@nifi.apache.org > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > A few thoughts: > > > > > > 1. I would love to see deprecation notices show up in the UI in > > > various ways to help motivate users to move off things to more > > > supportable things. That is not a prerequisite for anything happening > > > however. Just a good feature/nice thing to do for users when someone > > > is able to tackle it. > > > > > > 2. The decision to deprecate something and to further remove it need > > > not mean there is a superior solution available. If that thing itself > > > isn't getting the love/attention it needs to be > > > maintained/supported/healthy going forward that alone is enough to > > > remove it. That might well be the case with PostHTTP [1] and for > > > comparison you can see how much effort has gone into InvokeHTTP [2]. > > > > > > 3. When discussing a 2.0 release each thing we add as a 'must do' the > > > further away from reality such a release will become. We'll have to > > > get very specific about 'musts' vs 'wants'. > > > > > > [1] > > > > > > https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java > > > [2] > > > > > > https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java > > > > > > On Tue, Jul 27, 2021 at 8:47 AM David Handermann > > > wrote: > > > > > > > > Thanks Mark, providing a template or comparison statistics with Java > > > > versions and component configuration details would be very helpful. > If > > it > > > > is possible to run tests using a public API or deployable service, > that > > > > would also help confirm potential differences. > > > > > > > > Showing a deprecation notice in the UI could be helpful, perhaps as a > > > > configurable option. NIFI-8650 describes a general Flow Analysis > > > > capability, and it seems like that might be one possible way to > surface > > > > deprecation warnings. For something more specific to component > > > deprecation, > > > > it seems important to find a balance between making it obvious and > > making > > >
Re: [DISCUSS] NiFi 2.0 Release Goals
I'm not seeing the side thread that was going to discuss deprecation of PostHTTP. Has that thread started and I just don't see it? One (significant?) concern with PostHTTP is the smooth integration of NiFi-to-NiFi communication that is very transparently enabled with the ListenHTTP and PostHTTP processors. There's some special logic in there for handling flowfiles that InvokeHTTP doesn't really (nor should really) have. I know of several (large) NiFi installations that rely on the PostHTTP / ListenHTTP combination. It has enabled NiFi to NiFi communication for folks reluctant or unable to enable site-to-site type configuration. Honestly, I don't know why we'd want to "deprecate" any processor, as opposed to just moving it to a new location. If these processors can be ported and maintained to whatever the 2.0 API looks like, there's possibly little harm keeping them around. And by the way, what happened to the "marketplace" concept? Is this being considered for 2.0 as well? Because relocating the deprecated processors to an external nar might be the best solution. Losing PostHTTP entirely I think would be a mistake, but I'd conceptually support relocating it. Thanks, /Adam On Tue, Jul 27, 2021 at 2:11 PM Joe Witt wrote: > Looks like we just need to knock out 5 JIRAs :) [1] > > I felt like we had a label folks were using at one point but quickly > looking revealed nothing exciting. After this confluence page > stabilizes a bit we can probably knock out some JIRAs and such. > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599 > > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler > wrote: > > > > I find myself wishing I had a list of all the jiras / issues that have > > been put off for a 2.0 release because they required some change or > another > > :( > > > > From: Joe Witt > > Reply: dev@nifi.apache.org > > Date: July 27, 2021 at 12:30:35 > > To: dev@nifi.apache.org > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > A few thoughts: > > > > 1. I would love to see deprecation notices show up in the UI in > > various ways to help motivate users to move off things to more > > supportable things. That is not a prerequisite for anything happening > > however. Just a good feature/nice thing to do for users when someone > > is able to tackle it. > > > > 2. The decision to deprecate something and to further remove it need > > not mean there is a superior solution available. If that thing itself > > isn't getting the love/attention it needs to be > > maintained/supported/healthy going forward that alone is enough to > > remove it. That might well be the case with PostHTTP [1] and for > > comparison you can see how much effort has gone into InvokeHTTP [2]. > > > > 3. When discussing a 2.0 release each thing we add as a 'must do' the > > further away from reality such a release will become. We'll have to > > get very specific about 'musts' vs 'wants'. > > > > [1] > > > https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java > > [2] > > > https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java > > > > On Tue, Jul 27, 2021 at 8:47 AM David Handermann > > wrote: > > > > > > Thanks Mark, providing a template or comparison statistics with Java > > > versions and component configuration details would be very helpful. If > it > > > is possible to run tests using a public API or deployable service, that > > > would also help confirm potential differences. > > > > > > Showing a deprecation notice in the UI could be helpful, perhaps as a > > > configurable option. NIFI-8650 describes a general Flow Analysis > > > capability, and it seems like that might be one possible way to surface > > > deprecation warnings. For something more specific to component > > deprecation, > > > it seems important to find a balance between making it obvious and > making > > > it something that ends up getting ignored. > > > > > > Regards, > > > David Handermann > > > > > > On Tue, Jul 27, 2021 at 10:28 AM Mark Bean > wrote: > > > > > > > I'll start a new thread for PostHTTP when I get a template and/or > > detailed > > > > stats. > > > > > > > > I know the deprecation is noted in the documentation. That's a > > necessary > > > > and minimum level of notification. I was suggesting it be more > obvious > > in > > > > the UI. I think it would be beneficial to somehow be aware of the > > > > deprecation status simply by looking at the flow (perhaps even on the > > > > summary pages too), and without having to open the documentation for > > every > > > > processor to confirm whether or not a component is marked as > > deprecated. > > > > > > > > Thanks, > > > > Mark > > > > > > > > > > > > On Tue, Jul 27, 2021 at
