Re: Is there a roadmap for Nifi V2 ?

[Ryan Hendrickson]

If we wanted to evaluate if the NiFi 2.0-M2 release is sufficiently
production worthy for our individual use cases, is there a list of the
remaining items/known key gaps/etc to get in so we can make a decision if
those remaining items are noteworthy to our cases?

Thanks,
Ryan

On Mon, Mar 11, 2024 at 11:50 AM Pierre Villard 
wrote:

> Hi Robert,
>
> I'd expect an M3 release in the next couple of weeks and the first official
> 2.0 release should happen in the next couple of months I think. There is no
> official timeline as it depends on the community efforts around the
> remaining things we want to get it as well as the feedback from the
> community on the milestone releases.
>
> Having said that, if you're new with NiFi, I'd say it is safe to start with
> 2.0-M2 right now as long as you're ok with doing upgrades when the next
> releases are going out. Some companies are already using 2.0-M2 in
> production use cases.
>
> Hope this helps,
> Pierre
>
> Le lun. 11 mars 2024 à 15:14, Provencher, Robert <
> provencher.rob...@uqam.ca>
> a écrit :
>
> >
> > Hi!
> >
> >
> > I'm trying to find if there is a roadmap for Nifi V2.
> >
> > Are there dates, a calendar, to show the steps and to aim when the first
> > production-safe Nifi V2 version would be published ?
> >
> > Are there approximations of this date ?
> >
> > The reason I'm asking is that we are seriously considering using Nifi.
> > We are asking ourselves if it would be better to start using Nifi by
> > directly begin with V2, configuring our flows with V2 ?
> >
> > So a roadmap would help in taking our decision 
> >
> > Thanks for your help and your good work !!!
> >
> >
> > Robert
> >
>

Re: [discuss] nifi 2.0 and Java 21…

[Ryan Hendrickson]

Fantastic progress!  Looking forward to the new platform!
Wooohooo!

On Fri, Sep 15, 2023, 7:17 PM Joe Witt  wrote:

> Caveat to that - the build works right now as the Java release version in
> the pom is set to 17.  So we're building/running Java 21 but the form it is
> being built in is enforcing Java 17.  This is ensuring the Groovy compiler
> can work.  There is not current obvious path/option to build Groovy with
> Java 21 so we likely should focus on removal of any groovy things requiring
> compilation in our build.  Groovy runs on Java 21 just fine but the eclipse
> compiler as yet does not.  Either way we're on track to get those tests and
> groovy based source out.
>
> Still good progress on getting to NiFi 2.0 and Java 21 (which officially
> releases Sep 19th).
>
> Thanks
>
> On Fri, Sep 15, 2023 at 4:03 PM Joe Witt  wrote:
>
> > Team
> >
> > Long story short we are now able to build and run NiFi main/2.x line on
> > Java 21!
> >
> > We've made a lot of progress this week on top of all the great work
> > already done for getting to NiFi 2.0 but specifically to this thread of
> > making NiFi 2.0 be based on Java 21.
> >
> > There are a couple PRs outstanding but once they land I'll put up a PR
> for
> > this commit [1] and we will be building with Java 21-ea on the main line.
> > The full clean build with all tests and all profiles I could find is now
> > working locally and is also now running in my fork before I put up the PR
> > [2], [3].  NiFi also runs on Java 21.  We did have to make a bunch of
> > updates to all things Groovy and we're reducing/eliminating a lot of
> pieces
> > that are poorly maintained or need fundamentally different
> implementations
> > in Java 21.  The toolkit is likely to be removed it seems and we can
> later
> > introduce back specific pieces if/as needed but designed better/more
> easily
> > maintained.
> >
> > What would be ideal is we land a couple more key pieces like ensuring
> > every deprecated component is actually removed and ensuring the
> flow.xml.gz
> > is entirely gone.  Then we kick out a NiFi 2.0 M1 release for people to
> > work with.
> >
> > [1]
> >
> https://github.com/joewitt/nifi/commit/c3d16d949f8153441c64a1f98fd641cf80178f43
> > [2] https://github.com/joewitt/nifi/actions/runs/6203527626
> > [3] https://github.com/joewitt/nifi/actions/runs/6203527625
> >
> > Thanks
> > Joe
> >
> > On Fri, Sep 15, 2023 at 1:06 PM Pierre Villard <
> > pierre.villard...@gmail.com> wrote:
> >
> >> Templates are already out. Flow.xml being removed should be reviewed
> soon
> >> (PR was rebased today). I'm working on the removal of variables. I hope
> to
> >> get a PR for this in the next few days.
> >>
> >> Le ven. 15 sept. 2023, 19:22, Joe Witt  a écrit :
> >>
> >> > Timeline - we remain in full blitz mode to get things ready for 2.0.
> No
> >> > clear ETA but we need to be getting it out soon.  At least a milestone
> >> > release of it for people to work with.  There is a big change needed
> to
> >> get
> >> > rid of the flow.xml.gz in favor of the json form and that is in
> >> progress.
> >> > I am not sure offhand whether templates got the boot yet.
> >> >
> >> > Latest fun is wrestling our rather messy situation with Groovy in the
> >> build
> >> > as that seems not ready for Java 21 generally.
> >> >
> >> >
> >> >
> >> > On Fri, Sep 15, 2023 at 10:19 AM Ryan Hendrickson <
> >> > ryan.andrew.hendrick...@gmail.com> wrote:
> >> >
> >> > > I think NiFi 2.x going to Java 21 for all the reasons outlined
> makes a
> >> > lot
> >> > > of sense.
> >> > >
> >> > > Is there a timeline for 2.x?
> >> > >
> >> > > On Mon, Sep 11, 2023 at 5:00 AM Pierre Villard <
> >> > > pierre.villard...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Thanks Joe, it makes total sense and I agree that the ones that
> >> would
> >> > > > likely be slow at adopting Java 21 would not go to NiFi 2.0 super
> >> > quickly
> >> > > > anyway. Being able to bring the latest and greatest in NiFi is
> great
> >> > and
> >> > > > given all of the features announced in Java 21, I imagine a lot of
> >> > > projects
> >> > > > we depend on will be doing the same.

Re: [discuss] nifi 2.0 and Java 21…

[Ryan Hendrickson]

I think NiFi 2.x going to Java 21 for all the reasons outlined makes a lot
of sense.

Is there a timeline for 2.x?

On Mon, Sep 11, 2023 at 5:00 AM Pierre Villard 
wrote:

> Thanks Joe, it makes total sense and I agree that the ones that would
> likely be slow at adopting Java 21 would not go to NiFi 2.0 super quickly
> anyway. Being able to bring the latest and greatest in NiFi is great and
> given all of the features announced in Java 21, I imagine a lot of projects
> we depend on will be doing the same.
>
> Le jeu. 7 sept. 2023 à 19:36, Joe Witt  a écrit :
>
> > Pierre
> >
> > A few concerns you raised so want to address my view on each:
> >
> > Will users be able to adopt Java 21 fast enough?
> > I share Brandon's view on that in terms of their adoption timeline.  It
> > will likely align well with NiFi 2.0 itself in my estimation.
> >
> > Will this delay NiFi 2.0?
> > If it would then I'd not be supportive.  I don't think we need to bother
> > with adopting any of the features now.  What I would like us to have is
> the
> > option to adopt them as we progress.  We should get 2.0 done asap and if
> > this added delay then I'd be way less interested in this idea.
> >
> > Feature benefits of 21 and what will that bring?
> > Mark spoke well to the key one that stood out to me which was the new
> > threading model available.  It would be awfully nice to leverage that for
> > the efficiency it represents and especially if it can reduce some of our
> > heap usage which is valuable in cloud/shared compute contexts.
> >
> > Performance benefits of Java 21?
> > It appears from some analysis found with googling that Java 21 brings out
> > of the box 4-5% performance increases generally.  Not amazing but useful.
> >
> > User experience otherwise with Java 21?
> > I believe it would be consistent with Java 17 for their point of view in
> > terms of install/config/etc..
> >
> > My motivation for this is fairly pure honestly.  Since we're setting our
> > new minimum bar that lives for as long as the 2.x release line lives I'd
> > like to set it at the current LTS available when we ship that line as
> well.
> >
> > Thanks
> >
> >
> >
> > On Thu, Sep 7, 2023 at 4:22 AM Brandon DeVries <
> brandon.devr...@gmail.com>
> > wrote:
> >
> > > +1 to requiring java 21. Starting off as "up to date" as possibly
> makes a
> > > lot of sense, and some of the new features seem especially relevant to
> > NiFi.
> > >
> > > I definitely understand the concerns about organizations being willing
> /
> > > able to approve Java 21... But those same organizations might also be
> > > hesitant to move to NiFi 2.0. We will continue to support java 17 &
> NiFi
> > > 1.x for some time, so hopefully those groups will have the time they
> need
> > > to get approvals, do evaluations, and upgrade.
> > >
> > > Brandon
> > > 
> > > From: Pierre Villard 
> > > Sent: Thursday, September 7, 2023 6:15:58 AM
> > > To: dev@nifi.apache.org 
> > > Subject: Re: [discuss] nifi 2.0 and Java 21…
> > >
> > > Hi all,
> > >
> > > I share the concerns raised by Chris regarding how quickly users of
> NiFi
> > > will be able to adopt Java 21.
> > >
> > > While I'm definitely in favor of using the latest and greatest,
> > especially
> > > when it brings to the table such significant features, we need to be
> > > careful as it may significantly delay the adoption of NiFi 2.0 in big
> > > companies where deploying Java 21 will take time. I acknowledge that
> > going
> > > from Java 8 to Java 17 is certainly the same effort as going from Java
> 8
> > to
> > > Java 21 but how quickly security-sensitive environments will adopt a
> new
> > > release of Java that is completely new?
> > >
> > > In addition to that, it sounds like we would add a significant rework
> of
> > > the framework in NiFi 2.0 assuming we adopt Java 21 as the minimum
> > version.
> > > Do we think this is going to significantly delay the first release of
> > NiFi
> > > 2.0? We still have work to do but adding this on top could delay the
> > > release, no?
> > >
> > > Finally, the features that Java 21 are bringing sound super interesting
> > in
> > > the context of NiFi but do we already have an idea of what it will
> > improve?
> > > is it the user experience, and if so, how will it change? is it the
> > > performance, and if so, do we have an idea of how things will improve?
> > >
> > > Thanks,
> > > Pierre
> > >
> > > Le mer. 6 sept. 2023 à 23:07, Chris Sampson
> > >  a écrit :
> > >
> > > > Yeah, I understand the need to move to 21 as a minimum to take
> > advantage
> > > of
> > > > its features. Hopefully the wider java ecosystem won't be an issue in
> > the
> > > > short term.
> > > >
> > > > I just wanted the discussion to be clear about this being a change to
> > the
> > > > Java baseline/minimum for NiFi 2.0.
> > > >
> > > > It's a +1 from me.
> > > >
> > > > On Wed, 6 Sept 2023, 19:01 Joe Witt,  wrote:
> > > >
> > > > > Chris
> > > > >
> > > > > My suggestion is 

Re: [discuss] nifi 2.0 and Java 17...

[Ryan Hendrickson]

The major issue for our deployments would be the removal of Nashorn as
well.

Would GraalVM or an alternative be considered as a part of an initial NiFi
2.0 release?

Thanks,
Ryan

On Mon, Jun 5, 2023 at 12:38 PM Joe Witt  wrote:

> Team,
>
> Looking like we will update the NiFi 2.0 goals to be based on Java 17
> instead of 11.
>
> The noted concern around Java removing Nashorn in 11/17 we will need to
> identify an alternative plan for regardless and seems like David's proposal
> would do the trick.
>
> Let's give this thread a few more days and if still seems consensus is
> present lets just assume lazy consensus and update the NiFi 2.0 goals and
> make it happen.
>
> Thanks
>
> On Fri, Jun 2, 2023 at 8:46 AM David Handermann <
> exceptionfact...@apache.org>
> wrote:
>
> > I agree that moving forward with Java 17 as the minimum for NiFi 2.0 is
> the
> > best approach given the extended lifecycle of support for Java 17.
> >
> > With the removal of a number of legacy components, the current main
> branch
> > is in a much better position to make Java 17 the minimum.
> >
> > The deprecation and removal of Nashorn from the JDK is worth
> highlighting,
> > but it should not be a blocker for moving to Java 17. In this case, NiFi
> is
> > reflecting the deprecation of Nashorn that already exists in Java 11. I
> > have submitted a PR for NIFI-11630 to mark ECMAScript as deprecated for
> the
> > support branch in subsequent version 1 releases.
> >
> > With that background, there is ongoing maintenance of the Nashorn engine
> as
> > an external library, in addition to the GraalVM solution. However, this
> is
> > a good opportunity to take a different approach to scripting engine
> > integration. For maintenance and security purposes, it would be much
> better
> > to reduce the number of bundled scripting engines and make it easier to
> > bring your own. The current scripting bundle is around 100 MB, which is a
> > lot of weight for languages and solutions that do not apply across the
> > board. Providing an alternative that makes it easier to bring in a script
> > engine library should provide a better solution for the future. This also
> > should not be a blocker for an initial NiFi 2.0, but it is worth
> > highlighting given the general interest in scripted components.
> >
> > Regards,
> > David Handermann
> >
> > On Thu, Jun 1, 2023, 11:38 PM Dirk Arends 
> > wrote:
> >
> > > Hi Joe,
> > >
> > > > Who will be seriously impacted by the removal of Java 11 and what was
> > > your plan for upgrading to Java 17?
> > >
> > > >
> > >
> > > > thoughts?
> > >
> > > I would support moving the minimum Java version to 17 if it wasn’t for
> > the
> > > fact that Nashorn will be removed. Nashorn is already deprecated in
> Java
> > > 11, and was then fully removed in Java 15. I understand GraalVM is
> > intended
> > > to be its successor, however this has not yet been integrated into NiFi
> > and
> > > I’ve been unable to satisfactorily integrate it myself to date.
> > >
> > > In my NiFi usage, I make heavy use of the JavaScript engine in
> > > ExecuteScript and InvokeScriptedProcessor processors. To take advantage
> > of
> > > GraalVM supporting later ECMAScript versions than Nashorn, I have been
> > > attempting to use GraalVM as the JavaScript Engine for NiFi with
> limited
> > > success.
> > >
> > > Further details have been provided in JIRA ticket NIFI-6229 [1] and I’d
> > > welcome any assistance in trying to finalise GraalVM support, but
> > otherwise
> > > I’d consider the loss of Nashorn to be a potential blocker to adopting
> > Java
> > > 17.
> > >
> > > [1] https://issues.apache.org/jira/browse/NIFI-6229
> > >
> > >
> > > Regards,
> > >
> > > Dirk Arends
> > >
> > >
> > >
> > > On Thu, 1 Jun 2023 at 03:23, Joe Witt  wrote:
> > >
> > > > Team,
> > > >
> > > > We've discussed in the past having NiFi 2.0 move from Java 8 to Java
> 11
> > > as
> > > > the required minimum while also working on Java 17.
> > > >
> > > > As we move on in time though we are now 4 months (Sept) from. Java 11
> > > > openJDK going end of support.  Meanwhile, the Spring 5.x line goes
> end
> > of
> > > > support as of next year and Spring 6.x requires Java 17.  Also Java
> 21
> > > > comes out in Sept as well and is already the next LTS release.
> > > >
> > > > I am increasingly of the view that we should seriously
> discuss/consider
> > > > moving to Java 17 as our basis for NiFi 2.0 as otherwise it basically
> > > means
> > > > we'll be forced to move to NiFi 3.0 quite quickly.
> > > >
> > > > We already know we can build and run on Java 17 so we're good there.
> > > We'll
> > > > soon want to do the same for Java 21 ... and the more 'old stuff' we
> > hold
> > > > on to the harder it is.
> > > >
> > > > Who will be seriously impacted by the removal of Java 11 and what was
> > > your
> > > > plan for upgrading to Java 17?
> > > >
> > > > thoughts?
> > > >
> > > > Thanks
> > > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > --
> > > Dirk 

Re: [discuss] NiFi 1.20 and NiFi 2.0

[Ryan Hendrickson]

I second some calendar guessing.  There's been releases every 1-2 months
for a while now.  I'd be curious what a guess at a gap would be between
1.20.0 and a GA of 2.0.0.

Thanks,
Ryan

On Mon, Jan 9, 2023 at 4:50 PM Adam Taft  wrote:

> Joe / team,
>
> Question on this. I think it would be helpful to understand the desired
> timelines for the first 2.0.0 release. I know it's not strictly
> predictable, but having a sense of what the timing looks like is important
> to help understand the implications of a "maintenance only" 1.x line. The
> schedule would ideally come from the folks who are actively looking at /
> contributing to the 2.0 release. They probably have the best gauge as to
> "when" it might happen (under ideal conditions).
>
> One of the risks, of course, is if the 2.0 release stalls or delays. Having
> an idea of how 1.x might evolve for the users who are not necessarily
> early-adopters or those that need longer support tails. If 2.0 is delayed
> and 1.x looks unmaintained, there's a potential chance for the project to
> lose a bit of credibility. I know we don't anticipate this scenario, but if
> we had a plan for it, that would be reassuring.
>
> Maybe this was already addressed, I apologize if so. But if not, can we
> throw some darts on the calendar to help understand the ideal rollout of
> 2.0 on a timeline? And are there any adjustments for the scenario described
> above?
>
> Thanks in advance,
>
> /Adam
>
>
> On Mon, Jan 9, 2023 at 1:53 PM Joe Witt  wrote:
>
> > Team,
> >
> > As David mentioned in [1] following a successful NiFi 2.0 release goal
> > planning - we are now going to start moving the 'main' line to be the
> NiFi
> > 2.0 line which will allow for the key work to take place.  We will also
> > move niFi 1.x to its appropriate support line.
> >
> > It is also the case that we have nearly 100 JIRAs on NiFi 1.20 and we
> have
> > work in there including security items so it is time to make a release.
> > The intent then is to initiate 1.20 and immediate after that change
> 'main'
> > to 2.0.
> >
> > Going forward then all work on the 1.x line should be focused on
> > maintaining existing features, dependencies, and helping 1.x users
> migrate
> > to the 2.x line.  Otherwise, new feature work will happen on 'main' as it
> > normally does and will come out in the NiFi 2.x release line.
> >
> > Please flag key outstanding items as we narrow down the release candidate
> > for NiFi 1.20.
> >
> > Thanks
> > Joe
> >
> > [1] https://lists.apache.org/thread/qo4vvdw46235y7vy2crcd6l4m11wl7jz
> >
>

Re: [DISCUSS] Finalizing Release Goals for NiFi 2.0

[Ryan Hendrickson]

The Proposed Release Goals and Deprecated Components and Features pages
look great.

I appreciate the minor leap of Java 11 as a 2.x requirement vs Java 17.

Maybe once there is a timeline, the 2.x branch could be scheduled only to
be alive for a minor amount of time... a year, etc. Then a later 2.x or 3.0
release would bring about Java 17.

Ryan

On Wed, Dec 7, 2022 at 3:09 PM Mike Thomsen  wrote:

> > Mike - what do you mean by "controller service-based configuration for
>
> Using controller services for configuring bundles that connect an
> external service such as Cassandra, Elasticsearch, etc. and removing
> the option to configure connections on the processor.
>
> > I don't think you were suggesting the minimum version be Java 17, were
> you?
>
> I was. Partly as devil's advocate, partly because I actually want to
> use Java 17 as a daily driver.
>
> On Wed, Dec 7, 2022 at 2:20 PM Mark Bean  wrote:
> >
> > I agree this is a great start to a discussion with pointers to important
> > docs for the 2.0 transition. Thanks David!
> >
> > Mike - what do you mean by "controller service-based configuration for
> > connection details"?
> >
> > Also, the transition from Java 11 to 17 is not without potential issues.
> > I've discovered one already. [1] I support stepping up on Java version
> > requirements. Perhaps rather than the currently stated "Requires Java 8
> or
> > Java 11", the requirement can be "Requires Java 11 or Java 17". I don't
> > think you were suggesting the minimum version be Java 17, were you?
> Either
> > way, the issue with Java 17 needs to be identified and fixed as well as
> > more thorough testing to find other possible edge cases before we move
> > forward too aggressively.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-10958
> >
> > On Wed, Dec 7, 2022 at 1:33 PM Mike Thomsen 
> wrote:
> >
> > > Really good start on the discussion. One thing I'm curious about is
> > > Java 11 vs 17. Java 8 -> 11 is major jump that I can understand why
> > > businesses scoffed at that for a long time, but the lift from 11 to 17
> > > was about like 7 -> 8. A 2.0 release seems like a good time to jump
> > > straight to the latest official LTS for Java and start greenlighting
> > > new language features that might simplify things.
> > >
> > > I would also add (since I didn't see it) a design goal of forcing a
> > > complete shift in all bundles to using controller service-based
> > > configurations for connection details. 2.0 feels like a really good
> > > time for us to establish a community-wide best practice of
> > > centralizing configurations in dedicated components.
> > >
> > > On Wed, Dec 7, 2022 at 9:13 AM Mark Payne 
> wrote:
> > > >
> > > > Yeah, agreed. I am very supportive, as well.
> > > >
> > > > Thanks for taking the time to put this together, David.
> > > >
> > > > -Mark
> > > >
> > > >
> > > > > On Dec 7, 2022, at 4:07 AM, Pierre Villard <
> > > pierre.villard...@gmail.com> wrote:
> > > > >
> > > > > Thanks for putting this together David. This is an excellent
> writeup
> > > and
> > > > > it's great to have a release where we focus on tech debt as well as
> > > making
> > > > > sure we stay up to date with our dependencies and what we support.
> > > This is
> > > > > a great opportunity for us to clean a lot of things in our code
> and I
> > > can't
> > > > > wait for us to get started with this. I'm definitely a +1 to have a
> > > formal
> > > > > vote on this proposal.
> > > > >
> > > > > Thanks,
> > > > > Pierre
> > > > >
> > > > > Le mar. 6 déc. 2022 à 23:50, Joe Witt  a
> écrit :
> > > > >
> > > > >> David, All,
> > > > >>
> > > > >> This is an excellent writeup/good framing.  I am supportive of
> this
> > > > >> as-is since it is achievable and lays out a clear path.  We can
> make
> > > > >> milestone releases of NiFi 2.0.0 along the way until we achieve
> all
> > > > >> the stated goals. I assume migration bits will be the long pole
> and
> > > > >> once we have them sorted we can kick out a 2.0.0.   We already
> have a
> > > > >> version guide that governs how long we'd keep 1.x maintained
> though
> > > > >> the phase out is pretty natural as we move main to a 2.0.0 basis
> > > > >> anyway.
> > > > >>
> > > > >> Not to confuse this thread but it makes me think we could do a
> similar
> > > > >> framing for a NiFi 3.0 which lays out a potentially new approach
> to
> > > > >> NiFi decoupling the web/interface from the runtime/operations and
> one
> > > > >> which is more fundamentally K8S based.  But we can cross that
> bridge a
> > > > >> bit later.  Does seem more and more like folks in the community
> would
> > > > >> like to know more about the potential directions we can go.
> > > > >>
> > > > >> Thanks!
> > > > >> Joe
> > > > >>
> > > > >> On Tue, Dec 6, 2022 at 1:53 PM David Handermann
> > > > >>  wrote:
> > > > >>>
> > > > >>> Team,
> > > > >>>
> > > > >>> With the release of NiFi 1.19.0 deprecating support for Java 8,
> the
> > > end
> > > > >> of
> > > > >>> 

Re: NiFi Slack, IRC?

[Ryan Hendrickson]

Awesome, thanks.

On Wed, Aug 10, 2022 at 11:47 AM Joe Witt  wrote:

> Yep.  Just follow the invite link on the site.  There are about 2500
> users there now.
>
> [1]
> https://join.slack.com/t/apachenifi/shared_invite/zt-11njbtkdx-ZRU8FKYSWoEHRJetidy0zA
> [2] https://nifi.apache.org/mailing_lists.html
>
> On Wed, Aug 10, 2022 at 8:45 AM Ryan Hendrickson
>  wrote:
> >
> > Can anyone request access to the Slack channel?
> >
> > Ryan
> >
> > On Mon, Aug 8, 2022 at 3:11 PM Joe Witt  wrote:
> >
> > > All
> > >
> > > To close the loop here the invite link Keith was trying was sending
> > > him in circles.  We sent him the correct one and he is now in.  And
> > > we're fixing the invite link in the readme page which caused the
> > > confusion.  We're also dropping the IRC reference as that is
> > > unattended and I dont think we have enough people using IRC to treat
> > > that like a supported communication path anyway at this point.
> > >
> > > https://issues.apache.org/jira/browse/NIFI-10328
> > >
> > > Thanks
> > >
> > > On Mon, Aug 8, 2022 at 10:11 AM Joe Witt  wrote:
> > > >
> > > > Keith,
> > > >
> > > > 0. Please subscribe to dev list following the link provided in
> > > > https://nifi.apache.org/mailing_lists.html
> > > >
> > > > 1. IRC is definitely non active.  The mailing lists and slack are the
> > > > two mechanisms.
> > > >
> > > > 2.  Please do share what you're seeing when you try the slack link.
> > > > That is our most popular/active communication mechanism right now.
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Aug 8, 2022 at 9:51 AM Keith Miklas 
> > > wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > Where is the current active discussion for NiFi?
> > > > >
> > > > > 1. The IRC channel is empty
> > > > > 2. When I attempt to login to the NiFi Slack workspace, I'm
> getting a
> > > > > message saying that I don't have an account on the workspace.
> > > > > - The new user invite link doesn't work. I can't get in with my
> gmail
> > > > > account.
> > > > >
> > > > > Thanks,
> > > > > Keith
> > >
>

Re: NiFi Slack, IRC?

[Ryan Hendrickson]

Can anyone request access to the Slack channel?

Ryan

On Mon, Aug 8, 2022 at 3:11 PM Joe Witt  wrote:

> All
>
> To close the loop here the invite link Keith was trying was sending
> him in circles.  We sent him the correct one and he is now in.  And
> we're fixing the invite link in the readme page which caused the
> confusion.  We're also dropping the IRC reference as that is
> unattended and I dont think we have enough people using IRC to treat
> that like a supported communication path anyway at this point.
>
> https://issues.apache.org/jira/browse/NIFI-10328
>
> Thanks
>
> On Mon, Aug 8, 2022 at 10:11 AM Joe Witt  wrote:
> >
> > Keith,
> >
> > 0. Please subscribe to dev list following the link provided in
> > https://nifi.apache.org/mailing_lists.html
> >
> > 1. IRC is definitely non active.  The mailing lists and slack are the
> > two mechanisms.
> >
> > 2.  Please do share what you're seeing when you try the slack link.
> > That is our most popular/active communication mechanism right now.
> >
> > Thanks
> >
> > On Mon, Aug 8, 2022 at 9:51 AM Keith Miklas 
> wrote:
> > >
> > > Hello,
> > >
> > > Where is the current active discussion for NiFi?
> > >
> > > 1. The IRC channel is empty
> > > 2. When I attempt to login to the NiFi Slack workspace, I'm getting a
> > > message saying that I don't have an account on the workspace.
> > > - The new user invite link doesn't work. I can't get in with my gmail
> > > account.
> > >
> > > Thanks,
> > > Keith
>

Re: [EXT] Re: Dark mode

[Ryan Hendrickson]

I was in a meeting demo'ing a dark-mode tool with the lights off in a conf
rm.  Then we flipped to Nifi to show where the data came from and blinded
everyone by the light!


https://issues.apache.org/jira/browse/NIFI-5424

+1 for a dark mode.

Ryan


On Fri, Jul 13, 2018 at 12:54 PM Peter Wicks (pwicks) 
wrote:

> I did something similar back in 1.2.0 using CSS overrides and a chrome
> plugin : https://userstyles.org/styles/142978/dark-nifi-1-2-0.
>
>
> -Original Message-
> From: Brandon DeVries [mailto:b...@jhu.edu]
> Sent: Friday, July 13, 2018 9:42 AM
> To: dev@nifi.apache.org
> Subject: [EXT] Re: Dark mode
>
> I think there are a lot of people that would be a big +1 on this.  Maybe
> even more so if it was abstracted so there could be multiple / custom
> "themes" (e.g. dark, classic, high contrast / 508 compliant...).
>
> Brandon
>
> On Fri, Jul 13, 2018 at 7:29 AM Joe Witt  wrote:
>
> > Rich
> >
> > This could certainly be an interesting option.  Perhaps file a JIRA
> > with some details on where you're at in your PR and post that when
> > ready.  Identify things you think might be remaining and such..
> >
> > Thanks
> >
> > On Fri, Jul 13, 2018 at 7:25 AM, Rich M 
> wrote:
> > > Hi
> > >
> > > Going through some NiFi bits and pieces while I'm between projects;
> > > I've got a half-baked dark mode kicking around, is there any interest
> > > in me pushing this to a remote branch somewhere? There's some iframes
> > > in dialogs missing styling, it probably wants someone more familiar
> > > with Angular to look at it and a couple of places the styling isn't
> > > applied to so it'd need a little work to even consider merging.
> > >
> > > https://i.imgur.com/lSofSq5.jpg
> > >
> > > Rich
> >
>

Re: Configurable default connection Prioritizer

[Ryan Hendrickson]

Hi Mark,
I searched and didn't see a ticket.  I created one here:
https://issues.apache.org/jira/browse/NIFI-9974  | Default Prioritizer for
new Relationships

Thanks,
Ryan

On Fri, Apr 15, 2022 at 10:24 AM Mark Bean  wrote:

> I like the idea. I'm thinking about how it would be implemented in the UI
> (and API). Specifically, I'm comparing this feature of setting the
> prioritizer to setting default connection properties via the configuration
> of a process group. For the default connection properties, updates to the
> process group configuration only change the default settings for any new
> connection created; it does not affect existing connections. This was
> intentional because it seemed heavy-handed to apply such settings to all
> existing connections - especially expiration settings which could result in
> data loss.
>
> However, the recommendation here is for actively changing the prioritizer
> in all existing connections (and potentially nested connections in child
> process groups). I understand the benefit and use case, but it seems the
> two modification styles (new connections or existing connections) would
> easily become confused.
>
> Would a checkbox for "apply prioritizer to all existing connections" be
> appropriate? And, if so, we still need to somehow make it clear that this
> applies to just prioritizer settings. I do not believe we want the other
> connection settings to be recursively applied to existing connections.
>
> Do we want to introduce a new context menu item for process groups,
> "Connection Settings" or something similar?
>
> Is there a JIRA ticket for this proposed new feature?
>
> On Tue, Apr 12, 2022 at 11:48 PM Ryan Hendrickson <
> ryan.andrew.hendrick...@gmail.com> wrote:
>
> > @Bryan - Correct, everything is still per queue, just with that
> convenience
> > feature.
> >
> > Totally agree with @Salvatore too.  I hadn't even thought of nested
> process
> > groups.
> >
> > On Tue, Apr 12, 2022 at 10:14 PM Salvatore 
> > wrote:
> >
> > > #2 would definitely be convenient. Maybe also include an option whether
> > to
> > > recurse down through nested process groups, or just apply to the
> selected
> > > process group.
> > >
> > > On Wed, 13 Apr 2022 at 06:05, Bryan Bende  wrote:
> > >
> > > > Makes sense. For # 2, it is still per queue with an "Apply All"
> > > > convenience right? Just trying to differentiate with prioritizing
> > > > across all queues.
> > > >
> > > > On Tue, Apr 12, 2022 at 3:22 PM Ryan Hendrickson
> > > >  wrote:
> > > > >
> > > > > I see two things as particularly useful...
> > > > >
> > > > >  1) Default Prioritizer for new Relationships (Bound to a process
> > > group,
> > > > > similar to how the "Default FlowFile Expiration" can be changed).
> > > > >  2) Applying a prioritizer to an entire Process Group as a one-time
> > > > action.
> > > > >
> > > > > Some background... I'm hand-converting two super-legacy v0.7.3
> > canvases
> > > > to
> > > > > v1.15.3.  Part of that is applying flow priorities all over the
> place
> > > in
> > > > > the new system.  Probably not a common task, but I could see this
> > > feature
> > > > > being useful for other week-to-week work too.
> > > > >
> > > > > Ryan
> > > > >
> > > > > On Tue, Apr 12, 2022 at 1:32 PM Bryan Bende 
> > wrote:
> > > > >
> > > > > > I think there are two different concepts here... The original
> > > > > > discussion is just about default settings for new connections.
> The
> > > > > > idea in NIFI-6831 is about prioritizing data across multiple
> > queues,
> > > > > > either for all of nifi or within a given process group.
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 12, 2022 at 1:13 PM Mark Bean  >
> > > > wrote:
> > > > > > >
> > > > > > > We experimented with the idea of a custom "Global Prioritizer".
> > One
> > > > of
> > > > > > the
> > > > > > > problems with this approach is that it ran the risk of breaking
> > the
> > > > > > > multi-tenancy philosophy. If there were a truly global
> priority,
> > it
> > > > would
> 

Re: Configurable default connection Prioritizer

[Ryan Hendrickson]

@Bryan - Correct, everything is still per queue, just with that convenience
feature.

Totally agree with @Salvatore too.  I hadn't even thought of nested process
groups.

On Tue, Apr 12, 2022 at 10:14 PM Salvatore  wrote:

> #2 would definitely be convenient. Maybe also include an option whether to
> recurse down through nested process groups, or just apply to the selected
> process group.
>
> On Wed, 13 Apr 2022 at 06:05, Bryan Bende  wrote:
>
> > Makes sense. For # 2, it is still per queue with an "Apply All"
> > convenience right? Just trying to differentiate with prioritizing
> > across all queues.
> >
> > On Tue, Apr 12, 2022 at 3:22 PM Ryan Hendrickson
> >  wrote:
> > >
> > > I see two things as particularly useful...
> > >
> > >  1) Default Prioritizer for new Relationships (Bound to a process
> group,
> > > similar to how the "Default FlowFile Expiration" can be changed).
> > >  2) Applying a prioritizer to an entire Process Group as a one-time
> > action.
> > >
> > > Some background... I'm hand-converting two super-legacy v0.7.3 canvases
> > to
> > > v1.15.3.  Part of that is applying flow priorities all over the place
> in
> > > the new system.  Probably not a common task, but I could see this
> feature
> > > being useful for other week-to-week work too.
> > >
> > > Ryan
> > >
> > > On Tue, Apr 12, 2022 at 1:32 PM Bryan Bende  wrote:
> > >
> > > > I think there are two different concepts here... The original
> > > > discussion is just about default settings for new connections. The
> > > > idea in NIFI-6831 is about prioritizing data across multiple queues,
> > > > either for all of nifi or within a given process group.
> > > >
> > > >
> > > > On Tue, Apr 12, 2022 at 1:13 PM Mark Bean 
> > wrote:
> > > > >
> > > > > We experimented with the idea of a custom "Global Prioritizer". One
> > of
> > > > the
> > > > > problems with this approach is that it ran the risk of breaking the
> > > > > multi-tenancy philosophy. If there were a truly global priority, it
> > would
> > > > > affect all flows, each may have different priority rules. However,
> if
> > > > this
> > > > > could be applied only at the process group level, it might have
> legs.
> > > > >
> > > > > You can follow the initial approach to such a mechanism in the JIRA
> > > > ticket.
> > > > > https://issues.apache.org/jira/browse/NIFI-6831
> > > > >
> > > > >
> > > > > On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson <
> > > > > ryan.andrew.hendrick...@gmail.com> wrote:
> > > > >
> > > > > > I just went to the config button in my process group, hoping to
> > set all
> > > > > > relationships in there to priority first Lots of right
> > clicking &
> > > > > > dragging instead.
> > > > > >
> > > > > > +1 for an approach like that.
> > > > > >
> > > > > > On Tue, Apr 12, 2022 at 11:44 AM Joe Witt 
> > wrote:
> > > > > >
> > > > > > > Hello
> > > > > > >
> > > > > > > Certainly the spirit of this is a good idea.  Would likely need
> > to
> > > > > > approach
> > > > > > > it at a more flow/process group centric level.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> > > > > > > ryan.andrew.hendrick...@gmail.com> wrote:
> > > > > > >
> > > > > > > > This would be very helpful.
> > > > > > > >
> > > > > > > > Ryan
> > > > > > > >
> > > > > > > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <
> > > > > > salvatoref...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > Do you see much value in being able to specify an
> > instance-wide
> > > > (or
> > > > > > > > > cluster-wide) default prioritizer for all connections that
> > do not
> > > > > > have
> > > > > > > > one
> > > > > > > > > manually set?
> > > > > > > > >
> > > > > > > > > Along with the the following properties in nifi.properties:
> > > > > > > > > nifi.queue.backpressure.count=1
> > > > > > > > > nifi.queue.backpressure.size=1 GB
> > > > > > > > >
> > > > > > > > > I'd like to see see something like
> > > > > > > > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > > > > > > > PriorityAttributePrioritizer
> > > > > > > > >
> > > > > > > > > Thoughts? My only concern would be if connection
> prioritizers
> > > > have a
> > > > > > > > > noticeable impact on system resources.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
>

Re: Configurable default connection Prioritizer

[Ryan Hendrickson]

I see two things as particularly useful...

 1) Default Prioritizer for new Relationships (Bound to a process group,
similar to how the "Default FlowFile Expiration" can be changed).
 2) Applying a prioritizer to an entire Process Group as a one-time action.

Some background... I'm hand-converting two super-legacy v0.7.3 canvases to
v1.15.3.  Part of that is applying flow priorities all over the place in
the new system.  Probably not a common task, but I could see this feature
being useful for other week-to-week work too.

Ryan

On Tue, Apr 12, 2022 at 1:32 PM Bryan Bende  wrote:

> I think there are two different concepts here... The original
> discussion is just about default settings for new connections. The
> idea in NIFI-6831 is about prioritizing data across multiple queues,
> either for all of nifi or within a given process group.
>
>
> On Tue, Apr 12, 2022 at 1:13 PM Mark Bean  wrote:
> >
> > We experimented with the idea of a custom "Global Prioritizer". One of
> the
> > problems with this approach is that it ran the risk of breaking the
> > multi-tenancy philosophy. If there were a truly global priority, it would
> > affect all flows, each may have different priority rules. However, if
> this
> > could be applied only at the process group level, it might have legs.
> >
> > You can follow the initial approach to such a mechanism in the JIRA
> ticket.
> > https://issues.apache.org/jira/browse/NIFI-6831
> >
> >
> > On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson <
> > ryan.andrew.hendrick...@gmail.com> wrote:
> >
> > > I just went to the config button in my process group, hoping to set all
> > > relationships in there to priority first Lots of right clicking &
> > > dragging instead.
> > >
> > > +1 for an approach like that.
> > >
> > > On Tue, Apr 12, 2022 at 11:44 AM Joe Witt  wrote:
> > >
> > > > Hello
> > > >
> > > > Certainly the spirit of this is a good idea.  Would likely need to
> > > approach
> > > > it at a more flow/process group centric level.
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> > > > ryan.andrew.hendrick...@gmail.com> wrote:
> > > >
> > > > > This would be very helpful.
> > > > >
> > > > > Ryan
> > > > >
> > > > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <
> > > salvatoref...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Do you see much value in being able to specify an instance-wide
> (or
> > > > > > cluster-wide) default prioritizer for all connections that do not
> > > have
> > > > > one
> > > > > > manually set?
> > > > > >
> > > > > > Along with the the following properties in nifi.properties:
> > > > > > nifi.queue.backpressure.count=1
> > > > > > nifi.queue.backpressure.size=1 GB
> > > > > >
> > > > > > I'd like to see see something like
> > > > > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > > > > PriorityAttributePrioritizer
> > > > > >
> > > > > > Thoughts? My only concern would be if connection prioritizers
> have a
> > > > > > noticeable impact on system resources.
> > > > > >
> > > > >
> > > >
> > >
>

Re: Configurable default connection Prioritizer

[Ryan Hendrickson]

I just went to the config button in my process group, hoping to set all
relationships in there to priority first Lots of right clicking &
dragging instead.

+1 for an approach like that.

On Tue, Apr 12, 2022 at 11:44 AM Joe Witt  wrote:

> Hello
>
> Certainly the spirit of this is a good idea.  Would likely need to approach
> it at a more flow/process group centric level.
>
> Thanks
>
> On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> ryan.andrew.hendrick...@gmail.com> wrote:
>
> > This would be very helpful.
> >
> > Ryan
> >
> > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss 
> > wrote:
> >
> > > Hi,
> > >
> > > Do you see much value in being able to specify an instance-wide (or
> > > cluster-wide) default prioritizer for all connections that do not have
> > one
> > > manually set?
> > >
> > > Along with the the following properties in nifi.properties:
> > > nifi.queue.backpressure.count=1
> > > nifi.queue.backpressure.size=1 GB
> > >
> > > I'd like to see see something like
> > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > PriorityAttributePrioritizer
> > >
> > > Thoughts? My only concern would be if connection prioritizers have a
> > > noticeable impact on system resources.
> > >
> >
>

Re: Configurable default connection Prioritizer

[Ryan Hendrickson]

This would be very helpful.

Ryan

On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss 
wrote:

> Hi,
>
> Do you see much value in being able to specify an instance-wide (or
> cluster-wide) default prioritizer for all connections that do not have one
> manually set?
>
> Along with the the following properties in nifi.properties:
> nifi.queue.backpressure.count=1
> nifi.queue.backpressure.size=1 GB
>
> I'd like to see see something like
> nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> PriorityAttributePrioritizer
>
> Thoughts? My only concern would be if connection prioritizers have a
> noticeable impact on system resources.
>

Re: Nifi UI slow

[Ryan Hendrickson]

Deepak,
   We noticed moving from 1.11.x to 1.13.2 the UI within the cluster slowed
down dramatically.  I'd be interested in all the details Joe mentioned -
either here or the users's mailing list.

Thanks,
Ryan

On Mon, Aug 16, 2021 at 7:33 PM Joe Witt  wrote:

> Deepak
>
> This thread is probably more appropriate for the users mailing list.
> Feel free to start it over there or just keep it here at this point.
>
> So yeah we need to know the details you mentioned we'd want/need.
> Versions of nifi, OS, JVM, how many machines, what type of machines,
> etc...  What is the load average?   Is it always slow or only when
> flow X, Y, or Z are running?
>
> You can absolutely setup NiFi clusters that perform quite well and are
> responsive in the context of the REST/UI considerations even under
> high load.
>
> Thanks
>
> On Mon, Aug 16, 2021 at 4:13 PM Deepak Reddy 
> wrote:
> >
> > Hello,
> >
> > I've been struggling with a new issue for at least two weeks now. The
> issue
> > is that the Nifi UI response is very slow. I researched a lot on the
> > community and I disabled majority of the unused processors but still the
> > Nifi UI is very slow. Do you guys wanna have a peek into this issue? I
> know
> > you need a lot of other information about our cluster. But let me know so
> > that I can provide you with all those details. I tried, tried and tried
> and
> > finally reached for your help.
> >
> > Thanks
> > Deepak
>

Re: [DISCUSS] NiFi 2.0 Release Goals

[Ryan Hendrickson]

Thanks for the push Otto.  They are created:

[NIFI-9026 <https://issues.apache.org/jira/browse/NIFI-9026>] - Integration
with auto-scaling container environments
[NIFI-9027 <https://issues.apache.org/jira/browse/NIFI-9027>] - Integration
of S3 buckets with Remote Process Groups for outage events and data overruns
[NIFI-9028 <https://issues.apache.org/jira/browse/NIFI-9028>] - Large
Canvas Management

Best,
Ryan



On Fri, Aug 6, 2021 at 1:34 PM Otto Fowler  wrote:

>  You don’t have to assign it to yourself.  User submitted jiras are very
> important, because they include use cases and details that may not occur to
> a developer who just picks it up.  It is a great way to contribute!
>
> From: Ryan Hendrickson 
> 
> Reply: dev@nifi.apache.org  
> Date: August 6, 2021 at 03:23:43
> To: dev@nifi.apache.org  
> Subject:  Re: [DISCUSS] NiFi 2.0 Release Goals
>
> If I put Jira's in, then I'd have to admit I'm a developer vs a user
> :-)
>
> I'm just a power user at heart :-) ...
>
> Ryan
>
> On Thu, Aug 5, 2021 at 2:01 PM Otto Fowler 
> wrote:
>
> > Ryan, these are awesome, are there jiras for them? It would be best to
> > capture the requirements / use cases
> >
> > From: Ryan Hendrickson 
> > 
> > Reply: dev@nifi.apache.org  
> > Date: August 4, 2021 at 01:14:23
> > To: dev@nifi.apache.org  
> > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> >
> > I'll preface with what I'm about to say is not rooted in any real
> > understanding of the lift required or other customer experiences... We
> > have used NiFi for a long time and are excited for what new features may
> > lay ahead in the 2.x line.
> >
> > ---After having written this, I think our succinct pain points focus
> > around large data flow management, auto-scaling container environments,
> and
> > integration of external analytic/enrichment creation.---
> >
> > Thanks,
> > Ryan
> >
> >
> > 1 - Integration with auto-scaling container environments
> > We often struggle with scaling custom processors in large flows. We've
> > implemented a recent design pattern where we write custom python code
> > packaged into a docker image, fronted with a web-service, and deployed
> into
> > a container environment. In NiFi we use InvokeHTTP to call out to the
> > web-service. If the load becomes too great, we can easily spin-up more
> > than one external web-service without sacrificing the NiFi's overall
> > thruput and thread contention. In this scenario, we're using NiFi to
> > manage the overall dataflow, routes, ETL services, etc, but not so much
> > smarts around auto-scaling processing, etc.
> >
> > This makes it really easy to receive a dataflow analytic/enrichment from
> an
> > external team, wrap it in a docker image, and deploy it into an
> > auto-scaling container environment. This also lets us avoid conversations
> > about how performant the code is and if it's meeting a minimum throughput
> > requirement - typically XXGB/5minutes.
> >
> > The request for 2.x would be to make interacting with container deployed
> > services a more seamless, streamlined process, possibly through a
> > normalized API or as a "Remote Container Environment" that can bind to a
> > service with a health endpoint and verify it's alive and well, etc.
> >
> > 2 - Integration of S3 buckets with Remote Process Groups for outage
> events
> > and data overruns
> > Another frequent struggle is the occasional death of a server in a NiFi
> > flow. Example -- Our overall NiFi flow consists of ~15 stand-alone NiFis
> > and 2 NiFi clusters. The stand-alone NiFi's are linearly chained together
> > with Remote Process Groups to perform the overall dataflow goals. If any
> > one of the 15 experience an outage, data begins to backup on the previous
> > NiFi, hitting backpressure limits, etc. When the down server comes back
> > online, it receives a flood of data, often overwhelming it, necessitating
> > careful flowfile limits and backpressure points.
> >
> > The design pattern we're exploring, but would like to see baked into
> NiFi,
> > is that when a Remote Process Group becomes unavailable due to
> > server-to-server comms failures AND the backpressure limit is reached on
> a
> > relationship, the Remote Process Group would begin sending data to an S3
> > bucket as an automatic failover procedure. When the downed server comes
> > back-online, the Remote Process Group port would know to pull data from
> S3
> > AND receive data via the port from the chained server.
> >
> > 3 - Large Canvas Mana

Re: [DISCUSS] NiFi 2.0 Release Goals

[Ryan Hendrickson]

If I put Jira's in, then I'd have to admit I'm a developer vs a user
 :-)

I'm just a power user at heart :-) ...

Ryan

On Thu, Aug 5, 2021 at 2:01 PM Otto Fowler  wrote:

>  Ryan, these are awesome, are there jiras for them?  It would be best to
> capture the requirements / use cases
>
> From: Ryan Hendrickson 
> 
> Reply: dev@nifi.apache.org  
> Date: August 4, 2021 at 01:14:23
> To: dev@nifi.apache.org  
> Subject:  Re: [DISCUSS] NiFi 2.0 Release Goals
>
> I'll preface with what I'm about to say is not rooted in any real
> understanding of the lift required or other customer experiences... We
> have used NiFi for a long time and are excited for what new features may
> lay ahead in the 2.x line.
>
> ---After having written this, I think our succinct pain points focus
> around large data flow management, auto-scaling container environments, and
> integration of external analytic/enrichment creation.---
>
> Thanks,
> Ryan
>
>
> 1 - Integration with auto-scaling container environments
> We often struggle with scaling custom processors in large flows. We've
> implemented a recent design pattern where we write custom python code
> packaged into a docker image, fronted with a web-service, and deployed into
> a container environment. In NiFi we use InvokeHTTP to call out to the
> web-service. If the load becomes too great, we can easily spin-up more
> than one external web-service without sacrificing the NiFi's overall
> thruput and thread contention. In this scenario, we're using NiFi to
> manage the overall dataflow, routes, ETL services, etc, but not so much
> smarts around auto-scaling processing, etc.
>
> This makes it really easy to receive a dataflow analytic/enrichment from an
> external team, wrap it in a docker image, and deploy it into an
> auto-scaling container environment. This also lets us avoid conversations
> about how performant the code is and if it's meeting a minimum throughput
> requirement - typically XXGB/5minutes.
>
> The request for 2.x would be to make interacting with container deployed
> services a more seamless, streamlined process, possibly through a
> normalized API or as a "Remote Container Environment" that can bind to a
> service with a health endpoint and verify it's alive and well, etc.
>
> 2 - Integration of S3 buckets with Remote Process Groups for outage events
> and data overruns
> Another frequent struggle is the occasional death of a server in a NiFi
> flow. Example -- Our overall NiFi flow consists of ~15 stand-alone NiFis
> and 2 NiFi clusters. The stand-alone NiFi's are linearly chained together
> with Remote Process Groups to perform the overall dataflow goals. If any
> one of the 15 experience an outage, data begins to backup on the previous
> NiFi, hitting backpressure limits, etc. When the down server comes back
> online, it receives a flood of data, often overwhelming it, necessitating
> careful flowfile limits and backpressure points.
>
> The design pattern we're exploring, but would like to see baked into NiFi,
> is that when a Remote Process Group becomes unavailable due to
> server-to-server comms failures AND the backpressure limit is reached on a
> relationship, the Remote Process Group would begin sending data to an S3
> bucket as an automatic failover procedure. When the downed server comes
> back-online, the Remote Process Group port would know to pull data from S3
> AND receive data via the port from the chained server.
>
> 3 - Large Canvas Management
> Beginning with a NiFi flow that consists of ~15 stand-alone NiFis and 2
> NiFi clusters, each canvas is managed individually. To see the entire
> canvas, systems engineering diagrams need to be kept up-to-date identifying
> all the input/output ports. To move processors from server to server often
> requires the verification of the existence of scripts, configuration files,
> NiFi custom nars, and using the templating or Registry to migrate portions
> of the dataflow.
>
> Our current design pattern to accomplish some of these tasks include using
> Ansible, Jenkins, and Nexus to verify script & configuration files are up
> to date. We use Confluence's Gliffy to document the NiFi communication
> paths between the servers.
>
> Ideally, we'd be able to see an overall management view of the Canvases
> together in a single place - it would indicate which server the portions of
> the canvas are deployed on and tie together where the Remote Process Group
> ports are connected. Additionally, NiFi would allow the user to move
> Processors and Process Groups seamlessly from one server to another server
> to help manually distribute and balance data flow processing cycles.
>
>
>
> On Tue, Aug 3, 2021 at 12:45 PM Joe Witt  wrot

Re: [DISCUSS] NiFi 2.0 Release Goals

[Ryan Hendrickson]

I'll preface with what I'm about to say is not rooted in any real
understanding of the lift required or other customer experiences...  We
have used NiFi for a long time and are excited for what new features may
lay ahead in the 2.x line.

---After having written this, I think our succinct pain points focus
around large data flow management, auto-scaling container environments, and
integration of external analytic/enrichment creation.---

Thanks,
Ryan


1 - Integration with auto-scaling container environments
We often struggle with scaling custom processors in large flows.  We've
implemented a recent design pattern where we write custom python code
packaged into a docker image, fronted with a web-service, and deployed into
a container environment.  In NiFi we use InvokeHTTP to call out to the
web-service.  If the load becomes too great, we can easily spin-up more
than one external web-service without sacrificing the NiFi's overall
thruput and thread contention.  In this scenario, we're using NiFi to
manage the overall dataflow, routes, ETL services, etc, but not so much
smarts around auto-scaling processing, etc.

This makes it really easy to receive a dataflow analytic/enrichment from an
external team, wrap it in a docker image, and deploy it into an
auto-scaling container environment.  This also lets us avoid conversations
about how performant the code is and if it's meeting a minimum throughput
requirement - typically XXGB/5minutes.

The request for 2.x would be to make interacting with container deployed
services a more seamless, streamlined process, possibly through a
normalized API or as a "Remote Container Environment" that can bind to a
service with a health endpoint and verify it's alive and well, etc.

2 - Integration of S3 buckets with Remote Process Groups for outage events
and data overruns
Another frequent struggle is the occasional death of a server in a NiFi
flow.  Example -- Our overall NiFi flow consists of ~15 stand-alone NiFis
and 2 NiFi clusters.  The stand-alone NiFi's are linearly chained together
with Remote Process Groups to perform the overall dataflow goals.  If any
one of the 15 experience an outage, data begins to backup on the previous
NiFi, hitting backpressure limits, etc.  When the down server comes back
online, it receives a flood of data, often overwhelming it, necessitating
careful flowfile limits and backpressure points.

The design pattern we're exploring, but would like to see baked into NiFi,
is that when a Remote Process Group becomes unavailable due to
server-to-server comms failures AND the backpressure limit is reached on a
relationship, the Remote Process Group would begin sending data to an S3
bucket as an automatic failover procedure.  When the downed server comes
back-online, the Remote Process Group port would know to pull data from S3
AND receive data via the port from the chained server.

3 - Large Canvas Management
Beginning with a NiFi flow that consists of ~15 stand-alone NiFis and 2
NiFi clusters, each canvas is managed individually.  To see the entire
canvas, systems engineering diagrams need to be kept up-to-date identifying
all the input/output ports.  To move processors from server to server often
requires the verification of the existence of scripts, configuration files,
NiFi custom nars, and using the templating or Registry to migrate portions
of the dataflow.

Our current design pattern to accomplish some of these tasks include using
Ansible, Jenkins, and Nexus to verify script & configuration files are up
to date.  We use Confluence's Gliffy to document the NiFi communication
paths between the servers.

Ideally, we'd be able to see an overall management view of the Canvases
together in a single place - it would indicate which server the portions of
the canvas are deployed on and tie together where the Remote Process Group
ports are connected.  Additionally, NiFi would allow the user to move
Processors and Process Groups seamlessly from one server to another server
to help manually distribute and balance data flow processing cycles.



On Tue, Aug 3, 2021 at 12:45 PM Joe Witt  wrote:

> Kevin,
>
> This may well be a very interesting approach which opens up a lot of
> options for us in how we tackle a 2.x...
>
> Thanks
>
> On Tue, Aug 3, 2021 at 8:45 AM Kevin Doran 
> wrote:
> >
> > Outside of core NiFi, for MiNiFi, I think it would be beneficial to
> align MiNiFi Java and MiNiFI C++ to use the same method(s) for command and
> control / flow deployment. This was discussed when MiNiFi Java was merged
> into the NiFi codebase [1].
> >
> > My proposal would be to create a Java implementation of the C2 Protocol
> used by MiNiFi C++, both server-side and client-side, to allow deploying to
> MiNiFi Java instances from a centralize flow definition. Thanks to the
> unification of MiNiFi Java and NiFi, this could likely also be added as a
> capability of NiFi at that same time.
> >
> > If we are able to do this on the 1.x line, then I would also support
> 

Re: NIFI-7646 - Improve performance of MergeContent

[Ryan Hendrickson]

Thanks Mark!

On Wed, Apr 21, 2021 at 8:48 PM Mark Payne  wrote:

> Ryan,
>
> It gets a bit more complex than this, because the flowfiles may not always
> be accessed/read sequentially in exactly the same order that they live on
> disk, there’s concurrent threads/disk accessed to consider, etc. But in the
> best case scenarios, yes that is accurate.
>
> Keep in mind, though, that what you are comparing there is the performance
> of the disk accesses/reads, and that is, of course, not the entire picture.
> Lots more going on under the covers, so if you see a performance
> improvement of 20x in reading the content, that won’t mean a 20x
> improvement in overall throughout.
>
> But it sure won’t hurt! :)
>
> -Mark
>
> Sent from my iPhone
>
> > On Apr 21, 2021, at 8:34 PM, Ryan Hendrickson <
> ryan.andrew.hendrick...@gmail.com> wrote:
> >
> > https://issues.apache.org/jira/browse/NIFI-7646 - Improve performance
> of
> > MergeContent / others that read content of many small FlowFiles
> >
> > Hi,
> >   In reference to the ticket above, released in 1.13, the descriptions
> > says "if the FlowFile is small, say 200 bytes, the result is that we
> > perform 2+ disk accesses to read those 200 bytes (even though 4K - 8K is
> a
> > typical block size and could be read in the same amount of time as those
> > 200 bytes)."
> >
> >   To clarify, if the FlowFiles are never more than 1K, and the block size
> > is 4k, does that mean this improvement will read 4 FlowFiles with the
> > resources of 1?
> >
> >   This would be a 4:1 improvement.  Or in the 200 byte scenario, it would
> > be a 20:1 improvement?
> >
> > Thanks,
> > Ryan
>

Funnel Performance

[Ryan Hendrickson]

Hi everyone,
   I'm curious what the performance impact is of adding a number of funnels.

   For example:  We had 3 relationships with 10 million flow files in each
relationship going to a Funnel to create a single Relationship into a Merge
Content Processor.  Would it be 'more' performant to just send these 3
relationships directly to the Merge Content Processor?

   How does this work under-the-hood?  Does the funnel read from each
relationship and create a "new" one?  If so, how many flowfiles does it
pull at a time?  Does it respect priority across the many
relationships going into the funnel?

Thanks,
Ryan

NIFI-7646 - Improve performance of MergeContent

[Ryan Hendrickson]

https://issues.apache.org/jira/browse/NIFI-7646 - Improve performance of
MergeContent / others that read content of many small FlowFiles

Hi,
   In reference to the ticket above, released in 1.13, the descriptions
says "if the FlowFile is small, say 200 bytes, the result is that we
perform 2+ disk accesses to read those 200 bytes (even though 4K - 8K is a
typical block size and could be read in the same amount of time as those
200 bytes)."

   To clarify, if the FlowFiles are never more than 1K, and the block size
is 4k, does that mean this improvement will read 4 FlowFiles with the
resources of 1?

   This would be a 4:1 improvement.  Or in the 200 byte scenario, it would
be a 20:1 improvement?

Thanks,
Ryan

Re: Dropping support for Java 8

[Ryan Hendrickson]

Any idea of when a NiFi 2 line would happen - eg. within 2021, not til 2022?

Thanks,
Ryan

On Fri, Mar 26, 2021 at 4:34 PM José Luis Pedrosa 
wrote:

> Hi All
>
> thanks for the replies, I may say some quite incorrect things as I am new
> to NiFi. I see that there are two docker images:
>
> https://github.com/apache/nifi/tree/master/nifi-docker/dockermaven
> https://github.com/apache/nifi/tree/master/nifi-docker/dockerhub
>
> I am assuming what Joey is mentioning (mvn) applies only to "dockermaven"?
> Indeed the usual way is to have different tags for different versions of
> the underlying bases, which is also already present for us, as nifi chose
> (by omission) the default flavour for it (there are a lot of openjdk XX
> images, even windows), so one extreme we could generate one per the
> different openjdk on the other extreme we could generate only one.  Not
> sure if any of the JVM settings done in the scripts apply only to one Java
> version, potentially there would be the need to add flexibility to work
> with both on the scripts.
>
> From my perspective as a user: I use the container as a container, in this
> case I just want the recommended option (performance, stability...)  and
> after those criterias, the smaller the image the better.
>
> Happy to help if you have a clear idea if you (apache committers) of what
> is the destination desired.
>
> Kind regards
> JL
>
> On Fri, Mar 26, 2021 at 6:07 PM Kevin Doran  wrote:
>
> > That’s awesome, Joey. I was not aware of that new capability, and
> > certainly is an improvement over maintaining multiple Dockerfiles that
> only
> > differ in their base image. I’ll add that to our readme instructions for
> > building the Docker image, and if there is enough interest, we could use
> > that method to put out a java 11 based image in the future.
> >
> > > On Mar 26, 2021, at 13:40, Joey Frazee  .INVALID>
> > wrote:
> > >
> > > Kevin, it’s recently possible to specify the underlying openjdk image
> > tag as a property in the Maven build, e.g., -Pdocker
> > -Ddocker.image.tag=11-jre so it should be easier to start publishing
> those
> > now too if it’s decided it’s a good idea.
> > >
> > > The default remains 8 for the sorts of concerns being discussed, but
> > this provides the flexibility for people to use an official source
> release
> > to build 11-based images.
> > >
> > > -joey
> > >
> > >> On Mar 26, 2021, at 8:13 AM, Joe Witt  wrote:
> > >>
> > >> Good thread.  I'd say when a NiFi 2 line happens Java 8 would be gone
> > >> completely and we would/should consider only supporting the latest LTS
> > >> line perhaps.
> > >>
> > >>> On Fri, Mar 26, 2021 at 8:05 AM Kevin Doran 
> wrote:
> > >>>
> > >>> Hi JL,
> > >>>
> > >>> It’s worth discussing/considering changes such as this periodically,
> > so thanks for bringing it up.
> > >>>
> > >>> Personally, I would be hesitant to make such a large change. While it
> > would likely be a net-positive for the NiFi image itself, I think it
> would
> > impact a number of community members that have Dockerfile’s that use our
> > image as a starting point.
> > >>>
> > >>> A GitHub code search [1] seems to confirm this, showing >100
> > Dockerfiles that contain “FROM apache/nifi*”
> > >>>
> > >>> For NiFi 1.x, I think the best we could do is leverage tagging to
> > offer image variants that differ in layers we build upon, for example OS
> or
> > JDK/JRE variants. This seems to be a popular method, for example, Apache
> > Tomcat offers a multiple of combinations of version, JDK, and OS [2].
> > >>>
> > >>> So if it would be beneficial, we could add official images for other
> > jdk versions indicated by tags, for example apache/nifi:1.13.2,
> > apache/nifi:1.13.2-jre11, etc.
> > >>>
> > >>> I believe this was part of the plan for the (empty)
> > apache/nifi-container code repository [3]. I think the intention was
> always
> > to build out a richer set of diverse container images based on files in
> > this repository, which could be maintained/released decoupled from the
> NiFi
> > source code itself. With so many in the community running containerized
> > NiFi, perhaps it's worth reviving that discussion to see what, if
> anything,
> > would be most valuable to add to our container offerings.
> > >>>
> > >>> For NiFi 2 we can and should definitely consider what changes we want
> > to make to our “default” base image, including which JRE.
> > >>>
> > >>> [1]
> >
> https://github.com/search?l==%22FROM+apache%2Fnifi%22+language%3ADockerfile=code
> > <
> >
> https://github.com/search?l==%22FROM+apache/nifi%22+language:Dockerfile=code
> > >
> > >>> [2] https://hub.docker.com/_/tomcat?tab=description <
> > https://hub.docker.com/_/tomcat?tab=description>
> > >>> [3] https://github.com/apache/nifi-container <
> > https://github.com/apache/nifi-container>
> > >>>
> > >>> Thanks!
> > >>> Kevin
> > >>>
> > > On Mar 26, 2021, at 07:49, José Luis Pedrosa 
> > wrote:
> > 
> >  Hi All
> > 
> >  I see that the 

Re: [EXT] Re: NiFi 2.0 Roadmap

[Ryan Hendrickson]

Resurrecting an old thread here... Any updates on a 2.0 or new major
release of NiFi?

Thanks,
Ryan

On Fri, Aug 2, 2019 at 10:14 AM ski n  wrote:

> "a Flow Registry that can hold extensions and we can then make nifi 2.0
> fundamentally about distributing nifi as a kernel (small as possible) and
> all extensions come from a flow registry on demand. "
>
> Nice! Great to hear NiFi as an extensible platform. I would see that the
> NiFi Registry is linked to a repository. Where one is the official
> repository and another for community. At this repo/library people can
> public put Processors/Services/Templates/Extensions. Browsing/Installing
> should be similar to an app store (Jenkins does this nicely and also
> Node-Red Library https://flows.nodered.org).
>
> Another thing that I would like to see is that for developing/testing you
> don't need to leave NiFi. For example directly code/debug
> (codemirror/ace/monaco) and unit test it with input/output of a flowfile
> for a specific processor.
>
> Also a thing is making the GUI more responsive (rendering) and comfortable
> (like undo). Most big features are already on Confluence:
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
>
>
> Raymond
>
>
>
>
> Op vr 14 jun. 2019 om 22:59 schreef Peter Wicks (pwicks) <
> pwi...@micron.com
> >:
>
> > I've also heard something about a big change to the way relationships
> > work. Maybe grouping relationships into larger groupings "failure" as a
> > parent, with multiple optional/fine grained, children. Something like
> > that.  If that rings a bell, maybe add it to your list .
> >
> > -Original Message-
> > From: Joe Witt 
> > Sent: Friday, June 14, 2019 11:31 AM
> > To: dev@nifi.apache.org
> > Subject: [EXT] Re: NiFi 2.0 Roadmap
> >
> > It makes sense there would be an mvp set of registry capability and thus
> a
> > dependency for nifi 2.0 on registry readiness/version.  Otherwise I would
> > largely hope not.
> >
> >
> > On Fri, Jun 14, 2019 at 1:29 PM Otto Fowler 
> > wrote:
> >
> > > Will that effort or planning be across all the nifi projects?  minifi
> > > / cpp / registry etc?
> > >
> > >
> > > On June 14, 2019 at 13:01:36, Joe Witt (joe.w...@gmail.com) wrote:
> > >
> > > Peter,
> > >
> > > Yeah I think we're all circling around similar thoughts on things
> > > which are 'best for a major release' and we need to start codifying
> > > that. At the same time we need this to be focused on items which can
> > > only reasonably happen in a major release and not become a new kitchen
> > > sink for JIRAs/ideas. We should frame up a wiki page for this effort.
> > > I'm happy to kick that off soon (time permitting). In my mind the key
> > > domino here is having a Flow Registry that can hold extensions and we
> > > can then make nifi
> > > 2.0 fundamentally about distributing nifi as a kernel (small as
> > > possible) and all extensions come from a flow registry on demand.
> > > Other obvious things like Java 11 as the base requirement and killing
> > > off deprecated things come to mind.
> > >
> > > Thanks
> > >
> > > On Fri, Jun 14, 2019 at 11:45 AM Peter Wicks (pwicks)
> > > 
> > > wrote:
> > >
> > > > I've seen a lot of comments along the line of, "I don't think this
> > > > will happen before NiFi 2.0". Do we have a roadmap/list somewhere of
> > > > the big general changes planned for NiFi 2.0 or some kind of 2.0
> > roadmap?
> > > >
> > > > --Peter
> > > >
> > >
> >
>

Re: NiFi PriorityAttributePrioritizer doesn't prioritize all items in a queue.

[Ryan Hendrickson]

Hi Mark,
   Thanks for the confirmation.

   For the documentation, I'd ask that you consider adding something to the
"Configure Connection -> Available Prioritizer" Help Icon.. Right now it
says "Available prioritizers that could reprioritize FlowFiles in this work
queue."  And the Selected Prioritizers says: "Prioritizers that have been
selected to prioritize FlowFiles in this work queue."  Something to notify
users of the behavior and NiFi attribute that controls it would be helpful.

 Do you know if the default size of 20,000 is specific to the default
JVM size of 512MB in the bootstrap.conf?  Our JVMs are all set to 32GB.  If
we increase the JVM to something much larger, chosen for?  Would you think
that's a linear relationship, or more based on how many total connections
on the entire canvas?

   For a long term improvement..  What about providing an override?..
Similar to the box for "Size Threshold" and "Back Pressure Object
Threshold", could there be one with "Priority Queue Size", set to default
to 20,000, but could be modified on a connection-by-connection basis?
That'd help mitigate what I'm afraid is going to happen when we jacked the
size from 20,000 to 200,000 for the entire server instead of the single
queue.

Thanks again,
Ryan

On Thu, Jun 25, 2020 at 12:05 PM Mark Payne  wrote:

> Ryan,
>
> Yes, your analysis is correct. NiFi will prioritize things in the Active
> Queue. Once swapping starts happening, it can’t constantly rebalance all of
> the FlowFiles, as doing so would mean constantly re-reading the swap data
> from disk and re-writing all of the swap data, which would be prohibitively
> expensive.
>
> I went to find the section of the User Guide that explains this, but
> apparently this isn’t explained in the user guide like I thought it was :(
> Sorry about that. I created a Jira [1] to explain that.
>
> So yes, if you increase the swap threshold, that means that you’ll have a
> larger active queue and as a result you’ll be able to handle the
> prioritization better. But it means that you’ll be holding more FlowFiles
> in memory and as a result can encounter OutOfMemoryError easily if your
> heap is not large enough.
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-7583
>
>
> On Jun 24, 2020, at 2:01 PM, Ryan Hendrickson <
> ryan.andrew.hendrick...@gmail.com<mailto:ryan.andrew.hendrick...@gmail.com>>
> wrote:
>
> Hello,
>   We've noticed that the PriorityAttributePrioritizer doesn't prioritize
> all items in a queue.
>
>   We'll get a queue of 50,000 items in it, then a 50,001 item will be
> added with a #1 priority.  This item won't be prioritized above lower
> priority items in the queue.
>
>   We did some research into this and wanted to confirm what we found...
>
>For any Relationship there are 2 underlying queues:
>   (1) An Active Queue (java.util.PriorityQueue) for the first 20,000
> items, defined by the nifi.queue.swap.threshold in nifi.properties.
>   (2) A Swap Queue (java.util.ArrayList) for the rest of the
> queue's items.
>
>If the Active Queue is full, every new item, regardless of priority, is
> placed on the Swap Queue.  No item on the Swap Queue will be re-prioritized
> until the entire Active Queue is empty (Line 284
> <
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SwappablePriorityQueue.java#L284
> >).
> Thus, the PriorityAttributePrioritize only "actively" sorts the first
> 20,000 items on the relationship.  Once the queue is empty, it'll re-sort
> the rest and move the top 20,000 over.  Then the problem repeats.
>
>I hadn't read anywhere in the documentation that this is the case.
> Even in the admin guide definition of the nifi.queue.swap.threshold it
> doesn't mention that it affects the PriorityQueue system.  We just raised
> it from 20,000 to 200,000 but we're all assuming that's going to have
> detrimental effects elsewhere.
>
>   If you guys could confirm our suspicions here, we'd appreciate that.
> Also any suggestions on what to do here, or how an increased threshold size
> negatively impacts the JVM.
>
>
> Stack we followed to figure this out:
>
> SwappablePriorityQueue.java: doPoll()
> SwappablePriorityQueue.java: poll()
> StandardFlowFileQueue.java: queue.poll()
> StandardConnection.java flowFileQueue.poll()
> StandardProcessSession.java: conn.poll()
> InvokeHttp.java: session.get()
>
> Thanks,
> Ryan
>
>

Re: NiFi PriorityAttributePrioritizer doesn't prioritize all items in a queue.

[Ryan Hendrickson]

Hi Andy,
   Yea, it's the Apache NiFi 1.11.4 version.

Thanks,
Ryan

On Wed, Jun 24, 2020 at 7:06 PM Andy LoPresto  wrote:

> Hi Ryan,
>
> Thanks for writing such a detailed report. What version of NiFi are you
> observing this behavior against? I know there were some issues in older
> versions with queue swapping, but since you linked to current code, I’m
> assuming you’re experiencing this on 1.11.4?
>
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Jun 24, 2020, at 11:01 AM, Ryan Hendrickson <
> ryan.andrew.hendrick...@gmail.com> wrote:
> >
> > Hello,
> >   We've noticed that the PriorityAttributePrioritizer doesn't prioritize
> > all items in a queue.
> >
> >   We'll get a queue of 50,000 items in it, then a 50,001 item will be
> > added with a #1 priority.  This item won't be prioritized above lower
> > priority items in the queue.
> >
> >   We did some research into this and wanted to confirm what we found...
> >
> >For any Relationship there are 2 underlying queues:
> >   (1) An Active Queue (java.util.PriorityQueue) for the first 20,000
> > items, defined by the nifi.queue.swap.threshold in nifi.properties.
> >   (2) A Swap Queue (java.util.ArrayList) for the rest of the
> > queue's items.
> >
> >If the Active Queue is full, every new item, regardless of priority,
> is
> > placed on the Swap Queue.  No item on the Swap Queue will be
> re-prioritized
> > until the entire Active Queue is empty (Line 284
> > <
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SwappablePriorityQueue.java#L284
> >).
> > Thus, the PriorityAttributePrioritize only "actively" sorts the first
> > 20,000 items on the relationship.  Once the queue is empty, it'll re-sort
> > the rest and move the top 20,000 over.  Then the problem repeats.
> >
> >I hadn't read anywhere in the documentation that this is the case.
> > Even in the admin guide definition of the nifi.queue.swap.threshold it
> > doesn't mention that it affects the PriorityQueue system.  We just raised
> > it from 20,000 to 200,000 but we're all assuming that's going to have
> > detrimental effects elsewhere.
> >
> >   If you guys could confirm our suspicions here, we'd appreciate that.
> > Also any suggestions on what to do here, or how an increased threshold
> size
> > negatively impacts the JVM.
> >
> >
> > Stack we followed to figure this out:
> >
> > SwappablePriorityQueue.java: doPoll()
> > SwappablePriorityQueue.java: poll()
> > StandardFlowFileQueue.java: queue.poll()
> > StandardConnection.java flowFileQueue.poll()
> > StandardProcessSession.java: conn.poll()
> > InvokeHttp.java: session.get()
> >
> > Thanks,
> > Ryan
>
>

NiFi PriorityAttributePrioritizer doesn't prioritize all items in a queue.

[Ryan Hendrickson]

Hello,
   We've noticed that the PriorityAttributePrioritizer doesn't prioritize
all items in a queue.

   We'll get a queue of 50,000 items in it, then a 50,001 item will be
added with a #1 priority.  This item won't be prioritized above lower
priority items in the queue.

   We did some research into this and wanted to confirm what we found...

For any Relationship there are 2 underlying queues:
   (1) An Active Queue (java.util.PriorityQueue) for the first 20,000
items, defined by the nifi.queue.swap.threshold in nifi.properties.
   (2) A Swap Queue (java.util.ArrayList) for the rest of the
queue's items.

If the Active Queue is full, every new item, regardless of priority, is
placed on the Swap Queue.  No item on the Swap Queue will be re-prioritized
until the entire Active Queue is empty (Line 284
).
Thus, the PriorityAttributePrioritize only "actively" sorts the first
20,000 items on the relationship.  Once the queue is empty, it'll re-sort
the rest and move the top 20,000 over.  Then the problem repeats.

I hadn't read anywhere in the documentation that this is the case.
Even in the admin guide definition of the nifi.queue.swap.threshold it
doesn't mention that it affects the PriorityQueue system.  We just raised
it from 20,000 to 200,000 but we're all assuming that's going to have
detrimental effects elsewhere.

   If you guys could confirm our suspicions here, we'd appreciate that.
Also any suggestions on what to do here, or how an increased threshold size
negatively impacts the JVM.


Stack we followed to figure this out:

SwappablePriorityQueue.java: doPoll()
SwappablePriorityQueue.java: poll()
StandardFlowFileQueue.java: queue.poll()
StandardConnection.java flowFileQueue.poll()
StandardProcessSession.java: conn.poll()
InvokeHttp.java: session.get()

Thanks,
Ryan

Re: Potential 1.11.X showstopper

[Ryan Hendrickson]

Joe,
   We're running:

   - OpenJDK Java 1.8.0_242
   - NiFi 1.11.0
   - CentOS Linux 7.7.1908


   We're seeing this across a dozen NiFi's with the same setup.  To
reproduce the issue, Generate Flow Files 100GB across a couple million
files -> Site to Site -> Receive data -> Merge Content.  We had no issues
with this stack:

   - OpenJDK Java 1.8.0_232.
   - NiFi 1.9.2
   - CentOS Linux 7.7.1908

   Can your team setup a similar stack and test?

Ryan

On Thu, Feb 6, 2020 at 4:15 PM Joe Witt  wrote:

> received a direct reply - Elli cannot share.
>
> I think unless someone else is able to replicate the behavior there isn't
> much more we can tackle on this.
>
> Thanks
>
> On Thu, Feb 6, 2020 at 4:10 PM Joe Witt  wrote:
>
> > Yes Elli it is possible.  Can we please get those lsof outputs in a JIRA?
> > As well as more details about configuration?
> >
> > Thanks
> >
> > On Thu, Feb 6, 2020 at 2:44 PM Andy LoPresto 
> wrote:
> >
> >> I have no input on the specific issue you’re encountering, but a pattern
> >> we have seen to reduce the overhead of multiple remote input ports being
> >> required is to use a “central” remote input port and immediately follow
> it
> >> with a RouteOnAttribute to distribute specific flowfiles to the
> appropriate
> >> downstream flow / process group. Whatever sends data to this port can
> use
> >> an UpdateAttribute to add some “tracking/routing” attribute on the
> >> flowfiles before being sent. Inserting Merge/Split will likely affect
> your
> >> timing due to waiting for bins to fill, depending on your volume. S2S is
> >> pretty good at transmitting data on-demand with low overhead on one
> port;
> >> it’s when you have many remote input ports that there is substantial
> >> overhead.
> >>
> >>
> >> Andy LoPresto
> >> alopre...@apache.org
> >> alopresto.apa...@gmail.com
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
> >> > On Feb 6, 2020, at 2:34 PM, Elli Schwarz  .INVALID>
> >> wrote:
> >> >
> >> > We ran that command - it appears the site-to-sites that are causing
> the
> >> issue. We had a lot of remote process groups that weren't even being
> used
> >> (no data was being sent to that part of the dataflow), yet when running
> the
> >> lsof command they each had a large number of open files - almost 2k! -
> >> showing CLOSE_WAIT. Again, there were no flowfiles being sent to them,
> so
> >> can it be some kind of bug that keeping a remote process group open is
> >> somehow opening files and not closing them? (BTW, the reason we had to
> >> upgrade from 1.9.2 to 1.11.0 was because we had upgraded our Java
> version
> >> and that cause an IllegalBlockingModeException - is it possible that
> >> whatever fixed that problem is now causing an issue with open files?)
> >> >
> >> > We now disabled all of the unused remote process groups. We still have
> >> several remote process groups that we are using so if this is the issue
> it
> >> might be difficult to avoid, but at least we decreased the number of
> remote
> >> process groups we have. Another approach we are trying is a merge
> content
> >> before we send to the Nifi having the most issues, to have fewer flow
> files
> >> sent at once site to site, and then splitting them after they are
> received.
> >> > Thank you!
> >> >
> >> >On Thursday, February 6, 2020, 2:19:48 PM EST, Mike Thomsen <
> >> mikerthom...@gmail.com> wrote:
> >> >
> >> > Can you share a description of your flows in terms of average flowfile
> >> size, queue size, data velocity, etc.?
> >> > Thanks,
> >> > Mike
> >> >
> >> > On Thu, Feb 6, 2020 at 1:59 PM Elli Schwarz <
> eliezer_schw...@yahoo.com.invalid>
> >> wrote:
> >> >
> >> >  We seem to be experiencing the same problems. We recently upgraded
> >> several of our Nifis from 1.9.2 to 1.11.0, and now many of them are
> failing
> >> with "too many open files". Nothing else changed other than the upgrade,
> >> and our data volume is the same as before. The only solution we've been
> >> able to come up with is to run a script to check for this condition and
> >> restart the Nifi. Any other ideas?
> >> > Thank you!
> >> >
> >> > On Sunday, February 2, 2020, 9:11:34 AM EST, Mike Thomsen <
> >> mikerthom...@gmail.com> wrote:
> >> >
> >> >  Without further details, this is what I did to see if it was
> something
> >> > other than the usual issue of having not enough file handlers
> available.
> >> > Something like a legitimate case of someone forgetting to close file
> >> > objects or something in the code itself.
> >> >
> >> > 1. Setup a 8core/32GB VM on AWS w/ Amazon AMI.
> >> > 2. Pushed 1.11.1RC1
> >> > 3. Pushed the RAM settings to 6/12GB
> >> > 4. Disabled flowfile archiving because I only allocated 8GB of
> storage.
> >> > 5. Setup a flow that used 2 generateflow instances to generate massive
> >> > amounts of garbage data using all available cores. (All queues were
> >> setup
> >> > to hold 250k flow files)
> >> > 6. Kicked it off and let it run for probably about 

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

[Ryan Hendrickson]

We'd like to connect a NiFi to a Registry, Pull down a Process Group, and
get any NARs required for that without having to manually install them on
the NiFi.

That's what I'm understanding the Extension part of the 0.4.0 release to
be, am I on point, or way off?

Thanks,
Ryan

On Wed, Mar 27, 2019 at 4:17 PM Bryan Rosander 
wrote:

> Hey Pierre,
>
> You can version the metadata db in the root dir of the git repo with event
> hooks.
>
> With an h2 jdbc url of
>
> 'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE'
>
> You can connect to the db in a separate process, we're using that with a
> shell event hook to dump the db to sql script when mutations happen (now
> tenant events too :) ), storing that in the git repo using
> org.h2.tools.Script, and committing it.  Then we can restore from that sql
> script on startup before calling the stock startup script.
>
> Side benefit is you can see metadata state changes in the git history and
> manually inspect the sql.
>
>
> On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard <
> pierre.villard...@gmail.com>
> wrote:
>
> > Bryan,
> >
> > Thanks for the details. On my side, the main feature I (and others I
> > suppose) am looking for is the ability to rebuild the metadata DB from
> the
> > Git repo. But to be honest, this is not a blocker at all: I can live
> with a
> > custom Docker image containing this feature.
> >
> > So, from my point of view, I don't have any issue with #2 but maybe #1
> > would also give access to experimental features and have interesting
> > feedback from the community. But we would, indeed, need to make it
> crystal
> > clear that the APIs could change in ulterior release(s).
> >
> > Pierre
> >
> >
> >
> > Le mer. 27 mars 2019 à 18:11, Bryan Bende  a écrit :
> >
> > > Ryan,
> > >
> > > The short answer to your question is no, there isn't actually anything
> > > new in the UI. The registry UI is setup in a generic way to list
> > > "versioned items", so versioned extension bundles will just
> > > automatically show up in the list with versioned flows.
> > >
> > > Let me put together a wiki page that outlines all of the moving parts
> > > for the extension registry work and where we are currently at. I'll
> > > send the link on this thread when I have something.
> > >
> > >
> > > For Pierre and others,
> > >
> > > I was thinking more about this discussion of releasing 0.4.0, and I
> > > think there are two of approaches we could take...
> > >
> > > 1) If there is a desire for the community to release 0.4.0 sooner
> > > rather than later, then we can mark all of the extension bundle
> > > related APIs as experimental, which would then give us the freedom to
> > > continue evolving them until we have the full extension registry
> > > experience, and likely mark them as stable in 0.5.0, or whichever
> > > future release we decide. The reason for this is that there will
> > > likely be API changes needed while building out the NiFi side of
> > > things, and I don't want us to get stuck in a spot where we can't
> > > change some API since it has been released, and so far the NiFi side
> > > of the work hasn't been started yet.
> > >
> > > 2) Wait until more of the extension registry work is finalized and use
> > > that as the driver of when to release 0.4.0, with the obvious downside
> > > that it may take a bit longer to get a release out which then holds up
> > > anything else that is not related to extension registry.
> > >
> > > So I guess we just have to decide the driving factor for releasing
> > > 0.4.0... If its the non-extension related features/fixes, then #1 is
> > > probably the best approach. If its the extension related stuff, then
> > > I'd vote for #2 since it isn't really ready yet.
> > >
> > > -Bryan
> > >
> > > On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
> > >  wrote:
> > > >
> > > > Bryan,
> > > >Is there a new GUI part of the registry for the extension registry
> > > work
> > > > in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything
> > > immediately.
> > > > You mentioned docs and sys admin stuff... Could you point me in the
> > > > quick-start path if I wanted to try to kick-the-tires a little on
> this?
> > > >
> > > 

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

[Ryan Hendrickson]

Bryan,
   Is there a new GUI part of the registry for the extension registry work
in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything immediately.
You mentioned docs and sys admin stuff... Could you point me in the
quick-start path if I wanted to try to kick-the-tires a little on this?

Thanks,
Ryan

On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende  wrote:

> Mike,
>
> All record-oriented components are extensions and thus can make use of
> versioned components (i.e. nothing related to records is baked into
> the core framework of NiFi).
>
> The record API is essentially the module named
> nifi-record-serialization-services-api which at runtime is included in
> nifi-standard-services-api-nar. Any record oriented processors you
> build are dependent on a specific version of
> nifi-standard-services-api-nar since that is where the Java interface
> will be that your processor was compiled against.
>
> Right now, even without extension registry, you could deploy multiple
> versions of nifi-standard-services-api-nar to a NiFi instance, and
> then deploy multiple versions of your record processing NAR that
> correspond to the versions of your API NAR.
>
> Going forward with extension registry, we probably want to consider
> breaking up nifi-standard-services-api-nar into smaller API NARs, as
> well as nifi-standard-bundle into smaller processor bundles. For
> example, there could be a record API NAR and a record processors NAR,
> that would both be split out of from the standard NARs.
>
> -Bryan
>
>
> On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen 
> wrote:
> >
> > Great news about the extension registry! As we get closer to that being
> > ready, I'd like to add in some discussion about versioning the record API
> > separately. A lot of the custom processors we build now are Record
> > API-based, and it would be great to be able to decouple them from one
> > specific release of NiFi.
> >
> > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende  wrote:
> >
> > > Hi Pierre,
> > >
> > > I think we are definitely close to an 0.4.0 release. A major chunk of
> > > extension registry work has already landed in master, and I still have
> one
> > > other jira that I almost have ready and wanted to include, NIFIREG-233
> for
> > > generating extensions docs. Plus we probably need to make a few
> > > additions/updates to the admin guide.
> > >
> > > -Bryan
> > >
> > > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
> > > pierre.villard...@gmail.com>
> > > wrote:
> > >
> > > > All,
> > > >
> > > > Some really nice features have been included since the NiFi Registry
> > > 0.3.0
> > > > release and I'm wondering if it'd be a good time to consider a 0.4.0
> > > > release.
> > > >
> > > > There is a JIRA tagged for 0.4.0 and there are 2 opened pull
> requests,
> > > plus
> > > > interesting discussions / JIRAS on the mailing lists.
> > > >
> > > > Are there any reasons to hold off on a 0.4.0 release?  Are there
> > > particular
> > > > JIRAs that the community considers necessary to have as part of the
> > > > release?
> > > >
> > > > If not, happy to give it a try at performing RM duties!
> > > >
> > > > Thanks,
> > > > Pierre
> > > >
> > > --
> > > Sent from Gmail Mobile
> > >
>

Re: [DISCUSS] Deprecate processors who have Record oriented counterpart?

[Ryan Hendrickson]

We often don't use the Record Processors because of the Schema requirement
and complexity to use the LookupRecord processor.

I'll refer to this email in the NiFi mailing list: "GetMongo - Pass-on
Initial FlowFile?"... There were suggestions to use the LookupRecord
processor, but ultimately it couldn't do what we needed to be done, so we
had to string together a set of other processors.

For us, it was easier to string together a set of processors than to figure
out why LookupRecord, MongoDBLookupService, and InferAvroSchema wasn't
getting the job done for us.
 /---success---> *ReplaceText* (Prepend JSON Key)
---success-->  \
/
\
*GetMongo*
  ---> *Merge Content* (Combine
on Correlation Attribute Name, Binary Concat)
\
/
 \---original---> *ReplaceText*  (Prepend JSON Key)
---success--> /


If they're marked as deprecated, I'd really like to see barrier to entry
with the LookupRecord processors decreased.  The number 1 thing I don't
like about the Record processors is that they require a Schema, and the
complimentary processor(s?), specifically the GetMongo one, does not
require a schema.

Ryan

On Sat, Feb 23, 2019 at 11:39 AM Andrew Grande  wrote:

> I'm not sure deprecating is warranted. In my experience, record based
> processors are very powerful, but have a steep learning curve the way they
> are in NiFi today, and, frankly, simple things should be dead simple.
>
> Now, moving the record UX towards an easy extreme affects this equation,
> but e.g. I never open up a conversation with a new user by talking about
> records, Schema Registry or NiFi Registry.
>
> Maybe there's something coming up which I'm not aware yet? Please share.
>
> Andrew
>
> On Sat, Feb 23, 2019, 7:43 AM Sivaprasanna 
> wrote:
>
> > Team,
> >
> > Ever since the Record based processors were first introduced, there has
> > been active development in improving the Record APIs and constant
> interest
> > in introducing new set of Record oriented processors. It has gone to a
> > level where almost all the processors that deal with mainstream tech
> have a
> > Record based counterpart, such as the processors for MongoDB, Kafka,
> RDBMS,
> > HBase, etc., These record based processors have overcome the limitations
> of
> > the standard processors letting us build flows which are concise and
> > efficient especially when we are dealing with structured data. And more
> > over with the recent release of NiFi (1.9), we now have a new feature
> that
> > offers schema inference capability which even simplifies the process of
> > building flows with such processors. Having said that, I'm wondering if
> > this is a right time to raise the talk of deprecating processors which
> the
> > community believes has a much better record oriented counterpart,
> covering
> > all the functionalities currently offered by the standard processor.
> >
> > There are a few things that has to be talked about, like how should the
> > deprecated processor be displayed in the UI, etc., but even before going
> > through that route, I want to understand the community's thoughts on
> this.
> >
> > Thanks,
> > Sivaprasanna
> >
>

Include Section for Processor Breaking Changes in Changelog

[Ryan Hendrickson]

Hi,
   I recently upgraded a NiFi from 1.4 to 1.8 and found a breaking change.
The ExecuteStreamCommand processor, in NiFi 1.5, introduced a new
relationship for errors (NIFI-4559
).  The consequence:  We
missed that 1 of our many processors moved into an invalid state.  A week
later, we notice that part of our flow wasn't processing data because it's
waiting for the new relationship to be drawn.

   It'd be great if in the Release Notes

changes
like these are highlighted under a specific section, such as "Breaking
changes."

Thanks
Best,
Ryan

Re: [discuss] should we do a nifi 1.7.1 release?

[Ryan Hendrickson]

Ahh gotcha, and good point on grabbing the master.  I may do that...

Thanks,
Ryan

On Mon, Jul 9, 2018 at 3:18 PM Andy LoPresto  wrote:

> Hi Ryan,
>
> That sounds like a separate discussion the community should weigh in on.
> Right now, the release management process is fairly extensive and takes a
> few days of manual work to perform. In addition, releases need to be voted
> on by the community in order to be released, so this is an effort for
> community members as well.
>
> I’m not opposed to improving our RM process to make this work easier, but
> it’s not as simple as just cutting a release more frequently at this point
> in time. If you so desire, you can always checkout the current master or
> specific feature branches. It’s possible we could do something like a
> nightly tag, but officially releasing that through the Apache process is
> probably not doable in the near-term.
>
> The semantic versioning is also an issue, because 1.6.x releases are
> supposed to be bug fixes only, not feature releases [1].
>
> For the public API the Apache NiFi project aims to follow versioning
> principles as described at Semantic Versioning 2.0.0 <http://semver.org/>
>
> Consider the following scenarios in the context of the most recent
> 'example' release being 0.0.1 and with the understanding that these are
> about the public API as defined above.
>
>- For releases which are comprised solely of bug fixes or non-feature
>introducing or enhancing changes that requires only a 'patch' version bump
>(the Z part in X.Y.Z).  So the next release then is 0.0.2.
>- For releases which include backward compatible changes to introduce
>feature enhancements or new features that requires a 'minor' version change
>and the 'patch' version resets to '0' (the Y part in X.Y.0).  So the next
>release then is 0.1.0. A 'minor' version change is also required for any
>change that could result in an existing flow becoming invalid, such as the
>addition of a required property with no default or the addition of a
>relationship, or the removal of a property or relationship. Note: it is
>*NOT* acceptable in a 'minor' version to change anything that can
>result in an existing flow behaving differently (other than a component
>becoming invalid). Doing so would fundamentally alter the way in which
>organizations process data without them realizing it.
>- For releases which include non-backward compatible changes or
>changes deemed so substantive by the community that it is considered a
>'major' version change and the minor and patch versions reset to '0' (the X
>part in X.0.0).  So the next release then is 1.0.0.
>
> After a release occurs the 'patch' version will be automatically adjusted
> by maven without the release manager doing anything special.  So rarely
> will this value need to be manually set.  In the event of a 'major' or
> 'minor' bump though the entire relevant source tree will need to be
> adjusted.
>
>
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jul 9, 2018, at 12:11 PM, Ryan Hendrickson <
> ryan.andrew.hendrick...@gmail.com> wrote:
>
> As a user of NiFi, and someone converting things to use more standard
> processors, vs writing custom ones, I'd prefer smaller releases, or
> possible a release that only updates Processors with the bug fixes and
> improvements that went into them vs having to diff the configuration files
> each upgrade.  The bigger releases, like 1.6.0 (163 updates), and 1.7.0
> (191 updates) are great, but the waiting game for fixes to go into a
> release seems long.  I'd love a weekly release, or even just know that once
> a month there will be a 1.6.x release coming out with updates to
> processors.
>
> That said, I can't wait for this fix, slated for 1.8.0:
> https://issues.apache.org/jira/browse/NIFI-5334  (GetMongo should pass
> along NiFi FlowFile Attributes), love to see it in a 1.7.1.
>
> Ryan
>
> On Fri, Jul 6, 2018 at 10:55 AM Mike Thomsen 
> wrote:
>
> Aldrin and I got some Docker improvements in lately that might be good to
> throw in as well. They can definitely wait until 1.8 if everyone wants to
> KISS this release, but they could also add some real value for the docker
> users too.
>
> On Fri, Jul 6, 2018 at 1:31 AM V, Prashanth (Nokia - IN/Bangalore) <
> prashant...@nokia.com> wrote:
>
> Thanks Andy..  +1 for 1.7.1 release.
>
> Thanks & Regards,
> Prashanth
>
> From: Andy LoPresto [mailto:alopre...@apache.org ]
> 

Re: [discuss] should we do a nifi 1.7.1 release?

[Ryan Hendrickson]

As a user of NiFi, and someone converting things to use more standard
processors, vs writing custom ones, I'd prefer smaller releases, or
possible a release that only updates Processors with the bug fixes and
improvements that went into them vs having to diff the configuration files
each upgrade.  The bigger releases, like 1.6.0 (163 updates), and 1.7.0
(191 updates) are great, but the waiting game for fixes to go into a
release seems long.  I'd love a weekly release, or even just know that once
a month there will be a 1.6.x release coming out with updates to processors.

That said, I can't wait for this fix, slated for 1.8.0:
https://issues.apache.org/jira/browse/NIFI-5334  (GetMongo should pass
along NiFi FlowFile Attributes), love to see it in a 1.7.1.

Ryan

On Fri, Jul 6, 2018 at 10:55 AM Mike Thomsen  wrote:

> Aldrin and I got some Docker improvements in lately that might be good to
> throw in as well. They can definitely wait until 1.8 if everyone wants to
> KISS this release, but they could also add some real value for the docker
> users too.
>
> On Fri, Jul 6, 2018 at 1:31 AM V, Prashanth (Nokia - IN/Bangalore) <
> prashant...@nokia.com> wrote:
>
> > Thanks Andy..  +1 for 1.7.1 release.
> >
> > Thanks & Regards,
> > Prashanth
> >
> > From: Andy LoPresto [mailto:alopre...@apache.org]
> > Sent: Friday, July 06, 2018 1:34 AM
> > To: dev@nifi.apache.org
> > Subject: Re: [discuss] should we do a nifi 1.7.1 release?
> >
> > I’m working on the wildcard cert issue and would be able to put that
> along
> > with some other minor fixes into a 1.7.1 release.
> >
> > Andy LoPresto
> > alopre...@apache.org
> > alopresto.apa...@gmail.com
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > On Jul 5, 2018, at 11:01 AM, Robert R. Bruno  > rbru...@gmail.com>> wrote:
> >
> > +1 as well.  Any chance of this one as well?
> >
> > https://issues.apache.org/jira/browse/NIFI-5316
> >
> > On Thu, Jul 5, 2018, 11:33 Mark Bean  > mark.o.b...@gmail.com>> wrote:
> >
> >
> > +1 for a 1.7.1 release if it contains a fix for NIFI-5368 [1]. This bug
> is
> > breaking multiple unit tests on custom processors.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-5368
> >
> >
> >
> > On Thu, Jul 5, 2018 at 12:23 PM Joe Witt  > joe.w...@gmail.com>> wrote:
> >
> >
> > team,
> >
> > Wanted to kick off a thread to suggest we do a nifi 1.7.1 release.  It
> > sounds like we might have an issue handling wildcard certs in 1.7.0
> > [1] and it was reported again in an email today i think.  Also, if
> > this one is deemed legit it seems worth sorting out [2].  I'd imagine
> > there are a few other bug fixes as well we can pull in.
> >
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-5370
> > [2] https://issues.apache.org/jira/browse/NIFI-5377
> >
> > Thanks
> > Joe
> >
> >
> >
>

Threading in a NiFi Processor

[Ryan Hendrickson]

I have always stayed away from doing any threading inside a NiFi processor.

However, I recently came across a use-case where I'm calling a web service
from within a custom Nifi Processor and I don't want to overwhelm the web
service.

I'd like to instrument a sleep in the onTrigger() or process() method of
about 1 second, or a configurable amount.

Is there a pattern established for Nifi Processors to accomplish this
nicely?

Thanks,
Ryan

Re: AWS PutS3Object Inconsistent Logs vs Default Protocol

[Ryan Hendrickson]

Cool - https://issues.apache.org/jira/browse/NIFI-797

On Wed, Jul 29, 2015 at 11:08 AM, Joe Witt joe.w...@gmail.com wrote:

 Certainly.  Please do file a JIRA for it of course and if you run into
 anything needing help just fire away.

 Thanks!

 Joe

 On Wed, Jul 29, 2015 at 11:06 AM, Ryan Hendrickson
 rhendrickson.w...@gmail.com wrote:
  Hi all,
 If agreed upon, I don't mind putting the fix in, but it won't be til
  Sunday or next week..
 
 The PutS3Object reports to provenance that it uses an HTTP url, when
 the
  default for the S3 is actually HTTPS.  This makes it pretty misleading as
  to what is happening.
 
  This is verified HTTPS via wireshark and a code review.
 
  -Ryan
 
  --  Notes -
 
 
 https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=blob;f=nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java;h=24c82dd036f68b6cc1ed1df39d9a89b4cf7c0555;hb=HEAD
 
  171 final String url = http://; + bucket + .
 s3.amazonaws.com/
  + key;\r
  172 final long millis =
  TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);\r
  173 session.getProvenanceReporter().send(flowFile, url,
  millis);\r
 
 
 
 https://github.com/aws/aws-sdk-java/blob/8397ef2731c9ea57de1a529a2e3991eebb96b257/aws-java-sdk-core/src/main/java/com/amazonaws/ClientConfiguration.java
 
  Lines 104-110:
   /**
  * The protocol to use when connecting to Amazon Web Services.
  * p
  * The default configuration is to use HTTPS for all requests for
 increased
  * security.
  */
  private Protocol protocol = Protocol.HTTPS;

AWS PutS3Object Inconsistent Logs vs Default Protocol

[Ryan Hendrickson]

Hi all,
   If agreed upon, I don't mind putting the fix in, but it won't be til
Sunday or next week..

   The PutS3Object reports to provenance that it uses an HTTP url, when the
default for the S3 is actually HTTPS.  This makes it pretty misleading as
to what is happening.

This is verified HTTPS via wireshark and a code review.

-Ryan

--  Notes -

https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=blob;f=nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java;h=24c82dd036f68b6cc1ed1df39d9a89b4cf7c0555;hb=HEAD

171 final String url = http://; + bucket + .s3.amazonaws.com/
+ key;\r
172 final long millis =
TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);\r
173 session.getProvenanceReporter().send(flowFile, url,
millis);\r


https://github.com/aws/aws-sdk-java/blob/8397ef2731c9ea57de1a529a2e3991eebb96b257/aws-java-sdk-core/src/main/java/com/amazonaws/ClientConfiguration.java

Lines 104-110:
 /**
* The protocol to use when connecting to Amazon Web Services.
* p
* The default configuration is to use HTTPS for all requests for increased
* security.
*/
private Protocol protocol = Protocol.HTTPS;