Re: [DISCUSS] Release Timeline for NiFi 2.0.0-M3

[Joe Witt]

Also agreed there.  Profile should be there to exclude perhaps but include
should be default.

On Wed, Apr 24, 2024 at 6:30 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Thanks for the replies thus far!
>
> With the goal of including the new UI, it seems like a good time to
> move the module from an optional profile to part of the standard
> build. That would make the release candidate build and review process
> simpler, not requiring the optional build profile.
>
> Regards,
> David Handermann
>
> On Tue, Apr 23, 2024 at 2:42 PM Matt Gilman 
> wrote:
> >
> > I agree. Including the updated UI and getting some feedback would be
> great.
> >
> > On Mon, Apr 22, 2024 at 8:50 PM Joe Witt  wrote:
> >
> > > Id be a big fan of including the new UI.
> > >
> > > On Mon, Apr 22, 2024 at 5:45 PM Pierre Villard <
> > > pierre.villard...@gmail.com>
> > > wrote:
> > >
> > > > Thanks David, I'm happy to take care of a 1.26 release at the same
> time.
> > > >
> > > > Regarding the M3 release, should the new UI be included and
> instructions
> > > > given on how to access it to start collecting feedback? I'm under the
> > > > impression that almost all of the work has been done, no?
> > > >
> > > > Thanks,
> > > > Pierre
> > > >
> > > > Le mar. 23 avr. 2024, 02:03, David Handermann <
> > > exceptionfact...@apache.org
> > > > >
> > > > a écrit :
> > > >
> > > > > Team,
> > > > >
> > > > > We are approaching 300 Jira issues resolved for version 2.0.0-M3
> [1],
> > > > > including a number of important dependency upgrades and Python
> > > > > framework improvements.
> > > > >
> > > > > There are several open pull requests that are getting close to
> > > > > completion, which should be possible to include in an upcoming
> release
> > > > > version. There is also a significant pull request [2] for
> NIFI-12998
> > > > > [3] that includes some important changes to module directory
> > > > > hierarchy, providing a clearer distinction between framework and
> > > > > extensions, and implementing significant cleanup of dependency
> > > > > declarations across the repository.
> > > > >
> > > > > With these changes in view, we should be ready for another
> milestone
> > > > > release soon after merging the above pull request.
> > > > >
> > > > > Another milestone release seems to be the best course of action for
> > > > > now, providing the opportunity for additional review and testing
> > > > > before a full 2.0.0 release version.
> > > > >
> > > > > I would be glad to handle Release Manager responsibilities for
> > > > > 2.0.0-M3 when things are ready.
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%202.0.0-M3
> > > > > [2] https://github.com/apache/nifi/pull/8677
> > > > > [3] https://issues.apache.org/jira/browse/NIFI-12998
> > > > >
> > > >
> > >
>

Re: Desire to view more than 100 items within a queue

[Joe Witt]

Edward

Moved those cc'd to bcc including yourself.  To really send/receive notes
you'll want to subscribe to the mailing list.

We have to effectively draw the line somewhere at how much data is
retrieved and the click to content mechanism supported isn't meant to be
exhaustive as-is.  We can/should add some pagination but when you consider
most queues are actually changing/evolving so fast it didn't make sense.
For the pattern you use which is effectively a dead-letter queue though
your ask makes perfect sense.

Thanks

On Tue, Apr 23, 2024 at 2:38 PM Edward Wang
 wrote:

> To whom it may concern,
>
> We are using NiFi to pass FlowFiles into ExecuteStreamCommand processors.
> Occasionally, running the scripts using that processor results in errors
> that route into the nonzero status relationship.
> We are pointing the nonzero status relationships to non-functioning
> processors, so that they will remain within the connection indefinitely and
> we can track them.
>
> However, whenever we list the contents of a connection, we can only see
> the details of 100 FlowFiles at a time, despite the connection holding more
> than 100.
> This is the case using the web interface and through calling the API.
>
> I was wondering if anyone had experience or any ideas regarding using list
> queue to show all items.
>
> Thank you.
>
> Sincerely,
> Edward
>

Re: [DISCUSS] Release Timeline for NiFi 2.0.0-M3

[Joe Witt]

Id be a big fan of including the new UI.

On Mon, Apr 22, 2024 at 5:45 PM Pierre Villard 
wrote:

> Thanks David, I'm happy to take care of a 1.26 release at the same time.
>
> Regarding the M3 release, should the new UI be included and instructions
> given on how to access it to start collecting feedback? I'm under the
> impression that almost all of the work has been done, no?
>
> Thanks,
> Pierre
>
> Le mar. 23 avr. 2024, 02:03, David Handermann  >
> a écrit :
>
> > Team,
> >
> > We are approaching 300 Jira issues resolved for version 2.0.0-M3 [1],
> > including a number of important dependency upgrades and Python
> > framework improvements.
> >
> > There are several open pull requests that are getting close to
> > completion, which should be possible to include in an upcoming release
> > version. There is also a significant pull request [2] for NIFI-12998
> > [3] that includes some important changes to module directory
> > hierarchy, providing a clearer distinction between framework and
> > extensions, and implementing significant cleanup of dependency
> > declarations across the repository.
> >
> > With these changes in view, we should be ready for another milestone
> > release soon after merging the above pull request.
> >
> > Another milestone release seems to be the best course of action for
> > now, providing the opportunity for additional review and testing
> > before a full 2.0.0 release version.
> >
> > I would be glad to handle Release Manager responsibilities for
> > 2.0.0-M3 when things are ready.
> >
> > Regards,
> > David Handermann
> >
> > [1]
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%202.0.0-M3
> > [2] https://github.com/apache/nifi/pull/8677
> > [3] https://issues.apache.org/jira/browse/NIFI-12998
> >
>

Re: Support Nifi Install on Debian Linux

[Joe Witt]

Hello

NiFi 2.x line requires a minimum of Java 21.

Thanks

On Thu, Apr 18, 2024 at 10:42 PM Fábio Augusto Pássari 
wrote:

> Dear support,
>
> I am installing nifi-2.0.0-M2 on linux ubuntu but it does not initialize
> the Nifi service.
>
> *ls /usr/lib/jvm*
> default-java   java-11-openjdk-amd64 java-8-openjdk-amd64
> java-1.11.0-openjdk-amd64  java-17-openjdk-amd64 openjdk-11
> java-1.17.0-openjdk-amd64  java-1.8.0-openjdk-amd64
> java-1.18.0-openjdk-amd64  java-18-openjdk-amd64
>
> *echo $JAVA_HOME*
> /usr/lib/jvm/java-11-openjdk-amd64
>
> *bin/nifi.sh start*
>
> Java home: /usr/lib/jvm/java-11-openjdk-amd64
> NiFi home: /opt/nifi-2.0.0-M2
>
> Bootstrap Config File: /opt/nifi-2.0.0-M2/conf/bootstrap.conf
>
> Erro: ocorreu LinkageError ao carregar a classe principal
> org.apache.nifi.bootstrap.RunNiFi
> java.lang.UnsupportedClassVersionError:
> org/apache/nifi/bootstrap/RunNiFi has been compiled by a more recent
> version of the Java Runtime (class file version 65.0), this version of the
> Java Runtime only recognizes class file versions up to 55.0
>
> Can you help me?
> --
> Fábio Augusto Pássari
>

Re: Is there a need for the ConvertAvroToJson processor in 2.x?

[Joe Witt]

The best list is here

https://cwiki.apache.org/confluence/display/NIFI/Deprecated+Components+and+Features

But that is a great point.  We generally want to drop any ConvertXtoY bits
we have that aren't well covered by Records.

Thanks

On Wed, Apr 10, 2024 at 1:07 PM Dan S  wrote:

> I guess I am wondering why it was not included under NIFI-11172
> .
>
> On Wed, Apr 10, 2024 at 12:01 PM Mike Thomsen 
> wrote:
>
> > IIRC, the plan was to remove those ConvertToX processors in 2.X
> >
> > On Wed, Apr 10, 2024 at 11:36 AM Dan S  wrote:
> >
> > > I noticed that the ConvertAvroToJson processor is not marked as
> > deprecated
> > > in 2.x. Is there something it offers which cannot be done with the
> > > ConvertRecord processor configured with an AvroReader and
> > > JsonRecordSetWriter?
> > >
> >
>

Apr 24 - Apache NiFi board report

[Joe Witt]

NiFi Community

Thanks as always for the continued progress!



## Description:
The mission of NiFi is the creation and maintenance of software related to
providing an easy to use, powerful, and reliable system to process and
distribute data.

Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
integrate with and leverage the command and control of NiFi. There are both
Java and C++ implementations.

Apache NiFi Registry is a centralized registry for key configuration items
including flow versions, assets, and extensions for Apache NiFi and Apache
MiNiFi.

Apache NiFi Nar Maven Plugin is a release artifact used for supporting the
NiFi classloader isolation model.

Apache NiFi Flow Design System is a theme-able set of high quality UI
components and utilities for use across the various Apache NiFi web
applications in order to provide a more consistent user experience.

## Project Status:
Current project status: Ongoing. High.
Issues for the board: None.

## Membership Data:
Apache NiFi was founded 2015-07-14 (8 years ago)
There are currently 66 committers and 36 PMC members in this project.
The Committer-to-PMC ratio is roughly 2:1.

Community changes, past quarter:
- No new PMC members. Last addition was Csaba Bejan on 2023-10-25.
- Daniel Stieglitz was added as committer on 2024-02-09.
- Mark Bathori was added as committer on 2024-02-09

## Project Activity:
We released Apache  NiFi 2.0.0 milestone 2 January 29th with more than
200 completed JIRAs with new features.  The release has seen quick adoption
and found a variety of bugs and improvements.  To that end the community
is approaching a release of NiFi 2.0.0-M3 with already more than 235
resolved
JIRAs.  We anticipate a release for that soon as there are major
improvements
to the stability of the much anticipated first class integration with Python
components which has been highly valuable to NiFi's emerging role in data
pipelines to feed data to Generative AI applications.  We also released
Apache NiFi 1.25.0 with more than 100 JIRAs at the end of January and have
more than 100 already resolved on the upcoming 1.26.0 line.

## Community Health:
Consistent with past reports community health remains strong and growing.
Mailing list activity remains consistent and we saw a 30% increase on the
dev list in total emails.  The slack community continues to grow with
general hosting 3,133 users at the time of this report. A highly
desirable pattern of engagement is present where it isn't just committers
and PMC engaging to answer questions.  A new venture backed startup in
the US has formed as well which has led to an increase in traffic and
discussion regarding Apache NiFi.

Re: is there a trick to running Nifi on AWS

[Joe Witt]

Mark

I believe you will need to tell NiFi you want it to listen on more than the
localhost/loopback address.

nifi.web.https.host=localhost

Is a default in nifi.properties for instance.

Def take a look through the admin/install guide as well.

Thanks

On Thu, Mar 28, 2024 at 1:32 PM Mark Woodcock 
wrote:

> Howdy,
>
> Cranked up an EC2 instance.
> Installed Java 11.
> set up JAVA_HOME
> Downloaded Nifi 1.25.0
> unzipped Nifi
> set a nifi.sensitive.properties.key
> (https.port is default 8443)
>
> bin/nifi.sh start
>
> But, I can't even seem to access the most basic bit of the UI:
>
> curl -vvvk https://54.91.56.55:8443
> *   Trying 54.91.56.55:8443...
> * connect to 54.91.56.55 port 8443 failed: Connection refused
> * Failed to connect to 54.91.56.55 port 8443 after 17 ms: Connection
> refused
> * Closing connection 0
> curl: (7) Failed to connect to 54.91.56.55 port 8443 after 17 ms:
> Connection refused
>
> I have no doubt, I'm doing something astonishingly dumb.  Would someone be
> kind enough to point it out?
>
> thx,
>
> mew
>

Re: [discuss] What to do with the Cassandra components

[Joe Witt]

>> Regards,
> > > > >>> > > > > >>>>>> David Handermann
> > > > >>> > > > > >>>>>>
> > > > >>> > > > > >>>>>> [1]
> > https://github.com/apache/cassandra-java-driver/
> > > > >>> > > > > >>>>>>
> > > > >>> > > > > >>>>>> On Tue, Mar 19, 2024 at 12:37 PM Mike Thomsen <
> > > > >>> > > > > >> mikerthom...@gmail.com
> > > > >>> > > > > >>>>
> > > > >>> > > > > >>>>>> wrote:
> > > > >>> > > > > >>>>>>>
> > > > >>> > > > > >>>>>>> Matt,
> > > > >>> > > > > >>>>>>>
> > > > >>> > > > > >>>>>>> I got that. My point was that the Java changes
> > > appear to
> > > > >>> be a
> > > > >>> > > one
> > > > >>> > > > > >>> time
> > > > >>> > > > > >>>>>>> thing that DataStax did to make a better driver
> > with
> > > a
> > > > >>> much
> > > > >>> > > more
> > > > >>> > > > > >>>>>>> future-proof API. Since Scylla tracks them as
> > > closely as
> > > > >>> > > > > >> possible, I
> > > > >>> > > > > >>>>>>> suspect that we don't need to plan for a bunch of
> > > > >>> abstraction
> > > > >>> > > to
> > > > >>> > > > > >>> isolate
> > > > >>> > > > > >>>>>>> Java changes.
> > > > >>> > > > > >>>>>>>
> > > > >>> > > > > >>>>>>> On Tue, Mar 19, 2024 at 11:07 AM Steven Matison <
> > > > >>> > > > > >>>>>> steven.mati...@gmail.com>
> > > > >>> > > > > >>>>>>> wrote:
> > > > >>> > > > > >>>>>>>
> > > > >>> > > > > >>>>>>>> That was kinda where i got stuck and fell out on
> > my
> > > > >>> > > branch/jira.
> > > > >>> > > > > >>>>>> Mike and
> > > > >>> > > > > >>>>>>>> I wanted to make a new controller service ,
> > without
> > > > >>> backward
> > > > >>> > > > > >>>>>> compatibility;
> > > > >>> > > > > >>>>>>>> and remove the duplicate driver/connection
> > > properties
> > > > >>> found
> > > > >>> > in
> > > > >>> > > > > >>> some
> > > > >>> > > > > >>>>>> of the
> > > > >>> > > > > >>>>>>>> processors.
> > > > >>> > > > > >>>>>>>>
> > > > >>> > > > > >>>>>>>> I agree taking out all old stuff and making new
> > > > >>> controller
> > > > >>> > > > > >> service
> > > > >>> > > > > >>>>>> makes
> > > > >>> > > > > >>>>>>>> most sense.  4.x and 5.x should be mostly
> > backwards
> > > > >>> > compatible
> > > > >>> > > > > >> to
> > > > >>> > > > > >>>>>> 2&3.x
> > > > >>> > > > > >>>>>>>> with how it’s used within current processors.
> > > > >>> > > > > >>>>>>>>
> > > > >>> > > > > >>>>>>>>
> > > > >>> > > > > >>>>>>>>
> > > > >>> > > > > >>>>>>

Re: Nifi PutSFTP Processor Dot-Rename Error

[Joe Witt]

Hello

The image didnt come through

Please share nifi version and all settings for the component.

Thanks

On Wed, Mar 20, 2024 at 6:54 PM Prasad Patwardhan
 wrote:

> Dear Nifi Developer Support Team,
>
>
> I am writing to request assistance regarding an error encountered in the
> PutSFTP processor of our Apache NiFi data flow. Below are the details of
> the error:
>
>
> Error:
> ERROR [Timer-Driven Process Thread-223]
> o.a.nifi.processors.standard.PutSFTP
> PutSFTP[id=0a8613a0-018e-1000-ec12-8485106ac6fe] Unable to transfer
> StandardFlowFileRecord[uuid=f07ce855-1a60-45bb-845b-c56f77b8965d,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1710445580505-1, container=default,
> section=1], offset=98313,
> length=990],offset=0,name=20240209-203419_II0224SE45_894298_822585_313.tar.gz,size=990]
> to remote host nfs1
>
> due to org.apache.nifi.processor.exception.ProcessException: IOException
> thrown
>
> from PutSFTP[id=0a8613a0-018e-1000-ec12-8485106ac6fe]:
> java.io.IOException: Failed to rename dot-file to
> /remote-path/20240209-203419_II0224SE45_894298_822585_313.tar.gz due to 4:
> Failure: java.io.IOException: Failed to rename dot-file to
> /remote-path/20240209-203419_II0224SE45_894298_822585_313.tar.gz due to 4:
> Failure; routing to failure
>
> java.io.IOException: Failed to rename dot-file to
> /remote-path/20240209-203419_II0224SE45_894298_822585_313.tar.gz due to 4:
> Failure
>
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.put(SFTPTransfer.java:787)
>
> at
> org.apache.nifi.processors.standard.PutFileTransfer$1.process(PutFileTransfer.java:134)
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2692)
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2660)
>
> at
> org.apache.nifi.processors.standard.PutFileTransfer.onTrigger(PutFileTransfer.java:126)
>
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1360)
>
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
>
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
>
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:829)
>
> Caused by: net.schmizz.sshj.sftp.SFTPException: Failure
>
> at net.schmizz.sshj.sftp.Response.error(Response.java:140)
>
> at net.schmizz.sshj.sftp.Response.ensureStatusIs(Response.java:133)
>
> at
> net.schmizz.sshj.sftp.Response.ensureStatusPacketIsOK(Response.java:125)
>
> at net.schmizz.sshj.sftp.SFTPEngine.rename(SFTPEngine.java:250)
>
> at net.schmizz.sshj.sftp.SFTPClient.rename(SFTPClient.java:124)
>
> at net.schmizz.sshj.sftp.SFTPClient.rename(SFTPClient.java:119)
>
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.put(SFTPTransfer.java:783)
>
> ... 15 common frames omitted
>
>
> This error occurs when attempting to transfer a file to the remote host
> server that we have on running on Centos7 server called nfs1, specifically
> during the renaming process with the .LCK extension.
>
> In the Nifi PutSFTP processor we have the following settings marked as the
> following to rename the files to .LCK files prior to the completion of the
> transfer which will be renamed on transfer completion.
>
>
> The file does successfully transfer to folder that we have created on our
> Centos7 server however the error shown above pops up in the Nifi UI and the
> Nifi logs.
>
>
> [image: Screenshot 2024-03-20 at 1.36.19 PM.png]
>
> In order to ensure that it was not a permissions/certificate issue, I have
> used the cert to sftp commands in my terminal to the host machine and
> create, delete, and rename files that are inside the target directory.
>
>
> Could you please investigate this issue and provide guidance on how to
> resolve it? If additional information or logs are required, please let me
> know, and I will be happy to provide them.
>
> Thank you for your assistance!
>

Re: [discuss] What to do with the Cassandra components

[Joe Witt]

Matt

I dont know a ton about Cassandra but when I looked at client/driver notes
for 4+ it said it was compatible all the way back to 3.x.   Not sure what
that means but it surely seems worth exploring.  Also I dont know if the
4.x drivers get rid of the vulnerable bits.

Thanks

On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess  wrote:

> At the very least we should upgrade to Cassandra 3.11.6:
> https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt
>
> On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess  wrote:
>
> > If the community agrees to get rid of Cassandra 3 that'll save me effort
> > on the refactor after I add Cassandra 4 :) Otherwise those
> > vulnerabilities would only be in a "new" Cassandra 3 services NAR that
> > would not be included in the convenience binary.
> >
> > On Fri, Mar 15, 2024 at 1:28 PM Joe Witt  wrote:
> >
> >> Mike, Matt,
> >>
> >> Happy to hear you both have active efforts or are interested in doing
> so.
> >> Can you help me understand more specifically what that means for the
> >> current set of components?
> >>
> >> The CVE hits are concerning and long standing.  Supporting Cassandra 3
> >> implies the current set of dependencies would remain too right?
> >>
> >> Is the current set of components we have ones we want to retain?  We
> >> certainly need Cassandra components - but are the ones we have now the
> >> right ones?
> >>
> >> Thanks
> >> Joe
> >>
> >> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess 
> >> wrote:
> >>
> >> > I'm actively working this, I pushed my branch up in case anyone wants
> to
> >> > take a look [1]. The idea is to abstract the Cassandra API "up a
> couple
> >> > levels" and provide implementations for Cassandra 3, 4, and eventually
> >> 5.
> >> > For JDBC-like interfaces this is a PITA because of the API (Statement,
> >> > PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping we
> >> can
> >> > find a common pattern for abstracting the third-party library
> >> > implementation and API from the NiFi component (Processor,
> >> > ControllerService, etc.) API. I think we're doing something similar
> for
> >> > Kafka?
> >> >
> >> > Regards,
> >> > Matt
> >> >
> >> > [1] https://github.com/mattyb149/nifi/tree/cassy4
> >> >
> >> >
> >> > On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen 
> >> > wrote:
> >> >
> >> > > That’s been on my todo list for a little while but things kept
> coming
> >> up.
> >> > > I think I could get started on that now. Based on my initial
> research
> >> it
> >> > > appears that scylla uses the exact same api as datastax so
> supporting
> >> > both
> >> > > in a cql bundle should theoretically be fairly easy.
> >> > >
> >> > >
> >> > > Sent from my iPhone
> >> > >
> >> > > > On Mar 14, 2024, at 6:18 PM, Joe Witt  wrote:
> >> > > >
> >> > > > Team,
> >> > > >
> >> > > > Cassandra remains a really important system to be able to send
> data
> >> to.
> >> > > > However, it seems like we've not maintained these well.  We have
> >> what
> >> > > > appears to be at least a full generation behind on client versions
> >> (we
> >> > > are
> >> > > > on 3x vs 4x which is the latest stable with 5x apparently coming
> >> > > shortly).
> >> > > >
> >> > > > We have components to send data, query data, and use Cassandra as
> a
> >> > cache
> >> > > > store.  We have older mechanisms for json/avro and publish
> >> mechanisms
> >> > for
> >> > > > records.
> >> > > >
> >> > > > The libraries we do have depend on outdated versions of Guava and
> >> > result
> >> > > in
> >> > > > many CVE hits.
> >> > > >
> >> > > > I am inclined to think we should deprecate the 1.x components and
> >> > remove
> >> > > > them as-is from the 2.x line.  Then re-introduce them with record
> >> only
> >> > > > interfaces and built against the latest stable
> >> > > Cassandra/Datastax/ScyllaDB
> >> > > > interfaces.
> >> > > >
> >> > > > I'd love to hear thoughts from those closer to this space both as
> a
> >> > user
> >> > > > and developer so we can make good next steps.
> >> > > >
> >> > > > Thanks
> >> > >
> >> >
> >>
> >
>

Re: [discuss] What to do with the Cassandra components

[Joe Witt]

Mike, Matt,

Happy to hear you both have active efforts or are interested in doing so.
Can you help me understand more specifically what that means for the
current set of components?

The CVE hits are concerning and long standing.  Supporting Cassandra 3
implies the current set of dependencies would remain too right?

Is the current set of components we have ones we want to retain?  We
certainly need Cassandra components - but are the ones we have now the
right ones?

Thanks
Joe

On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess  wrote:

> I'm actively working this, I pushed my branch up in case anyone wants to
> take a look [1]. The idea is to abstract the Cassandra API "up a couple
> levels" and provide implementations for Cassandra 3, 4, and eventually 5.
> For JDBC-like interfaces this is a PITA because of the API (Statement,
> PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping we can
> find a common pattern for abstracting the third-party library
> implementation and API from the NiFi component (Processor,
> ControllerService, etc.) API. I think we're doing something similar for
> Kafka?
>
> Regards,
> Matt
>
> [1] https://github.com/mattyb149/nifi/tree/cassy4
>
>
> On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen 
> wrote:
>
> > That’s been on my todo list for a little while but things kept coming up.
> > I think I could get started on that now. Based on my initial research it
> > appears that scylla uses the exact same api as datastax so supporting
> both
> > in a cql bundle should theoretically be fairly easy.
> >
> >
> > Sent from my iPhone
> >
> > > On Mar 14, 2024, at 6:18 PM, Joe Witt  wrote:
> > >
> > > Team,
> > >
> > > Cassandra remains a really important system to be able to send data to.
> > > However, it seems like we've not maintained these well.  We have what
> > > appears to be at least a full generation behind on client versions (we
> > are
> > > on 3x vs 4x which is the latest stable with 5x apparently coming
> > shortly).
> > >
> > > We have components to send data, query data, and use Cassandra as a
> cache
> > > store.  We have older mechanisms for json/avro and publish mechanisms
> for
> > > records.
> > >
> > > The libraries we do have depend on outdated versions of Guava and
> result
> > in
> > > many CVE hits.
> > >
> > > I am inclined to think we should deprecate the 1.x components and
> remove
> > > them as-is from the 2.x line.  Then re-introduce them with record only
> > > interfaces and built against the latest stable
> > Cassandra/Datastax/ScyllaDB
> > > interfaces.
> > >
> > > I'd love to hear thoughts from those closer to this space both as a
> user
> > > and developer so we can make good next steps.
> > >
> > > Thanks
> >
>

[discuss] What to do with the Cassandra components

[Joe Witt]

Team,

Cassandra remains a really important system to be able to send data to.
However, it seems like we've not maintained these well.  We have what
appears to be at least a full generation behind on client versions (we are
on 3x vs 4x which is the latest stable with 5x apparently coming shortly).

We have components to send data, query data, and use Cassandra as a cache
store.  We have older mechanisms for json/avro and publish mechanisms for
records.

The libraries we do have depend on outdated versions of Guava and result in
many CVE hits.

I am inclined to think we should deprecate the 1.x components and remove
them as-is from the 2.x line.  Then re-introduce them with record only
interfaces and built against the latest stable Cassandra/Datastax/ScyllaDB
interfaces.

I'd love to hear thoughts from those closer to this space both as a user
and developer so we can make good next steps.

Thanks

Re: website: PMC members also listed as committers?

[Joe Witt]

For anyone interested in improving the logic here is where the page lives:

https://github.com/apache/nifi-site/blob/main/content/community/_index.md

It utilizes this directive/entry such as
{{< project-members project="nifi" >}}

That is powered by a short-code in hugo terms (I'm told) which can be found
here
https://github.com/apache/nifi-site/blob/main/themes/nifi/layouts/shortcodes/project-members.html

That short-code leverages information which is found in
https://github.com/apache/nifi-site/blob/main/config.toml
on lines 31 and 32.

Perhaps the solution is to improve the logic of the short-code such that it
allows a second argument such as 'exclude-project' which might look like
{{< project-members project="nifi" exclude-project="nifi-pmc">}}

Then update the short code such that if the second argument is provided we
remove names in the 'nifi' list that are also in the 'nifi-pmc' list.

Not sure what mechanisms/etc.. are available but this should give anyone
motivated good context to work with.

Thanks

On Wed, Feb 21, 2024 at 1:18 PM Joe Witt  wrote:

> Marton
>
> I wasn't sure why so I checked the source [1] and found a very nice
> improvement which is the page is now dynamically generated reflecting our
> actual membership/roster in ASF records.  So people don't need to mess with
> this anymore which is great.
>
> It is duplicative between PMC and Committership listing and ideally we
> only show the delta since being a PMC member implies committership.  I'm
> sure someone could improve that logic to only list the committers.
>
> [1]
> https://github.com/apache/nifi-site/blob/main/content/community/_index.md
>
> On Wed, Feb 21, 2024 at 10:17 AM Marton Szasz  wrote:
>
>> Hi all,
>>
>> I just noticed that on the people page of the nifi website, PMC members
>> are also listed under committers. This seems to be a recent change,
>> maybe since the redesign?
>>
>> Old website without name duplication:
>>
>> https://web.archive.org/web/20231208050808/https://nifi.apache.org/people.html
>> New website with name duplication: https://nifi.apache.org/community/
>>
>> Is this change intentional? If so, why? I find it a bit confusing.
>>
>> Thanks,
>> Marton
>>
>>

Re: website: PMC members also listed as committers?

[Joe Witt]

Marton

I wasn't sure why so I checked the source [1] and found a very nice
improvement which is the page is now dynamically generated reflecting our
actual membership/roster in ASF records.  So people don't need to mess with
this anymore which is great.

It is duplicative between PMC and Committership listing and ideally we only
show the delta since being a PMC member implies committership.  I'm sure
someone could improve that logic to only list the committers.

[1]
https://github.com/apache/nifi-site/blob/main/content/community/_index.md

On Wed, Feb 21, 2024 at 10:17 AM Marton Szasz  wrote:

> Hi all,
>
> I just noticed that on the people page of the nifi website, PMC members
> are also listed under committers. This seems to be a recent change,
> maybe since the redesign?
>
> Old website without name duplication:
>
> https://web.archive.org/web/20231208050808/https://nifi.apache.org/people.html
> New website with name duplication: https://nifi.apache.org/community/
>
> Is this change intentional? If so, why? I find it a bit confusing.
>
> Thanks,
> Marton
>
>

Re: [DISCUSS] New Project Repository for Python Extensions?

[Joe Witt]

David

Good and important thread for NiFi.  Some quick thoughts on this more
generally.  Perhaps we ...

0. Have a repo for the 'nifi-api'.  Any true extension should be against a
specific version of the nifi api only.  This should change far less
frequently than the other bits.  Probably should review that this is true
from history.  We probably also have to include things that are
'effectively' part of the true API we need to honor which includes
expression language, records, and hopefully relatively few other things.
1. Have the mono repo like we do now for the nifi framework, nifi
application, minifi, registry and such.  And notably the nifi-assembly
remains here.
2. Have a repo for 'nifi-java-extensions' and we move everything there
which is meant as an extension component against the nifi-api specifically
(not the framework things like provenance/etc. - frankly anything that is
related to the 'nifi-framework-api' is NOT a true extension and doesn't
belong in this 'nifi-java-extension' repo)
3. Have a repo for 'nifi-python-extensions'

We can release 0 whenever we need to improve the nifi-api and then 1-3 at
the same time like we do today.  Or we can when needed release them
independently like fixing specific bugs or vulnerabilities/etc..

Worth really digging in and understanding what this could/should look like
to give ourselves cleaner air going forward.

Thanks
Joe

On Thu, Feb 1, 2024 at 1:03 PM Gábor Gyimesi  wrote:

> Hi David,
>
> MiNiFi C++ had had a Python API before, using Python's stable C API, but
> the processors had a different, simpler format like this following example:
>
> https://github.com/apache/nifi-minifi-cpp/blob/main/extensions/python/pythonprocessors/examples/GaussianDistributionWithNumpy.py
>
> Our goal was to be able to support NiFi's Python API, with the ease of only
> copying the Python processor file to MiNiFi C++'s configured python
> processor path, and use them in the flow config the same way as the
> original MiNiFi C++ style python processors.
>
> There is already an open PR for supporting this for NiFi's
> FlowFileTransform processor types (as of now MiNiFi C++ does not support
> record based flow file processing):
> https://github.com/apache/nifi-minifi-cpp/pull/1712 and also an open PR
> for
> supporting virtual environments:
> https://github.com/apache/nifi-minifi-cpp/pull/1721 as previously MiNiFi
> C++ only supported system installed Python packages. The implementation
> uses the same C API bindings as before, importing NiFi's nifiapi adapted to
> MiNiFi C++'s python API to be able to use NiFi's Python processors.
>
> There are still a few limitations due to the differences between NiFi and
> MiNiFi C++ implementations which are listed here:
>
> https://github.com/apache/nifi-minifi-cpp/blob/d27430260c8c35dac52011bdb31b22b36e10539d/extensions/python/PYTHON.md
> Some of these limitations are being addressed by these jira tickets in this
> epic: https://issues.apache.org/jira/browse/MINIFICPP-2272
>
> I tested all the available Python processors (aside from RecordTransform
> processors) of NiFi 2.0.0-M2 and they seem to be working with MiNiFi C++
> with these PRs, so it looks promising.
>
> Regards,
> Gabor Gyimesi
>
>
>
> On Thu, 1 Feb 2024 at 19:03, David Handermann  >
> wrote:
>
> > Hi Gabor,
> >
> > Thanks for the reply.
> >
> > It is helpful to know about the progress of Python Processor support
> > in MiNiFi C++. Is the goal to support the same NiFi Python API as
> > implemented for NiFi itself?
> >
> > The goal of a separate repository for Python extensions would be to
> > keep it self-contained for testing and releasing. From that
> > perspective, it would have a dependency on a declared version of the
> > NiFi Python API, and would include automated build workflows for
> > testing.
> >
> > For the NiFi framework components, there would still need to be
> > internal components that support testing implementations of Python
> > APIs, but the Python Extensions repository would have its own
> > decoupled set of tests.
> >
> > Regards,
> > David Handermann
> >
> > On Thu, Feb 1, 2024 at 11:05 AM Gábor Gyimesi 
> wrote:
> > >
> > > Hi David,
> > >
> > > Currently we are in the process of implementing support for the NiFi
> > python
> > > processors in MiNiFi C++. Probably in the next open source release this
> > > feature will be available, so the available NiFi Python processors will
> > be
> > > usable in MiNiFi C++ as well. I think this idea would help with the
> > > collaboration of supporting these processors in both Java and C++
> > projects.
> > > It would certainly make release verification easier to be able to
> > > concentrate only on the Python processors if they are released
> > separately.
> > >
> > > My concern is how would the automatic testing and verification would
> work
> > > in this scenario? Would all the testing of the Python processors be
> moved
> > > to the new repository and would be tested there separately, with both
> > NiFi
> > > 

Re: UpdateAttribute Failure Relationship

[Joe Witt]

Lucas

The change proposed is specific to the original intent, implementation,
evolution, and bridging to new intent for certain cases of that specific
processor.

I remain supportive of processors using the capabiltities of the api and
framework to best meet the user needs.  “failure” is just a string. The
meaning of that depends on a processor and what it does.

Thanks

On Fri, Feb 9, 2024 at 5:04 PM Lucas Ottersbach 
wrote:

> I think that's a good approach which actually addresses the underlying
> issue. Thank you Joe, Mark and all others.
>
> As far as I know the default last resort behaviour of rollback + yield,
> that a lot processors exhibit, is due to them being based on
> AbstractProcessor.
>
> Does it make sense to incorporate the outlined approach into
> AbstractProcessor instead of UpdateAttribute?
> This way, other processors can undergo the same (opt-in) behavioural change
> without having to re-implement it on a per processor basis.
> Every processor in question only would need to add the property declared by
> AbstractProcessor to its list of properties. Processors that do not include
> the property wouldn't support to configure the behaviour and thus default
> to the current behaviour instead.
>
> I think that would provide a good middle ground, both for users that want a
> explicit failure relationship and for those that rather want a simpler flow
> and ensure those kinds of errors won't happen another way.
>
> What do you think?
>
> Lucas
>
> Joe Witt  schrieb am Sa., 10. Feb. 2024, 00:06:
>
> > Lots of good commentary and great focus on minimizing impact to the users
> > while fixing what is admittedly not the most desired behavior for some
> > cases as it relates to the very popular UpdateAttribute.
> >
> > We do not want to enforce that all processors have failure relationships.
> > Presumably that notion is specific to processors which take an input
> > flowfile.  Even then though we have many examples where having a failure
> > relationship does not add value.  A few examples such as DistributeLoad,
> > DuplicateFlowFile and others which have concepts such as 'unmatched'
> etc..
> > Certainly most things can and should but we dont need a strict policy
> > here.  We should still let people building processors be thoughtful about
> > what is best.
> >
> > UpdateAttribute specifically... There is history to why it works the way
> it
> > does.  Things changed and it didn't evolve or couldn't because its use
> was
> > so widespread and we didnt want to create too much pain for users.  But
> > because of 2.0 and some improvements like the ability to code up
> migration
> > behaviors on a per extension basis we can work our way out of this
> without
> > causing pain for the users.
> >
> > For NiFi 1.x it should stay as is.
> > For NiFi 2.x a solution is outlined in NIFI-6344.  It reads:
> > Given the new capabilities for migrating configs in NiFi 2.0 we can fix
> > this.
> >
> > Add a property to UpdateAttribute that is 'Failure Strategy' and the
> > options are 'rollback' or 'route to failure'. If that property is set
> with
> > rollback it behaves like it does now and I recommend that remain the
> > default. If that property is set to 'route to failure' then we add a
> > relationship which needs to be set which is of course called 'failure'.
> For
> > flows being migrated from a version before this behavior was available
> to a
> > version that has this capability we just set the value of this parameter
> to
> > our default.
> >
> > This lets existing flows migrate over just fine. It lets us give users a
> > failure path for the cases they want one. It lets us keep the vast
> majority
> > of flows and uses of this where failure is not relevant stay clean. And
> it
> > handles migration.
> >
> > The processor needs to be updated to catch the exceptions and then follow
> > this logic. Today it just lets it fly to the framework which causes the
> > processor to yield and penalizes the flowfile for the default time. When
> > now catching the problem we should just avoid yielding and instead
> penalize
> > the specific offending flowfile which lets everything else operate super
> > fast.
> >
> > Thanks to Mark Payne for the chat on this.
> >
> > This can be done at any time by anyone that wants to take it on.  It is
> not
> > a blocker for nifi 2.x.  The migration capabilities give us really nice
> > options for many cases we've hit over the years going forward.
> >
> > Thanks
> >
> >
> > On Fri, Feb 9, 2024 at 2:53 PM Adam Taft  wrote:
> >

Re: UpdateAttribute Failure Relationship

[Joe Witt]

Lots of good commentary and great focus on minimizing impact to the users
while fixing what is admittedly not the most desired behavior for some
cases as it relates to the very popular UpdateAttribute.

We do not want to enforce that all processors have failure relationships.
Presumably that notion is specific to processors which take an input
flowfile.  Even then though we have many examples where having a failure
relationship does not add value.  A few examples such as DistributeLoad,
DuplicateFlowFile and others which have concepts such as 'unmatched' etc..
Certainly most things can and should but we dont need a strict policy
here.  We should still let people building processors be thoughtful about
what is best.

UpdateAttribute specifically... There is history to why it works the way it
does.  Things changed and it didn't evolve or couldn't because its use was
so widespread and we didnt want to create too much pain for users.  But
because of 2.0 and some improvements like the ability to code up migration
behaviors on a per extension basis we can work our way out of this without
causing pain for the users.

For NiFi 1.x it should stay as is.
For NiFi 2.x a solution is outlined in NIFI-6344.  It reads:
Given the new capabilities for migrating configs in NiFi 2.0 we can fix
this.

Add a property to UpdateAttribute that is 'Failure Strategy' and the
options are 'rollback' or 'route to failure'. If that property is set with
rollback it behaves like it does now and I recommend that remain the
default. If that property is set to 'route to failure' then we add a
relationship which needs to be set which is of course called 'failure'. For
flows being migrated from a version before this behavior was available to a
version that has this capability we just set the value of this parameter to
our default.

This lets existing flows migrate over just fine. It lets us give users a
failure path for the cases they want one. It lets us keep the vast majority
of flows and uses of this where failure is not relevant stay clean. And it
handles migration.

The processor needs to be updated to catch the exceptions and then follow
this logic. Today it just lets it fly to the framework which causes the
processor to yield and penalizes the flowfile for the default time. When
now catching the problem we should just avoid yielding and instead penalize
the specific offending flowfile which lets everything else operate super
fast.

Thanks to Mark Payne for the chat on this.

This can be done at any time by anyone that wants to take it on.  It is not
a blocker for nifi 2.x.  The migration capabilities give us really nice
options for many cases we've hit over the years going forward.

Thanks


On Fri, Feb 9, 2024 at 2:53 PM Adam Taft  wrote:

> I mean, this really speaks to the principal that (in my humble opinion) All
> Processors Shalt Have Failure Relationships as best practice.
>
> So I think the decision is really, how much pain do we want to impose on
> NiFi 2.0 adopters. UpdateAttribute and RouteOnAttribute are used literally
> everywhere (especially on large installations), and it would be quite the
> chore(!) to update these if we were to make a non-compatible change for
> 2.0.  But 2.0 is also a very logical time/place to make such backwards
> breaking changes to UpdateAttribute, RouteOnAttribute and/or other
> processors that are missing failure relationships.
>
> I just fundamentally have never liked the lack of control for flowfiles
> getting requeued if a processor exception is left uncaught. You are almost
> always just going to repeat the failure condition, it's rarely useful to
> retry without some sort of deliberate flow manager action. The enhancements
> to relationships (to enable retries, backoffs, etc.) even reinforces my
> point of view that requeuing is almost always wrong.
>
> So I think the decision here is how much pain is this going to cause for
> NiFi 2.0 adoption? And if it's too much pain, then we should do something
> in a backwards compatible kind of way (like offering a dynamic failure
> relationship for those target processors). And if it's not too much pain,
> just add failure relationships directly to the processors and rip the
> bandaid off.
>
> A solution like a "try/catch" option to the expression language is not that
> exciting to me. I do believe that adoption will be hindered if we make a
> breaking change because of the high usage of Update and Route processors.
> And so 'm concluding that a dynamic relationship approach, leaving the EL
> as designed, is the best path forward. But I'm hoping there's even a more
> clever solution out there waiting for someone to bring forward.
>
> I do believe whatever we do should be a 2.0 thing. Let's not consider any
> changes on this for 1.x (unless it somehow enables better adoption).
>
> /Adam
>
> On Fri, Feb 9, 2024 at 1:27 PM u...@moosheimer.com 
> wrote:
>
> > Yes, that certainly involves a lot of effort.
> > I wonder whether it's a good idea to fix a 

Re: NiFi 1.25.0 - Bootstrap Notification Services are deprecated - no replacement?

[Joe Witt]

Martin

Correct. They are deprecated in the 1.x line (still usable) and are removed
in the 2.x line.  With more and more deployments happening in things like
K8S environments there are far better ways of offering these which reduce
the burden on NiFi itself and the dependency management it represents.
Similarly in traditional linux style service installs there are other
options that monitor services better that can be used as well.  We're
trying to strike a better balance than we have in the past of
building/providing all the things ourselves vs helping/integrating with
common/popular services as we go forward.

Thanks

On Thu, Feb 1, 2024 at 11:05 AM Martin Fong  wrote:

> We get the following WARN from the deprecated log and soon will be gone.
> So this feature will be gone and no replacement?
>
> 2024-02-01 11:37:44,048 WARN [main]
> deprecation.org.apache.nifi.bootstrap.RunNiFi Bootstrap Notification
> Services are deprecated [email-notification]
> org.apache.nifi.deprecation.log.DeprecationException: Reference Class
> [org.apache.nifi.bootstrap.RunNiFi] ClassLoader
> [jdk.internal.loader.ClassLoaders$AppClassLoader@55054057]
> at
> org.apache.nifi.deprecation.log.StandardDeprecationLogger.getExtendedArguments(StandardDeprecationLogger.java:63)
> at
> org.apache.nifi.deprecation.log.StandardDeprecationLogger.warn(StandardDeprecationLogger.java:54)
> at
> org.apache.nifi.bootstrap.RunNiFi.registerNotificationServices(RunNiFi.java:422)
> at org.apache.nifi.bootstrap.RunNiFi.loadServices(RunNiFi.java:410)
> at org.apache.nifi.bootstrap.RunNiFi.(RunNiFi.java:177)
> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:294)
>
> We have followed this in the past and still using it:
> https://pierrevillard.com/2017/05/11/monitoring-nifi-bootstrap-notifier/
>
> Please advise,
> Martin Fong
> Enterprise Technical Support Specialist, Infrastructure & Platform (IAG)
> Technology Services Division, Technology Infrastructure Services
> City of Toronto
> 703 Don Mills Road, 2nd Floor
> Toronto, ON
> M3C 3N3
> Tel:   416-397-7565
> e-mail: martin.f...@toronto.ca
>
> This e-mail message is confidential and subject to copyright. Any
> unauthorized use or disclosure is prohibited. If you have received this
> email and are not the intended recipient, please advise and delete it.
> Thank you.
>
>

Re: [VOTE] Release Apache NiFi 2.0.0-M2 (RC4)

[Joe Witt]

+1 binding

Thanks again David

On Fri, Jan 26, 2024 at 1:46 PM Bryan Bende  wrote:

> +1 binding
>
> Ran through release helper
> Ran simple flow
> Saved flows to registry, verified DB persistence is fixed
>
> Thanks David!
>
> On Fri, Jan 26, 2024 at 1:33 PM Robert Fellows 
> wrote:
> >
> > +1 non-binding
> >
> > verified hashes
> > build with java 21
> > ran simple flow
> > built with new UI
> >
> >
> > On Fri, Jan 26, 2024 at 12:11 PM Nissim Shiman  >
> > wrote:
> >
> > >  +1 (non-binding)
> > >
> > > verified source release sha256/512 checksums
> > >
> > > successfully ran build using:
> > > Apache Maven 3.9.6
> > > Java 21 2023-09-19 LTS
> > > linux kernel 3.10.0-1160
> > >
> > > Ran various simple flows successfully.  Migrated registry from 1.24 and
> > > 2.0.0-M1 to this RC successfully.  (i.e. and tested importing PGs,
> > > committing new versions of PGs to registry successfully, both from
> nifi and
> > > filesystem).  Verified flows were persisted in registry after registry
> > > restart as well.
> > >
> > > Noticed non-blocking issue where reporting tasks' link to referenced
> > > controller service is not completely working.  Created NIFI-12678 for
> it.
> > >
> > > Tested:
> > > NIFI-11389 Controller Services's link to referencing Controller
> Services
> > > components not always working
> > > NIFI-12387 Flow Configuration History can record a Comments change for
> > > Controller Service when no changes have been made
> > >
> > > NIFI-12394 when importing versioned flow with component that migrates
> > > properties, controller service reference is invalid
> > > NIFI-12666 Correct NiFi Registry Data Source Configuration
> > >
> > > Thank you David and team for the upcoming release!
> > >
> > > Nissim Shiman
> > >
> > > On Friday, January 26, 2024 at 02:30:45 AM UTC, David Handermann <
> > > exceptionfact...@apache.org> wrote:
> > >
> > >  Team,
> > >
> > > I am pleased to be calling this vote for the source release of Apache
> > > NiFi 2.0.0-M2.
> > >
> > > Please review the following guide for how to verify a release candidate
> > > build:
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> > >
> > > The source being voted on and the convenience binaries are available
> > > on the Apache Distribution Repository:
> > >
> > > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M2
> > >
> > > The build artifacts are available on the Apache Nexus Repository:
> > >
> > > https://repository.apache.org/content/repositories/orgapachenifi-1264
> > >
> > > Git Tag: nifi-2.0.0-M2-RC4
> > > Git Commit ID: 640b7bdfbbb8842f057a9bf49dc2b9b5d092abda
> > > GitHub Commit Link:
> > >
> > >
> https://github.com/apache/nifi/commit/640b7bdfbbb8842f057a9bf49dc2b9b5d092abda
> > >
> > > Checksums of nifi-2.0.0-M2-source-release.zip
> > >
> > > SHA256:
> 1946eceb3ae4f541545e93ddc6dd2cbe2a3302a6a46e6c584f3ffc1c1bd1e18b
> > > SHA512:
> > >
> e8e67e1e67359553479c0721a1c49ae6706cc6882eadf92e1f5ccc2619637ab87119a06221980d4c34d99b7b6d9a2138c77440b407074090e727b5d4447ab799
> > >
> > > Release artifacts are signed with the following key:
> > >
> > > https://people.apache.org/keys/committer/exceptionfactory.asc
> > >
> > > KEYS file is available on the Apache Distribution Repository:
> > >
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > Issues resolved for this version: 214
> > >
> > >
> > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353861
> > >
> > > Release note highlights can be found on the project wiki:
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M2
> > >
> > > The vote will be open for 72 hours.
> > >
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build from source, and test.
> > > Then please vote:
> > >
> > > [] +1 Release this package as nifi-2.0.0-M2
> > > [] +0 no opinion
> > > [] -1 Do not release this package because...
> > >
> >
> >
> >
> > --
> > ---
> > Rob Fellows
>

Re: [VOTE] Release Apache NiFi 1.25.0 (RC1)

[Joe Witt]

+1 binding.

Full clean build with contrib check and the usual checks works out.  Ran
simple local flow and checks out as well.  Checked latest commit.

Thanks Pierre!

On Thu, Jan 25, 2024 at 11:53 AM Pierre Villard 
wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.25.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.25.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1263
>
> Git Tag: nifi-1.25.0-RC1
> Git Commit ID: 6ecc398d3f92425447e43242af4992757e25b3c5
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/6ecc398d3f92425447e43242af4992757e25b3c5
>
> Checksums of nifi-1.25.0-source-release.zip
>
> SHA256: 6a14b35bf5beb4ae3fcf76df8d09676e61c6bc309a97dc6785eae84b7cbaef78
> SHA512:
>
> 1ffb2586ce9591ce2c5aa39fdec43a89ffd29081a31d51dc95dd953cb7f94584d0a0171bb1ba7c9495f431aec3770d324dbabb319b01bb6fdce5a0a00426fffa
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/pvillard.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 103
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353860
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.25.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [] +1 Release this package as nifi-1.25.0
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 2.0.0-M2 (RC3)

[Joe Witt]

+1 binding

All the usual checks.  Build/Integration Tests/Contrib check all clean. Ran
local flow on osx.  Built/deployed in docker and was able to
reattach/continue from a flow four months old like nothing happened.  Very
slick.

Also built with the new UI included and that is coming along super nicely!

Thanks everyone and as always David for RC efforts!

On Thu, Jan 25, 2024 at 10:10 AM Gábor Gyimesi  wrote:

> +1 non-binding
>
> - Verified hashes and signatures
> - Successfully built NiFi with contrib-check in the following environment:
>   - Ubuntu 22.04 6.5.0-14-generic
>   - java 21 2023-09-19 LTS
> Java(TM) SE Runtime Environment (build 21+35-LTS-2513)
> Java HotSpot(TM) 64-Bit Server VM (build 21+35-LTS-2513, mixed mode,
> sharing)
>   - Apache Maven 3.6.3
> - Designed and ran a flow with S3 upload successfully
> - Built a containerized version of NiFi and tested site to site use cases
> and http communication between NiFi and MiNiFi C++
>
> Thanks for RMing David!
>
> Regards,
> Gabor
>
>
>
> On Thu, 25 Jan 2024 at 15:46, Robert Fellows 
> wrote:
>
> > +1 non-binding
> >
> > *verified hashes*
> >
> > *built with:*
> > openjdk version "21.0.1" 2023-10-17 LTS
> > OpenJDK Runtime Environment Zulu21.30+15-CA (build 21.0.1+12-LTS)
> > OpenJDK 64-Bit Server VM Zulu21.30+15-CA (build 21.0.1+12-LTS, mixed
> mode,
> > sharing)
> >
> > General usage of the application looks good. Added a couple of the new
> > processors and services without issue.
> >
> >
> > On Thu, Jan 25, 2024 at 8:11 AM David Handermann <
> > exceptionfact...@apache.org> wrote:
> >
> > > Team,
> > >
> > > I am pleased to be calling this vote for the source release of Apache
> > > NiFi 2.0.0-M2.
> > >
> > > Please review the following guide for how to verify a release candidate
> > > build:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> > >
> > > The source being voted on and the convenience binaries are available
> > > on the Apache Distribution Repository:
> > >
> > > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M2
> > >
> > > The build artifacts are available on the Apache Nexus Repository:
> > >
> > > https://repository.apache.org/content/repositories/orgapachenifi-1262
> > >
> > > Git Tag: nifi-2.0.0-M2-RC3
> > > Git Commit ID: 439ac5f596fe90d2591376caec1499ed86abfd6b
> > > GitHub Commit Link:
> > >
> > >
> >
> https://github.com/apache/nifi/commit/439ac5f596fe90d2591376caec1499ed86abfd6b
> > >
> > > Checksums of nifi-2.0.0-M2-source-release.zip
> > >
> > > SHA256:
> 0637647ba7b6b8f0743cf00d95dd3e99250f2737ae9583daee511477cad22847
> > > SHA512:
> > >
> >
> 56d6c196ec64884c56a65278f7dd32cc62f1595ff5b2ec37403dcd54b7757a9ac70aa75a50ee6f61237e37b486d32b1ebbf95a4e38a2ea5dc7f694c779d5c6c4
> > >
> > > Release artifacts are signed with the following key:
> > >
> > > https://people.apache.org/keys/committer/exceptionfactory.asc
> > >
> > > KEYS file is available on the Apache Distribution Repository:
> > >
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > Issues resolved for this version: 207
> > >
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353861
> > >
> > > Release note highlights can be found on the project wiki:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M2
> > >
> > > The vote will be open for 72 hours.
> > >
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build from source, and test.
> > > Then please vote:
> > >
> > > [] +1 Release this package as nifi-2.0.0-M2
> > > [] +0 no opinion
> > > [] -1 Do not release this package because...
> > >
> >
> >
> > --
> > ---
> > Rob Fellows
> >
>

Re: [CANCEL][VOTE] Release Apache NiFi 2.0.0-M2 (RC1)

[Joe Witt]

; could
> > > >> not be resolved: org.apache.nifi:nifi-py4j-bridge:jar:2.0.0-M2
> > (absent),
> > > >> org.apache.nifi:nifi-standard-shared-nar:nar:2.0.0-M2 (absent):
> Could
> > not
> > > >> find artifact org.apache.nifi:nifi-py4j-bridge:jar:2.0.0-M2 in
> > central (
> > > >> https://repo.maven.apache.org/maven2) -> [Help 1]
> > > >> [ERROR] Failed to execute goal on project nifi-opentelemetry-nar:
> > Could
> > > >> not resolve dependencies for project
> > > >> org.apache.nifi:nifi-opentelemetry-nar:nar:2.0.0-M2: The following
> > > >> artifacts could not be resolved:
> > > >> org.apache.nifi:nifi-opentelemetry-processors:jar:2.0.0-M2 (absent),
> > > >> org.apache.nifi:nifi-standard-shared-nar:nar:2.0.0-M2 (absent):
> Could
> > not
> > > >> find artifact
> > org.apache.nifi:nifi-opentelemetry-processors:jar:2.0.0-M2 in
> > > >> central (https://repo.maven.apache.org/maven2) -> [Help 1]
> > > >> [ERROR] Failed to execute goal on project nifi-jolt-nar: Could not
> > > >> resolve dependencies for project
> > > >> org.apache.nifi:nifi-jolt-nar:nar:2.0.0-M2: The following artifacts
> > could
> > > >> not be resolved:
> org.apache.nifi:nifi-standard-shared-nar:nar:2.0.0-M2
> > > >> (absent), org.apache.nifi:nifi-jolt-processors:jar:2.0.0-M2
> (absent):
> > > >> org.apache.nifi:nifi-standard-shared-nar:nar:2.0.0-M2 was not found
> in
> > > >> https://repo.maven.apache.org/maven2 during a previous attempt.
> This
> > > >> failure was cached in the local repository and resolution is not
> > > >> reattempted until the update interval of central has elapsed or
> > updates are
> > > >> forced -> [Help 1]
> > > >
> > > >
> > > > without an option to resume. Please advise. Thanks!
> > > >
> > > >
> > > >
> > > > On Mon, Jan 22, 2024 at 10:25 AM David Handermann <
> > > > exceptionfact...@apache.org> wrote:
> > > >
> > > >> RC1 for 2.0.0-M2 is cancelled.
> > > >>
> > > >> On Mon, Jan 22, 2024 at 9:23 AM David Handermann
> > > >>  wrote:
> > > >> >
> > > >> > Joe,
> > > >> >
> > > >> > Thanks for highlighting the build issue related to the optional
> > > >> > profile for the new UI. In the interest of maintaining consistency
> > in
> > > >> > the source release, I agree this issue is worth resolving.
> > > >> >
> > > >> > A new version of the json-path library was also released over the
> > > >> > weekend, resolving CVE-2023-51074. Although this does not appear
> to
> > > >> > have a direct impact on NiFi usage, it would be helpful to include
> > as
> > > >> > well.
> > > >> >
> > > >> > I am cancelling RC1 and I will follow up with an RC2 build after
> > > >> > getting the json-path 2.9.0 upgrade reviewed and merged.
> > > >> >
> > > >> > Regards,
> > > >> > David Handermann
> > > >> >
> > > >> > On Mon, Jan 22, 2024 at 9:15 AM Joe Witt 
> > wrote:
> > > >> > >
> > > >> > > -1
> > > >> > >
> > > >> > > David,
> > > >> > >
> > > >> > > First of all thanks for continuing to plow through the magic of
> > > >> release
> > > >> > > management.
> > > >> > >
> > > >> > > I noticed the static check failed in github and it was due to a
> > > >> snapshot
> > > >> > > reference. I then tried a local build of the RC and it failed
> for
> > me
> > > >> for
> > > >> > > the same reason.  The difference here is that I was wanting to
> > > >> include the
> > > >> > > new UI so it could also be evaluated in the RC.
> > > >> > >
> > > >> > > I think this is worth resolving and doing and RC2 for but if you
> > > >> disagree
> > > >> > > that is fair as well.
> > > >> > >
> > > >> > > Thanks
> > > >> > >
> > > >> > > On Sat, Jan 20, 2024 at 9:3

Re: [VOTE] Release Apache NiFi 2.0.0-M2 (RC1)

[Joe Witt]

-1

David,

First of all thanks for continuing to plow through the magic of release
management.

I noticed the static check failed in github and it was due to a snapshot
reference. I then tried a local build of the RC and it failed for me for
the same reason.  The difference here is that I was wanting to include the
new UI so it could also be evaluated in the RC.

I think this is worth resolving and doing and RC2 for but if you disagree
that is fair as well.

Thanks

On Sat, Jan 20, 2024 at 9:36 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0-M2.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on and the convenience binaries are available
> on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M2
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1254
>
> Git Tag: nifi-2.0.0-M2-RC1
> Git Commit ID: b462c7051d004be70fba34f2795bd5c682cd1124
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/b462c7051d004be70fba34f2795bd5c682cd1124
>
> Checksums of nifi-2.0.0-M2-source-release.zip
>
> SHA256: c42655212abacef6f901f2a43ad7f43d3172c631652c517d822a36321199d411
> SHA512:
> b1a4d6790f85719a46f321f64cfc436da459795f03315a2f578940e266aa98a3d4a6d50d5be83583e7ac7adbf4a10f6050be314acc62a27fd247d082100718f4
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 197
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353861
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M2
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-2.0.0-M2
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [DISCUSS] Preparing for NiFi 2.0.0-M2

[Joe Witt]

Wonderful progress. Definitely time and an M2 seems like a good idea for
the reasons noted.

Happy to help take RM if you get squeezed on time.

Thanks

On Mon, Jan 15, 2024 at 12:46 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> We have had some great feedback and many improvements since the
> release of NiFi 2.0.0-M1 at the end of November. With over 160 [1]
> Jira issues already resolved for the next iteration, we are in a good
> position to prepare for another milestone release version.
>
> The main branch now includes significant framework dependency upgrades
> such as Spring 6 and Jetty 12, along with several new components and
> other bug fixes. Considering the significant number of changes to
> framework libraries already in place, it would be useful to release a
> second milestone version before a full general availability release.
>
> There is a pull request for NIFI-9458 [2] that includes impactful
> lower-level changes to date and time parsing and formatting, which is
> part of the NiFi 2.0 Release Goals [3]. Following the review and
> incorporation of these changes, I would be glad to handle Release
> Manager responsibilities for a NiFi 2.0.0-M2 version, ideally at some
> point this week.
>
> Regards,
> David Handermann
>
> [1] https://issues.apache.org/jira/projects/NIFI/versions/12353861
> [2] https://github.com/apache/nifi/pull/8248
> [3]
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Release+Goals
>

board report for Jan 2024

[Joe Witt]

Team,

Congrats again on all the great work.  Thanks to everyone involved.  Here
is our board report for Jan 2024.

## Description:
The mission of NiFi is the creation and maintenance of software related to
providing an easy to use, powerful, and reliable system to process and
distribute data.

Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
integrate with and leverage the command and control of NiFi. There are both
Java and C++ implementations.

Apache NiFi Registry is a centralized registry for key configuration items
including flow versions, assets, and extensions for Apache NiFi and Apache
MiNiFi.

Apache NiFi Nar Maven Plugin is a release artifact used for supporting the
NiFi classloader isolation model.

Apache NiFi Flow Design System is a theme-able set of high quality UI
components and utilities for use across the various Apache NiFi web
applications in order to provide a more consistent user experience.

## Project Status:
Current project status: Ongoing. High.
Issues for the board: None.

## Membership Data:
Apache NiFi was founded 2015-07-14 (8 years ago)
There are currently 64 committers and 36 PMC members in this project.
The Committer-to-PMC ratio is roughly 8:5.

Community changes, past quarter:
- Csaba Bejan was added to the PMC on 2023-10-25
- No new committers.  Last addition was Timea Barna on 2023-08-02.

## Project Activity:
The community conducted its first major (semantic versioning) milestone 1
release of Apache NiFi 2.0.0 M1 in November of 2023.  The first new major
release since 2016!  This release is a total game changer for NiFi.  In
addition to the more than 900 JIRAs included which comprise of features, bug
fixes, and security related changes this release makes major steps forward.
The NiFi 2.0 means we are up to date with and based on Java 21 and the
latest
of many great projects like Apache Maven, Spring, Jetty, and many more.
Users
can now write legit NiFi components in Python and enjoy a first class
integrated experience in NiFi as if they were written in Java. We eliminated
many defunct code areas, improved and clarified APIs, eliminated poorly
maintained code, and much more.  We anticipate having a full blown 2.0
production release soon though we already know of several production users
of
the 2.0 M1 release now.

We also kept the momentum up with the NiFi 1.x line releasing, also in
November, Apache NiFi 1.24.0.  Again this comes with improvements, security
changes, and bug fixes.

We have also at long last finally launched a revamped website for
nifi.apache.org.

## Community Health:
Community health remains strong and growing.  Our mailing list activity
as measured by the dev list dropped by 17% but overall activity remained
busy.  The dev list drop likely relates to the Holidays and the activity
on Slack.  PR activity, reviews, and merges remain highly active with more
than 31 authors contributing code in this quarter.  We added another 150
or so net Slack users this quarter now sitting at 2980 in the general
channel. We continue to ensure discussions that should turn into JIRAs or
other documented mailing lists do get sent to those mechanisms. We continue
to see plenty of commercial activity around NiFi in the form of vendor
material, blogs, social media posts, etc..

Re: New Project Website Design Staged for Deployment

[Joe Witt]

Super appreciative David and James.  A good step forward to refresh and
retool and grow from.

Thanks!

On Wed, Jan 3, 2024 at 8:39 PM David Handermann 
wrote:

> Team,
>
> Thanks to background work from several designers, and significant
> recent effort from James Mingardi-Elliott [1], we have a refreshed
> project website design deployed to staging and ready for production.
>
> The combined set of changes can be reviewed in the following pull request:
>
> https://github.com/apache/nifi-site/pull/82
>
> The staging site provides a functional way to view the new design,
> based on the main-staging branch [2] of the nifi-site repository.
>
> https://nifi.staged.apache.org
>
> The new design includes a modernized home page and streamlined
> navigation, with a focus on making the most popular pages quickly
> accessible.
>
> There is room for improvement in areas like generated project
> documentation, but the primary goal of this new design is to provide a
> fresh foundation for future improvements.
>
> The pull request provides an opportunity for correcting any notable
> problems, with the goal of pushing the new version to production early
> next week.
>
> Regards,
> David Handermann
>
> [1] https://github.com/james-elliott
> [2] https://github.com/apache/nifi-site/tree/main-staging
>

Re: Error in the expression-language-guide page.

[Joe Witt]

Looks like Alex is planning to address it
https://issues.apache.org/jira/browse/NIFI-12535

Thanks


On Thu, Dec 21, 2023 at 8:40 AM Dan S  wrote:

> Can you please add a little more detail which examples you are referring to
> and what the errors are?
>
> On Wed, Dec 20, 2023 at 11:32 PM A S  wrote:
>
> > Hi,
> > There are errors on the page in Table 13  14. Also, the tables are put
> > with same name 'PadLeft'.
> >
> > Thanks,
> > Ankura
> >
>

Re: NiFi SSO Login Using OIDC

[Joe Witt]

Hello Lourdes

It is extremely hard to find someone to respond meaningfully to questions
related to a NiFi release that is so many years back.  There have been
thousands of JIRAs to fix/improve/change the way such components behave.
If you can please get updated to something more recent I think you'll find
getting help from the community easier.

Thanks

On Mon, Nov 6, 2023 at 6:58 AM Lourdes Ursela Carmen Kuskanto
 wrote:

> Dear NiFi Developer Team,
>
> I want to enable SSO login in NiFi version 1.4 in my Compute Engine in GCP
> using OIDC. I want to allowed all user from my Azure Active Directory group
> that i have made before to be able to login to NiFi using their account as
> registered. But i can't seem to make it work.
>
> I have followed this step :
> https://github.com/benkelly/NiFi-Authentication-with-Azure-Active-Directory-Setup-Guide
>
> This is my configuration:
>
> nifi.properties
>
> # Core Properties #
> nifi.flow.configuration.file=./conf/flow.xml.gz
> nifi.flow.configuration.archive.enabled=true
> nifi.flow.configuration.archive.dir=./conf/archive/
> nifi.flow.configuration.archive.max.time=30 days
> nifi.flow.configuration.archive.max.storage=500 MB
> nifi.flow.configuration.archive.max.count=
> nifi.flowcontroller.autoResumeState=true
> nifi.flowcontroller.graceful.shutdown.period=10 sec
> nifi.flowservice.writedelay.interval=500 ms
> nifi.administrative.yield.duration=30 sec
> # If a component has no work to do (is "bored"), how long should we wait
> before checking again for work?
> nifi.bored.yield.duration=10 millis
> nifi.queue.backpressure.count=1
> nifi.queue.backpressure.size=1 GB
>
> nifi.authorizer.configuration.file=./conf/authorizers.xml
>
> nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
> nifi.templates.directory=./conf/templates
> nifi.ui.banner.text=
> nifi.ui.autorefresh.interval=30 sec
> nifi.nar.library.directory=./lib
> nifi.nar.library.autoload.directory=./extensions
> nifi.nar.working.directory=./work/nar/
> nifi.documentation.working.directory=./work/docs/components
>
> 
> # State Management #
> 
> nifi.state.management.configuration.file=./conf/state-management.xml
> # The ID of the local state provider
> nifi.state.management.provider.local=local-provider
> # The ID of the cluster-wide state provider. This will be ignored if NiFi
> is not clustered but must be populat>
> nifi.state.management.provider.cluster=zk-provider
> # Specifies whether or not this instance of NiFi should run an embedded
> ZooKeeper server
> nifi.state.management.embedded.zookeeper.start=false
> # Properties file that provides the ZooKeeper properties to use if
> 
>
> nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
>
>
> # H2 Settings
> nifi.database.directory=./database_repository
> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>
> # FlowFile Repository
>
> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
>
> nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
> nifi.flowfile.repository.directory=./flowfile_repository
> nifi.flowfile.repository.checkpoint.interval=20 secs
> nifi.flowfile.repository.always.sync=false
> nifi.flowfile.repository.encryption.key.provider.implementation=
> nifi.flowfile.repository.encryption.key.provider.location=
> nifi.flowfile.repository.encryption.key.provider.password=
> nifi.flowfile.repository.encryption.key.id=
> nifi.flowfile.repository.encryption.key=
> nifi.flowfile.repository.retain.orphaned.flowfiles=true
>
>
> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
> nifi.queue.swap.threshold=2
>
> # Content Repository
>
> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
> nifi.content.claim.max.appendable.size=1 MB
> nifi.content.repository.directory.default=./content_repository
> nifi.content.repository.archive.max.retention.period=7 days
> nifi.content.repository.archive.max.usage.percentage=50%
> nifi.content.repository.archive.enabled=true
> nifi.content.repository.always.sync=false
> nifi.content.viewer.url=../nifi-content-viewer/
> nifi.content.repository.encryption.key.provider.implementation=
> nifi.content.repository.encryption.key.provider.location=
> nifi.content.repository.encryption.key.provider.password=
> nifi.content.repository.encryption.key.id=
> nifi.content.repository.encryption.key=
>
> # Provenance Repository Properties
>
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
> nifi.provenance.repository.encryption.key.provider.implementation=
> nifi.provenance.repository.encryption.key.provider.location=
> nifi.provenance.repository.encryption.key.provider.password=
> nifi.provenance.repository.encryption.key.id=
> nifi.provenance.repository.encryption.key=
>
> # 

Re: Docker container for 2.0.0-M1

[Joe Witt]

Hello

Yeah pls do submit a PR.

Thanks

On Wed, Nov 29, 2023 at 9:34 PM Matthew Hawkins  wrote:

> Hi Joe,
>
> I have it running against Bellsoft's v21 alpine-musl base fine, with python
> enabled. Happy to submit the Dockerfile if you want. In a nutshell,
>
> apk add --no-cache --no-interactive --no-progress coreutils unzip curl jq
> xmlstarlet procps-ng python3 py3-pip py3-virtualenv ripgrep
>
> And other than obviously Alpine not Debian, Busybox wants adduser/addgroup
> not the util-linux variants. I moved the apk call to the scripts section so
> only one apk update is needed, and deleted the apt cache cleanup. I added
> the ripgrep package to grok logs.
>
> I initially missed the VOLUME line in the dockerfile. Unfortunately I can't
> seem to get the -v flag to work to use local folders instead of anonymous
> volumes. I appreciate it's probably done for performance but it would be
> nice if they had sensible names. Host is Win11 with docker desktop and
> wsl2. For brevity, I'm adding a -v
> c:\users\me\nifi\conf:/opt/nifi/nifi-current/conf format line for each of
> the exported volumes to the docker run invocation (wrote a small cmd to
> launch as I'm not typing that monstrosity out daily by hand)
> I know I can manually docker volume some nice named ones and alter the
> VOLUME line accordingly. Should a docker compose be used instead? Because
> this is all feeling hackish.
>
> KR,
>
> On Wed, 29 Nov 2023, 17:24 Joe Witt,  wrote:
>
> > Matt
> >
> > Alpine is still desirable but it was about availability of Alpine w Java
> 21
> > at that time.  I think Alpine also required some different command and
> such
> > in the dockerfile.
> >
> > As far as the repo volumes they can be mapped as you wish now.  It is
> how I
> > keep updating my own flow deployments I test with when I want to retain
> > state across deployment.  Are you not able to map them?
> >
> > Thanks
> >
> >
> >
> > On Tue, Nov 28, 2023 at 9:41 PM Matthew Hawkins 
> > wrote:
> >
> > > Two Q's regarding the docker container;
> > >
> > > 1. Why the Debian version instead of Alpine? Was it compatibility with
> > > external stuff? I can confirm 2.0.0-M1 works fine with the alpine
> version
> > > of liberica-jdk on at least basic flows, with python enabled. It'll
> save
> > > disk space. I love Debian but Alpine is king of containers!
> > >
> > > 2. If we shift the repository folders out of NIFI_HOME per the regular
> > best
> > > practice doco, then a persistent  docker volume could easily be used to
> > > house these, making upgrades painless. I guess the flow.json, keystores
> > and
> > > configs could move too.
> > >
> > > Kind regards,
> > > Matt
> > >
> >
>

Re: Docker container for 2.0.0-M1

[Joe Witt]

Matt

Alpine is still desirable but it was about availability of Alpine w Java 21
at that time.  I think Alpine also required some different command and such
in the dockerfile.

As far as the repo volumes they can be mapped as you wish now.  It is how I
keep updating my own flow deployments I test with when I want to retain
state across deployment.  Are you not able to map them?

Thanks



On Tue, Nov 28, 2023 at 9:41 PM Matthew Hawkins  wrote:

> Two Q's regarding the docker container;
>
> 1. Why the Debian version instead of Alpine? Was it compatibility with
> external stuff? I can confirm 2.0.0-M1 works fine with the alpine version
> of liberica-jdk on at least basic flows, with python enabled. It'll save
> disk space. I love Debian but Alpine is king of containers!
>
> 2. If we shift the repository folders out of NIFI_HOME per the regular best
> practice doco, then a persistent  docker volume could easily be used to
> house these, making upgrades painless. I guess the flow.json, keystores and
> configs could move too.
>
> Kind regards,
> Matt
>

Re: [VOTE] Release Apache NiFi 1.24.0 (RC5)

[Joe Witt]

+1 binding

All the usual build checks.  Fired up NiFi.  Amazing to see the 1.x line
only 1.1GB now!

Thanks Pierre for the great RM work on a tough release cycle and to all for
continued help with votes!

Thanks
Joe


On Thu, Nov 23, 2023 at 9:36 AM Gábor Gyimesi  wrote:

> +1 (non-binding)
>
> Went through the same verification steps as per the previous RC, and LGTM:
>
> - Verified binary hashes and signatures
> - Successfully built NiFi with contrib-check using:
>   - Apache Maven 3.6.3
>   - openjdk version "11.0.20.1" 2023-08-24
>   - Ubuntu 22.04 6.2.0-36-generic
> - Designed and ran some simple flows successfully
> - Tested minifi-c2 server heartbeats and flow update successfully with
> MiNiFi C++
> - Tested communication between MiNiFi C++ and NiFi using InvokeHTTP -
> ListenHTTP processors and using Site 2 Site functionality
> with RemoteProcessGroup
>
> Thanks Pierre!
>
> BR,
> Gabor
>
> On Thu, 23 Nov 2023 at 16:14, Pierre Villard 
> wrote:
>
> > Team,
> >
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > 1.24.0.
> >
> > Please review the following guide for how to verify a release candidate
> > build:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> >
> > The source being voted on the and the convenience binaries are available
> on
> > the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.24.0
> >
> > The build artifacts are available on the Apache Nexus Repository:
> >
> > https://repository.apache.org/content/repositories/orgapachenifi-1251
> >
> > Git Tag: nifi-1.24.0-RC5
> > Git Commit ID: 5241f434829ca46a26a475600ef7c00e1e271e37
> > GitHub Commit Link:
> >
> >
> https://github.com/apache/nifi/commit/5241f434829ca46a26a475600ef7c00e1e271e37
> >
> > Checksums of nifi-1.24.0-source-release.zip
> >
> > SHA256: 715eb61cdc017a5f7ba4d845ae962fdf83d389829db2a8948be14f99f2cc8272
> > SHA512:
> >
> >
> 574147002b905ef64447edec0c7308f4fc3a97b3c7f01edca05464b2b418bcd3922f956d093736014443eec88ceba36af81df37398c5fe23a1288235b79d7306
> >
> > Release artifacts are signed with the following key:
> >
> > https://people.apache.org/keys/committer/pvillard.asc
> >
> > KEYS file is available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > Issues resolved for this version: 285
> >
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353443
> >
> > Release note highlights can be found on the project wiki:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.24.0
> >
> > The vote will be open for 72 hours.
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test. Then
> > please vote:
> >
> > [] +1 Release this package as nifi-1.24.0
> > [] +0 no opinion
> > [] -1 Do not release this package because...
> >
>

Re: [VOTE] Release Apache NiFi 2.0.0-M1 (RC6)

[Joe Witt]

+1 binding

Checked all the things.  Fired it up.  Looks great.

Huge thanks to you David for continuing through this challenging release as
an RM.  And also to all who keep participating in this vote.  NiFi 2.0 is
super important to the future of NiFi and this milestone release is a great
step forward.

Thanks

On Thu, Nov 23, 2023 at 2:21 AM Gábor Gyimesi  wrote:

> +1 (non-binding)
>
> Went through the same verification process as per the previous RC, LGTM:
>
> - Verified binary hashes and signatures
> - Successfully built NiFi with contrib-check using:
>   - Apache Maven 3.6.3
>   - Java 21 2023-09-19 LTS
>   - Ubuntu 22.04 6.2.0-36-generic
> - Designed and ran some simple flows successfully
> - Tested minifi-c2 server successfully with MiNiFi C++ (using a draft of
> the change conforming the new header requirements
> https://issues.apache.org/jira/browse/MINIFICPP-2244)
> - Built and started docker container successfully from
> nifi-docker/dockerhub/Dockerfile
>
> Thanks David!
>
> Best Regards,
> Gabor
>
> On Thu, 23 Nov 2023 at 03:49, David Handermann <
> exceptionfact...@apache.org>
> wrote:
>
> > Team,
> >
> > Thank you for your patience and continued support in reviewing these
> > release candidates.
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi 2.0.0-M1.
> >
> > Please review the following guide for how to verify a release candidate
> > build:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> >
> > The source being voted on the and the convenience binaries are
> > available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M1
> >
> > The build artifacts are available on the Apache Nexus Repository:
> >
> > https://repository.apache.org/content/repositories/orgapachenifi-1250
> >
> > Git Tag: nifi-2.0.0-M1-RC6
> > Git Commit ID: 49fa0d86746f544e294b4ba04b2795d426ba0271
> > GitHub Commit Link:
> >
> >
> https://github.com/apache/nifi/commit/49fa0d86746f544e294b4ba04b2795d426ba0271
> >
> > Checksums of nifi-2.0.0-M1-source-release.zip
> >
> > SHA256: 1a0271c35f585a8e49c5e578422a8e3910fbfc50069a7c875bc6cec5a056eb91
> > SHA512:
> >
> 99e824a0bce109bde84218b914487fd44ba344d5ae9d8fbed9982b41dc2ee7e57b6dd30550c17a4868d43afd900705c48ec48d5e9e63a1b4130c79858f64963c
> >
> > Release artifacts are signed with the following key:
> >
> > https://people.apache.org/keys/committer/exceptionfactory.asc
> >
> > KEYS file is available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > Issues resolved for this version: 959
> >
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12339599
> >
> > Release note highlights can be found on the project wiki:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M1
> >
> > The vote will be open for 72 hours.
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test.
> > Then please vote:
> >
> > [] +1 Release this package as nifi-2.0.0-M1
> > [] +0 no opinion
> > [] -1 Do not release this package because...
> >
>

Re: [VOTE] Release Apache NiFi 2.0.0-M1 (RC3)

[Joe Witt]

+1 binding

Full clean build with contrib check.  All integration tests.  All clear.
Sigs/hashes good.
Build and deploy docker and run containers with python and friends all
good.  Though it is 1.7GB so we'll want to improve.
Build and run the convenience binary and all checks out.   Only 870MB now!
So awesome.
Run various flows/processors including custom python processors. All good.

This release is packed and a great step toward a full blown 2.0 release.

Thanks every and thanks David!

Joe

On Sat, Nov 18, 2023 at 5:45 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0-M1.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are
> available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M1
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1245
>
> Git Tag: nifi-2.0.0-M1-RC3
> Git Commit ID: 5f7e637082d3ab2c45ce3d10e3bc31344e7581da
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/5f7e637082d3ab2c45ce3d10e3bc31344e7581da
>
> Checksums of nifi-2.0.0-M1-source-release.zip
>
> SHA256: d620aec234e14aaf0ca1da58b3c465b23e7a97a7ab4af3e6eb199f56509aaf39
> SHA512:
> 32713d4039104ae163201638933e6d7d35a681611c40353fa1707514d0bf3ea6af6627f54d2b4a4263933242293abe20755803cd5b4c0d1de655ccc978bd98b1
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 945
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12339599
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M1
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-2.0.0-M1
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.24.0 (RC3)

[Joe Witt]

+1 binding

Thanks everyone.  Thanks Pierre!

On Fri, Nov 17, 2023 at 1:39 PM Nissim Shiman 
wrote:

>  +1 (non-binding)
>
> verified source release sha256/512 checksums
>
> successfully ran build using:
> Apache Maven 3.9.5
> Java openjdk version: 1.8.0_382
> linux kernel 3.10.0-1160
>
> Ran various simple flows successfully.  Migrated registry from 1.23.2 to
> this RC successfully.  (i.e. and tested importing PGs, committing new
> versions of PGs to registry successfully.)
>
> Tested/verified:
> NIFI-11782 NPE when adding Snippet with label into Process Group
> NIFI-12084 Logging by process group is not workin on 1.x support branch
> (thank you to all those who worked on this and its parent ticket,
> NIFI-3065.  This new granular level of logging is really nice)
>
> While testing, noticed/filed non holdup issues:
> NIFI-12387 Flow Configuration History can record a Comments change for
> Controller Service when no changes have been made
> NIFI-12388 Process Group log file suffix can become out of sync with its
> name when PG was copied from a version controlled PG
>
> Thank you Pierre and team for the upcoming release!
>
> Nissim Shiman
>
>
> On Friday, November 17, 2023 at 08:10:20 PM UTC, Dan S <
> dsti...@gmail.com> wrote:
>
>  +1 (non-binding)
>
>   - Went through the Release Candidate Verification
>   <
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> >
>   - Verified signatures and hashes
>   - Built on CentOS Linux 7
>   - Java version Oracle 17.0.7
>   - Maven version  3.9.5
>   - Executed simple flow to exercise NIFI-11197
>   
>   - Confirmed fix for NIFI-12165
>   
>   - Confirmed fix for NIFI-11959
>   
>
> Thanks for RM'ing Pierre!
>
> On Fri, Nov 17, 2023 at 1:01 PM Marton Szasz  wrote:
>
> > +1 (binding)
> >
> > Compiled and ran with Java 17 on Ubuntu 22.04, executing a simple flow.
> >
> > On Fri, Nov 17, 2023 at 6:59 PM Matt Burgess 
> wrote:
> > >
> > > +1 (binding)
> > >
> > > Ran through release helper, ran NiFi standalone with Java 8, tested
> > > various flows and components.  Thanks for RM'ing Pierre!
> > >
> > > On Thu, Nov 16, 2023 at 2:01 PM Pierre Villard
> > >  wrote:
> > > >
> > > > Team,
> > > >
> > > > I am pleased to be calling this vote for the source release of Apache
> > NiFi
> > > > 1.24.0.
> > > >
> > > > Please review the following guide for how to verify a release
> candidate
> > > > build:
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> > > >
> > > > The source being voted on the and the convenience binaries are
> > available on
> > > > the Apache Distribution Repository:
> > > >
> > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.24.0
> > > >
> > > > The build artifacts are available on the Apache Nexus Repository:
> > > >
> > > >
> https://repository.apache.org/content/repositories/orgapachenifi-1241
> > > >
> > > > Git Tag: nifi-1.24.0-RC3
> > > > Git Commit ID: 586a1a8789e9c39914f4a6088ac26e22d60eeb91
> > > > GitHub Commit Link:
> > > >
> >
> https://github.com/apache/nifi/commit/586a1a8789e9c39914f4a6088ac26e22d60eeb91
> > > >
> > > > Checksums of nifi-1.24.0-source-release.zip
> > > >
> > > > SHA256:
> > 590cf9b1219f9fd66c81cc1740b2e245d85e341cdc280c769d354084dc27ee8a
> > > > SHA512:
> > > >
> >
> 8d3b9cb9c4686242620ad40ad83fadb972403ee70a101cbb6fa0116b54ad7793702da230871281c0de40322ddfdbfc89dacba7b690465e7b2329850dca5132e8
> > > >
> > > > Release artifacts are signed with the following key:
> > > >
> > > > https://people.apache.org/keys/committer/pvillard.asc
> > > >
> > > > KEYS file is available on the Apache Distribution Repository:
> > > >
> > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >
> > > > Issues resolved for this version: 280
> > > >
> > > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353443
> > > >
> > > > Release note highlights can be found on the project wiki:
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.24.0
> > > >
> > > > The vote will be open for 72 hours.
> > > >
> > > > Please download the release candidate and evaluate the necessary
> items
> > > > including checking hashes, signatures, build from source, and test.
> > Then
> > > > please vote:
> > > >
> > > > [] +1 Release this package as nifi-1.24.0
> > > > [] +0 no opinion
> > > > [] -1 Do not release this package because...
> >
>

Re: Removing JRuby?

[Joe Witt]

Team,

Coming back to this as I've been reviewing all the great output of OWASP
that we're getting daily now (thanks David) as well as some findings from a
nice vendor tool by JFrog.  And what is popping up left and right are
issues in terms of vulnerabilities tied to Jython-standalone.  It is very
clear this component needs to go and is not actively maintained.  It is
also as noted tied to Python 2 which is eol.  I'll file a JIRA to deprecate
in 1.x and remove in 2.x.

The discussion about Javascript/Ruby, etc.. I think the best course going
forward is a specific component appropriate to each desired mechanism is
introduced in 2.x as interest/effort dictates.  The native Python support
will eliminate the need for Jython obviously so not worried about that
coming back.

Thanks

On Tue, Nov 7, 2023 at 5:17 AM Mike Thomsen  wrote:

> Matt,
>
> I don't use JRuby, but the reason it raised my eyebrow a little was that
> AFAIK JRuby is the complete opposite of Jython in terms of enterprise
> readiness and parity with its C-based counterpart.
>
> Edward,
>
> The problem with Lua here is that the Java versions of Lua aren't being
> maintained. I think the most recent release on any branch of them was 3
> years ago. They're basically dead in the water as projects.
>
> On Mon, Nov 6, 2023 at 2:57 PM Edward Armes 
> wrote:
>
> > While I do like the ExecuteScript processors as they are great at digging
> > you out of a hole. The performance of them isn't that great.
> >
> > That being said I would ve careful about dropping Lua support as there
> is a
> > growing list of software that supports end user/administrator extensions
> > via Lua for those that don't want to have to re-compile software
> > themselves. On the other hand given that Jython doesn't yet have a
> Python 3
> > implementation it could be argued dropping Jython support is a must given
> > that the Python 2.x line is basically end of life.
> >
> > Now I wonder if its worth re-factoring the ExecuteScript processors to be
> > per language implementations inheriting from a common base like a few
> > processors do already.
> >
> > Edward
> >
> > On Mon, 6 Nov 2023, 16:24 Matt Burgess,  wrote:
> >
> > > I believe it is because in both ExecuteScript and ExecuteGroovyScript
> > > you can do "regular" groovy, but EGS has extra built-ins such as easy
> > > access to controller services, Groovy SQL stuff, etc, and we could
> > > keep building it out. But IMO we'd have to port the rest of the
> > > scripted components (ScriptedReader/Writer, etc.) over to the Groovy
> > > bundle and make sure there's a drop-in replacement in the Python stuff
> > > before we'd want to deprecate the scripted bundle.
> > >
> > > On the JRuby front, is that something you use actively? This question
> > > is for you and the entire community of course.
> > >
> > > Regards,
> > > Matt
> > >
> > > On Mon, Nov 6, 2023 at 7:12 AM Mike Thomsen 
> > > wrote:
> > > >
> > > > If we deprecate ExecuteScript, I think we need to have groovyx be
> ready
> > > to
> > > > function as a drop-in replacement if it's not there already.
> > > >
> > > > On Sun, Nov 5, 2023 at 9:21 PM Matt Burgess 
> > > wrote:
> > > >
> > > > > IIRC the removal of these engines was mostly due to lack of use or
> at
> > > > > least the perception thereof. If JRuby is being used by the
> community
> > > > > actively, I'm happy to revisit that discussion. Luaj's JSR-223
> > > > > interface left something to be desired, but JRuby just needed a
> > system
> > > > > variable set or something like that.
> > > > >
> > > > > For the groovyx bundle, because it is Groovy-specific I tend to
> think
> > > > > we could make better use of that than ExecuteScript, especially if
> we
> > > > > do get rid of all the engines. We have a Groovy-specific
> processor, a
> > > > > "real" Python SDK, and no more Nashorn. Perhaps we move all the
> > > > > scripted components to the Groovy bundle, although I believe some
> > > > > folks still make use of Jython for these. Of course if we reinstate
> > > > > JRuby for ExecuteScript it's probably best to keep things the way
> > they
> > > > > are, or create a jruby bundle. The original scripting bundle was
> > > > > aiming to support several engines, but if it turns out only one or
> > two
> > > > > will be useful, it may not be worth shoehorning all that JSR-223
> > logic
> > > > > when engine-specific components could be simpler, more easily
> > > > > maintained, and allow for the idioms of the language to be better
> > used
> > > > > (as is done in the groovyx bundle).
> > > > >
> > > > > Just my two cents, looking forward to everyone's thoughts!
> > > > >
> > > > > - Matt
> > > > >
> > > > > On Sun, Nov 5, 2023 at 8:31 PM Mike Thomsen <
> mikerthom...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/NIFI-11646
> > > > > >
> > > > > > I get the removal of Lua, but not the removal of JRuby. It's a
> > clean
> > > > > > reimplementation of Ruby native to the 

Re: [DISCUSS] Release Timing for NiFi 2.0.0 M1 and 1.24.0

[Joe Witt]

Very cool - thanks David and Pierre for tackling RM duties!

On Mon, Nov 13, 2023 at 11:59 AM Pierre Villard 
wrote:

> Thanks David, sounds good, will get started on the 1.24 release soon.
>
> Le lun. 13 nov. 2023 à 19:37, David Handermann <
> exceptionfact...@apache.org>
> a écrit :
>
> > Thanks for replies Pierre and Mark!
> >
> > With several pull requests merged last week addressing the Python
> > concerns Mark raised, I believe we are ready to proceed with the
> > release candidate process.
> >
> > Based on a conversation Pierre, we should be able to collaborate on
> > preparing candidate builds for both 2.0.0-M1 and 1.24.0.
> >
> > We will need to make a few website adjustments for the 2.0.0-M1
> > version in terms of documentation and downloads, but it should be
> > possible to handle that with some alternative navigation links for
> > now.
> >
> > I plan on starting the release candidate build process for 2.0.0-M1 soon.
> >
> > Regards,
> > David Handermann
> >
> > On Wed, Nov 8, 2023 at 5:53 PM Mark Payne  wrote:
> > >
> > > Hey Pierre,
> > >
> > > I ran into a couple of issues (they appear to be timing issues) around
> > the Python stuff on startup. I think we need to hold off on the release
> > until these are resolved but I think I should have a PR up some time
> > tomorrow hopefully.
> > >
> > > Thanks
> > > -Mark
> > >
> > >
> > >
> > > > On Nov 8, 2023, at 10:30 AM, Pierre Villard <
> > pierre.villard...@gmail.com> wrote:
> > > >
> > > > Hey David,
> > > >
> > > > With all of the recent changes we merged, I believe we are in the
> right
> > > > spot to start the process with 1.24 and 2.0M1 releases.
> > > >
> > > > Happy to help with this,
> > > > Pierre
> > > >
> > > > Le ven. 3 nov. 2023 à 22:46, David Handermann <
> > exceptionfact...@apache.org>
> > > > a écrit :
> > > >
> > > >> Team,
> > > >>
> > > >> We have made great progress on a large number of features and fixes
> > > >> for NiFi 2.0.0. The Jira version page for 2.0.0 [1] lists almost 900
> > > >> issues! At the same time, we have 250 issues tagged for version
> 1.24.0
> > > >> [2].
> > > >>
> > > >> Although there is still more work to do for a full release of
> 2.0.0, I
> > > >> believe we are at a good point to release a milestone version.
> > > >> Releasing 1.24.0 at the same time will also provide the latest
> > > >> upgrades, plus new features and deprecations that will help prepare
> > > >> for upgrading to 2.0.0.
> > > >>
> > > >> We have informally referred to the initial release as a milestone
> > > >> version, so it seems like officially naming it 2.0.0-M1 would be the
> > > >> most straightforward option.
> > > >>
> > > >> There are a couple more open issues that I believe we can wrap up in
> > > >> the next few days, but otherwise we should be able to target early
> > > >> next week for preparing a release build.
> > > >>
> > > >> With the goal of coordinating a release of multiple versions, I
> would
> > > >> be glad to handle release manager duties for one of the versions.
> > > >>
> > > >> Regards,
> > > >> David Handermann
> > > >>
> > > >> [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
> > > >> [2] https://issues.apache.org/jira/projects/NIFI/versions/12353443
> > > >>
> > >
> >
>

Re: Processor Help

[Joe Witt]

Geoff

To establish some shared understanding lets summarize a few of the
components at play.

SourceSystem: This is where the data you want to grab lives and it sounds
like it is accessible via some protocol and as files.

DataSourceProcess: This sounds like some process which is active
creating/writing data to files in the SourceSystem.  It is very important
to understand how this process behaves as it actively writes data because
you don't want to consume the data while it is still being
written/renamed/etc..  Common techniques are that such processes write data
with a special filenaming technique to indicate it is still working on it
such as starting with a '.' or something like that then you could configure
the List* processors in nifi to ignore such filenames.  Often though this
is not possible so you'll end up having to use things like checking file
modification age and waiting to pick it up after a while with the hope this
means the file is done being written.  That can fail too for fun reasons
and so you can resort to monitoring file age.  All these various joyful
processes are understood and baked into the nifi processes where possible
for List/Fetch/etc.. because this is just the joys of file based IO.

Protocol: How the NiFi server will communicate with the SourceSystem.
Could be FTP, SFTP, file share if accessible to the nifi server, etc.. That
will be important to specify.

List*: These processors in nifi tend to execute the appropriate protocol to
then conduct a listing of what files are present meeting some criteria in
the destination system.  This processor just pulls metadata not the actual
content of the files.

Fetch*:  These processors given a list of one or more files it will then go
and actually pull the content of those files to the NiFi server itself.  It
optionally is usually configured to delete the raw source data once
successfully pulled.

Put*: These processors given a flowfile with content/metdata will
write/send/etc.. the file/metadata to the appropriate target using the
appropriate protocol.  That is data is copied from nifi's internal
repositories to the target.

Get*: These processors tend to be a combination of the logic in Listing and
Fetching and are used much less often these days because List/Fetch tends
to be more powerful and performant.

You mention wanting to have a way to pull from the initial location the
DataSourceProcess writes to then write that data to some holding point.  I
don't recommend this as it does not make the problem easier.  Instead just
have NiFi be the staging/holding point.  That is its job in this equation.
>From there you can send the data wherever you like and yes Put* processors
output relationships (often success) can go wherever you like.  Once you
have run a Fetch* processor the mental model you want to have is the data
is IN nifi and it is responsible to keep it safe and move it to your next
part of the flow.

Thanks
Joe

On Wed, Nov 1, 2023 at 1:25 PM Buthorne, Geoffrey <
gbutho...@epsilonsystems.com> wrote:

> Dev Team,
>
> You have a wonderful product, and I use it every day to move files to and
> from a variety of servers.  It works well and thank you.
> Recently, I've come across a situation that I can't seem to find a
> processor for and I'm hoping you can help point me in the right direction.
>
> I am trying to move files that are in an active directory.
> The number of files and their size, dictate that I first copy the files
> out of the active directory while on a remote server to some other
> directory on that remote server that is not actively moving/deleting the
> files from.  I would like to create a directory tree to help keep files
> organized and manageable, as I prepare to pick them up and bring to my
> server.
> Once I have created the needed directories, and copied the files out of
> the active directory(ies) - then I can safely begin to move them.
>
> I have looked at the various get/fetch/put processors to help with this -
> but so far - the processors will always first pick up the files and bring
> them to my server from the remote system.  It will then place them back on
> the remote server, to whatever directory I configure it to. - I could then
> pick it up again, but I'm waiting bandwidth and resources transferring the
> file up the same file 3 times.
> I thought a putfile - once on a remote server - would work however, after
> the put file, that task chain is done and my only options appear to be
> successfully terminate or failure and retry.  I don't seem to be able to
> add another task/processor after a put.
>
> Is there a processor that will allow remote directory management (make
> directories at remote server, copy/move remote files to those directories)
> and then Get/Fetch?
>
> I'm still a little new at this and I certainly do not have experience will
> all the processors available - I'm hoping there is a processor that will
> allow some remote file management/set up before bring the files to my
> 

Re: discuss NiFi 1.24 release

[Joe Witt]

David

I think we can hold off for a few weeks - I'll respond to the Slack message
on that.

Will be sad to see H2 go.  The original nifi flowfile repository ran on
H2.  Surprisingly fast and stable actually.  But happy to hear there is
such progress underway.

Thanks

On Tue, Oct 10, 2023 at 4:27 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Joe,
>
> Thanks for initiating the discussion on a 1.24 release. There are a
> handful of new features, plus additional deprecations and dependency
> upgrades that would be very useful to release sooner rather than
> later.
>
> If we are considering a 1.24 release within the next week, then I
> would expect needing a 1.25 release to incorporate some additional
> deprecations around the same time we are ready for a 2.0 milestone
> release version.
>
> I am in the process of implementing a replacement for H2 to store flow
> configuration history, and I plan to have a pull request ready in the
> next few days. There will be some additional work to support migration
> on version 1 support branch.
>
> If we want to move forward with a 1.24 release within a week or so,
> then these changes could be targeted for 1.25. If we want to wait a
> few weeks, then I could see this being incorporated in 1.24.
>
> Regards,
> David Handermann
>
> On Tue, Oct 10, 2023 at 6:14 PM Mark Payne  wrote:
> >
> > Thanks Joe. I wouldn’t argue against doing a 1.24 release.
> >
> > Thanks
> > -Mark
> >
> > > On Oct 10, 2023, at 7:01 PM, Joe Witt  wrote:
> > >
> > > Team,
> > >
> > > We have plenty of bits out there to kick a release.  I'm happy to RM
> this
> > > if nobody else wants to.
> > >
> > > Had a user on Slack ask for it.
> > >
> > > Thanks
> > > Joe
> >
>

discuss NiFi 1.24 release

[Joe Witt]

Team,

We have plenty of bits out there to kick a release.  I'm happy to RM this
if nobody else wants to.

Had a user on Slack ask for it.

Thanks
Joe

Re: [discuss] NiFi 2 and Node.js version

[Joe Witt]

My two cents on whether this is a 2.x tied thing is that it doesn't seem
like we're talking about changing the user's experience in either case here
so it should not really be a 2.x blocker.  More than anything it sounds
like we should/need/want to do this to be on a more
maintainable/supportable implementation of the UI and there sounds like a
path to bring this in which should be a non issue for the user.  So if that
understanding is correct I'd say there is no need to correlate this with a
2.x release.Definitely if we were talking about a different user
experience that would change my thoughts though.

Thanks

On Tue, Oct 10, 2023 at 2:15 PM Chris Sampson
 wrote:

> Matt,
>
> Thanks for correcting my misunderstanding around the utilities in the
> build. It sounds like updating the versions of node & npm used for the
> build should be a separate effort to updating the UI, which it appears
> you're making good headway with.
>
> It seems the question for NiFi 2.x is whether to upgrade these tool
> versions in the build.
>
> Exciting to see the move to newer UI technologies, and hopefully a simpler
> dev experience around UI changes.
>
> On Tue, 10 Oct 2023, 21:44 Matt Gilman,  wrote:
>
> > Chris,
> >
> > Thanks for raising this. Node.js (and npm) are used at build time to
> > package the front-end applications. Bumping those versions should be
> > doable. However, the front end still uses a lot of legacy and now
> > unsupported dependencies and also requires updating. This aspect is a
> > much larger effort. There is a JIRA [1] filed for migrating the front end
> > to current versions of these dependencies. That work is underway and
> things
> > are progressing nicely. The JIRA also lays out a high-level plan for
> > introducing this change without impacting the existing UI until it's
> fully
> > ready.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-11481
> >
> > On Tue, Oct 10, 2023 at 3:16 PM Chris Sampson
> >  wrote:
> >
> > > The main focus of NiFi 2.x has so far been to upgrade Java to JDK 21
> and
> > > remove deprecated/unmaintained code and artefacts.
> > >
> > > The NiFi & NifI Registry UIs are currently using Node.js v16, which is
> an
> > > LTS release, but is imminently due to be end of life [1]. Node.js v18
> has
> > > been the current LTS release since October 2022, although v20 is about
> to
> > > take over that spot.
> > >
> > > Should the NiFi 2.x milestone include an upgrade of the UI frameworks
> to
> > > (at least) Node.js v20, or v18 if there are compatibility issues with
> > > dependencies used?
> > >
> > > If this is already on somebody’s roadmap, then great, but I just
> noticed
> > > this in the pom.xml files and thought it was worth raising.
> > >
> > >
> > > [1]: https://nodejs.dev/en/about/releases/
> > >
> > > Cheers,
> > >
> > > ---
> > > Chris Sampson
> > > IT Consultant
> > > chris.samp...@naimuri.com
> > >
> > >
> > >
> >
>

nifi oct 23 board report

[Joe Witt]

Team,

Thanks as always.  Here is the board report I submitted for this month.


## Description:
The mission of NiFi is the creation and maintenance of software related to
providing an easy to use, powerful, and reliable system to process and
distribute data.

Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
integrate with and leverage the command and control of NiFi. There are both
Java and C++ implementations.

Apache NiFi Registry is a centralized registry for key configuration items
including flow versions, assets, and extensions for Apache NiFi and Apache
MiNiFi.

Apache NiFi Nar Maven Plugin is a release artifact used for supporting the
NiFi classloader isolation model.

Apache NiFi Flow Design System is a theme-able set of high quality UI
components and utilities for use across the various Apache NiFi web
applications in order to provide a more consistent user experience.

## Project Status:
Current project status: Ongoing. High.
Issues for the board: None.

## Membership Data:
Apache NiFi was founded 2015-07-14 (9 years ago)
There are currently 64 committers and 35 PMC members in this project.
The Committer-to-PMC ratio is roughly 8:5.

Community changes, past quarter:
- Chris Sampson was added to the PMC on 2023-08-19
- Timea Barna was added as committer on 2023-08-02

## Project Activity:
The community conducted at least four key feature, improvement, security
related
releases during the reporting period.  These include MiNiFi CPP 0.15.0
on September 3rd and NIFi 1.23.2, 1.23.1, and 1.23.0 on Aug 22, 18, and Jul
28.
The MiNiFi release adds support to TLS v1.3, PutS3 for multipart object
upload,
among other capabilities and several bug fixes.
The NiFi 1.23.x line makes it easier to work Microsoft Excel documents in
the
flow and leverage AWS Glue Schema Registry and fixes a large range of
defects
or reported vulnerable libs across Spring Framework, Bouncy Castle and more.
But the big news for the community remains all about the heavy push to
Apache NiFi 2.0.0.  With more than 740 JIRAs resolved and counting this
major
release will indeed be major.  NiFi 2 now builds and runs and depends on
Java 21.  We've started adopting many of the nice language features
including lightweight threads and cleaner syntax options.  We're reducing
far more code than ever before that is no longer maintained, no longer
necessary, etc.  NiFi 2.0 also makes running real processes in legit
Python super easy which allows NiFi to help data engineers far more
as they often found the Java development processes too slow for their
use cases.  We don't have a clear ETA for 2.0 but this calendar year is
still in sight.

## Community Health:
Community health is stronger than ever.  Our mailing list activity
has picked up in the past quarter with 36%, 26% and 70% increase across
dev, users, and issues lists.  There are more PRs coming in and getting
reviews and merges - specifically a 10% increase in the period.
Meanwhile, usage and popularity of the Slack channel is growing. We
reported 2700 in our general slack channel last quarter and now
we are at 2839.  The depth of conversation on there is excellent spanning
simple user questions, developer discussions which we do push to JIRA
or mailing list as needed. And importantly we still see active blogging,
videos, webinars, conference talks on NiFi.

Re: Custom-processor configuration suggestions

[Joe Witt]

Russ

It sounds like what you have is a case of significant reference data you
need made available to various instances of this processor that knows how
to use that reference state to do its function.

This is similar to cases like IP geo enrichment where the dataset on which
you'd make the decision is larger and more importantly subject to change
over time.  In such cases the ideal state is:
(A) The reference dataset(s) is hosted at a RESTful endpoint and can be
periodically pulled and stored some place local/easily accessible.
(B) The processor knows where to look for this reference dataset download
and is able to hot reload it on the fly to include understanding that the
needed datasets might not yet be made available and it should yield until
it sees them and loads them.

Thanks

On Wed, Sep 27, 2023 at 11:51 AM Russell Bateman 
wrote:

> I'm posting this plea for suggestions as I'm short on imagination here.
>
> We have some custom processors that need extraordinary amounts of
> configuration of the sort a flow writer would have to copy and paste
> in--huge amounts of Yaml, regular expressions, etc. This is what our
> flow writers are already doing. It would be easier to insert a filename
> or -path, but...
>
> ...asking a custom processor to perform filesystem I/O is icky because
> of unpredictable filesystem access post installation. Thinking about how
> installation is beyond my control, I don't want to make installation
> messy, etc. Containers, Kubernetes deployment, etc. complicate this.
>
> I thought of wiring /GetFile/ to a subdirectory (problematic, but less
> so?) and accepting files as input to pass on to needy processors who
> would recognize, adopt and incorporate configuration based on
> higher-level and simpler cues posted by flow writers as property values.
>
> Assuming you both grok and are interested in what I'm asking, do you
> have thoughts, cautionary statements or even cat-calls to offer? Maybe
> there are obvious answers I'm just not thinking of.
>
> Profuse thanks,
>
> Russ

Re: [discuss] Time for a NiFi 2.0 M1 release?

[Joe Witt]

The Packager thing looks pretty close to being ready anyway.  I dont see
that holding anything up on any line at this point.

I don't have any heartburn waiting for the flowxml and templates to get
tossed out as indeed that needs to happen regardless.

As far as M1 being a "the most disruptive" variant.  While not strictly
your point - that I'd say is a non goal for any of the releases meaning
we're not trying to have a release where we pretend we're sure it is the
most disruptive.  The purpose of the M1 will need to be squarely rooted in
getting to a production/stable release.  The purpose of any subsequent Mn
or the actual official 2.0.0 release will be ensuring it is the best
possible migration path we intend to make available.

On Tue, Sep 26, 2023 at 12:04 PM Adam Taft  wrote:

> I'm also hoping that both 1.x and 2.x lines can receive the PackageFlowFile
> processor that Mike Moser recently proposed. That way, the M1 release and
> the most recent 1.x release will have a simple (or logical) replacement for
> PostHTTP.
>
> In general, it would be nice to have 1.x lined up with 2.0-M1 so that the
> transitional experience is as disruptive as it's going to be when 2.0-final
> is released. That is, I want all the things that can break to break, once a
> 2.0 milestone is released. From that perspective, I agree with Pierre that
> waiting for the flow.xml work to finalize makes the most sense, because
> then users can start getting a feel for how it will affect them. Lots of
> deployment scripts (think Ansible or equivalent) rely on the flow.xml.gz
> file specifically.
>
> The most disruptive parts of the 1.x to 2.x transition would ideally be
> realized as early as possible. Understand and agree with the urgency to get
> 2.0-M1 released, but also concerned that it doesn't allow a proper
> evaluation of all breaking changes just yet.
>
> /Adam
>
> On Tue, Sep 26, 2023 at 9:18 AM Pierre Villard <
> pierre.villard...@gmail.com>
> wrote:
>
> > Hey Joe,
> >
> > Definitely a +1 to get a M1 release ASAP. I'd still recommend waiting on
> > the flow.xml removal work to be merged. The reason being that users may
> > give useful feedback when they'll try NiFi 2.0 with existing flows coming
> > from NiFi 1.x and getting rid of all of the XML based stuff. There is
> also
> > a PR coming soon for the frontend work of the templates removal.
> Hopefully
> > both can be completed this week or next week.
> >
> > Pierre
> >
> > Le mar. 26 sept. 2023 à 17:35, Joe Witt  a écrit :
> >
> > > Team,
> > >
> > > The NiFi 2.0 release has more than 700 resolved JIRAs on it [1] and
> > growing
> > > every day.
> > >
> > > The NiFi 2.0 deprecation plan is well underway and largely complete
> [2].
> > >
> > > We still need to remove a lot of now deprecated code, tests which are
> > never
> > > run and largely don't work, eliminate the flow.xml which has a JIRA/PR
> > > underway.  And more.  But we're getting close and we need to start
> > getting
> > > this in the hands of users.
> > >
> > > The docker image can now be built in 'nifi-docker/dockermaven' after a
> > full
> > > build from root with 'mvn install -Pdocker'.  And it comes up with
> > Ubuntu,
> > > Java 21, Python 3.9, and NiFi 2.0 ready to roll with Python processors
> > > enabled.
> > >
> > > I propose we start closing down soon to make a NiFi 2.0 M1 release
> happen
> > > even before we have all the things done.  We need to start getting
> > feedback
> > > and giving people a chance to work with it.
> > >
> > > Lastly, a huge thank you to the folks in the community that have been
> > > helping push towards 2.x with code changes, removals, reviews, bug
> > reports,
> > > etc..  Super awesome to see.  NiFi 2.x is shaping up nicely to be
> useful
> > > not only for our well established user base which spans the globe and
> > every
> > > industry but now we are also seeing a lot of opportunity and fit for
> NiFi
> > > in these exciting AI use cases particularly involving orchestrating the
> > > data flows with embeddings, vector stores, and LLMs.  And the Python
> > > capabilities in NiFi 2.x make NiFi far easier to use for the very
> > important
> > > data engineer user base.
> > >
> > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
> > > [2]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Deprecated+Components+and+Features
> > >
> > > Thanks
> > > Joe
> > >
> >
>

Re: Processor stuck in yielding

[Joe Witt]

Yeah that description of behavior means there was a bug in the processor
(for sure).  I have to believe in the last 5 years that it has been made
better.  You should never have to stop/start the processor to get it to
behave right.

Thanks

On Tue, Sep 26, 2023 at 10:39 AM Phillip Lord 
wrote:

> Honestly if it were me… I’d chalk it up to a glitch in the
> matrix.  Troubleshooting step 1 for me would be to restart that processor,
> hopefully it works and you can move on with your day.
> If it’s still occurring then maybe try adding some debug logging for that
> processor?
>
> As mentioned though, I’d also definitely look into upgrading to a newer
> version… I haven’t dug into any issues re that component.  But a lot has
> changed since 1.8.
> On Sep 26, 2023 at 1:02 PM -0400, Ivan Dolinin
> , wrote:
>
> Understood.
>
> I stopped the processor and started it back. The processor successfully
> read all 7+k messages which were in the queue. Was there something I could
> do to see why the processor believed that the queue was empty?
>
> Thanks,
>
> Ivan Dolinin / (416) 583-5833 x2012 / idoli...@cleverdevices.com idoli...@cleverdevices.com>
>
> From: Joe Witt 
> Sent: Tuesday, September 26, 2023 12:54 PM
> To: Ivan Dolinin ; dev@nifi.apache.org
> Subject: Re: Processor stuck in yielding
>
> Unfortunately that does not tell us why yield was called. But it likely
> means the processor believes the queue it is talking with contains no data.
> The code shows that as the most likely culprit. http
> External (joew...@apache.org<mailto:joew...@apache.org>)
> Report This Email<
> https://protection.inkyphishfence.com/report?id=c2tvdXQtY2xldmVyLWRldmljZXMvaWRvbGluaW5AY2xldmVyZGV2aWNlcy5jb20vNTRmZTg2ZmQ0YTFjOWIxMTUyYzBmZjk4NjI2ZDE0YzkvMTY5NTc0NzI3Ny40OQ==#key=a8d6f5821ef32eee25a2b7063049fd6d>
> FAQ<https://getskout.com/emailprotectionfaq/> Skout Email Protection<
> https://getskout.com/emailprotection/>
>
> Unfortunately that does not tell us why yield was called. But it likely
> means the processor believes the queue it is talking with contains no data.
> The code shows that as the most likely culprit.
>
>
> https://github.com/apache/nifi/blob/98aabf2c50f857efc72fd6f2bfdd9965b97fa195/nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/ConsumeAMQP.java#L130-L133
> <
> https://shared.outlook.inky.com/link?domain=github.com=h.eJxVj71ygzAQhF_FQ9rAIRkh5MqZtMlM8gj6NbJBIpIgRSbvHmRckOaK73Znd3-KOQzF6VD0KU3xBHCxqZ9FJf0IfOKy1-CssSAGL4B1nAuDJalNR6g2kmKjWoOFUYqxlghGDUeM3C2l46EUs1ODjhvg49f0IDswBS91jD5EiEHCyK2DK184-HD5VyGrYad-9S7Oo355__yosuHpDR3rcj3H4vlQ3PKoePNzKuWgFx1KpRe7esEqP1hn3XnjD3xfTBqju9aohiPJBEIEy9oY1rW4VaiRDFDLCG0oprRqWI7ROebq9bdN6by1rdbi-aXya4d-_wB0XHiT.MEQCIHfldm6PVe5TryWRj4uzK6s48lz95tME0s_ZGSVejUcAAiBSfrnPqI6Kio7IplZOa3uez4MZxuT8NejbMeTHaD6c3g
> >
> if (lastReceived == null) {
> // If no messages received, then yield.
> context.yield();
> }
>
> Thanks
>
> On Tue, Sep 26, 2023 at 9:48 AM Ivan Dolinin  <mailto:idoli...@cleverdevices.com>> wrote:
> Adding log in text. Looks like the image didn’t go through.
>
> 2023-09-26 08:16:58,207 DEBUG [Timer-Driven Process Thread-7]
> o.a.nifi.amqp.processors.ConsumeAMQP
> ConsumeAMQP[id=78f68d05-f59d-3b9b-746e-5d480be723df] has chosen to yield
> its resources; will not be scheduled to run again for 1000 milliseconds
> 2023-09-26 08:16:58,207 DEBUG [Timer-Driven Process Thread-9]
> o.a.nifi.amqp.processors.ConsumeAMQP
> ConsumeAMQP[id=85c40dd5-3f41-3895-25e9-ee3569331185] has chosen to yield
> its resources; will not be scheduled to run again for 1000 milliseconds
>
>
> Ivan Dolinin / (416) 583-5833 x2012 / idoli...@cleverdevices.com idoli...@cleverdevices.com>
>
> From: Ivan Dolinin  idoli...@cleverdevices.com.INVALID>>
> Sent: Tuesday, September 26, 2023 12:20 PM
> To: Joe Witt mailto:joew...@apache.org>>;
> dev@nifi.apache.org<mailto:dev@nifi.apache.org>
> Subject: RE: Processor stuck in yielding
>
>
> Joe,
>
> thank you for a very quick response.
>
> Here is the screenshot of the log. There is nothing else recorded for the
> UUID 78f68d05-f59d-3b9b-746e-5d480be723df. I could not trace the beginning
> of this state. We have 3 ConsumeAMQP processors reading from RabbitMQ, only
> this one is yielding, other 2 reading messages as expected.
>
>
>
>
> Ivan Dolinin / (416) 583-5833 x2012 / idoli...@cleverdevices.com idoli...@cleverdevices.com>
>
> From: Joe Witt mailto:joew...@apache.org>>
> Sent: Tuesday, September 26, 2023 11:49 AM
> To: dev@nifi.apache.org<mailto:dev@nifi.apache.org>
> Subject: Re: Processor stuck in yielding
>
>
> Ivan,
>
> Please show what the logs stat

Re: Processor stuck in yielding

[Joe Witt]

Unfortunately that does not tell us why yield was called.  But it likely
means the processor believes the queue it is talking with contains no
data.  The code shows that as the most likely culprit.

https://github.com/apache/nifi/blob/98aabf2c50f857efc72fd6f2bfdd9965b97fa195/nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/ConsumeAMQP.java#L130-L133
if (lastReceived == null) {
// If no messages received, then yield.
context.yield();
}

Thanks

On Tue, Sep 26, 2023 at 9:48 AM Ivan Dolinin 
wrote:

> Adding log in text. Looks like the image didn’t go through.
>
>
>
> 2023-09-26 08:16:58,207 DEBUG [Timer-Driven Process Thread-7]
> o.a.nifi.amqp.processors.ConsumeAMQP
> ConsumeAMQP[id=78f68d05-f59d-3b9b-746e-5d480be723df] has chosen to yield
> its resources; will not be scheduled to run again for 1000 milliseconds
>
> 2023-09-26 08:16:58,207 DEBUG [Timer-Driven Process Thread-9]
> o.a.nifi.amqp.processors.ConsumeAMQP
> ConsumeAMQP[id=85c40dd5-3f41-3895-25e9-ee3569331185] has chosen to yield
> its resources; will not be scheduled to run again for 1000 milliseconds
>
>
>
>
>
> Ivan Dolinin / (416) 583-5833 x2012 / idoli...@cleverdevices.com
>
>
>
> *From:* Ivan Dolinin 
> *Sent:* Tuesday, September 26, 2023 12:20 PM
> *To:* Joe Witt ; dev@nifi.apache.org
> *Subject:* RE: Processor stuck in yielding
>
>
>
>
>
> Joe,
>
>
>
> thank you for a very quick response.
>
>
>
> Here is the screenshot of the log. There is nothing else recorded for the
> UUID 78f68d05-f59d-3b9b-746e-5d480be723df. I could not trace the beginning
> of this state. We have 3 ConsumeAMQP processors reading from RabbitMQ, only
> this one is yielding, other 2 reading messages as expected.
>
>
>
>
>
>
>
> Ivan Dolinin / (416) 583-5833 x2012 / idoli...@cleverdevices.com
>
>
>
> *From:* Joe Witt 
> *Sent:* Tuesday, September 26, 2023 11:49 AM
> *To:* dev@nifi.apache.org
> *Subject:* Re: Processor stuck in yielding
>
>
>
>
>
> Ivan,
>
>
>
> Please show what the logs state specifically if you can.
>
>
>
> That processor could yield for the following reasons from a quick look:
>
> 1. The connection between it and the next processor in the flow is a full
> queue not being processed fast enough.
>
> 2. The data connection to AMQP is working but it is told there is no data
> currently to pull.
>
> 3. ...
>
>
>
> That is about it actually.  I am looking at the current code though.  NiFi
> 1.8.0 is 5 years old (today ironically) and that processor and the
> associated libraries and even framework have changed a lot since then.  You
> will definitely need to try your flow on the latest 1.23.x line if the
> issue being seen is not a simple config problem.
>
>
>
> Thanks
>
>
>
>
>
> On Tue, Sep 26, 2023 at 8:40 AM Ivan Dolinin <
> idoli...@cleverdevices.com.invalid> wrote:
>
> Hello,
>
> In one of our production instances running NiFi 1.8.0, the ConsumeAMQP
> processor is stuck in yielding. All the log is saying that it has chosen to
> yield. No errors are happening in the downstream processors. There are
> thousands of messages in the queue it is reading from. The processor is
> connecting to the queue.  What could be the reason?
>
>
> Thanks,
>
> Ivan Dolinin
>
>

Re: Processor stuck in yielding

[Joe Witt]

Ivan,

Please show what the logs state specifically if you can.

That processor could yield for the following reasons from a quick look:
1. The connection between it and the next processor in the flow is a full
queue not being processed fast enough.
2. The data connection to AMQP is working but it is told there is no data
currently to pull.
3. ...

That is about it actually.  I am looking at the current code though.  NiFi
1.8.0 is 5 years old (today ironically) and that processor and the
associated libraries and even framework have changed a lot since then.  You
will definitely need to try your flow on the latest 1.23.x line if the
issue being seen is not a simple config problem.

Thanks


On Tue, Sep 26, 2023 at 8:40 AM Ivan Dolinin
 wrote:

> Hello,
>
> In one of our production instances running NiFi 1.8.0, the ConsumeAMQP
> processor is stuck in yielding. All the log is saying that it has chosen to
> yield. No errors are happening in the downstream processors. There are
> thousands of messages in the queue it is reading from. The processor is
> connecting to the queue.  What could be the reason?
>
>
> Thanks,
>
> Ivan Dolinin
>
>

[discuss] Time for a NiFi 2.0 M1 release?

[Joe Witt]

Team,

The NiFi 2.0 release has more than 700 resolved JIRAs on it [1] and growing
every day.

The NiFi 2.0 deprecation plan is well underway and largely complete [2].

We still need to remove a lot of now deprecated code, tests which are never
run and largely don't work, eliminate the flow.xml which has a JIRA/PR
underway.  And more.  But we're getting close and we need to start getting
this in the hands of users.

The docker image can now be built in 'nifi-docker/dockermaven' after a full
build from root with 'mvn install -Pdocker'.  And it comes up with Ubuntu,
Java 21, Python 3.9, and NiFi 2.0 ready to roll with Python processors
enabled.

I propose we start closing down soon to make a NiFi 2.0 M1 release happen
even before we have all the things done.  We need to start getting feedback
and giving people a chance to work with it.

Lastly, a huge thank you to the folks in the community that have been
helping push towards 2.x with code changes, removals, reviews, bug reports,
etc..  Super awesome to see.  NiFi 2.x is shaping up nicely to be useful
not only for our well established user base which spans the globe and every
industry but now we are also seeing a lot of opportunity and fit for NiFi
in these exciting AI use cases particularly involving orchestrating the
data flows with embeddings, vector stores, and LLMs.  And the Python
capabilities in NiFi 2.x make NiFi far easier to use for the very important
data engineer user base.

[1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
[2]
https://cwiki.apache.org/confluence/display/NIFI/Deprecated+Components+and+Features

Thanks
Joe

Re: [discuss] nifi 2.0 and Java 21…

[Joe Witt]

Java 21 officially just launched:
https://www.prnewswire.com/news-releases/oracle-releases-java-21-and-extends-support-roadmap-301930953.html

Java 21 official releases available such as:
https://www.azul.com/downloads/?version=java-21-lts=jdk#zulu

As soon as the temurin/azul builds we need are available on Github CI we'll
have our build updated.

As soon as the docker images we need are ready for the same we'll have
nifi-docker updated.

We should be Java 21 ready on main/NiFi 2.x and a full 10 years of a
supported LTS feature line to work with.

Thanks all!
Joe

On Sat, Sep 16, 2023 at 9:03 PM Ryan Hendrickson <
ryan.andrew.hendrick...@gmail.com> wrote:

> Fantastic progress!  Looking forward to the new platform!
> Wooohooo!
>
> On Fri, Sep 15, 2023, 7:17 PM Joe Witt  wrote:
>
> > Caveat to that - the build works right now as the Java release version in
> > the pom is set to 17.  So we're building/running Java 21 but the form it
> is
> > being built in is enforcing Java 17.  This is ensuring the Groovy
> compiler
> > can work.  There is not current obvious path/option to build Groovy with
> > Java 21 so we likely should focus on removal of any groovy things
> requiring
> > compilation in our build.  Groovy runs on Java 21 just fine but the
> eclipse
> > compiler as yet does not.  Either way we're on track to get those tests
> and
> > groovy based source out.
> >
> > Still good progress on getting to NiFi 2.0 and Java 21 (which officially
> > releases Sep 19th).
> >
> > Thanks
> >
> > On Fri, Sep 15, 2023 at 4:03 PM Joe Witt  wrote:
> >
> > > Team
> > >
> > > Long story short we are now able to build and run NiFi main/2.x line on
> > > Java 21!
> > >
> > > We've made a lot of progress this week on top of all the great work
> > > already done for getting to NiFi 2.0 but specifically to this thread of
> > > making NiFi 2.0 be based on Java 21.
> > >
> > > There are a couple PRs outstanding but once they land I'll put up a PR
> > for
> > > this commit [1] and we will be building with Java 21-ea on the main
> line.
> > > The full clean build with all tests and all profiles I could find is
> now
> > > working locally and is also now running in my fork before I put up the
> PR
> > > [2], [3].  NiFi also runs on Java 21.  We did have to make a bunch of
> > > updates to all things Groovy and we're reducing/eliminating a lot of
> > pieces
> > > that are poorly maintained or need fundamentally different
> > implementations
> > > in Java 21.  The toolkit is likely to be removed it seems and we can
> > later
> > > introduce back specific pieces if/as needed but designed better/more
> > easily
> > > maintained.
> > >
> > > What would be ideal is we land a couple more key pieces like ensuring
> > > every deprecated component is actually removed and ensuring the
> > flow.xml.gz
> > > is entirely gone.  Then we kick out a NiFi 2.0 M1 release for people to
> > > work with.
> > >
> > > [1]
> > >
> >
> https://github.com/joewitt/nifi/commit/c3d16d949f8153441c64a1f98fd641cf80178f43
> > > [2] https://github.com/joewitt/nifi/actions/runs/6203527626
> > > [3] https://github.com/joewitt/nifi/actions/runs/6203527625
> > >
> > > Thanks
> > > Joe
> > >
> > > On Fri, Sep 15, 2023 at 1:06 PM Pierre Villard <
> > > pierre.villard...@gmail.com> wrote:
> > >
> > >> Templates are already out. Flow.xml being removed should be reviewed
> > soon
> > >> (PR was rebased today). I'm working on the removal of variables. I
> hope
> > to
> > >> get a PR for this in the next few days.
> > >>
> > >> Le ven. 15 sept. 2023, 19:22, Joe Witt  a écrit :
> > >>
> > >> > Timeline - we remain in full blitz mode to get things ready for 2.0.
> > No
> > >> > clear ETA but we need to be getting it out soon.  At least a
> milestone
> > >> > release of it for people to work with.  There is a big change needed
> > to
> > >> get
> > >> > rid of the flow.xml.gz in favor of the json form and that is in
> > >> progress.
> > >> > I am not sure offhand whether templates got the boot yet.
> > >> >
> > >> > Latest fun is wrestling our rather messy situation with Groovy in
> the
> > >> build
> > >> > as that seems not ready for Java 21 generally.
> > >> >
> > >> &g

Re: [discuss] nifi 2.0 and Java 21…

[Joe Witt]

Caveat to that - the build works right now as the Java release version in
the pom is set to 17.  So we're building/running Java 21 but the form it is
being built in is enforcing Java 17.  This is ensuring the Groovy compiler
can work.  There is not current obvious path/option to build Groovy with
Java 21 so we likely should focus on removal of any groovy things requiring
compilation in our build.  Groovy runs on Java 21 just fine but the eclipse
compiler as yet does not.  Either way we're on track to get those tests and
groovy based source out.

Still good progress on getting to NiFi 2.0 and Java 21 (which officially
releases Sep 19th).

Thanks

On Fri, Sep 15, 2023 at 4:03 PM Joe Witt  wrote:

> Team
>
> Long story short we are now able to build and run NiFi main/2.x line on
> Java 21!
>
> We've made a lot of progress this week on top of all the great work
> already done for getting to NiFi 2.0 but specifically to this thread of
> making NiFi 2.0 be based on Java 21.
>
> There are a couple PRs outstanding but once they land I'll put up a PR for
> this commit [1] and we will be building with Java 21-ea on the main line.
> The full clean build with all tests and all profiles I could find is now
> working locally and is also now running in my fork before I put up the PR
> [2], [3].  NiFi also runs on Java 21.  We did have to make a bunch of
> updates to all things Groovy and we're reducing/eliminating a lot of pieces
> that are poorly maintained or need fundamentally different implementations
> in Java 21.  The toolkit is likely to be removed it seems and we can later
> introduce back specific pieces if/as needed but designed better/more easily
> maintained.
>
> What would be ideal is we land a couple more key pieces like ensuring
> every deprecated component is actually removed and ensuring the flow.xml.gz
> is entirely gone.  Then we kick out a NiFi 2.0 M1 release for people to
> work with.
>
> [1]
> https://github.com/joewitt/nifi/commit/c3d16d949f8153441c64a1f98fd641cf80178f43
> [2] https://github.com/joewitt/nifi/actions/runs/6203527626
> [3] https://github.com/joewitt/nifi/actions/runs/6203527625
>
> Thanks
> Joe
>
> On Fri, Sep 15, 2023 at 1:06 PM Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>> Templates are already out. Flow.xml being removed should be reviewed soon
>> (PR was rebased today). I'm working on the removal of variables. I hope to
>> get a PR for this in the next few days.
>>
>> Le ven. 15 sept. 2023, 19:22, Joe Witt  a écrit :
>>
>> > Timeline - we remain in full blitz mode to get things ready for 2.0.  No
>> > clear ETA but we need to be getting it out soon.  At least a milestone
>> > release of it for people to work with.  There is a big change needed to
>> get
>> > rid of the flow.xml.gz in favor of the json form and that is in
>> progress.
>> > I am not sure offhand whether templates got the boot yet.
>> >
>> > Latest fun is wrestling our rather messy situation with Groovy in the
>> build
>> > as that seems not ready for Java 21 generally.
>> >
>> >
>> >
>> > On Fri, Sep 15, 2023 at 10:19 AM Ryan Hendrickson <
>> > ryan.andrew.hendrick...@gmail.com> wrote:
>> >
>> > > I think NiFi 2.x going to Java 21 for all the reasons outlined makes a
>> > lot
>> > > of sense.
>> > >
>> > > Is there a timeline for 2.x?
>> > >
>> > > On Mon, Sep 11, 2023 at 5:00 AM Pierre Villard <
>> > > pierre.villard...@gmail.com>
>> > > wrote:
>> > >
>> > > > Thanks Joe, it makes total sense and I agree that the ones that
>> would
>> > > > likely be slow at adopting Java 21 would not go to NiFi 2.0 super
>> > quickly
>> > > > anyway. Being able to bring the latest and greatest in NiFi is great
>> > and
>> > > > given all of the features announced in Java 21, I imagine a lot of
>> > > projects
>> > > > we depend on will be doing the same.
>> > > >
>> > > > Le jeu. 7 sept. 2023 à 19:36, Joe Witt  a
>> écrit :
>> > > >
>> > > > > Pierre
>> > > > >
>> > > > > A few concerns you raised so want to address my view on each:
>> > > > >
>> > > > > Will users be able to adopt Java 21 fast enough?
>> > > > > I share Brandon's view on that in terms of their adoption
>> timeline.
>> > It
>> > > > > will likely align well with NiFi 2.0 itself in my estimation.
>> > > > >

Re: [discuss] nifi 2.0 and Java 21…

[Joe Witt]

Team

Long story short we are now able to build and run NiFi main/2.x line on
Java 21!

We've made a lot of progress this week on top of all the great work already
done for getting to NiFi 2.0 but specifically to this thread of making NiFi
2.0 be based on Java 21.

There are a couple PRs outstanding but once they land I'll put up a PR for
this commit [1] and we will be building with Java 21-ea on the main line.
The full clean build with all tests and all profiles I could find is now
working locally and is also now running in my fork before I put up the PR
[2], [3].  NiFi also runs on Java 21.  We did have to make a bunch of
updates to all things Groovy and we're reducing/eliminating a lot of pieces
that are poorly maintained or need fundamentally different implementations
in Java 21.  The toolkit is likely to be removed it seems and we can later
introduce back specific pieces if/as needed but designed better/more easily
maintained.

What would be ideal is we land a couple more key pieces like ensuring every
deprecated component is actually removed and ensuring the flow.xml.gz is
entirely gone.  Then we kick out a NiFi 2.0 M1 release for people to work
with.

[1]
https://github.com/joewitt/nifi/commit/c3d16d949f8153441c64a1f98fd641cf80178f43
[2] https://github.com/joewitt/nifi/actions/runs/6203527626
[3] https://github.com/joewitt/nifi/actions/runs/6203527625

Thanks
Joe

On Fri, Sep 15, 2023 at 1:06 PM Pierre Villard 
wrote:

> Templates are already out. Flow.xml being removed should be reviewed soon
> (PR was rebased today). I'm working on the removal of variables. I hope to
> get a PR for this in the next few days.
>
> Le ven. 15 sept. 2023, 19:22, Joe Witt  a écrit :
>
> > Timeline - we remain in full blitz mode to get things ready for 2.0.  No
> > clear ETA but we need to be getting it out soon.  At least a milestone
> > release of it for people to work with.  There is a big change needed to
> get
> > rid of the flow.xml.gz in favor of the json form and that is in progress.
> > I am not sure offhand whether templates got the boot yet.
> >
> > Latest fun is wrestling our rather messy situation with Groovy in the
> build
> > as that seems not ready for Java 21 generally.
> >
> >
> >
> > On Fri, Sep 15, 2023 at 10:19 AM Ryan Hendrickson <
> > ryan.andrew.hendrick...@gmail.com> wrote:
> >
> > > I think NiFi 2.x going to Java 21 for all the reasons outlined makes a
> > lot
> > > of sense.
> > >
> > > Is there a timeline for 2.x?
> > >
> > > On Mon, Sep 11, 2023 at 5:00 AM Pierre Villard <
> > > pierre.villard...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Joe, it makes total sense and I agree that the ones that would
> > > > likely be slow at adopting Java 21 would not go to NiFi 2.0 super
> > quickly
> > > > anyway. Being able to bring the latest and greatest in NiFi is great
> > and
> > > > given all of the features announced in Java 21, I imagine a lot of
> > > projects
> > > > we depend on will be doing the same.
> > > >
> > > > Le jeu. 7 sept. 2023 à 19:36, Joe Witt  a écrit
> :
> > > >
> > > > > Pierre
> > > > >
> > > > > A few concerns you raised so want to address my view on each:
> > > > >
> > > > > Will users be able to adopt Java 21 fast enough?
> > > > > I share Brandon's view on that in terms of their adoption timeline.
> > It
> > > > > will likely align well with NiFi 2.0 itself in my estimation.
> > > > >
> > > > > Will this delay NiFi 2.0?
> > > > > If it would then I'd not be supportive.  I don't think we need to
> > > bother
> > > > > with adopting any of the features now.  What I would like us to
> have
> > is
> > > > the
> > > > > option to adopt them as we progress.  We should get 2.0 done asap
> and
> > > if
> > > > > this added delay then I'd be way less interested in this idea.
> > > > >
> > > > > Feature benefits of 21 and what will that bring?
> > > > > Mark spoke well to the key one that stood out to me which was the
> new
> > > > > threading model available.  It would be awfully nice to leverage
> that
> > > for
> > > > > the efficiency it represents and especially if it can reduce some
> of
> > > our
> > > > > heap usage which is valuable in cloud/shared compute contexts.
> > > > >
> > > > > Performance benefits of Java 21?
> > > > > It appears from some ana

Re: [discuss] nifi 2.0 and Java 21…

[Joe Witt]

Timeline - we remain in full blitz mode to get things ready for 2.0.  No
clear ETA but we need to be getting it out soon.  At least a milestone
release of it for people to work with.  There is a big change needed to get
rid of the flow.xml.gz in favor of the json form and that is in progress.
I am not sure offhand whether templates got the boot yet.

Latest fun is wrestling our rather messy situation with Groovy in the build
as that seems not ready for Java 21 generally.



On Fri, Sep 15, 2023 at 10:19 AM Ryan Hendrickson <
ryan.andrew.hendrick...@gmail.com> wrote:

> I think NiFi 2.x going to Java 21 for all the reasons outlined makes a lot
> of sense.
>
> Is there a timeline for 2.x?
>
> On Mon, Sep 11, 2023 at 5:00 AM Pierre Villard <
> pierre.villard...@gmail.com>
> wrote:
>
> > Thanks Joe, it makes total sense and I agree that the ones that would
> > likely be slow at adopting Java 21 would not go to NiFi 2.0 super quickly
> > anyway. Being able to bring the latest and greatest in NiFi is great and
> > given all of the features announced in Java 21, I imagine a lot of
> projects
> > we depend on will be doing the same.
> >
> > Le jeu. 7 sept. 2023 à 19:36, Joe Witt  a écrit :
> >
> > > Pierre
> > >
> > > A few concerns you raised so want to address my view on each:
> > >
> > > Will users be able to adopt Java 21 fast enough?
> > > I share Brandon's view on that in terms of their adoption timeline.  It
> > > will likely align well with NiFi 2.0 itself in my estimation.
> > >
> > > Will this delay NiFi 2.0?
> > > If it would then I'd not be supportive.  I don't think we need to
> bother
> > > with adopting any of the features now.  What I would like us to have is
> > the
> > > option to adopt them as we progress.  We should get 2.0 done asap and
> if
> > > this added delay then I'd be way less interested in this idea.
> > >
> > > Feature benefits of 21 and what will that bring?
> > > Mark spoke well to the key one that stood out to me which was the new
> > > threading model available.  It would be awfully nice to leverage that
> for
> > > the efficiency it represents and especially if it can reduce some of
> our
> > > heap usage which is valuable in cloud/shared compute contexts.
> > >
> > > Performance benefits of Java 21?
> > > It appears from some analysis found with googling that Java 21 brings
> out
> > > of the box 4-5% performance increases generally.  Not amazing but
> useful.
> > >
> > > User experience otherwise with Java 21?
> > > I believe it would be consistent with Java 17 for their point of view
> in
> > > terms of install/config/etc..
> > >
> > > My motivation for this is fairly pure honestly.  Since we're setting
> our
> > > new minimum bar that lives for as long as the 2.x release line lives
> I'd
> > > like to set it at the current LTS available when we ship that line as
> > well.
> > >
> > > Thanks
> > >
> > >
> > >
> > > On Thu, Sep 7, 2023 at 4:22 AM Brandon DeVries <
> > brandon.devr...@gmail.com>
> > > wrote:
> > >
> > > > +1 to requiring java 21. Starting off as "up to date" as possibly
> > makes a
> > > > lot of sense, and some of the new features seem especially relevant
> to
> > > NiFi.
> > > >
> > > > I definitely understand the concerns about organizations being
> willing
> > /
> > > > able to approve Java 21... But those same organizations might also be
> > > > hesitant to move to NiFi 2.0. We will continue to support java 17 &
> > NiFi
> > > > 1.x for some time, so hopefully those groups will have the time they
> > need
> > > > to get approvals, do evaluations, and upgrade.
> > > >
> > > > Brandon
> > > > 
> > > > From: Pierre Villard 
> > > > Sent: Thursday, September 7, 2023 6:15:58 AM
> > > > To: dev@nifi.apache.org 
> > > > Subject: Re: [discuss] nifi 2.0 and Java 21…
> > > >
> > > > Hi all,
> > > >
> > > > I share the concerns raised by Chris regarding how quickly users of
> > NiFi
> > > > will be able to adopt Java 21.
> > > >
> > > > While I'm definitely in favor of using the latest and greatest,
> > > especially
> > > > when it brings to the table such significant features, we need to be
> > > >

Re: [DISCUSS] Deprecate TLS Toolkit for Removal?

[Joe Witt]

David

Fully supportive.

I'll take it a step further and indicate that I am now an advocate of
removing anything we have in our build related to running things or tests
written in Groovy.  We should maintain the Groovy support for scripted
components but we should eliminate or replace any tools/tests written in
Groovy.  I didn't quite realize how unique the setup seems to be as it
relates to codehaus/vs apache groovy and the versions associated with
integrating with Maven vs versions of Java/etc.. This is adding brittleness
and build complexity we don't need.  To that end I'd even suggest we
consider dropping even the encrypt-config on main so we can work to a 2.x
M1 release then get that rewritten with tests in pure Java for a NiFi 2.x
release.

Thanks

On Wed, Sep 13, 2023 at 12:58 PM Kevin Doran  wrote:

>  I agree that tis toolkit can probably be removed in 2.0, and I would add
> that tinycert.org provides another option for teams that need to setup
> dev/test environments with trusted certificates.
>
> Thanks,
> Kevin
>
> On Sep 13, 2023 at 11:46:19, David Handermann  >
> wrote:
>
> > Hi Isha,
> >
> > Thanks for the helpful reply. I agree that the TLS Toolkit is most
> > convenient for development and lab deployments, and that's where we
> > should be able to provide some documentation for alternatives. The
> > existing Secure Cluster Walkthrough is a helpful reference for TLS
> > Toolkit usage, so if we updated that to provide similar guidance using
> > other tools, that would be useful.
> >
> > Getting everything right with TLS can be challenging, but when it
> > comes to project maintenance, it seems better to focus on core
> > capabilities and not maintain the TLS Toolkit if the primary use case
> > is for non-production deployments.
> >
> > The encrypt-config command is a different question, but a very good
> > question. It includes functionality that is very specific to NiFi, and
> > it is also in need of refactoring. The threat model for containerized
> > deployments may be somewhat different than running directly on
> > physical hardware. With the need to support various approaches,
> > however, some type of configuration encryption remains a relevant
> > concern.
> >
> > Regards,
> > David Handermann
> >
> > On Wed, Sep 13, 2023 at 10:19 AM Isha Lamboo
> >  wrote:
> >
> >
> > Hi David,
> >
> >
> > My primary use for the TLS toolkit is for lab deployments, mostly during
> > in-house trainings. I will miss the convenience of having a full set of
> > keystores and truststores ready to go with a single command, but then
> > again, a few commands in a script should replicate this well enough,
> > without the need for maintaining the toolkit.
> >
> >
> > I see no obstacles to adopting NiFi 2.0 if the TLS toolkit is phased out,
> > from the perspective of the deployments I manage.
> >
> >
> > On a side note: How relevant is the encrypt config part of the toolkit
> > still in a mostly containerized world?
> >
> >
> > Regards,
> >
> >
> > Isha
> >
> >
> > -Oorspronkelijk bericht-
> >
> > Van: David Handermann 
> >
> > Verzonden: woensdag 13 september 2023 15:16
> >
> > Aan: dev@nifi.apache.org
> >
> > Onderwerp: [DISCUSS] Deprecate TLS Toolkit for Removal?
> >
> >
> > Team,
> >
> >
> > The TLS Toolkit provides a number of useful features for securing NiFi
> > server communication, but it also presents several maintenance concerns.
> In
> > light of other available tools, I am raising the question of removing the
> > TLS Toolkit from the repository as part of NiFi 2.0 technical debt
> > reduction.
> >
> >
> > With the addition of automatic self-signed certificate generation in NiFi
> > 1.14.0, the TLS Toolkit is much less relevant to standalone or
> development
> > deployments. The validity period of the automatic certificate is limited,
> > but it provides a method of getting started without any need for the TLS
> > Toolkit.
> >
> >
> > On the other end of the spectrum, orchestrated deployments of Kubernetes
> > can take advantage of cert-manager [1] for declarative and configurable
> > certificate generation and distribution.
> >
> >
> > Cluster deployments on physical hardware or virtual machines may have
> > organization-specific Certificate Authorities, which require certificate
> > request processing external to NiFi itself. For this scenario,
> documenting
> > several standard OpenSSL commands may help to describe converting between
> > PEM and PKCS12 files for common use cases.
> >
> >
> > Back to standalone deployments, Let's Encrypt provides automated
> > certificate provisioning with many tools for managing updates. For a
> > self-signed solution, the mkcert [2] tool is a popular and simple option
> > that works across modern operating systems.
> >
> >
> > With these alternatives, the use cases for TLS Toolkit seem limited.
> >
> > The Toolkit code is not well-structured, and includes several modes that
> > involve custom configuration files with a Jetty web server. There are a
> > 

Re: new PackageFlowFile processor

[Joe Witt]

Ok.  Certainly simplifies it but likely makes it applicable to larger
flowfiles only.  The format is meant to allow appending and result in large
sets of flowfiles for io efficiency and specifically for storage as the
small files/tons of files thing can cause poor performance pretty quickly
(10s of thousands of files in a single directory).

But maybe that simplicity is fine and we just link to the MergeContent
packaging option if users need more.

On Fri, Sep 8, 2023 at 7:06 AM Michael Moser  wrote:

> I was thinking 1 file in -> 1 flowfile-v3 file out.  No merging of multiple
> files at all.  Probably change the mime.type attribute.  It might not even
> have any config properties at all if we only support flowfile-v3 and not v1
> or v2.
>
> -- Mike
>
>
> On Fri, Sep 8, 2023 at 9:56 AM Joe Witt  wrote:
>
> > Mike
> >
> > In user terms this makes sense to me. Id only bother with v3 or whatever
> is
> > latest. We want to dump the old code. And if there are seriously older
> > versions v1,v2 then nifi 1.x can be used.
> >
> > The challenge is that you end up needing some of the same complexity in
> > implementation and config of merge content i think. What did you have in
> > mind for that?
> >
> > Thanks
> >
> > On Fri, Sep 8, 2023 at 6:53 AM Michael Moser  wrote:
> >
> > > Devs,
> > >
> > > I can't find if this was suggested before, so here goes.  With the
> demise
> > > of PostHTTP in NiFi 2.0, the recommended alternative is to
> MergeContent 1
> > > file into FlowFile-v3 format then InvokeHTTP.  What does the community
> > > think about supporting a new PackageFlowFile processor that is simple
> to
> > > configure (compared to MergeContent!) and simply packages flowfile
> > > attributes + content into a FlowFile-v[1,2,3] format?  This would also
> > > offer a simple way to export flowfiles from NiFi that could later be
> > > re-ingested and recovered using UnpackContent.  I don't want to submit
> a
> > PR
> > > for such a processor without first asking the community whether this
> > would
> > > be acceptable.
> > >
> > > Thanks,
> > > -- Mike
> > >
> >
>

Re: new PackageFlowFile processor

[Joe Witt]

Mike

In user terms this makes sense to me. Id only bother with v3 or whatever is
latest. We want to dump the old code. And if there are seriously older
versions v1,v2 then nifi 1.x can be used.

The challenge is that you end up needing some of the same complexity in
implementation and config of merge content i think. What did you have in
mind for that?

Thanks

On Fri, Sep 8, 2023 at 6:53 AM Michael Moser  wrote:

> Devs,
>
> I can't find if this was suggested before, so here goes.  With the demise
> of PostHTTP in NiFi 2.0, the recommended alternative is to MergeContent 1
> file into FlowFile-v3 format then InvokeHTTP.  What does the community
> think about supporting a new PackageFlowFile processor that is simple to
> configure (compared to MergeContent!) and simply packages flowfile
> attributes + content into a FlowFile-v[1,2,3] format?  This would also
> offer a simple way to export flowfiles from NiFi that could later be
> re-ingested and recovered using UnpackContent.  I don't want to submit a PR
> for such a processor without first asking the community whether this would
> be acceptable.
>
> Thanks,
> -- Mike
>

Re: [discuss] nifi 2.0 and Java 21…

[Joe Witt]

Pierre

A few concerns you raised so want to address my view on each:

Will users be able to adopt Java 21 fast enough?
I share Brandon's view on that in terms of their adoption timeline.  It
will likely align well with NiFi 2.0 itself in my estimation.

Will this delay NiFi 2.0?
If it would then I'd not be supportive.  I don't think we need to bother
with adopting any of the features now.  What I would like us to have is the
option to adopt them as we progress.  We should get 2.0 done asap and if
this added delay then I'd be way less interested in this idea.

Feature benefits of 21 and what will that bring?
Mark spoke well to the key one that stood out to me which was the new
threading model available.  It would be awfully nice to leverage that for
the efficiency it represents and especially if it can reduce some of our
heap usage which is valuable in cloud/shared compute contexts.

Performance benefits of Java 21?
It appears from some analysis found with googling that Java 21 brings out
of the box 4-5% performance increases generally.  Not amazing but useful.

User experience otherwise with Java 21?
I believe it would be consistent with Java 17 for their point of view in
terms of install/config/etc..

My motivation for this is fairly pure honestly.  Since we're setting our
new minimum bar that lives for as long as the 2.x release line lives I'd
like to set it at the current LTS available when we ship that line as well.

Thanks



On Thu, Sep 7, 2023 at 4:22 AM Brandon DeVries 
wrote:

> +1 to requiring java 21. Starting off as "up to date" as possibly makes a
> lot of sense, and some of the new features seem especially relevant to NiFi.
>
> I definitely understand the concerns about organizations being willing /
> able to approve Java 21... But those same organizations might also be
> hesitant to move to NiFi 2.0. We will continue to support java 17 & NiFi
> 1.x for some time, so hopefully those groups will have the time they need
> to get approvals, do evaluations, and upgrade.
>
> Brandon
> 
> From: Pierre Villard 
> Sent: Thursday, September 7, 2023 6:15:58 AM
> To: dev@nifi.apache.org 
> Subject: Re: [discuss] nifi 2.0 and Java 21…
>
> Hi all,
>
> I share the concerns raised by Chris regarding how quickly users of NiFi
> will be able to adopt Java 21.
>
> While I'm definitely in favor of using the latest and greatest, especially
> when it brings to the table such significant features, we need to be
> careful as it may significantly delay the adoption of NiFi 2.0 in big
> companies where deploying Java 21 will take time. I acknowledge that going
> from Java 8 to Java 17 is certainly the same effort as going from Java 8 to
> Java 21 but how quickly security-sensitive environments will adopt a new
> release of Java that is completely new?
>
> In addition to that, it sounds like we would add a significant rework of
> the framework in NiFi 2.0 assuming we adopt Java 21 as the minimum version.
> Do we think this is going to significantly delay the first release of NiFi
> 2.0? We still have work to do but adding this on top could delay the
> release, no?
>
> Finally, the features that Java 21 are bringing sound super interesting in
> the context of NiFi but do we already have an idea of what it will improve?
> is it the user experience, and if so, how will it change? is it the
> performance, and if so, do we have an idea of how things will improve?
>
> Thanks,
> Pierre
>
> Le mer. 6 sept. 2023 à 23:07, Chris Sampson
>  a écrit :
>
> > Yeah, I understand the need to move to 21 as a minimum to take advantage
> of
> > its features. Hopefully the wider java ecosystem won't be an issue in the
> > short term.
> >
> > I just wanted the discussion to be clear about this being a change to the
> > Java baseline/minimum for NiFi 2.0.
> >
> > It's a +1 from me.
> >
> > On Wed, 6 Sept 2023, 19:01 Joe Witt,  wrote:
> >
> > > Chris
> > >
> > > My suggestion is rooted in making Java 21 the minimum of the NiFi 2.0
> > > line.  It would not work on Java 17.  The reason for this is so that we
> > can
> > > leverage the longest duration of a given LTS line while also benefiting
> > > from the language improvements that affords.  Maintaining compatibility
> > > with future versions we generally have to do.  But whatever the minimum
> > > version we accept dictates which language features we can leverage.  So
> > if
> > > it is 17 then we can't leverage anything from the 21 line for example.
> > >
> > > GIven the nature and timelines of LTS I don't really think there is the
> > > same burn in logic that we'd have all known in the past before the
> > &g

Re: [discuss] nifi 2.0 and Java 21…

[Joe Witt]

Chris

My suggestion is rooted in making Java 21 the minimum of the NiFi 2.0
line.  It would not work on Java 17.  The reason for this is so that we can
leverage the longest duration of a given LTS line while also benefiting
from the language improvements that affords.  Maintaining compatibility
with future versions we generally have to do.  But whatever the minimum
version we accept dictates which language features we can leverage.  So if
it is 17 then we can't leverage anything from the 21 line for example.

GIven the nature and timelines of LTS I don't really think there is the
same burn in logic that we'd have all known in the past before the
LTS/STS/etc.. release constructs existed.

Thanks

On Wed, Sep 6, 2023 at 10:53 AM Chris Sampson
 wrote:

> To be clear, is the discussion one of making Java 21 the minimum
> requirement for NiFi 2.0.0, or rather NiFi 2.x be compatible with Java 21,
> while retaining Java 17 as a minimum?
>
> If we moved straight to a Java 21 requirement, will we run into
> compatibility issues that delay initial NiFi 2 release? Will the move to
> Java 21 mean some organisations delay their migration to NiFi 2 through not
> wanting to move to the latest Java LTS version until it's had a time for
> "settling" through security/bug patches, etc.? And are either of these
> sufficient concern to pause Java 21 becoming the requirement, as we may
> then need to extend NiFi 1.x maintenance for longer into the future?
>
> Generally, I'm in favour of moving to "latest and greatest", particularly
> for LTS versions of technologies, but the rate of Java version adoption
> across the community gives me pause.
>
> I certainly see the advantage of new Java features for NiFi in Java 21,
> such as the already mentioned virtual threads.
>
> On Wed, 6 Sept 2023, 18:34 Mike Thomsen,  wrote:
>
> > +1 100%
> >
> > On Wed, Sep 6, 2023 at 11:48 AM Adam Taft  wrote:
> >
> > > Yes, please. +1 Exactly what Mark said. Virtual threads have potential
> to
> > > be extremely impactful to applications like NiFi.
> > >
> > > /Adam
> > >
> > > On Wed, Sep 6, 2023 at 7:26 AM Mark Payne 
> wrote:
> > >
> > > > Thanks for bringing his up, Joe.
> > > >
> > > > I would definitely be a +1. I think the new virtual thread concept
> > would
> > > > have great impact on us.
> > > > It would allow us to significantly simplify our scheduling logic,
> which
> > > > would help with code maintainability
> > > > but would also make configuration simpler. This is one of the most
> > > > difficult things for users to configure,
> > > > and I would very much welcome the ability to simplify this. It would
> > > > likely also yield better off-heap memory
> > > > utilization by reducing the number of native threads necessary.
> > > >
> > > > Thanks
> > > > -Mark
> > > >
> > > >
> > > > > On Sep 6, 2023, at 10:20 AM, Joe Witt  wrote:
> > > > >
> > > > > Team
> > > > >
> > > > > Thought it might be worth relighting this thread with Java 21 GA
> > > > imminent.
> > > > > Given the timing we should give consideration to having Java 21 as
> > the
> > > > > basis for nifi 2.x to buy maximum time with LTS alignment.  There
> are
> > > > also
> > > > > a couple interesting language features we can likely take advantage
> > of.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Thanks
> > > > > Joe
> > > > >
> > > > > On Wed, Jun 21, 2023 at 6:21 AM David Handermann <
> > > > > exceptionfact...@apache.org> wrote:
> > > > >
> > > > >> Hi Dirk,
> > > > >>
> > > > >> Thanks for summarizing your findings in the referenced Jira
> issues.
> > It
> > > > >> sounds like subsequent discussion of Nashorn support may be better
> > on
> > > a
> > > > new
> > > > >> thread.
> > > > >>
> > > > >> The Spring 6 and Jetty 11 upgrades are going to require
> significant
> > > > work.
> > > > >> One incremental step in that direction was making Java 17 the
> > minimum
> > > > >> version, and upgrading to Jetty 10 should also help move things
> > > forward.
> > > > >>
> > > > >> Although compiling NiFi modules with a reference t

Re: [discuss] nifi 2.0 and Java 21…

[Joe Witt]

Team

Thought it might be worth relighting this thread with Java 21 GA imminent.
Given the timing we should give consideration to having Java 21 as the
basis for nifi 2.x to buy maximum time with LTS alignment.  There are also
a couple interesting language features we can likely take advantage of.

What do you think?

Thanks
Joe

On Wed, Jun 21, 2023 at 6:21 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Hi Dirk,
>
> Thanks for summarizing your findings in the referenced Jira issues. It
> sounds like subsequent discussion of Nashorn support may be better on a new
> thread.
>
> The Spring 6 and Jetty 11 upgrades are going to require significant work.
> One incremental step in that direction was making Java 17 the minimum
> version, and upgrading to Jetty 10 should also help move things forward.
>
> Although compiling NiFi modules with a reference to the standalone Nashorn
> library may introduce issues, there should be other options for referencing
> the library at runtime. That requires custom class loading, which some
> Processors support, so that seems like the general direction to go.
>
> If you have additional findings, feel free to start a new developer list
> thread and that may gather additional feedback.
>
> Regards,
> David Handermann
>
> On Wed, Jun 21, 2023 at 12:17 AM Dirk Arends 
> wrote:
>
> > Since initially raising concerns about the move to Java 17 losing
> Nashorn,
> >  I have been investigating the suggestion to use Nashorn as a standalone
> > package as potential easier alternative to GraalVM. [1]
> >
> > While making some progress, a number of issues have been encountered
> which
> > I haven't been able to resolve as yet. More details are included in
> > relevant JIRA tickets, but summarising:
> >
> > - Building NiFi with a recent Nashorn dependency leads to errors
> > "Unsupported class file major version 61" [2]
> > - Building NiFi using Java 17 highlights issues with the current Jetty
> > version, which I believe would require an upgrade from 9.4.51 to 11.0.15
> > [3]
> > - Jetty 11 then requires an upgrade of the Spring Framework version 5 to
> 6.
> > [4]
> >
> > The current steps to remove references to "Javascript" as a preinstalled
> > scripting language [5] are understandable, but it does seem there is
> still
> > quite a bit to do before Nashorn or another external scripting engine
> could
> > be used.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-11700: Java 17 Nashorn
> > standalone support
> > [2] https://issues.apache.org/jira/browse/NIFI-11701: Support building
> > with
> > version 61 class files
> > [3] https://issues.apache.org/jira/browse/NIFI-11702: Upgrade Jetty to
> > version 11
> > [4] https://issues.apache.org/jira/browse/NIFI-11703: Upgrade Spring
> > Framework to version 6
> > [5] https://issues.apache.org/jira/browse/NIFI-11713: Remove Deprecated
> > ECMAScript Support
> >
> > Regards,
> > Dirk Arends
> >
>

Re: Help with NiFi transfer speeds

[Joe Witt]

Jordan

Some of the references to Gbps are in reference to network terms where
rates are truly measured in bits and on a base of 1000 where as some
reference Gbps referring to actual throughput/sizes and usually those are
in Bytes and on a base of 1024.  I'm making some assumptions on what is
meant from the writeup.

Rate/Capabilities:
- NIC on the machine NiFi runs on: 10Gbps = 1.22GB/sec theoretical max.
Factor in some overhead and such and good to assume around 1GB/sec is safe
with 1.2GB being feasible in some cases (like you mentioned).
- 100 GB files then in theory should take close to 2 mins to transfer
assuming the NIC and network is otherwise available.

We'd need to know more about the type of disks involved.  They're network
mapped which is important but sounds like the network is pretty solid
design.  The type of disks then do matter at least in terms of the rate
they're meant to sustain.  As you noted though you can fix some variables
and achieve 1+GB/sec copies so likely these disks are quite fast.

Prob would need to better understand the setup such as where nifi is, vs
where the storage is, vs where the data comes from vs where it goes to.
Just to consider the various factors at play there.

All that said...

"usually we see transfer speeds of more than 1gbps and files move over
> pretty quickly. In the last two months5MB/sec"

Nothing in NiFi changed but the rates went from around 1GB to 5MB/sec?  Is
there any sign of packet loss between the systems?  That can certainly do
it.  Ways to check this vary but an old school technique is to send various
sized ICMP messages and just monitor their health.  In a LAN setup anything
less than 100% would be interesting.

The *most common* culprit for sudden performance changes like this though
have to do with virus/malware scanning software being put on machines like
this.  It is certainly a good idea but the performance penalty for some of
these tools is severe.  They engage as every bit of data is read/written in
many cases and do their analysis.  These are often turned on by others
sometimes without the admins of a given system even knowing.

Any chance that can be a factor here?

Thanks

On Tue, Sep 5, 2023 at 3:17 AM POMEROY, Jordan 7094
 wrote:

> Hello
>
> Wonder if you can help me with an issue that is being reported to us via
> our users.
>
> We use NiFi to transfer evidential phone data across our network and place
> it on to our networked file storage server. Most files are in excess of
> 100gb, and we have allocated the server for this work a 10gb NIC. Although
> we do not expect the transfers to reach these speeds, usually we see
> transfer speeds of more than 1gbps and files move over pretty quickly.
>
> In the last two months, our users have reported to us that it is taking
> significantly longer for files to reach the destination. Upon looking, the
> transfer speeds have dropped to roughly 5mbps.
>
> I have asked our Network team to look at firewall rules to see if there is
> a bottleneck on the Network anywhere, but there isn't. we've also migrated
> the virtual server which NiFi is configured on and changed network switch
> ports etc to see if this has made any difference, but nothing.
>
> To test the network even further, I mapped from the NiFi virtual server to
> the file storage server and transferred over a large file to test speeds.
> This moved over at 1.2gbps with no issues at all.
>
> The "get file" and "put file" commands have not been touched since they
> were originally configured, and we are currently using NiFi version 1.5.0.
> The server is running OS Windows Server 2022.
>
> Any help you can provide as to what we can check next to get this working
> would be very much appreciated.
>
> Thanks,
> Jordan.
>
> Jordan Pomeroy
> Server and Storage Engineer
> Infrastructure team
> Herts, Beds and Cambs Police ICT Department
> The best way to contact me is via Teams or Email as I have no fixed desk.
> jordan.pome...@herts.pnn.police.uk jordan.pome...@herts.pnn.police.uk>
>
> 
> Internet e-mail is not to be treated as a secure means of communication.
> Hertfordshire Constabulary monitors all internet e-mail activity and
> content. This communication is confidential and intended for the
> addressee(s) only. Please notify the sender if you have received this in
> error. Unauthorised use or disclosure of the contents may be unlawful.
> Opinions expressed in this document may not be official policy.
>
> For more details please see Hertfordshire Constabulary Privacy Policy <
> https://www.herts.police.uk/hyg/fpnbch/privacy-notice/>
>

Re: Cannot create connectionpoolablefactory communication link faliure

[Joe Witt]

Hello

You may want to share more details about the controller service/processors
being configured and their values.  And then also show what is resulting in
the nifi-app.log file when you try to start these components.

Thanks

On Mon, Sep 4, 2023 at 7:56 AM Bandana Kumari 
wrote:

>
> Hello There,
> I am using Apachenifi for first time while connecting with remote mysql
> database from a ubuntu server getting error :
> Cannot create connectionpoolablefactory communication link faliure.
> I have done everything from myside but unable to connect .
>
> Please help me to resolve this issue if possible, can we arrange a call
> for this.
>
> Thanks & Regards
> Bandana Kumari
> DBA ,Gurgaon
> Goolean Technologies  Limited.
>

Re: [DISCUSS] MiNiFi C++ 0.15.0 release

[Joe Witt]

+1 a lot of nice things in there!

On Thu, Aug 24, 2023 at 7:52 AM Martin Zink  wrote:

> Hi community,
>
> I'd like to initiate a discussion about the next release of MiNiFi C++. The
> last release was more than four months ago, and there have been many new
> features, bug fixes and stability improvements committed to the development
> branch
> since then: 71 tickets closed, over 96 commits as of today.
>
> I would be happy to take on RM duties for this release.
>
> Notable features and improvements since the 0.14.0 release:
>
> - ConsumeWindowsEventLog can work from log files
> - ConsumeWindowsEventLog resolve Security/UserID attribute
> - TLS v1.3 support
> - PutS3Object multipart upload support
> - Use systemd service management on Linux
> - Add ProcessContext::getStateManager to Lua/Python
> - Reworked GetTCP to be more inline with ListenTCP
> - SSL support for Prometheus reporter
> - Documentation improvements
> - Multiarch docker support
> - RFC3339 parsing with expression language
> - Reworked Minifi controller
> - gcc-13 support
>
> - Fix for waking up prematurely after processor yields
> - Fix system certificate store usage in SSLContextService on Linux
> - Fix inconsistent naming in C2 machineArch
> - Fix default CA path for S3 on CentOS
> - Removed CronScheduler locale requirements
>
> We've upgraded our third party dependencies (notable mentions)
> - Replaced LibreSSL with OpenSSL 3.1.1
> - Upgraded RocksDB to v8.1.1
> - Upgraded LibCurl to v8.1.0
> - Upgraded CivetWeb to v1.16
> - Upgraded OpenCV to v4.7.0
> - Upgraded GoogleCloud SDK to v2.10.1
> - Upgraded Azure SDK to v12.7.0
>
> The core API is still not mature enough to be able to commit to it, so
> in line with previous discussions I suggest releasing it as 0.15.0.
>
> Do you agree it is time for a new release? Are there any blockers that we
> should definitely include in this release?
>
> Thanks,
> Martin
>

Re: [VOTE] Release Apache NiFi 1.23.2 (RC1)

[Joe Witt]

+1 (binding)
Verified the sigs/hashes

Note; Link for release helper is broken now as the name is different just.
need to update email template.

Build/tested on Java 8

Verified code change was there.

Thanks David.  Thanks All.



On Tue, Aug 22, 2023 at 9:46 AM Mark Payne  wrote:

> +1 (binding)
>
> - Verified hashes
> - Performed full build with contrib-check using Java 17:
> nifi-1.23.2 $ mvn -v
> Apache Maven 3.9.4 (dfbb324ad4a7c8fb0bf182e6d91b0ae20e3d2dd9)
> Maven home: /usr/local/Cellar/maven/3.9.4/libexec
> Java version: 17.0.8, vendor: Homebrew, runtime:
> /usr/local/Cellar/openjdk@17/17.0.8/libexec/openjdk.jdk/Contents/Home
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "11.7.8", arch: "x86_64", family: “mac"
>
> - Performed some quick verification that the app was functional
> - Tested to ensure that the identified Jira NIFI-11971 was addressed.
>
> Thanks for handling the RM duties again, David!
>
> -Mark
>
>
> > On Aug 22, 2023, at 11:00 AM, Matt Burgess  wrote:
> >
> > +1 (binding)
> >
> > Ran through release helper and my usual flows verifying no
> > regressions. Thanks for RM'ing David!
> >
> > On Mon, Aug 21, 2023 at 4:16 PM David Handermann
> >  wrote:
> >>
> >> Team,
> >>
> >> NOTE: This is a shortened vote window given the narrow scope of changes.
> >>
> >> I am pleased to be calling this vote for the source release of Apache
> >> NiFi 1.23.2.
> >>
> >> Please review the following guide for how to verify a release candidate
> build:
> >>
> >>
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >>
> >> The source being voted on the and the convenience binaries are
> >> available on the Apache Distribution Repository:
> >>
> >> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.2
> >>
> >> The build artifacts are available on the Apache Nexus Repository:
> >>
> >> https://repository.apache.org/content/repositories/orgapachenifi-1235
> >>
> >> Git Tag: nifi-1.23.2-RC1
> >> Git Commit ID: dec043e590f26ba2f3594f4f297dcd2b7e565ab7
> >> GitHub Commit Link:
> >>
> https://github.com/apache/nifi/commit/dec043e590f26ba2f3594f4f297dcd2b7e565ab7
> >>
> >> Checksums of nifi-1.23.2-source-release.zip
> >>
> >> SHA256: cfaa374c383a316de6d9230a0786ebe7dd97c03f8903b36365faa27bf6424fac
> >> SHA512:
> a63fa624532aa3f1769013d67499caf48b9771bc3515ffca4daba2dd379722c3b25bea4519b56bc647d4e05db122e7a41519a52a703a0179b431fa881dadb4cf
> >>
> >> Release artifacts are signed with the following key:
> >>
> >> https://people.apache.org/keys/committer/exceptionfactory.asc
> >>
> >> KEYS file is available on the Apache Distribution Repository:
> >>
> >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> >>
> >> Issues resolved in this version: 1
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353568
> >>
> >> Release note highlights can be found on the project wiki:
> >>
> >>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.2
> >>
> >> The vote will be open for 24 hours.
> >>
> >> Please download the release candidate and evaluate the necessary items
> >> including checking hashes, signatures, build from source, and test.
> >> Then please vote:
> >>
> >> [] +1 Release this package as nifi-1.23.2
> >> [] +0 no opinion
> >> [] -1 Do not release this package because...
>
>

Re: [DISCUSS] Release Apache NiFi 1.23.2 for NIFI-11971

[Joe Witt]

+1

On Mon, Aug 21, 2023 at 6:58 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> Thanks for the work on release for NiFi 1.23.1 last week.
>
> As it happens, another important bug surfaced last week with Content
> Repository handling of empty files, described in NIFI-11971 [1]. This
> bug was introduced due to changes for NIFI-11670 [2] released in NiFi
> 1.23.0.
>
> This issue is serious enough that we should release a new patch
> version 1.23.2, limited to this particular fix.
>
> I plan to prepare a release candidate build and send out a
> short-duration vote thread later today.
>
> Regards,
> David Handermann
>
> [1] https://issues.apache.org/jira/browse/NIFI-11971
> [2] https://issues.apache.org/jira/browse/NIFI-11670
>

Re: [VOTE] Release Apache NiFi 1.23.1 (RC1)

[Joe Witt]

+1 (binding)

Built on Java 8, build size good, hashes/sigs, it runs.

Good stuff all!

Thanks for the RM DH.

Thanks

On Wed, Aug 16, 2023 at 10:13 AM Steven Matison 
wrote:

> +1 non binding
>
> Verified source release checksums
> Built w/ java version 17 *Apache Maven 3.9.2*
> Checks completed
>
> NiFi 1.23.1 started and imported test flow
>
> Thanks David and Team for another release!!
>
> On Tue, Aug 15, 2023 at 2:07 PM David Handermann <
> exceptionfact...@apache.org> wrote:
>
> > Team,
> >
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > 1.23.1.
> >
> > Please review the following guide for how to verify a release candidate
> > build:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >
> > The source being voted upon and the convenience binaries are available on
> > the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.1/
> >
> > The build artifacts are available on the Apache Nexus Repository:
> >
> > https://repository.apache.org/content/repositories/orgapachenifi-1234
> >
> > Git Tag: nifi-1.23.1-RC1
> > Git Commit ID: f739e9dd2d476b3c0df5a806ff1dffd24be52916
> > GitHub Commit Link:
> >
> >
> https://github.com/apache/nifi/commit/f739e9dd2d476b3c0df5a806ff1dffd24be52916
> >
> > Checksums of nifi-1.23.1-source-release.zip:
> >
> > SHA256: 01f5f67a9f9703232f6fe260f6c1da73f3e3a0764b8e8464f915cfad168278e6
> > SHA512:
> >
> >
> f8a010ad5a5dd1f71fe04e5d2bf1c6637d2d0a8a7c580a0ff4dbd76c12c2e5ab4ac43e1f5314f9fca85cebe1606bd5e7ae0a8b62f577ddc68696ebd0155d
> >
> > Release artifacts are signed with the following key:
> >
> > https://people.apache.org/keys/committer/exceptionfactory.asc
> >
> > KEYS file available here:
> >
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 51 issues were closed/resolved for this release:
> >
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353541
> >
> > Release note highlights can be found here:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.1
> >
> > The vote will be open for 72 hours.
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test. Then
> > please vote:
> >
> > [ ] +1 Release this package as nifi-1.23.1
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
>

Re: Processing JSON Files Embedded with Base64 Encoded Files

[Joe Witt]

Also depending on the setup you could first use
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apache.nifi.processors.standard.EncodeContent/index.html
in decode mode then do normal json handling.

Thanks

On Tue, Aug 15, 2023 at 7:42 AM Yolanda Davis 
wrote:

> Hello Chris,
>
> Depending on the structure of your Json file, you may be able to achieve
> what you need using processors such as JoltTransformJSON
> <
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.22.0/org.apache.nifi.processors.standard.JoltTransformJSON/index.html
> >
> paired with NiFi's expression language which supports base64
> <
> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64encode
> >
> encoding and decoding.  JoltTransform is very useful if you're looking to
> convert or transform incoming JSON data to another json structure. Also
> EvaluateJsonPath
> <
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.22.0/org.apache.nifi.processors.standard.EvaluateJsonPath/index.html
> >
> may
> be useful as well if you're looking to extract data first from json then
> potentially decode/encode
>
> Hope this helps!
>
> -yolanda
>
> On Tue, Aug 15, 2023 at 9:45 AM Reid, Chris 
> wrote:
>
> > Hello.
> >
> > I understand that there's currently no Apache NiFi processor that can
> > process JSON files embedded with Base64 encoded files.
> >
> > I'd like to know if it's possible to develop a custom Apache NiFi
> > processor to process JSON files embedded with Base64 files?
> >
> > Thank you.
> >
> > Chris
> >
>
>
> --
> --
> yolanda.m.da...@gmail.com
> @YolandaMDavis
>

Re: Java 17 features in 2.x

[Joe Witt]

Matt

Yeah that is a good summary of it.  We want to help the community move NiFi
forward along with Java's advances and these major line breaks are the best
opportunity to do so.  We're in effect making a bet on a given line or set
of Java lines for quite some time so we should take advantage of that.

Joe

On Mon, Aug 7, 2023 at 5:51 PM Matt Burgess  wrote:

> That's a fair point and I think represents the way we want to go
> forward. If I understand correctly, you're saying bug fixes meant for
> both lines shouldn't need/use Java 17 features but new capabilities
> for 2.0 should encourage the use of Java 17 features when prudent?
>
> Thanks,
> Matt
>
> On Mon, Aug 7, 2023 at 8:33 PM Joe Witt  wrote:
> >
> > The views shared thus far are certainly reasonable but I do want to add
> > a different take.
> >
> > The reason we want to do major releases from time to time is so that we
> can
> > take advantage of leaps in the language and frameworks we rely on.  To
> that
> > end it would seem unfortunate to not start aggressively taking advantage
> of
> > that.  In particular we've been held to Java 8 standards for at least 7
> > years.  I would advocate we allow and even encourage usage of Java 17
> > features/syntax to help move forward.
> >
> > The point about backporting is important and I agree the 'easiest' way is
> > if changes made to main are backportable.  Then again we don't really
> need
> > everything to be backportable and for sure that will start happening less
> > and less.  If we're talking about 'bug fixes' then it probably makes
> sense
> > to prefer for now they avoid Java 17 assuming a given fix should target
> > both 2.x and 1.x lines.  But if we're talking about new features or even
> > improvements I think we should be free to move on to Java 17
> > idioms/benefits.  If a contrib does this then it just won't go to the 1.x
> > line.  This atrophy is natural/ok I think and lets us give the 2.x line
> the
> > attention/growth it deserves.
> >
> > Thanks
> >
> > On Mon, Aug 7, 2023 at 2:19 PM David Handermann <
> exceptionfact...@apache.org>
> > wrote:
> >
> > > Mike,
> > >
> > > Thanks for raising the question. Following Matt's comments, I recommend
> > > minimizing use of Java 17 features to make it easier to backport
> changes
> > > for now.
> > >
> > > Incremental adjustments can be done when backporting, but it requires
> the
> > > author and reviewer to pay careful attention since the GitHub automated
> > > builds for the main branch run on Java 17.
> > >
> > > As Matt recommended, the alternative is to provide separate pull
> requests
> > > for the main and support branches.
> > >
> > > We already have a few Java 11 and 17 references on the main branch for
> > > things like List.of(), and most of these are easy to adjust when
> > > backporting, but they do require careful attention.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Mon, Aug 7, 2023 at 4:04 PM Matt Burgess 
> wrote:
> > >
> > > > In my opinion that's ok, but I think it would be helpful if a PR is
> > > > going to be backported to support/nifi-1.x that the PR author
> provides
> > > > two PRs, one against main with Java 17 features and one against
> > > > support/nifi-1.x with Java 8 features. That being said, allowing Java
> > > > 17 features may make maintenance tougher while there's an active 1.x
> > > > branch. Maybe we should wait until we only support NiFi 2.x?
> > > >
> > > > On Mon, Aug 7, 2023 at 4:59 PM Mike Thomsen 
> > > > wrote:
> > > > >
> > > > > Since we're standardizing on 17, we're free and clear to use Java
> 17
> > > > > features, right?
> > > >
> > >
>

Re: Java 17 features in 2.x

[Joe Witt]

The views shared thus far are certainly reasonable but I do want to add
a different take.

The reason we want to do major releases from time to time is so that we can
take advantage of leaps in the language and frameworks we rely on.  To that
end it would seem unfortunate to not start aggressively taking advantage of
that.  In particular we've been held to Java 8 standards for at least 7
years.  I would advocate we allow and even encourage usage of Java 17
features/syntax to help move forward.

The point about backporting is important and I agree the 'easiest' way is
if changes made to main are backportable.  Then again we don't really need
everything to be backportable and for sure that will start happening less
and less.  If we're talking about 'bug fixes' then it probably makes sense
to prefer for now they avoid Java 17 assuming a given fix should target
both 2.x and 1.x lines.  But if we're talking about new features or even
improvements I think we should be free to move on to Java 17
idioms/benefits.  If a contrib does this then it just won't go to the 1.x
line.  This atrophy is natural/ok I think and lets us give the 2.x line the
attention/growth it deserves.

Thanks

On Mon, Aug 7, 2023 at 2:19 PM David Handermann 
wrote:

> Mike,
>
> Thanks for raising the question. Following Matt's comments, I recommend
> minimizing use of Java 17 features to make it easier to backport changes
> for now.
>
> Incremental adjustments can be done when backporting, but it requires the
> author and reviewer to pay careful attention since the GitHub automated
> builds for the main branch run on Java 17.
>
> As Matt recommended, the alternative is to provide separate pull requests
> for the main and support branches.
>
> We already have a few Java 11 and 17 references on the main branch for
> things like List.of(), and most of these are easy to adjust when
> backporting, but they do require careful attention.
>
> Regards,
> David Handermann
>
> On Mon, Aug 7, 2023 at 4:04 PM Matt Burgess  wrote:
>
> > In my opinion that's ok, but I think it would be helpful if a PR is
> > going to be backported to support/nifi-1.x that the PR author provides
> > two PRs, one against main with Java 17 features and one against
> > support/nifi-1.x with Java 8 features. That being said, allowing Java
> > 17 features may make maintenance tougher while there's an active 1.x
> > branch. Maybe we should wait until we only support NiFi 2.x?
> >
> > On Mon, Aug 7, 2023 at 4:59 PM Mike Thomsen 
> > wrote:
> > >
> > > Since we're standardizing on 17, we're free and clear to use Java 17
> > > features, right?
> >
>

Re: NiFi 2.0 - QuestDB

[Joe Witt]

Hello

Here are a couple examples where it seems like we definitely have some work
to do for the QuestDB bits.  Right now it is blocking full clean builds on
Linux machines in my case though works on OSX.

See:
https://issues.apache.org/jira/browse/NIFI-11896
https://issues.apache.org/jira/browse/NIFI-11897

Thanks

On Wed, Jul 19, 2023 at 8:15 AM Mark Payne  wrote:

> Sounds good. Thanks Bence.
>
> > On Jul 19, 2023, at 11:07 AM, Simon Bence 
> wrote:
> >
> > Thanks for the feedback from everyone!
> >
> > As I understand the intention is supported and with some preparation
> (covering the cases mentioned) it can be done. I will raise some PR in the
> foreseeable future to target these questions.
> >
> > Regards,
> > Bence
> >
> >> On 2023. Jul 19., at 16:01, David Handermann <
> exceptionfact...@apache.org> wrote:
> >>
> >> Thanks for the suggestion and additional background Bence, that is very
> >> helpful in evaluating the default inclusion approach.
> >>
> >> I agree with Joe's concern about handling potential corruption. We have
> >> recently reduced dependency on the H2 file-backed database driver so
> that
> >> it is now limited to Flow Configuration History. Based on experience
> there,
> >> NiFi can fail to start when the database file is corrupted, which is not
> >> ideal. We should look into improving that behavior, allowing NiFi to
> start
> >> and saving off the corrupted file instead of failing to start. If we go
> >> forward with QuestDB as the default strategy for status history, we
> should
> >> build in the resilient approach as a prerequisite to enabling it in the
> >> default configuration.
> >>
> >> Regards,
> >> David Handermann
> >>
> >> On Wed, Jul 19, 2023 at 8:31 AM Simon Bence 
> >> wrote:
> >>
> >>> Thanks for the quick feedback!
> >>>
> >>> Joe: your concerns are relevant, let me provide some details:
> >>>
> >>> The database uses some disk space, determined by the number of
> components
> >>> and the number of covered days. During adding it I was checking for
> time
> >>> usage and however I don’t have the numbers any more, the usage seemed
> >>> reasonable. I can do a bit of testing and bring some numbers to improve
> >>> confidence with it. Additionally the necessary disk space is limited:
> we
> >>> have rollover handling capability, which limits the amount of stored
> data,
> >>> to the target number plus one days. This is due to the limitations of
> >>> QuestDB with partitioning: at the time of development the smallest
> >>> partition strategy way day based if I remember correctly so the unit of
> >>> deletion was the partition just shifted out from the threshold. (Now it
> >>> looks to be the hour based partitoning which might worth the effort to
> >>> upgrade to)
> >>>
> >>> The current rollover deletes all the data older than the threshold,
> but I
> >>> am thinking on adding a new implementation which keeps some aggregated
> >>> information about the components. That of course needs some more space,
> >>> again: depending on the number of components and the time.
> >>>
> >>> In case the disk is full, we have no way to push down metrics to the
> >>> database and currently there is no fallback strategy for it. A
> possible way
> >>> would be to temporarily keep the data in memory (similar to the
> >>> VolatileComponentStatusRepository in that regard) but I am not
> convinced
> >>> that if a node with resources close to the limitations it would be
> >>> necessarily a good strategy to write data into the memory instead of
> the
> >>> disk. This is something to consider.
> >>>
> >>> If the database becomes corrupted than we loose the status information.
> >>> This I think is true for most of the persisted storage however I would
> >>> think if the database files are not changed by using external tools
> there
> >>> is an insignificant chance for this. Fallback strategies might be added
> >>> (like if NiFi considers the database corrupted, it might start a new
> one)
> >>> but even without this I think the QuestDB based solution has its merits
> >>> compared to the in memory storage.
> >>>
> >>> Manual intervention should not be needed. Currently in order to use
> this
> >>> capability, the co

Re: [VOTE] Release Apache NiFi 1.23.0 (RC3)

[Joe Witt]

+1 (binding)

sigs/hashes
Full clean build with basic flow up and running
JDK 17

On Wed, Jul 26, 2023 at 5:37 PM Emilio Setiadarma <
emiliosetiada...@gmail.com> wrote:

> +1 (non-binding)
>
> Verified signature and SHA256/512 hashes of the release.
> OS: MacOS Monterrey 12.6.5
> Java version: OpenJDK version 17.0.7
> Ran the latest build, booted up NiFi with no issues and started up a simple
> NiFi flow
>
> Thanks David!
>
> On Wed, Jul 26, 2023 at 2:10 PM Matt Burgess  wrote:
>
> > +1 (binding)
> >
> > Ran through release helper, verified H2 migration on Registry,
> > ExtractRecordSchema, various other flows to ensure no regressions.
> >
> > Thanks for RM'ing David!
> >
> > On Tue, Jul 25, 2023 at 5:33 PM David Handermann
> >  wrote:
> > >
> > > Team,
> > >
> > > I am pleased to be calling this vote for the source release of Apache
> > NiFi
> > > 1.23.0.
> > >
> > > Please review the following guide for how to verify a release candidate
> > > build:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> > >
> > > The source being voted upon and the convenience binaries are available
> on
> > > the Apache Distribution Repository:
> > >
> > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.0/
> > >
> > > The build artifacts are available on the Apache Nexus Repository:
> > >
> > > https://repository.apache.org/content/repositories/orgapachenifi-1233
> > >
> > > Git Tag: nifi-1.23.0-RC3
> > > Git Commit ID: 27a690a30a6ae77c217a47ac0601e85970777ca2
> > > GitHub Commit Link:
> > >
> >
> https://github.com/apache/nifi/commit/27a690a30a6ae77c217a47ac0601e85970777ca2
> > >
> > > Checksums of nifi-1.23.0-source-release.zip:
> > >
> > > SHA256:
> 39c97d89804b005cc995c56a87a3df6f68c44ee42114dffe756bbac90a3593cf
> > > SHA512:
> > >
> >
> f256ca731a67435e9883626931abc58f28cda9deb6a7d0a84ed97b78104e43b3b638ee2297d79f92bf5a1e19f62cc78e0a886fa0094593ab34b21d658c59eadd
> > >
> > > Release artifacts are signed with the following key:
> > >
> > > https://people.apache.org/keys/committer/exceptionfactory.asc
> > >
> > > KEYS file available here:
> > >
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > 132 issues were closed/resolved for this release:
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353320
> > >
> > > Release note highlights can be found here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.0
> > >
> > > The vote will be open for 72 hours.
> > >
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build from source, and test.
> Then
> > > please vote:
> > >
> > > [ ] +1 Release this package as nifi-1.23.0
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> >
>

Re: [DISCUSS] NiFi 1.23.0 Release Timeline

[Joe Witt]

Sounds good David.  I'll have your Spring PR merged shortly if it
isnt already.

Any chance you want to take the RM duties for this run?  I can do it but my
Internet access isn't awesome right now so it could be pretty slow.

Created the RM JIRA https://issues.apache.org/jira/browse/NIFI-11832

Thanks

On Wed, Jul 19, 2023 at 11:21 AM Mark Payne  wrote:

> Thanks David. I agree it makes sense to put together a 1.23 release. I did
> just push up a PR for another bug that will be important to get into this
> release: NIFI-11783 [1].
>
> Here, it can cause the framework to not properly handle retries for some
> processors.
>
> Thanks
> -Mark
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-11783
>
> On 2023/07/19 14:45:33 David Handermann wrote:
> > Team,
> >
> > Although we just released NiFi 1.22.0 last month, we already have over
> 100
> > Jira issues resolved for 1.23.0 [1]. Many of these are dependency
> upgrades,
> > some of which address flagged vulnerabilities, although most do not
> appear
> > to have a direct impact on NiFi usage patterns. There are also some
> > important bug fixes, and the new ExcelReader.
> >
> > There are two open issues related to Parameter Context handling [2] and
> > QueryRecord behavior [3] which should be included before going forward,
> but
> > otherwise it seems like we have sufficient changes to warrant another
> > release on the version 1 branch.
> >
> > Anything else worth highlighting?
> >
> > Regards,
> > David Handermann
> >
> > [1]
> >
> https://issues.apache.org/jira/issues/?jql=project+%3D+NIFI+AND+fixVersion+%3D+1.23.0
> > [2] https://issues.apache.org/jira/browse/NIFI-11814
> > [3] https://issues.apache.org/jira/browse/NIFI-11825
> >
>

Re: NiFi 2.0 - QuestDB

[Joe Witt]

Bence

It no doubt is superior to the current default in terms feature/benefit.
We need to just address any fault scenarios.  I personally dont care how
much space it uses.  All good there.  I am only focused on fault scenarios
and recovery of those.  This isnt data we need to protect like
content/metadata of the flow itself.  We can and should be more ruthless in
automating recovery.

Thanks

On Wed, Jul 19, 2023 at 9:31 AM Simon Bence 
wrote:

> Thanks for the quick feedback!
>
> Joe: your concerns are relevant, let me provide some details:
>
> The database uses some disk space, determined by the number of components
> and the number of covered days. During adding it I was checking for time
> usage and however I don’t have the numbers any more, the usage seemed
> reasonable. I can do a bit of testing and bring some numbers to improve
> confidence with it. Additionally the necessary disk space is limited: we
> have rollover handling capability, which limits the amount of stored data,
> to the target number plus one days. This is due to the limitations of
> QuestDB with partitioning: at the time of development the smallest
> partition strategy way day based if I remember correctly so the unit of
> deletion was the partition just shifted out from the threshold. (Now it
> looks to be the hour based partitoning which might worth the effort to
> upgrade to)
>
> The current rollover deletes all the data older than the threshold, but I
> am thinking on adding a new implementation which keeps some aggregated
> information about the components. That of course needs some more space,
> again: depending on the number of components and the time.
>
> In case the disk is full, we have no way to push down metrics to the
> database and currently there is no fallback strategy for it. A possible way
> would be to temporarily keep the data in memory (similar to the
> VolatileComponentStatusRepository in that regard) but I am not convinced
> that if a node with resources close to the limitations it would be
> necessarily a good strategy to write data into the memory instead of the
> disk. This is something to consider.
>
> If the database becomes corrupted than we loose the status information.
> This I think is true for most of the persisted storage however I would
> think if the database files are not changed by using external tools there
> is an insignificant chance for this. Fallback strategies might be added
> (like if NiFi considers the database corrupted, it might start a new one)
> but even without this I think the QuestDB based solution has its merits
> compared to the in memory storage.
>
> Manual intervention should not be needed. Currently in order to use this
> capability, the configuration must be changed but if we would make this the
> default, it should work without any additional interaction.
>
> Regards,
> Bence
>
> > On 2023. Jul 19., at 14:57, Joe Witt  wrote:
> >
> > Agree functionally
> >
> > How does this handle disk usage?   Any manual intervention needed?  What
> if
> > the disk is full where it writes?  What if the db somehow becomes
> > corrupted?
> >
> > Id like to ensure this thing is zero ops as much as possible such that in
> > error conditions it resets and gets going again.
> >
> > Thanks
> >
> > On Wed, Jul 19, 2023 at 8:55 AM Pierre Villard <
> pierre.villard...@gmail.com>
> > wrote:
> >
> >> I do think this provides great value. The possibility to get access to
> >> status history of the components and at system level across restart is a
> >> great improvement for NiFi troubleshooting. It also gives the ability to
> >> store this information for a longer period of time. I'm definitely in
> favor
> >> of making this the default starting with NiFi 2.0.
> >>
> >> Le mer. 19 juil. 2023 à 13:49, Simon Bence  a
> >> écrit :
> >>
> >>> Hi Community,
> >>>
> >>> I was thinking if it would make sense to set the QuestDB as default for
> >>> status history backend in 2.0? It is there for a while and I would
> >> consider
> >>> it as a step forward so the new major version might be a good time for
> >> the
> >>> wider audience. It comes with less memory usage for bigger flows, the
> >>> possibility of checking status information when the node is not running
> >> or
> >>> restarted so I think it worth consideration. Any insight or improvement
> >>> point is appreciated, thanks!
> >>>
> >>> Regards,
> >>> Bence
> >>
>
>

Re: NiFi 2.0 - QuestDB

[Joe Witt]

Agree functionally

How does this handle disk usage?   Any manual intervention needed?  What if
the disk is full where it writes?  What if the db somehow becomes
corrupted?

Id like to ensure this thing is zero ops as much as possible such that in
error conditions it resets and gets going again.

Thanks

On Wed, Jul 19, 2023 at 8:55 AM Pierre Villard 
wrote:

> I do think this provides great value. The possibility to get access to
> status history of the components and at system level across restart is a
> great improvement for NiFi troubleshooting. It also gives the ability to
> store this information for a longer period of time. I'm definitely in favor
> of making this the default starting with NiFi 2.0.
>
> Le mer. 19 juil. 2023 à 13:49, Simon Bence  a
> écrit :
>
> > Hi Community,
> >
> > I was thinking if it would make sense to set the QuestDB as default for
> > status history backend in 2.0? It is there for a while and I would
> consider
> > it as a step forward so the new major version might be a good time for
> the
> > wider audience. It comes with less memory usage for bigger flows, the
> > possibility of checking status information when the node is not running
> or
> > restarted so I think it worth consideration. Any insight or improvement
> > point is appreciated, thanks!
> >
> > Regards,
> > Bence
>

board report for July 2023

[Joe Witt]

Team,

Here is the submitted board report.  Thanks again to all for
continuing to make this a great project and useful tool for people and
organizations all over the world!

## Description:
The mission of NiFi is the creation and maintenance of software related to
providing an easy to use, powerful, and reliable system to process and
distribute data.

Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
integrate with and leverage the command and control of NiFi. There are both
Java and C++ implementations.

Apache NiFi Registry is a centralized registry for key configuration items
including flow versions, assets, and extensions for Apache NiFi and Apache
MiNiFi.

Apache NiFi Nar Maven Plugin is a release artifact used for supporting the
NiFi classloader isolation model.

Apache NiFi Flow Design System is a theme-able set of high quality UI
components and utilities for use across the various Apache NiFi web
applications in order to provide a more consistent user experience.

## Project Status:
Current project status: Ongoing. High.
Issues for the board: None.

## Membership Data:
Apache NiFi was founded 2015-07-14 (9 years ago)
There are currently 63 committers and 34 PMC members in this project.
The Committer-to-PMC ratio is roughly 2:1.

Community changes, past quarter:
- Nándor Soma Abonyi was added to the PMC on 2023-06-25
- No new committers. Last addition was Ferenc Kis on 2023-03-23.

## Project Activity:
The NiFi community released NiFi 1.22.0 Jun 2023 and NiFi 1.21.0 in Apr 2023.
Each release provided new features, stability, and addressed reported
vulnerabilities within NiFi and dependencies.  Meanwhile the community is
rapidly approaching a NiFi 2.0 major release already containing more than 466
changes. The MiNiFiCPP project also released 0.14.0 in April and we've
released the latest NiFi Nar Maven Plugin version 1.5.1 in June.


## Community Health:
Community health remains strong and we see high activity across the various
channels of contribution and engagement including JIRA, Github, Slack, Mailing
Lists and so on.  Meetups are less common these days but we still see good
activity in conferences, blog posts, and more. We also continue to enjoy
excellent engagement for discussions, voting threads, and release candidate
evaluation.  Slack remains a super popular point of engagement and we've grown
again this quarter to more than 2700 people in our 'general' slack channel
alone. The dev mailing list activity was 20% higher this quarter than last
while the user list is less active.  Much of the user list activity has
moved to slack as we've reported for a while now but still the user list is
an effective tool for collaboration with more than 100 emails in this quarter.
Via JIRA, commits, and our issues list we still see a lot of activity and even
growth in activity this quarter.  Importantly our rate of PR activity remains
high with more than 350+ PRs in the quarter again but importantly we have more
closed than opened showing we're finally turning the corner in this regard.
Some of the activity levels and commit levels reflect that we're managing
actively both the 1.x and 2.x release lines now so we do anticipate many of
these productivity metrics will drop in the coming quarters as we get the 2.0
release out and stable.

[ANNOUNCE] New nifi PMC Member Nandor Soma Abonyi

[Joe Witt]

On behalf of the Apache NiFI PMC, I am very pleased to announce that Nandor
has accepted the PMC's invitation to join the Apache NiFi PMC. We greatly
appreciate the PR reviews and code contributions and release management
work of Nandor.

Please join us in congratulating and welcoming Nandor to the Apache NiFi
PMC.

Congratulations Nandor!

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.5.1

[Joe Witt]

+1 (binding) thanks!

On Thu, Jun 15, 2023 at 9:45 AM Matt Burgess  wrote:
>
> +1 (binding)
>
> Verified signatures and hashes, built with the system below, built
> NiFi NARs and verified the issue I was encountering is fixed. Thanks
> for RM'ing Nandor!
>
> Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39)
> Maven home: /Users/mburgess/.sdkman/candidates/maven/current
> Java version: 11.0.19, vendor: Eclipse Adoptium, runtime:
> /Users/mburgess/.sdkman/candidates/java/11.0.19-tem
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "13.4", arch: "aarch64", family: "mac"
>
> On Wed, Jun 14, 2023 at 6:55 PM Nandor Soma Abonyi
>  wrote:
> >
> > Hello Apache NiFi Community,
> >
> > I am pleased to be calling this vote for the source release of Apache NiFi 
> > NAR Maven Plugin 1.5.1.
> >
> > The source being voted upon, including signatures, digests, etc. can be 
> > found at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.5.1/
> >
> > A helpful reminder on how the release candidate verification process works:
> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+NAR+Maven+Plugin+release+candidate
> >
> > The Git tag is nifi-nar-maven-plugin-1.5.1-rc1
> > The Git commit ID is 39fc959426ea405df6360969b55ae2adad47e1aa
> > https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=39fc959426ea405df6360969b55ae2adad47e1aa
> >
> > Checksums of nifi-nar-maven-plugin-1.5.1-source-release.zip:
> > SHA256: 0ddc4efbfe504f9ed6477b8c572f2c6e5ba0c953e2e5b063bdfbd1f934eda6bf
> > SHA512: 
> > a7281c8a3769db37e3491f3a5e54533b5f26bdcad99f8adb1e5609f1de17309aefb3d49eb9231e75a814e42566525b9afe4a11bda2c4ab48e8bab5a298b72b24
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/nsabonyi.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 3 issues were closed/resolved for this release:
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353009
> >
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.5.1
> >
> > The vote will be open for 72 hours. Please download the release
> > candidate and evaluate the necessary items including checking hashes,
> > signatures, build from source, and test. Then please vote:
> >
> > [ ] +1 Release this package as nifi-nar-maven-plugin-1.5.1
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because …
> >

Fwd: [ANNOUNCE] Apache NiFi 1.22.0 release.

[Joe Witt]

-- Forwarded message -
From: Joe Witt 
Date: Sun, Jun 11, 2023 at 9:27 PM
Subject: [ANNOUNCE] Apache NiFi 1.22.0 release.
To: 


Hello

The Apache NiFi team would like to announce the release of Apache NiFi 1.22.0.

Apache NiFi is an easy to use, powerful, and reliable system to
process and distribute
data.  Apache NiFi was made for dataflow.  It supports highly
configurable directed graphs
of data routing, transformation, and system mediation logic.

More details on Apache NiFi can be found here:
https://nifi.apache.org/

The release artifacts can be downloaded from here:
https://nifi.apache.org/download.html

Maven artifacts have been made available and mirrored as per normal
ASF artifact processes.

Issues closed/resolved for this list can be found here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353069

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.22.0

Thank you
The Apache NiFi team

[RESULT][VOTE] Release Apache NiFi 1.22.0

[Joe Witt]

Apache NiFi Community,

I am pleased to announce that the 1.22.0 release of Apache NiFi passes with
8 +1 (binding) votes
8 +1 (non-binding) votes
0 0 votes
0 -1 votes

Thanks to all who helped make this release possible.

Here is the PMC vote thread:
https://lists.apache.org/thread/vbvfp3dvy7drg780mok72cvhb7xnv9pz

On Sun, Jun 11, 2023 at 8:38 PM Joe Witt  wrote:

> +1 binding
>
>
> On Sun, Jun 11, 2023 at 10:40 AM Kevin Doran  wrote:
>
>>  +1 binding.
>>
>> Ran through the steps in the Release Helper Guide to verify the
>> sigs/hashes, build, build docker, and run.
>>
>> Congrats to the community on this release, and thanks for RM’ing, Joe!
>>
>> Kevin
>>
>> On Jun 6, 2023 at 17:38:25, Joe Witt  wrote:
>>
>> > Hello,
>> >
>> > I am pleased to be calling this vote for the source release of Apache
>> NiFi
>> > 1.22.0.
>> >
>> > The source zip, including signatures, digests, etc. can be found at:
>> > https://repository.apache.org/content/repositories/orgapachenifi-1228
>> >
>> > The source being voted upon and the convenience binaries can be found
>> at:
>> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.22.0/
>> >
>> > A helpful reminder on how the release candidate verification process
>> works:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>> >
>> > The Git tag is nifi-1.22.0-RC1
>> > The Git commit ID is 71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
>> >
>> >
>> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
>> >
>> >
>> https://github.com/apache/nifi/commit/71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
>> >
>> > Checksums of nifi-1.22.0-source-release.zip:
>> > SHA256: d63a5f1db95160434670f112a0ee6e06fa141182bd0f12f0e58e3229991f3743
>> > SHA512:
>> >
>> >
>> 5f6da64e75a2d5446f1f274fe3de7e97290f5b916eabbc0491a99c6db33f74fbdea001b403044e75bc5c2183fb0500dfc143d0f4cba19d3511f866fff774d7f5
>> >
>> > Release artifacts are signed with the following key:
>> > https://people.apache.org/keys/committer/joewitt.asc
>> >
>> > KEYS file available here:
>> > https://dist.apache.org/repos/dist/release/nifi/KEYS
>> >
>> > 186 issues were closed/resolved for this release:
>> >
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353069
>> >
>> > Release note highlights can be found here:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.22.0
>> >
>> > The vote will be open for 72 hours.
>> > Please download the release candidate and evaluate the necessary items
>> > including checking hashes, signatures, build
>> > from source, and test. Then please vote:
>> >
>> > [ ] +1 Release this package as nifi-1.22.0
>> > [ ] +0 no opinion
>> > [ ] -1 Do not release this package because...
>> >
>>
>

Re: [VOTE] Release Apache NiFi 1.22.0 (RC1)

[Joe Witt]

+1 binding


On Sun, Jun 11, 2023 at 10:40 AM Kevin Doran  wrote:

>  +1 binding.
>
> Ran through the steps in the Release Helper Guide to verify the
> sigs/hashes, build, build docker, and run.
>
> Congrats to the community on this release, and thanks for RM’ing, Joe!
>
> Kevin
>
> On Jun 6, 2023 at 17:38:25, Joe Witt  wrote:
>
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > 1.22.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1228
> >
> > The source being voted upon and the convenience binaries can be found at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.22.0/
> >
> > A helpful reminder on how the release candidate verification process
> works:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >
> > The Git tag is nifi-1.22.0-RC1
> > The Git commit ID is 71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
> >
> >
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
> >
> >
> https://github.com/apache/nifi/commit/71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
> >
> > Checksums of nifi-1.22.0-source-release.zip:
> > SHA256: d63a5f1db95160434670f112a0ee6e06fa141182bd0f12f0e58e3229991f3743
> > SHA512:
> >
> >
> 5f6da64e75a2d5446f1f274fe3de7e97290f5b916eabbc0491a99c6db33f74fbdea001b403044e75bc5c2183fb0500dfc143d0f4cba19d3511f866fff774d7f5
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 186 issues were closed/resolved for this release:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353069
> >
> > Release note highlights can be found here:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.22.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test. Then please vote:
> >
> > [ ] +1 Release this package as nifi-1.22.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
>

[VOTE] Release Apache NiFi 1.22.0 (RC1)

[Joe Witt]

Hello,

I am pleased to be calling this vote for the source release of Apache NiFi
1.22.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1228

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.22.0/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.22.0-RC1
The Git commit ID is 71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
https://github.com/apache/nifi/commit/71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8

Checksums of nifi-1.22.0-source-release.zip:
SHA256: d63a5f1db95160434670f112a0ee6e06fa141182bd0f12f0e58e3229991f3743
SHA512:
5f6da64e75a2d5446f1f274fe3de7e97290f5b916eabbc0491a99c6db33f74fbdea001b403044e75bc5c2183fb0500dfc143d0f4cba19d3511f866fff774d7f5

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

186 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353069

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.22.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test. Then please vote:

[ ] +1 Release this package as nifi-1.22.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: [discuss] nifi 2.0 and Java 17...

[Joe Witt]

Team,

Looking like we will update the NiFi 2.0 goals to be based on Java 17
instead of 11.

The noted concern around Java removing Nashorn in 11/17 we will need to
identify an alternative plan for regardless and seems like David's proposal
would do the trick.

Let's give this thread a few more days and if still seems consensus is
present lets just assume lazy consensus and update the NiFi 2.0 goals and
make it happen.

Thanks

On Fri, Jun 2, 2023 at 8:46 AM David Handermann 
wrote:

> I agree that moving forward with Java 17 as the minimum for NiFi 2.0 is the
> best approach given the extended lifecycle of support for Java 17.
>
> With the removal of a number of legacy components, the current main branch
> is in a much better position to make Java 17 the minimum.
>
> The deprecation and removal of Nashorn from the JDK is worth highlighting,
> but it should not be a blocker for moving to Java 17. In this case, NiFi is
> reflecting the deprecation of Nashorn that already exists in Java 11. I
> have submitted a PR for NIFI-11630 to mark ECMAScript as deprecated for the
> support branch in subsequent version 1 releases.
>
> With that background, there is ongoing maintenance of the Nashorn engine as
> an external library, in addition to the GraalVM solution. However, this is
> a good opportunity to take a different approach to scripting engine
> integration. For maintenance and security purposes, it would be much better
> to reduce the number of bundled scripting engines and make it easier to
> bring your own. The current scripting bundle is around 100 MB, which is a
> lot of weight for languages and solutions that do not apply across the
> board. Providing an alternative that makes it easier to bring in a script
> engine library should provide a better solution for the future. This also
> should not be a blocker for an initial NiFi 2.0, but it is worth
> highlighting given the general interest in scripted components.
>
> Regards,
> David Handermann
>
> On Thu, Jun 1, 2023, 11:38 PM Dirk Arends 
> wrote:
>
> > Hi Joe,
> >
> > > Who will be seriously impacted by the removal of Java 11 and what was
> > your plan for upgrading to Java 17?
> >
> > >
> >
> > > thoughts?
> >
> > I would support moving the minimum Java version to 17 if it wasn’t for
> the
> > fact that Nashorn will be removed. Nashorn is already deprecated in Java
> > 11, and was then fully removed in Java 15. I understand GraalVM is
> intended
> > to be its successor, however this has not yet been integrated into NiFi
> and
> > I’ve been unable to satisfactorily integrate it myself to date.
> >
> > In my NiFi usage, I make heavy use of the JavaScript engine in
> > ExecuteScript and InvokeScriptedProcessor processors. To take advantage
> of
> > GraalVM supporting later ECMAScript versions than Nashorn, I have been
> > attempting to use GraalVM as the JavaScript Engine for NiFi with limited
> > success.
> >
> > Further details have been provided in JIRA ticket NIFI-6229 [1] and I’d
> > welcome any assistance in trying to finalise GraalVM support, but
> otherwise
> > I’d consider the loss of Nashorn to be a potential blocker to adopting
> Java
> > 17.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-6229
> >
> >
> > Regards,
> >
> > Dirk Arends
> >
> >
> >
> > On Thu, 1 Jun 2023 at 03:23, Joe Witt  wrote:
> >
> > > Team,
> > >
> > > We've discussed in the past having NiFi 2.0 move from Java 8 to Java 11
> > as
> > > the required minimum while also working on Java 17.
> > >
> > > As we move on in time though we are now 4 months (Sept) from. Java 11
> > > openJDK going end of support.  Meanwhile, the Spring 5.x line goes end
> of
> > > support as of next year and Spring 6.x requires Java 17.  Also Java 21
> > > comes out in Sept as well and is already the next LTS release.
> > >
> > > I am increasingly of the view that we should seriously discuss/consider
> > > moving to Java 17 as our basis for NiFi 2.0 as otherwise it basically
> > means
> > > we'll be forced to move to NiFi 3.0 quite quickly.
> > >
> > > We already know we can build and run on Java 17 so we're good there.
> > We'll
> > > soon want to do the same for Java 21 ... and the more 'old stuff' we
> hold
> > > on to the harder it is.
> > >
> > > Who will be seriously impacted by the removal of Java 11 and what was
> > your
> > > plan for upgrading to Java 17?
> > >
> > > thoughts?
> > >
> > > Thanks
> > >
> >
> >
> > --
> > Regards,
> >
> > --
> > Dirk Arends
> >
>

Re: [discuss] nifi 1.22 looking worthy at this point

[Joe Witt]

Team,

I plan to kick off the RC as soon as
https://issues.apache.org/jira/browse/NIFI-11593 lands.

Thanks

On Tue, May 30, 2023 at 6:21 PM Nissim Shiman 
wrote:

>  I see NIFI-11392 has already been taken care of.
>
> Thank you Joe and markap14
> On Tuesday, May 30, 2023 at 10:07:45 AM EDT, Joe Witt <
> joe.w...@gmail.com> wrote:
>
>  Thanks Michael and Nissim. Will flag both for follow up as part of 1.22.
>
> On Wed, May 24, 2023 at 12:35 PM Michael Moser  wrote:
>
> > Team,
> >
> > I observed something wonky with the UI in 1.22.0-SNAPSHOT (using the
> latest
> > support/nifi-1.x branch)  and I wrote NIFI-11560 [1] for it.  I would be
> > sad if we release 1.22.0 and then decide this is significant, so any
> extra
> > eyes on it beforehand would be appreciated.
> >
> > [1] - https://issues.apache.org/jira/browse/NIFI-11560
> >
> > Thanks,
> > -- Mike
> >
> >
> > On Wed, May 24, 2023 at 2:13 PM Nissim Shiman  >
> > wrote:
> >
> > >  Joe and team,
> > >
> > > Thank you for the coming release.
> > > I am hoping someone can review/commit
> > > https://issues.apache.org/jira/browse/NIFI-11392for this as well.
> > > This resolves an issue where nifi gets hung on shutdown.
> > > I have reviewed and tested this and it looks good.
> > >
> > > (and thank you very much markap14 and mosermw for previous recent
> > > reviews/merging of tickets I was working on)
> > > Thanks,
> > > Nissim Shiman
> > >
> > >
> > >On Wednesday, May 24, 2023 at 12:57:42 PM EDT, David Handermann <
> > > exceptionfact...@apache.org> wrote:
> > >
> > >  Joe,
> > >
> > > Thanks for initiating the discussion, I agree that the number of bug
> > fixes,
> > > dependency upgrades, and other improvements warrants a new 1.22.0
> release
> > > version. This would also incorporate some additional deprecations,
> which
> > > will be helpful as we continue making progress on NiFi 2.0 goals.
> > >
> > > The system tests have had sporadic failures recently, so it seems best
> to
> > > wrap up NIFI-11591 [1] before proceeding.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > [1] https://issues.apache.org/jira/browse/NIFI-11591
> > >
> > > On Wed, May 24, 2023 at 11:39 AM Joe Witt  wrote:
> > >
> > > > Team,
> > > >
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%201.22.0
> > > >
> > > > It appears NiFi 1.22 has enough weight to be worth kicking out a
> > release.
> > > > 120+ JIRAs included several performance enhancers, bug fixes, a few
> new
> > > > features like ModifyCompression, PutGCP, etc..
> > > >
> > > > Anything we're specifically waiting on?
> > > >
> > > > Thanks
> > > >
> > >
> >
>

[discuss] nifi 2.0 and Java 17...

[Joe Witt]

Team,

We've discussed in the past having NiFi 2.0 move from Java 8 to Java 11 as
the required minimum while also working on Java 17.

As we move on in time though we are now 4 months (Sept) from. Java 11
openJDK going end of support.  Meanwhile, the Spring 5.x line goes end of
support as of next year and Spring 6.x requires Java 17.  Also Java 21
comes out in Sept as well and is already the next LTS release.

I am increasingly of the view that we should seriously discuss/consider
moving to Java 17 as our basis for NiFi 2.0 as otherwise it basically means
we'll be forced to move to NiFi 3.0 quite quickly.

We already know we can build and run on Java 17 so we're good there.  We'll
soon want to do the same for Java 21 ... and the more 'old stuff' we hold
on to the harder it is.

Who will be seriously impacted by the removal of Java 11 and what was your
plan for upgrading to Java 17?

thoughts?

Thanks

Re: [discuss] nifi 1.22 looking worthy at this point

[Joe Witt]

Thanks Michael and Nissim. Will flag both for follow up as part of 1.22.

On Wed, May 24, 2023 at 12:35 PM Michael Moser  wrote:

> Team,
>
> I observed something wonky with the UI in 1.22.0-SNAPSHOT (using the latest
> support/nifi-1.x branch)  and I wrote NIFI-11560 [1] for it.  I would be
> sad if we release 1.22.0 and then decide this is significant, so any extra
> eyes on it beforehand would be appreciated.
>
> [1] - https://issues.apache.org/jira/browse/NIFI-11560
>
> Thanks,
> -- Mike
>
>
> On Wed, May 24, 2023 at 2:13 PM Nissim Shiman 
> wrote:
>
> >  Joe and team,
> >
> > Thank you for the coming release.
> > I am hoping someone can review/commit
> > https://issues.apache.org/jira/browse/NIFI-11392for this as well.
> > This resolves an issue where nifi gets hung on shutdown.
> > I have reviewed and tested this and it looks good.
> >
> > (and thank you very much markap14 and mosermw for previous recent
> > reviews/merging of tickets I was working on)
> > Thanks,
> > Nissim Shiman
> >
> >
> > On Wednesday, May 24, 2023 at 12:57:42 PM EDT, David Handermann <
> > exceptionfact...@apache.org> wrote:
> >
> >  Joe,
> >
> > Thanks for initiating the discussion, I agree that the number of bug
> fixes,
> > dependency upgrades, and other improvements warrants a new 1.22.0 release
> > version. This would also incorporate some additional deprecations, which
> > will be helpful as we continue making progress on NiFi 2.0 goals.
> >
> > The system tests have had sporadic failures recently, so it seems best to
> > wrap up NIFI-11591 [1] before proceeding.
> >
> > Regards,
> > David Handermann
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-11591
> >
> > On Wed, May 24, 2023 at 11:39 AM Joe Witt  wrote:
> >
> > > Team,
> > >
> > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%201.22.0
> > >
> > > It appears NiFi 1.22 has enough weight to be worth kicking out a
> release.
> > > 120+ JIRAs included several performance enhancers, bug fixes, a few new
> > > features like ModifyCompression, PutGCP, etc..
> > >
> > > Anything we're specifically waiting on?
> > >
> > > Thanks
> > >
> >
>

[discuss] nifi 1.22 looking worthy at this point

[Joe Witt]

Team,

https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%201.22.0

It appears NiFi 1.22 has enough weight to be worth kicking out a release.
120+ JIRAs included several performance enhancers, bug fixes, a few new
features like ModifyCompression, PutGCP, etc..

Anything we're specifically waiting on?

Thanks

Re: OpenTelemetry Integration

[Joe Witt]

Brian Putt, All

Are you aware of any good tools/services that can ingest the traces and
provide an interesting view/story/reporting on it?

I could see us emitting otel events instead of our current provenance
mechanism and using that both internally to do what we already do but also
have a clear/spec friendly way of exporting it to others.

Thanks

On Sat, Jul 30, 2022 at 7:43 AM u...@moosheimer.com 
wrote:

> Hello Brian, Bryan, Greg, NiFi devs,
>
> Integrating OpenTelemetry is a very good idea, especially since the major
> cloud providers also rely on it. This could also be interesting for
> Stateless NiFi.
>
> I have a suggestion that I would like to put up for discussion.
>
> Would it be useful to make a list of what extensions or new development
> would be helpful for a complete integration of OpenTelemetry?
>
> I'm thinking of ConsumeMQTT and PublishMQTT, for example. Currently these
> can do max. MQTT version 3.11, but since version 5 the User Properties
> exist, which are similar to the HTTP header fields.
> Thus one could implement OpenTelemetry in the MQTT processors similarly as
> in HTTP.
>
> With a list we could make an overview of the "necessary" adjustments and
> advertise for support.
>
> If what I write is nonsense, then I may not have understood something and
> I take it all back :)
>
> Mit freundlichen Grüßen / best regards
> Kay-Uwe Moosheimer
>
> > Am 29.07.2022 um 05:09 schrieb Brian Putt :
> >
> > Hello Bryan / Greg / NiFi devs,
> >
> > Distributed tracing (DT) is similar to provenance in that it shows the
> path
> > a particular flowfile travels, but its core selling point is that it
> > supports tracing across multiple systems/services regardless of what's
> > receiving the data. Provenance is a fantastic feature and there are
> > instances where one might want to draw that bigger picture of identifying
> > bottlenecks as data flows from one system to another and that system
> > may/may not be using NiFi.
> >
> > DT utilizes three ids: traceId, parentId, and spanId. While a tree can be
> > built using two ids, the third id (traceId) helps bring all of the
> relevant
> > information out of a datastore more easily.
> > DT is focused more on performance and identifying bottlenecks in one or
> > more systems. Imagine if NiFi were receiving data from various sources
> > (i.e. HTTP, Kafka, SQS) and NiFi egressed to other sources (HTTP, Kafka,
> > NiFi).
> > DT provides a spec that we'd be able to follow and correlate the data as
> it
> > traverses from system to system. Each system that participates in the DT
> > ecosystem would simply emit information (a trace is made up of one or
> more
> > spans) and there'd be a collection system which would aggregate all of
> > these spans and would draw a bigger picture of the path that data went
> > through and could help identify key bottlenecks.
> >
> > OpenTelemetry (OTEL) provides clients (across many languages, including
> > java) where developers can instrument their library's APIs and
> participate
> > in a DT ecosystem as it adheres to the tracing spec. Egressing trace data
> > is possible without using OTEL, but then we may find ourselves having to
> > recreate the wheel, but could be optimized for NiFi.
> >
> > Creating a reporting task could certainly be a path, mainly have a few
> > concerns with that:
> >
> > 1. If provenance is disabled, will provenance events still be emitted and
> > be collected by a new reporting task?
> > 2. There'll be an impact on performance, how much is unknown. OTEL is
> > gaining traction across industry and there are ways to mitigate
> > performance, mainly sampling and the fact that *tracing is best effort*.
> > Spans would be emitted from NiFi via UDP to a collector on the same
> network
> > 3. Would there be any issues with appending a flowfile attribute that is
> > carried throughout the flow where it maintains the traceId, parentSpanId,
> > and trace flags? See below for more details
> >
> > There's a W3C spec (Trace context) which includes a formatted string that
> > would be propagated to services (HTTP, Kafka, etc...). So if NiFi were to
> > put information onto kafka, any consumers of that data would be able to
> > continue the trace and help draw the bigger picture.
> >
> > W3C Spec: https://www.w3.org/TR/trace-context/#traceparent-header
> >
> > For #2, since DT is focused on performance, sampling can help alleviate
> > chatter over the wire and ideally, 0.01% would draw the same picture as
> 1%
> > or 10%+. This is certainly different from provenance as DT is focused on
> > performance over quality of the data and should not be thought of as
> > auditing.
> >
> https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampler
> >
> >> On Thu, Jul 28, 2022 at 5:01 PM Bryan Bende  wrote:
> >>
> >> Hi Greg,
> >>
> >> I don't really know anything about OpenTelemetry, but from the
> >> perspective of integrating something into the framework, some things
> 

Re: Help in creating a custom record reader

[Joe Witt]

Hello

To Dan's point this request for assistance is probably too generic for
anyone to offer a meaningful response just now.  I'd point out you don't
need to be thinking about how to integrate it with specific processors such
as those you list.  By virtue of creating your own RecordReader
implementation all such processors will automatically be able to use it.
So you can simplify your question to "How do I build a record reader?".  To
that end it is probably best to start with an existing RecordReader and
work it from there.

Thanks

On Thu, May 11, 2023 at 7:35 AM Dan S  wrote:

> Abhik Lodh, do you have specific questions?
>
> On Thu, May 11, 2023 at 10:29 AM Abhik Lodh 
> wrote:
>
> > Hi,
> > I am writing this mail because I wanted to create a custom Record Reader
> > in NiFi that parses ASCII files (or any other currently unsupported file
> > format) but I’m having a very hard time creating a custom controller
> > service that does this while integrating with ConvertRecord, UpdateRecord
> > and ValidateRecord just like other RecordReaders.
> >
> > Could anyone help me out with this?
> >
> > Thanks and regards,
> > Abhik Lodh
> >
>

Re: [VOTE] Release Apache NiFi MiNiFi C++ 0.14.0

[Joe Witt]

+1 binding.  Was able to do full build on a linux system.  Hashes/sigs
check out.

The L look good with basic spot check.  The Notice needs to be updated to
reflect the 2023 copyright year.

Thanks!

On Fri, Apr 14, 2023 at 9:59 AM Marton Szasz  wrote:

> +1 (binding)
> The package, hashes, signatures, etc were all good. Release helper
> guide for future reference:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173087303
>
> Tested compilation on Ubuntu 22.04 with both GCC and Clang. Tested
> both the self-built and the convenience binaries in a log collection
> use case, forwarding messages via HTTP to NiFi.
> Tested compilation on Windows x64, and a security event log collection
> use case using ConsumeWindowsEventLog, and forwarding via
> PutUDP/ListenUDP to NiFi.
>
> Thank you for preparing the RC!
>
> Marton
>
> On Fri, 14 Apr 2023 at 18:55, Ferenc Gerlits  wrote:
> >
> > +1 (non-binding)
> >
> > Verified signatures and hashes, compared the source to the git tag.
> > Built from source on Ubuntu Linux 22.04 and Windows.
> > Ran the convenience binary on Ubuntu Linux 22.04 with a simple
> > ConsumeJournald -> PublishKafka flow.
> > Installed and ran the binary built from the source on Windows with a
> > simple ConsumeWindowsEventLog -> PublishKafka flow.
> > Everything worked fine.
> >
> > Thanks,
> > Ferenc
>

Re: Are all 1.21 NARs uploaded as Maven artifacts?

[Joe Witt]

Works great out of Apache's nexus repository so I'm guessing it is just
cloning/taking time.

On Mon, Apr 10, 2023 at 10:09 AM Bryan Bende  wrote:

> You are right, I didn't actually try to download it before, that is
> strange.
>
> On Mon, Apr 10, 2023 at 1:07 PM Mike Thomsen 
> wrote:
> >
> > When I clicked on it, I got a 404. Weird.
> >
> > On Mon, Apr 10, 2023 at 12:53 PM Bryan Bende  wrote:
> >
> > > The Azure NAR looks like it is there:
> > >
> > >
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-azure-nar/1.21.0/
> > >
> > > On Mon, Apr 10, 2023 at 12:40 PM Mike Thomsen 
> > > wrote:
> > > >
> > > > It’s failing on the azure one which is why I am raising the issue. I
> > > know we are still pushing that because it’s part of the convenient
> binary.
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Apr 10, 2023, at 12:26 PM, Joe Witt  wrote:
> > > > >
> > > > > Hello
> > > > >
> > > > > Pretty sure all the normal stuff/paths followed and uploaded.  It
> > > might be
> > > > > that we're not pushing Atlas nars anymore?
> > > > >
> > > > > I dont recall but that is possible.
> > > > >
> > > > > Thanks
> > > > >
> > > > >> On Mon, Apr 10, 2023 at 9:21 AM Mike Thomsen <
> mikerthom...@gmail.com>
> > > wrote:
> > > > >>
> > > > >> Getting this error on two different machines on two different
> > > networks:
> > > > >>
> > > > >> ownloaded from central:
> > > > >>
> > > > >>
> > >
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-sql-reporting-nar/1.21.0/nifi-sql-reporting-nar-1.21.0.nar
> > > > >> (26 MB at 183 kB/s)
> > > > >>
> > > > >> Downloaded from central:
> > > > >>
> > > > >>
> > >
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-other-graph-services-nar/1.21.0/nifi-other-graph-services-nar-1.21.0.nar
> > > > >> (49 MB at 332 kB/s)
> > > > >>
> > > > >> Downloaded from central:
> > > > >>
> > > > >>
> > >
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-atlas-nar/1.21.0/nifi-atlas-nar-1.21.0.nar
> > > > >> (74 MB at 491 kB/s)
> > > > >>
> > > > >> [*INFO*]
> > > > >>
> > >
> **
> > > > >>
> > > > >> [*INFO*] *BUILD FAILURE*
> > > > >>
> > > > >> [*INFO*]
> > > > >>
> > >
> **
> > > > >>
> > > > >> [*INFO*] Total time:  02:54 min
> > > > >>
> > > > >> [*INFO*] Finished at: 2023-04-10T12:20:08-04:00
> > > > >>
> > > > >> [*INFO*]
> > > > >>
> > >
> **
> > > > >>
> > > > >> [*ERROR*] Failed to execute goal on project nifi-assembly: *Could
> not
> > > > >> resolve dependencies for project
> > > org.apache.nifi:nifi-assembly:pom:1.21.0:
> > > > >> Could not find artifact org.apache.nifi:nifi-azure-nar:nar:1.21.0
> in
> > > > >> central (https://repo.maven.apache.org/maven2
> > > > >> <https://repo.maven.apache.org/maven2>)* -> *[Help 1]*
> > > > >>
> > > > >> [*ERROR*]
> > > > >>
> > > > >> [*ERROR*] To see the full stack trace of the errors, re-run Maven
> > > with the
> > > > >> *-e* switch.
> > > > >>
> > > > >> [*ERROR*] Re-run Maven using the *-X* switch to enable full debug
> > > logging.
> > > > >>
> > > > >> [*ERROR*]
> > > > >>
> > > > >> [*ERROR*] For more information about the errors and possible
> > > solutions,
> > > > >> please read the following articles:
> > > > >>
> > > > >> [*ERROR*] *[Help 1]*
> > > > >>
> > > > >>
> > >
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> > > > >>
> > >
>

Re: Are all 1.21 NARs uploaded as Maven artifacts?

[Joe Witt]

I get a 404 for that link as well.  Give it time maybe...perhaps it is
still cloning slowly over time.

On Mon, Apr 10, 2023 at 10:08 AM Mike Thomsen 
wrote:

> By it, I mean this:
>
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-azure-nar/1.21.0/nifi-azure-nar-1.21.0.nar
>
> On Mon, Apr 10, 2023 at 1:07 PM Mike Thomsen 
> wrote:
>
> > When I clicked on it, I got a 404. Weird.
> >
> > On Mon, Apr 10, 2023 at 12:53 PM Bryan Bende  wrote:
> >
> >> The Azure NAR looks like it is there:
> >>
> >>
> >>
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-azure-nar/1.21.0/
> >>
> >> On Mon, Apr 10, 2023 at 12:40 PM Mike Thomsen 
> >> wrote:
> >> >
> >> > It’s failing on the azure one which is why I am raising the issue. I
> >> know we are still pushing that because it’s part of the convenient
> binary.
> >> >
> >> > Sent from my iPhone
> >> >
> >> > > On Apr 10, 2023, at 12:26 PM, Joe Witt  wrote:
> >> > >
> >> > > Hello
> >> > >
> >> > > Pretty sure all the normal stuff/paths followed and uploaded.  It
> >> might be
> >> > > that we're not pushing Atlas nars anymore?
> >> > >
> >> > > I dont recall but that is possible.
> >> > >
> >> > > Thanks
> >> > >
> >> > >> On Mon, Apr 10, 2023 at 9:21 AM Mike Thomsen <
> mikerthom...@gmail.com>
> >> wrote:
> >> > >>
> >> > >> Getting this error on two different machines on two different
> >> networks:
> >> > >>
> >> > >> ownloaded from central:
> >> > >>
> >> > >>
> >>
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-sql-reporting-nar/1.21.0/nifi-sql-reporting-nar-1.21.0.nar
> >> > >> (26 MB at 183 kB/s)
> >> > >>
> >> > >> Downloaded from central:
> >> > >>
> >> > >>
> >>
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-other-graph-services-nar/1.21.0/nifi-other-graph-services-nar-1.21.0.nar
> >> > >> (49 MB at 332 kB/s)
> >> > >>
> >> > >> Downloaded from central:
> >> > >>
> >> > >>
> >>
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-atlas-nar/1.21.0/nifi-atlas-nar-1.21.0.nar
> >> > >> (74 MB at 491 kB/s)
> >> > >>
> >> > >> [*INFO*]
> >> > >>
> >>
> **
> >> > >>
> >> > >> [*INFO*] *BUILD FAILURE*
> >> > >>
> >> > >> [*INFO*]
> >> > >>
> >>
> **
> >> > >>
> >> > >> [*INFO*] Total time:  02:54 min
> >> > >>
> >> > >> [*INFO*] Finished at: 2023-04-10T12:20:08-04:00
> >> > >>
> >> > >> [*INFO*]
> >> > >>
> >>
> **
> >> > >>
> >> > >> [*ERROR*] Failed to execute goal on project nifi-assembly: *Could
> not
> >> > >> resolve dependencies for project
> >> org.apache.nifi:nifi-assembly:pom:1.21.0:
> >> > >> Could not find artifact org.apache.nifi:nifi-azure-nar:nar:1.21.0
> in
> >> > >> central (https://repo.maven.apache.org/maven2
> >> > >> <https://repo.maven.apache.org/maven2>)* -> *[Help 1]*
> >> > >>
> >> > >> [*ERROR*]
> >> > >>
> >> > >> [*ERROR*] To see the full stack trace of the errors, re-run Maven
> >> with the
> >> > >> *-e* switch.
> >> > >>
> >> > >> [*ERROR*] Re-run Maven using the *-X* switch to enable full debug
> >> logging.
> >> > >>
> >> > >> [*ERROR*]
> >> > >>
> >> > >> [*ERROR*] For more information about the errors and possible
> >> solutions,
> >> > >> please read the following articles:
> >> > >>
> >> > >> [*ERROR*] *[Help 1]*
> >> > >>
> >> > >>
> >>
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> >> > >>
> >>
> >
>

Re: Are all 1.21 NARs uploaded as Maven artifacts?

[Joe Witt]

Hello

Pretty sure all the normal stuff/paths followed and uploaded.  It might be
that we're not pushing Atlas nars anymore?

I dont recall but that is possible.

Thanks

On Mon, Apr 10, 2023 at 9:21 AM Mike Thomsen  wrote:

> Getting this error on two different machines on two different networks:
>
> ownloaded from central:
>
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-sql-reporting-nar/1.21.0/nifi-sql-reporting-nar-1.21.0.nar
> (26 MB at 183 kB/s)
>
> Downloaded from central:
>
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-other-graph-services-nar/1.21.0/nifi-other-graph-services-nar-1.21.0.nar
> (49 MB at 332 kB/s)
>
> Downloaded from central:
>
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-atlas-nar/1.21.0/nifi-atlas-nar-1.21.0.nar
> (74 MB at 491 kB/s)
>
> [*INFO*]
> **
>
> [*INFO*] *BUILD FAILURE*
>
> [*INFO*]
> **
>
> [*INFO*] Total time:  02:54 min
>
> [*INFO*] Finished at: 2023-04-10T12:20:08-04:00
>
> [*INFO*]
> **
>
> [*ERROR*] Failed to execute goal on project nifi-assembly: *Could not
> resolve dependencies for project org.apache.nifi:nifi-assembly:pom:1.21.0:
> Could not find artifact org.apache.nifi:nifi-azure-nar:nar:1.21.0 in
> central (https://repo.maven.apache.org/maven2
> )* -> *[Help 1]*
>
> [*ERROR*]
>
> [*ERROR*] To see the full stack trace of the errors, re-run Maven with the
> *-e* switch.
>
> [*ERROR*] Re-run Maven using the *-X* switch to enable full debug logging.
>
> [*ERROR*]
>
> [*ERROR*] For more information about the errors and possible solutions,
> please read the following articles:
>
> [*ERROR*] *[Help 1]*
>
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
>

[ANNOUNCE] Apache NiFi 1.21.0 release.

[Joe Witt]

Hello

The Apache NiFi team would like to announce the release of Apache NiFi 1.21.0.

Apache NiFi is an easy to use, powerful, and reliable system to
process and distribute data.  Apache NiFi was made for dataflow.  It
supports highly configurable directed graphs of data routing,
transformation, and system mediation logic.

More details on Apache NiFi can be found here:
https://nifi.apache.org/

The release artifacts can be downloaded from here:
https://nifi.apache.org/download.html

Maven artifacts have been made available and mirrored as per normal
ASF artifact processes.

Issues closed/resolved for this list can be found here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352899

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.21.0

Thank you
The Apache NiFi team

[RESULT][VOTE] Release Apache NiFi 1.21.0 (RC2)

[Joe Witt]

Apache NiFi Community,

I am pleased to announce that the 1.21.0 release of Apache NiFi passes with
8 +1 (binding) votes
11 +1 (non-binding) votes
0 0 votes
0 -1 votes

Thanks to all who helped make this release possible.

Here is the PMC vote thread:
https://lists.apache.org/thread/fkpvjqnf48ghb025clq4sxzy2zzkh362

On Fri, Apr 7, 2023 at 8:17 AM Mark Payne  wrote:
>
> +1 (binding)
>
> Performed full build with contrib-check profile successfully
> Ran all system tests successfully
> Verified signature
> Verified hash
> Tested several specific flows in registry, around using embedded PG’s, some 
> versioned and some not.
>
> I did encounter a bug that I filed a Jira [1]  for. It was fairly minor as 
> its scope is quite narrow and there’s a simple enough work around. Not 
> concerning enough to change my vote.
>
> Thanks for performing the RM Joe!
>
> -Mark
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-11394
>
> On Apr 6, 2023, at 2:48 PM, Emilio Setiadarma  
> wrote:
>
> +1 (non-binding)
>
> Verified signature and SHA256/512 hashes of the release.
> OS: MacOS Ventura 13.2.1
> Java version: OpenJDK version 11.0.18
> Ran the build, and booted up NiFi with no issues, and was able to start up
> a simple flow
>
>
>
> On Thu, Apr 6, 2023 at 9:19 AM Michael Moser  wrote:
>
> +1 (binding)
>
> Verified checksums and signature of source release
> Built (with empty local repo) on CentOS Linux release 7.9.2009
> Apache Maven 3.8.6
> Java version: 1.8.0_362, vendor: Red Hat, Inc., runtime:
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.362.b08-1.el7_9.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-1160.76.1.el7.x86_64", arch: "amd64",
> family: "unix"
>
> Ran NiFi and NiFi Registry with no issues.
>
> Hello again team!
>
> -- Mike
>
> On Thu, Apr 6, 2023 at 9:47 AM Paul Grey  wrote:
>
> +1 (non-binding)
>
> - Verified signatures and hashes of [
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.21.0/] binaries
>
> - Build of source successful (macOS 12 / Azul Java 1.8.0_322)
> - Application startup successful
> - Verified successful flow of data using known good flows (Kafka,
> PutSlack)
> - verify NIFI-4890 - OIDC refactor (backport) (Okta dev account)
>
> - Application startup successful (set-single-user-credentials, three node
> dev cluster)
> - Verified successful flow of data using known good flows (Kafka,
> PutSlack)
> - Testing of cluster disconnect / offload / connect operations
>
> - Build of source successful (Debian 10.13 / OpenJDK 1.8.0_362)
> - Application startup successful
> - Verified successful flow of data using known good flows (PutSlack)
>
> - Captured test failure (Linux) as NIFI-11391 to investigate.
>
> On Thu, Apr 6, 2023 at 8:31 AM Gábor Gyimesi 
> wrote:
>
> +1 (non-binding)
>
> Verified signatures and hashes
> Built NiFi on ubuntu 22.04
> Ran a simple flow and a flow integrated with MiNiFi C++ through
> InvokeHTTP
>
> LGTM
>
> Thanks Joe!
>
> BR,
> Gabor
>
> On Thu, 6 Apr 2023 at 14:15, Martin Zink 
> wrote:
>
> +1 (non-binding)
>
> Went through the helper guide verified signatures and hashes.
> I've also verified that integration with the latest MiNiFi C++ works
> as
> expected (InvokeHTTP based flows)
>
> Everything looks good
>
> Thanks Joe for RMing!
>
> On Thu, Apr 6, 2023 at 9:47 AM Ferenc Kis 
> wrote:
>
> +1 (non-binding)
>
> - Went through the helper guide, full clean build with contrib
> check,
> verified signatures and hashes
>  Maven home: /Users/fkis/.sdkman/candidates/maven/current
>  Java version: 11.0.18, vendor: Eclipse Adoptium, runtime:
> /Users/fkis/.sdkman/candidates/java/11.0.18-tem
>  Default locale: en_HU, platform encoding: UTF-8
>  OS name: "mac os x", version: "13.2.1", arch: "aarch64", family:
> "mac"
> - Started NiFi, created a simple flow with ListenHTTP
> - Started Nifi Registry, set up integration with NiFi, and saved
> the
> previous flow to NiFi Registry
> - Started MiNiFi Java:
>  * created a simple GenerateFlowFile -> InvokeHttp flow and pushed
> to
> MiNiFi via C2 protocol
>  * managed to push data from MiNiFi to NiFi
>
> Thanks Joe for RMing!
>
> Regards
> Ferenc
>
> On Thu, Apr 6, 2023 at 9:06 AM Pierre Villard <
> pierre.villard...@gmail.com
>
> wrote:
>
> +1 binding
>
> Went through the usual steps. Deployed a 3 node secured cluster
> on
> GCP
> with
> a secure NiFi Registry. Tested some flows. All good.
>
> Thanks Joe for taking care o

Re: [VOTE] Release Apache NiFi 1.21.0 (RC2)

[Joe Witt]

+1 (binding)

On Thu, Apr 6, 2023 at 11:48 AM Emilio Setiadarma
 wrote:
>
> +1 (non-binding)
>
> Verified signature and SHA256/512 hashes of the release.
> OS: MacOS Ventura 13.2.1
> Java version: OpenJDK version 11.0.18
> Ran the build, and booted up NiFi with no issues, and was able to start up
> a simple flow
>
>
>
> On Thu, Apr 6, 2023 at 9:19 AM Michael Moser  wrote:
>
> > +1 (binding)
> >
> > Verified checksums and signature of source release
> > Built (with empty local repo) on CentOS Linux release 7.9.2009
> > Apache Maven 3.8.6
> > Java version: 1.8.0_362, vendor: Red Hat, Inc., runtime:
> > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.362.b08-1.el7_9.x86_64/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "linux", version: "3.10.0-1160.76.1.el7.x86_64", arch: "amd64",
> > family: "unix"
> >
> > Ran NiFi and NiFi Registry with no issues.
> >
> > Hello again team!
> >
> > -- Mike
> >
> > On Thu, Apr 6, 2023 at 9:47 AM Paul Grey  wrote:
> >
> > > +1 (non-binding)
> > >
> > > - Verified signatures and hashes of [
> > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.21.0/] binaries
> > >
> > > - Build of source successful (macOS 12 / Azul Java 1.8.0_322)
> > > - Application startup successful
> > > - Verified successful flow of data using known good flows (Kafka,
> > PutSlack)
> > > - verify NIFI-4890 - OIDC refactor (backport) (Okta dev account)
> > >
> > > - Application startup successful (set-single-user-credentials, three node
> > > dev cluster)
> > > - Verified successful flow of data using known good flows (Kafka,
> > PutSlack)
> > > - Testing of cluster disconnect / offload / connect operations
> > >
> > > - Build of source successful (Debian 10.13 / OpenJDK 1.8.0_362)
> > > - Application startup successful
> > > - Verified successful flow of data using known good flows (PutSlack)
> > >
> > > - Captured test failure (Linux) as NIFI-11391 to investigate.
> > >
> > > On Thu, Apr 6, 2023 at 8:31 AM Gábor Gyimesi 
> > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Verified signatures and hashes
> > > > Built NiFi on ubuntu 22.04
> > > > Ran a simple flow and a flow integrated with MiNiFi C++ through
> > > InvokeHTTP
> > > >
> > > > LGTM
> > > >
> > > > Thanks Joe!
> > > >
> > > > BR,
> > > > Gabor
> > > >
> > > > On Thu, 6 Apr 2023 at 14:15, Martin Zink 
> > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Went through the helper guide verified signatures and hashes.
> > > > > I've also verified that integration with the latest MiNiFi C++ works
> > as
> > > > > expected (InvokeHTTP based flows)
> > > > >
> > > > > Everything looks good
> > > > >
> > > > > Thanks Joe for RMing!
> > > > >
> > > > > On Thu, Apr 6, 2023 at 9:47 AM Ferenc Kis 
> > > > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > - Went through the helper guide, full clean build with contrib
> > check,
> > > > > > verified signatures and hashes
> > > > > >   Maven home: /Users/fkis/.sdkman/candidates/maven/current
> > > > > >   Java version: 11.0.18, vendor: Eclipse Adoptium, runtime:
> > > > > > /Users/fkis/.sdkman/candidates/java/11.0.18-tem
> > > > > >   Default locale: en_HU, platform encoding: UTF-8
> > > > > >   OS name: "mac os x", version: "13.2.1", arch: "aarch64", family:
> > > > "mac"
> > > > > > - Started NiFi, created a simple flow with ListenHTTP
> > > > > > - Started Nifi Registry, set up integration with NiFi, and saved
> > the
> > > > > > previous flow to NiFi Registry
> > > > > > - Started MiNiFi Java:
> > > > > >   * created a simple GenerateFlowFile -> InvokeHttp flow and pushed
> > > to
> > > > > > MiNiFi via C2 protocol
> > > > > >   * managed to push data from MiNiFi to NiFi
> > > > > >
> > > > > > Thanks Joe for RMing!
> > > > > >

April board report for NiFi

[Joe Witt]

Team,

Here is what I submitted to the board for our April report.  Thanks
again for all the great work and progress.

## Description:
The mission of NiFi is the creation and maintenance of software related to
providing an easy to use, powerful, and reliable system to process and
distribute data.

Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
integrate with and leverage the command and control of NiFi. There are both
Java and C++ implementations.

Apache NiFi Registry is a centralized registry for key configuration items
including flow versions, assets, and extensions for Apache NiFi and Apache
MiNiFi.

Apache NiFi Nar Maven Plugin is a release artifact used for supporting the
NiFi classloader isolation model.

Apache NiFi Flow Design System is a theme-able set of high quality UI
components and utilities for use across the various Apache NiFi web
applications in order to provide a more consistent user experience.

## Issues:
There are no issues requiring board attention at this time.

## Membership Data:
Apache NiFi was founded 2015-07-14 (8 years ago)
There are currently 63 committers and 33 PMC members in this project.
The Committer-to-PMC ratio is roughly 2:1.

Community changes, past quarter:
- No new PMC members. Last addition was Nathan Gough on 2022-09-06.
- Ferenc Kis was added as committer on 2023-03-23
- Nándor Soma Abonyi was added as committer on 2023-03-15

## Project Activity:
The NiFi community released NiFi 1.20.0 in Feb 2023 and is in the RC voting
process for NiFi 1.21.0 now expected to be available in early April 2023.
More than 170 JIRAs have already landed toward our upcoming NiFi 2.0.0
release as discussed in previous board reports.  The 1.21 release is
largely focused on stability and continuing to prepare to move 1.x users
to 2.x with as minimal disruption as possible. As always we remain super
focused on dependency maintenance as it relates to security vulnerabilities.

## Community Health:
Community health remains a point of strength and emphasis.  We have long
enjoyed excellent engagement on release verification and voting. We have added
a couple new committers in this cycle and there are more in the pipeline and
discussions. Mailing list activity is stable if not a little slowed but again
is countered by consistent growth of 100 or so users per quarter into Slack
where we now see more than 2600 users in the general channel up by more than
100 from last report. Our JIRA and commit activity has increased and some of
this is likely as we're working now two major feature lines both 1.x and 2.x.

[VOTE] Release Apache NiFi 1.21.0 (RC2)

[Joe Witt]

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi 1.21.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1226

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.21.0/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.21.0-RC2
The Git commit ID is 892f822107da84ca0dcfefdb5c91c5bc11dc3dc1
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=892f822107da84ca0dcfefdb5c91c5bc11dc3dc1

Checksums of nifi-1.21.0-source-release.zip:
SHA256: 598bf8e90f871f4ca25709dd4ecf3da16c814c08c0d8b2c8744dbd34df34dea5
SHA512: 
58c10976bc22fb5406d9d1d9b7ca7d90c2dbed99565d00f1172bb33b375e9e068da5003d9dbbd87d2b17626e4e310b999c8581718532934e855c2134be763f04

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

126 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352899

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.21.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test. Then please vote:

[ ] +1 Release this package as nifi-1.21.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

[CANCEL][VOTE] Release Apache NiFi 1.21.0 (RC1)

[Joe Witt]

Will get RC2 up soon

On Thu, Mar 30, 2023 at 10:11 AM David Handermann
 wrote:
>
> -1 binding
>
> Thanks to macdoor615 testing 1.21.0-RC1 and reporting NIFI-11365 [1] I am
> voting down the release candidate to correct OpenID Connect integration
> with reverse proxy servers.
>
> I have submitted a pull request [2] to correct the behavior.
>
> Regards,
> David Handermann
>
> [1] https://issues.apache.org/jira/browse/NIFI-11365
> [2] https://github.com/apache/nifi/pull/7104
>
> On Wed, Mar 29, 2023 at 5:01 PM Joe Witt  wrote:
>
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi 1.21.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1224
> >
> > The source being voted upon and the convenience binaries can be found at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.21.0/
> >
> > A helpful reminder on how the release candidate verification process works:
> >
> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >
> > The Git tag is nifi-1.21.0-RC1
> > The Git commit ID is eef5ca1afe8532521e5dd8f49029f8d5a3ac6028
> >
> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=eef5ca1afe8532521e5dd8f49029f8d5a3ac6028
> >
> > Checksums of nifi-1.21.0-source-release.zip:
> > SHA256: 744bb0931d0965cfb882741300ee7fe2fa6ef896eb952a31929bec7f4f5ce46f
> > SHA512:
> > cbca6cf5f9a970efff37a2ddaadc92d1c7b6c1ed1abc4c0113d1413198c0798586b4587bb8beb3382c8831b4890c42154a34ba8f1354eba414c59cb4ede4e64a
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 110 issues were closed/resolved for this release:
> >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352899
> >
> > Release note highlights can be found here:
> >
> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.21.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test. Then please vote:
> >
> > [ ] +1 Release this package as nifi-1.21.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >

[VOTE] Release Apache NiFi 1.21.0 (RC1)

[Joe Witt]

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi 1.21.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1224

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.21.0/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.21.0-RC1
The Git commit ID is eef5ca1afe8532521e5dd8f49029f8d5a3ac6028
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=eef5ca1afe8532521e5dd8f49029f8d5a3ac6028

Checksums of nifi-1.21.0-source-release.zip:
SHA256: 744bb0931d0965cfb882741300ee7fe2fa6ef896eb952a31929bec7f4f5ce46f
SHA512: 
cbca6cf5f9a970efff37a2ddaadc92d1c7b6c1ed1abc4c0113d1413198c0798586b4587bb8beb3382c8831b4890c42154a34ba8f1354eba414c59cb4ede4e64a

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

110 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352899

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.21.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test. Then please vote:

[ ] +1 Release this package as nifi-1.21.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: [discuss] NiFi support for Hadoop ecosystem components

[Joe Witt]

> technologies seem to still depend on it. For example, Iceberg, in which
> > case I'm worried about the consequences of removing the support for an
> > increasingly popular technology.
> >
> > So I wonder whether it is possible to find a forward-looking solution that
> > could serve all projects. I've always found configuring Hadoop and friends
> > too tricky and I thought it was primarily for historical reasons. The
> > issues you describe could easily result from such a thing. I assume that
> > over time, new and new things have been added on top of the existing
> > implementation without significant refactoring.
> >
> > My - probably utopistic - idea would be to contact the Hadoop and Hive
> > teams and share the issues we are dealing with. Probably we are not alone
> > in these problems, but I don't know whether they are aware of them. Even if
> > they are, I think approaching them is worth the chance. Who knows where we
> > will end up if somebody representing the NiFi project does that?
> >
> > Regards,
> > Nandor Soma Abonyi
> >
> >
> > > On Mar 24, 2023, at 10:40 PM, Jeremy Dyer  wrote:
> > >
> > > I think option 2 is the best way to handle this.
> > >
> > > Technology naturally changes over time and some components of Nifi might
> > not make the most sense to keep around in the main line for the masses
> > anymore. However I really like still having them there for people to very
> > simply add if they so desire too. I see other platforms do this by adding a
> > “contrib” repo. What if we had something like a “nifi-contrib” or
> > “nifi-emeritus” repo in GitHub, Apache GitHub repo, where the community can
> > still be involved as desired but also keep things readily available to
> > those who might not even be heavily involved in the community?
> > >
> > > I even see this as a sustainable pattern for any components that need
> > “moved out”.
> > >
> > > I wouldn’t even think those components in the “contrib” repo would
> > require voting on for releases but someone, or a vendor, could update them
> > via PRs after the official release.
> > >
> > > Jeremy Dyer
> > >
> > > Get Outlook for
> > > iOS<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> > > aka.ms%2Fo0ukef=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C7a74
> > > 6c107132419b7ec808db2eae6b08%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C
> > > 0%7C638155098878698642%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ
> > > QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=xXRy
> > > LdqqQND5lG1MaBEonKblKwlpmMdKvOH34FouBPI%3D=0>
> > > 
> > > From: Chakravarty, G 
> > > Sent: Friday, March 24, 2023 4:36:43 PM
> > > To: dev@nifi.apache.org 
> > > Subject: Re: [discuss] NiFi support for Hadoop ecosystem components
> > >
> > > I am wondering if the standard Nifi jdbc/odbc processors with some basic
> > testing with the common drivers like Simba etc. Hive drivers can help to
> > alleviate the issue without having separate HiveQL processors.
> > >
> > > GC
> > > 
> > > From: Bryan Bende 
> > > Sent: Friday, March 24, 2023 4:05 PM
> > > To: dev@nifi.apache.org 
> > > Subject: Re: [discuss] NiFi support for Hadoop ecosystem components
> > >
> > > I lean towards option 2 with the caveat that maybe we don't have to
> > > retain every Hadoop related component when creating this separate set
> > > of components. Mainly I'm thinking that Hive has been the most
> > > problematic to maintain so maybe that is dropped all together. I think
> > > it would be unfortunate to not have publicly available HDFS
> > > processors.
> > >
> > > On Fri, Mar 24, 2023 at 3:23 PM Matt Burgess 
> > wrote:
> > >>
> > >> As one of the small number of people that fight the battle, I like
> > >> the idea of Option 1 (full disclosure: I work for a vendor). From a
> > >> community standpoint (I'm on the PMC) I'm not strongly opposed to
> > >> Option 2 although I wouldn't want to be the one managing and
> > >> releasing the artifacts :) Having said that, unless it remained
> > >> maintained and released, I feel like it would just be a component
> > >> graveyard (or maybe more like the Apache Attic), in which case it
> > >> seems unnecessary and that's why I'm behind Option 1. Interested to
> > >> hear others

deleting a bunch of not Apache NiFi tags

[Joe Witt]

Team,

It was just noticed we have a ton of tags pushed by someone in the
nifi repo (cant tell who/why yet) that relate to tag releases of nifi
that are vendor oriented (not apache nifi).  We'll be cleaning these
up.

Thanks

Re: [discuss] NiFi support for Hadoop ecosystem components

[Joe Witt]

James

Some are definitely less fun than others with Hive being the most notable.

I should rephrase my vendor thing on point one: It is as far as I know all
vendor supported Hadoop components.  Whether NiFi is or not is a different
point.

Option 2 is the most realistic I suspect but still want to see what people
think.

Basically anything which depends on the ‘hadoop-client’ maven artifact is
where the games begin.

Thanks

On Fri, Mar 24, 2023 at 2:34 PM James Srinivasan 
wrote:

> I'm a Hadoop and Nifi user without vendor support so unsurprisingly aren't
> keen on #1, but then relying on community support and development is always
> going to be a risk for us. If it came to it, we'd probably stop using Nifi
> rather than pay a vendor which would be a real shame.
>
> Are certain Hadoop processors more maintenance heavy than others? Its a
> rather wide ecosystem.
>
> On Fri, 24 Mar 2023, 18:07 Joe Witt,  wrote:
>
> > Team,
> >
> > For the full time NiFi has been in Apache we've built with support for
> > various Hadoop ecosystem components like HDFS, Hive, HBase, others,
> > and more recently formats/serialization modes like necessary for
> > Parquet, Orc, Iceberg, etc..
> >
> > All of these things however present endless challenges with
> > compatibility across different versions (Hive being the most difficult
> > by far), vendors (hadoop vendors, cloud vendors, etc..).  And also
> > super notably the incredible number of dependencies, dependency
> > conflicts, inclusions/exclusions, old log libs, vulnerability updates,
> > etc..  And last but certainly not least a big reason why our build has
> > grown so much.
> >
> > We have a couple options:
> > 1. Deprecate these components in NiFi 1.x and drop them entirely in
> > NiFi 2.x.  Leave this as a problem for vendors to deal with.  NiFi
> > users interacting with such components are nearly exclusively doing so
> > with vendors anyway.
> >
> > 2. Remove the components from NiFi main code line and create a
> > separate repo for 'nifi-hadoop-extensions'.  We manage those
> > independently and release them periodically.  They would be available
> > for people to grab the nars if they want to use them.  We include none
> > of them in the convenience binary going forward by default.
> >
> > 3. Change nothing.  Continue to battle with the above listed items.
> > This is admittedly a bit of a non-option.  We can't keep spending the
> > same time/energy on these we have.  It is a very small number of
> > people that fight this battle.
> >
> > Look forward to hearing thoughts on these options or others we might
> > consider.
> >
> > Thanks
> >
>

[discuss] NiFi support for Hadoop ecosystem components

[Joe Witt]

Team,

For the full time NiFi has been in Apache we've built with support for
various Hadoop ecosystem components like HDFS, Hive, HBase, others,
and more recently formats/serialization modes like necessary for
Parquet, Orc, Iceberg, etc..

All of these things however present endless challenges with
compatibility across different versions (Hive being the most difficult
by far), vendors (hadoop vendors, cloud vendors, etc..).  And also
super notably the incredible number of dependencies, dependency
conflicts, inclusions/exclusions, old log libs, vulnerability updates,
etc..  And last but certainly not least a big reason why our build has
grown so much.

We have a couple options:
1. Deprecate these components in NiFi 1.x and drop them entirely in
NiFi 2.x.  Leave this as a problem for vendors to deal with.  NiFi
users interacting with such components are nearly exclusively doing so
with vendors anyway.

2. Remove the components from NiFi main code line and create a
separate repo for 'nifi-hadoop-extensions'.  We manage those
independently and release them periodically.  They would be available
for people to grab the nars if they want to use them.  We include none
of them in the convenience binary going forward by default.

3. Change nothing.  Continue to battle with the above listed items.
This is admittedly a bit of a non-option.  We can't keep spending the
same time/energy on these we have.  It is a very small number of
people that fight this battle.

Look forward to hearing thoughts on these options or others we might consider.

Thanks