Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

David beat me to it :) IMO the only NAR that should have any dependencies
on Cassandra is the services NAR, not the processors or services API.

On Fri, Mar 22, 2024 at 11:10 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Mike,
>
> Thanks for sharing the branch, it is helpful to have that as a
> reference example. Have you been able to exercise any of that approach
> at runtime?
>
> Based on what is there right now, attempting to mark the DataStax
> java-driver-core as provided does not look like it will work. It may
> pass unit tests, but runtime NAR class loading requires that classes
> be available in the same NAR, or in a parent NAR. That means when NiFi
> tries to load the Controller Service interface, it must have access to
> a version of the relevant Cassandra driver classes. By marking the
> dependency as provided, it will not be available in the API NAR, and
> thus not available when loading the service interface. Including it in
> the API NAR won't work either, because it conflicts with the ScyllaDB
> java-driver-core in the implementation NAR.
>
> This is the reason Matt and I highlighted for providing a layer of
> abstraction at the Controller Service API level.
>
> Regards,
> David Handermann
>
> On Fri, Mar 22, 2024 at 8:13 AM Mike Thomsen 
> wrote:
> >
> > Work so far: https://github.com/MikeThomsen/nifi/tree/cql-changes
> >
> > On Thu, Mar 21, 2024 at 9:52 AM Mike Thomsen 
> wrote:
> >
> > > Matt/David,
> > >
> > > By this evening, I should be at a point where I can share my branch. It
> > > should be far enough along that y'all can see what I mean about how
> most of
> > > the changes really weren't that complicated. My sense is that if we
> > > collaborate on it, we can probably get it ready for a PR within a week
> or
> > > two.
> > >
> > > It would probably be a good idea to plan to revisit the Cassandra DMC's
> > > design and make it more flexible.
> > >
> > > One nice thing about the new DataStax driver is that it supports
> > > configuration by a very detailed configuration file format, so we can
> give
> > > users that option + combine it with EL/parameters (I envision an option
> > > where the user puts EL in the file, we load the file, preprocess the
> EL and
> > > load that into the driver)
> > >
> > > On Wed, Mar 20, 2024 at 4:01 PM Mike Thomsen 
> > > wrote:
> > >
> > >> If it were that simple, they would probably have just gone with that
> > >> solution. That said, the API is functionally vendor agnostic at this
> point
> > >> at the Java API level. So I see no need to add abstraction above
> that. I've
> > >> got probably 2/3 of nifi-cassandra-bundle converted. Hitting a few
> pain
> > >> points where I'm having to dig deep into the docs to make progress,
> but so
> > >> far, so good.
> > >>
> > >> On Wed, Mar 20, 2024 at 2:38 PM Matt Burgess 
> > >> wrote:
> > >>
> > >>> It would be interesting to see if you exclude the Scylla API JAR
> from the
> > >>> Scylla implementation and instead include DataStax's, if that works.
> > >>> However I'm still leaning towards a vendor-agnostic API.
> > >>>
> > >>> On Wed, Mar 20, 2024 at 11:26 AM Mike Thomsen <
> mikerthom...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > At first glance, the package names look identical to me:
> > >>> >
> > >>> >
> https://java-driver.docs.scylladb.com/scylla-4.15.0.x/api/index.html
> > >>> >
> > >>> > So I see no reason to not take them at their word that it's drop-in
> > >>> >
> > >>> > On Wed, Mar 20, 2024 at 11:04 AM David Handermann <
> > >>> > exceptionfact...@apache.org> wrote:
> > >>> >
> > >>> > > Mike,
> > >>> > >
> > >>> > > One important thing to mention about the DataStax vs ScyllaDB
> driver
> > >>> > > is that the Maven coordinates are different, and managing the
> > >>> > > dependencies correctly will make or break the implementation.
> > >>> > >
> > >>> > > In other words, if it is possible to use the DataStax 4 core JAR
> in
> > >>> > > the Controller Service API, but use the ScyllaDB 3 query JAR in
> the
> > >>> > > ScyllaDB implementation, then that could avoid th

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

It would be interesting to see if you exclude the Scylla API JAR from the
Scylla implementation and instead include DataStax's, if that works.
However I'm still leaning towards a vendor-agnostic API.

On Wed, Mar 20, 2024 at 11:26 AM Mike Thomsen 
wrote:

> At first glance, the package names look identical to me:
>
> https://java-driver.docs.scylladb.com/scylla-4.15.0.x/api/index.html
>
> So I see no reason to not take them at their word that it's drop-in
>
> On Wed, Mar 20, 2024 at 11:04 AM David Handermann <
> exceptionfact...@apache.org> wrote:
>
> > Mike,
> >
> > One important thing to mention about the DataStax vs ScyllaDB driver
> > is that the Maven coordinates are different, and managing the
> > dependencies correctly will make or break the implementation.
> >
> > In other words, if it is possible to use the DataStax 4 core JAR in
> > the Controller Service API, but use the ScyllaDB 3 query JAR in the
> > ScyllaDB implementation, then that could avoid the need for additional
> > abstraction. Without taking a closer look, however, I would be
> > surprised if this worked.
> >
> > Although ScyllaDB highlights their forked driver as a drop-in
> > replacement for the DataStax version, and maintains the same Java
> > package names, there is a difference between a complete replacement
> > and a shared API JAR. Without a common API JAR, that both
> > implementations can use, it will be necessary to provide an
> > abstraction in NiFi that avoids depending on either library at the
> > Controller Service API level.
> >
> > Regards,
> > David Handermann
> >
> > On Wed, Mar 20, 2024 at 8:25 AM Mike Thomsen 
> > wrote:
> > >
> > > Matt/David,
> > >
> > > I got some time this morning to take a crack at directly migrating it
> > over
> > > to the DataStax 4.17 driver. Definitely got a lot of work to do, but so
> > far
> > > I haven't hit any real snags. This is a branch that reverts the commit
> to
> > > remove the cassandra bundle and reuses the existing features as a
> > > foundation. From what I'm seeing so far (feels like I'm about 25% done)
> > it
> > > should be doable to reuse the existing bundle, but rename it to the
> "CQL
> > > Bundle" and just add a second controller service for Scylla that is
> > > otherwise 100% the same codewise.
> > >
> > > On Tue, Mar 19, 2024 at 6:41 PM Mike Thomsen 
> > wrote:
> > >
> > > > A cursory look at the Cassandra 5 stuff didn’t indicate any
> > > > incompatibility. So yeah, I think we are likely pretty safe to use
> the
> > 4.17
> > > > driver
> > > > Sent from my iPhone
> > > >
> > > > > On Mar 19, 2024, at 3:35 PM, Matt Burgess 
> > wrote:
> > > > >
> > > > > Is it likely now (due to the refactor) that we will simply be able
> > to
> > > > > upgrade the driver when Cassandra 5 is GA? Also does anyone use
> > Netflix's
> > > > > Astyanax [1]?
> > > > >
> > > > > [1]
> > > > >
> > > >
> >
> https://cassandra.apache.org/doc/stable/cassandra/getting_started/drivers.html#java
> > > > >
> > > > >> On Tue, Mar 19, 2024 at 3:10 PM Mike Thomsen <
> > mikerthom...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> Realistically, I think we are only likely to see two drivers:
> > > > >>
> > > > >> * DataStax
> > > > >> * ScyllaDB
> > > > >>
> > > > >> The latter makes a selling point of being a binary compatible,
> > drop-in
> > > > >> replacement for the former.
> > > > >>
> > > > >> That's why I don't see a need to have an abstraction layer per
> se. I
> > > > think
> > > > >> we only need "DataStaxConnectionProviderImpl" and
> > > > >> "ScyllaDBConnectionProviderImpl" with the difference being which
> > jar is
> > > > >> imported by maven.
> > > > >>
> > > > >> On Tue, Mar 19, 2024 at 2:59 PM David Handermann <
> > > > >> exceptionfact...@apache.org> wrote:
> > > > >>
> > > > >>> Mike,
> > > > >>>
> > > > >>> Thanks for the reply and clarification.
> > > > >>>
> > > > >>> I agree there is no need to m

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

; > > >>
> > > >> On Tue, Mar 19, 2024 at 12:37 PM Mike Thomsen <
> mikerthom...@gmail.com
> > >
> > > >> wrote:
> > > >> >
> > > >> > Matt,
> > > >> >
> > > >> > I got that. My point was that the Java changes appear to be a one
> > time
> > > >> > thing that DataStax did to make a better driver with a much more
> > > >> > future-proof API. Since Scylla tracks them as closely as
> possible, I
> > > >> > suspect that we don't need to plan for a bunch of abstraction to
> > isolate
> > > >> > Java changes.
> > > >> >
> > > >> > On Tue, Mar 19, 2024 at 11:07 AM Steven Matison <
> > > >> steven.mati...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > That was kinda where i got stuck and fell out on my branch/jira.
> > > >> Mike and
> > > >> > > I wanted to make a new controller service , without backward
> > > >> compatibility;
> > > >> > > and remove the duplicate driver/connection properties found in
> > some
> > > >> of the
> > > >> > > processors.
> > > >> > >
> > > >> > > I agree taking out all old stuff and making new controller
> service
> > > >> makes
> > > >> > > most sense.  4.x and 5.x should be mostly backwards compatible
> to
> > > >> 2&3.x
> > > >> > > with how it’s used within current processors.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Tue, Mar 19, 2024 at 10:49 AM Matt Burgess <
> > mattyb...@apache.org>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > The abstraction is to isolate Java API changes, not protocol
> > > >> > > compatibility
> > > >> > > > Changing to the java-driver comes with a number of changes to
> > the
> > > >> code
> > > >> > > (see
> > > >> > > > Steven's and my branches), if we can abstract that API it
> should
> > > >> lead to
> > > >> > > > more maintainable code in the future by not having to change
> any
> > > >> > > > processors, just the controller service implementation.
> > > >> > > >
> > > >> > > >
> > > >> > > > On Tue, Mar 19, 2024 at 10:14 AM Mike Thomsen <
> > > >> mikerthom...@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >>
> >
> https://opensource.docs.scylladb.com/stable/using-scylla/drivers/cql-drivers/scylla-java-driver.html
> > > >> > > > >
> > > >> > > > > Directly quoting Scylla docs here:
> > > >> > > > >
> > > >> > > > > > The Scylla Java Driver is a drop-in replacement for the
> > > >> DataStax Java
> > > >> > > > > Driver. As such, no code changes are needed to use this
> > driver.
> > > >> > > > >
> > > >> > > > > On Tue, Mar 19, 2024 at 10:13 AM Mike Thomsen <
> > > >> mikerthom...@gmail.com>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Matt,
> > > >> > > > > >
> > > >> > > > > > I don't think we need to really "abstract above" the
> drivers
> > > >> because
> > > >> > > > the
> > > >> > > > > > Java DataStax driver appears to support 4.X all the way
> > back to
> > > >> 2.X,
> > > >> > > as
> > > >> > > > > > well as the enterprise versions from DataStax
> > > >> > > > > >
> > > >> > > > > >
> > > >> https://docs.datastax.com/en/driver-matrix/docs/java-drivers.html
> > > >> > > > > >
> > > >> > > > > > Similar situation with Scylla. When I loo

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

The abstraction is to isolate Java API changes, not protocol compatibility.
Changing to the java-driver comes with a number of changes to the code (see
Steven's and my branches), if we can abstract that API it should lead to
more maintainable code in the future by not having to change any
processors, just the controller service implementation.


On Tue, Mar 19, 2024 at 10:14 AM Mike Thomsen 
wrote:

>
> https://opensource.docs.scylladb.com/stable/using-scylla/drivers/cql-drivers/scylla-java-driver.html
>
> Directly quoting Scylla docs here:
>
> > The Scylla Java Driver is a drop-in replacement for the DataStax Java
> Driver. As such, no code changes are needed to use this driver.
>
> On Tue, Mar 19, 2024 at 10:13 AM Mike Thomsen 
> wrote:
>
> > Matt,
> >
> > I don't think we need to really "abstract above" the drivers because the
> > Java DataStax driver appears to support 4.X all the way back to 2.X, as
> > well as the enterprise versions from DataStax
> >
> > https://docs.datastax.com/en/driver-matrix/docs/java-drivers.html
> >
> > Similar situation with Scylla. When I looked at the driver, it appeared
> to
> > copy verbatim the entire public API of that driver. So I think before we
> > dive into abstractions, it's worth doing a bit more validation of these
> > details. IMHO, this might be a much lighter lift than anticipated.
> >
> >
> > On Mon, Mar 18, 2024 at 4:30 PM Matt Burgess 
> wrote:
> >
> >> Totally agree, that's what my branch does (see link in previous email).
> >> The
> >> more I work with it, the more I think I can abstract it further from
> their
> >> JDBC-like API but I started with a bunch of delegate classes then I
> figure
> >> I'll see where I can consolidate to more abstract concepts. If I don't
> >> have
> >> to support Cassandra 3 with the new API, so much the better.
> >>
> >> Regards,
> >> Matt
> >>
> >> On Mon, Mar 18, 2024 at 4:14 PM David Handermann <
> >> exceptionfact...@apache.org> wrote:
> >>
> >> > Matt et al,
> >> >
> >> > It is good to see the background effort on moving Cassandra
> >> > capabilities in a supportable direction.
> >> >
> >> > I think new Cassandra components will require a significant departure
> >> > from current Controller Service abstractions. Right now, the existing
> >> > service interface does not provide a clean abstraction from the
> >> > Cassandra library, which is part of the reason for the current
> >> > coupling to the legacy driver version.
> >> >
> >> > Following up from Joe's comments, it seems like the cleanest way
> >> > forward is to deprecate the current bundle on the 1.x branch, and
> >> > remove the current bundle from the main branch. That will provide a
> >> > clean slate for new Service and Processor implementations, without
> >> > concern for uncertain compatibility questions.
> >> >
> >> > Regards,
> >> > David Handermann
> >> >
> >> > On Mon, Mar 18, 2024 at 2:35 PM Matt Burgess 
> >> wrote:
> >> > >
> >> > > What do y'all think about removing the individual connection
> >> properties
> >> > > from the Cassandra processors for NiFi 2.0 and requiring a
> >> > > CassandraSessionProvider instead? I think we started doing that
> >> elsewhere
> >> > > (Elasticsearch maybe?), I noticed duplicate code in the
> >> > > CassandraSessionProvider and AbstractCassandraProcessor, if we keep
> >> those
> >> > > properties I can refactor them into a utility class.
> >> > >
> >> > > Thanks,
> >> > > Matt
> >> > >
> >> > >
> >> > > On Fri, Mar 15, 2024 at 2:44 PM Steven Matison <
> >> steven.mati...@gmail.com
> >> > >
> >> > > wrote:
> >> > >
> >> > > > I got through quite a bit of work to enable 4.x…
> >> > > >
> >> > > > The 3.x pieces that were not backwards compatible is very edge use
> >> > case and
> >> > > > could have been done slightly differently but with work around.
> >> > > >
> >> > > > https://github.com/steven-matison/nifi/tree/nifi-10120-1
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> 

Re: DB Dynamic Connection

[Matt Burgess]

Eduardo,

I believe if you have a Parameter Context set on a process group then those
values will be available when you evaluate Expression Language anywhere in
the processor / controller code, as long as your properties are referring
to the parameters and not variables.

Regards,
Matt

On Sun, Mar 17, 2024 at 9:38 PM Eduardo Fontes 
wrote:

> Hi Matt,
>
> I encountered some issues while attempting to implement what I had in mind.
> The main obstacle is that I'm unsure how to pass a "Parameter Context"
> parameter name as a variable, as the evaluation of P.C. occurs during the
> startup phase of the processor/controller. I require this functionality to
> transmit sensitive values, such as passwords, without exposing them in
> flowfile properties as plain text. My initial idea was to obtain a
> parameter context from within a controller code, which is invoked from
> within the onTrigger function of a processor.
>
> Perhaps it would be better if the Parameter Context acted as Azure Key
> Vault or AWS Secrets Manager!
>
> Any thoughts?
>
> PS.: My plan B is to implement it using a Scripted Processor that: 1) get
> values for db connection from a secret vault in the cloud; 2) make a DB
> connection; 3) Read the data and send to processor relationship; 4) close
> connection.
>
> On Fri, Mar 15, 2024 at 5:23 PM Matt Burgess  wrote:
>
> > True, but my concern is that you might see performance issues with a new
> > connection each time, especially if the same value(s) come in many times
> in
> > a row (i.e. choosing the same connection config). Having a small cache
> > might afford you some speedups.
> >
> > Regards,
> > Matt
> >
> > On Sun, Mar 10, 2024 at 9:17 AM Eduardo Fontes  >
> > wrote:
> >
> > > Hi Matt,
> > >
> > > I don't think I need a pool or a cache, since DB connection will be
> used
> > > once for an object (table/view). So I think that won't be a problem
> > create
> > > a DB connection, read object and destroy connection, for each object.
> > >
> > > I'll try to implement this using DBCPService Controller Interface.
> > >
> > > Thanks for your consideration.
> > >
> > > Eduardo Fontes
> > >
> > > On Tue, Mar 5, 2024 at 11:10 PM Matt Burgess 
> > wrote:
> > >
> > > > Eduardo,
> > > >
> > > > It doesn't sound like DBCPConnectionPoolLookup will work for you
> > because
> > > of
> > > > all the different connection strings. I don't know if there's a good
> > > reason
> > > > why we couldn't create the BasicDataSource when getConnection() is
> > > called,
> > > > passing in a Map of FlowFile attributes (that's how the Lookup
> version
> > > > works). One issue I do see is with "churn" if we're recreating the
> data
> > > > source each time. At that point it's not pooling connections. I
> suppose
> > > you
> > > > could have an internal cache of data sources but it would have to be
> > > > bounded and/or configurable and have a least-recently-used (LRU)
> > eviction
> > > > strategy.
> > > >
> > > > DBCPService is the name of the controller service interface that the
> > > > database processors use, but that's a misnomer since the API doesn't
> > > > mention pooling specifically. Instead you could have an
> implementation
> > > that
> > > > uses a cache vs a pooling approach. But Apache DBCP does handle a lot
> > of
> > > > the management (validation, eviction, idle timeouts, etc.)  so unless
> > > > there's no way to avoid the potential memory/performance issues (like
> > > > having 50+ controller services in a PG) you could try to wrangle
> > smaller
> > > > pools per data source and cache those if that's ok for your use case.
> > > >
> > > > My two cents,
> > > > Matt
> > > >
> > > > On Tue, Mar 5, 2024 at 7:25 PM Eduardo Fontes <
> > eduardo.fon...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Everybody!
> > > > >
> > > > > I'm thinking about make a generic ingestor with Apache NiFi but I
> > found
> > > > > some difficulties because of the DataBase Connection Pool
> controller.
> > > It
> > > > > doesn't accept flowfiles parameters for its properties, specially
> > > > > connection string, username and password (for security reasons,
> some
> > > > > sensitive parameter name instead password itself).
> > > > >
> > > > > This is important because, as a generic ingestor, I might have
> > hundreds
> > > > of
> > > > > different connection strings, and I had a lot of problems when I
> > tried
> > > to
> > > > > put 50 DBCP controllers in a Process Group.
> > > > >
> > > > > I wouldn't like to create a flow for each ingestion, but one flow
> for
> > > > each
> > > > > database vendor.
> > > > >
> > > > > Does anyone have any suggestions on how I can achieve this? Would
> it
> > be
> > > > > easy to create a parameterized DBCP controller? (That I could do it
> > > > myself)
> > > > >
> > > > > Best regards.
> > > > >
> > > > > Eduardo Fontes
> > > > > Data Eng / System Analyst Sr.
> > > > >
> > > >
> > >
> >
>

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

Sounds like a plan, thanks much!

On Mon, Mar 18, 2024 at 4:49 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Matt,
>
> Thanks, that makes sense on further review of the branch you mentioned
> previously.
>
> It sounds like we have a general consensus on a way forward.
>
> I will proceed with writing up the Jira issues and putting together
> pull requests for deprecation and removal of the existing Cassandra 3
> components. That should put things in a good place to land the new
> capabilities when they are ready. It also resolves the current
> vulnerability findings with the legacy driver, so this approach is
> helpful on several fronts.
>
> Regards,
> David Handermann
>
> On Mon, Mar 18, 2024 at 3:30 PM Matt Burgess  wrote:
> >
> > Totally agree, that's what my branch does (see link in previous email).
> The
> > more I work with it, the more I think I can abstract it further from
> their
> > JDBC-like API but I started with a bunch of delegate classes then I
> figure
> > I'll see where I can consolidate to more abstract concepts. If I don't
> have
> > to support Cassandra 3 with the new API, so much the better.
> >
> > Regards,
> > Matt
> >
> > On Mon, Mar 18, 2024 at 4:14 PM David Handermann <
> > exceptionfact...@apache.org> wrote:
> >
> > > Matt et al,
> > >
> > > It is good to see the background effort on moving Cassandra
> > > capabilities in a supportable direction.
> > >
> > > I think new Cassandra components will require a significant departure
> > > from current Controller Service abstractions. Right now, the existing
> > > service interface does not provide a clean abstraction from the
> > > Cassandra library, which is part of the reason for the current
> > > coupling to the legacy driver version.
> > >
> > > Following up from Joe's comments, it seems like the cleanest way
> > > forward is to deprecate the current bundle on the 1.x branch, and
> > > remove the current bundle from the main branch. That will provide a
> > > clean slate for new Service and Processor implementations, without
> > > concern for uncertain compatibility questions.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Mon, Mar 18, 2024 at 2:35 PM Matt Burgess 
> wrote:
> > > >
> > > > What do y'all think about removing the individual connection
> properties
> > > > from the Cassandra processors for NiFi 2.0 and requiring a
> > > > CassandraSessionProvider instead? I think we started doing that
> elsewhere
> > > > (Elasticsearch maybe?), I noticed duplicate code in the
> > > > CassandraSessionProvider and AbstractCassandraProcessor, if we keep
> those
> > > > properties I can refactor them into a utility class.
> > > >
> > > > Thanks,
> > > > Matt
> > > >
> > > >
> > > > On Fri, Mar 15, 2024 at 2:44 PM Steven Matison <
> steven.mati...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > I got through quite a bit of work to enable 4.x…
> > > > >
> > > > > The 3.x pieces that were not backwards compatible is very edge use
> > > case and
> > > > > could have been done slightly differently but with work around.
> > > > >
> > > > > https://github.com/steven-matison/nifi/tree/nifi-10120-1
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Mar 15, 2024 at 2:30 PM Matt Burgess  >
> > > wrote:
> > > > >
> > > > > > Oops used the wrong email address so if there have been
> responses to
> > > the
> > > > > > Cassandra thread since mine I missed them, my bad!
> > > > > >
> > > > > > On Fri, Mar 15, 2024 at 2:00 PM Matt Burgess <
> mattyb...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > I believe the CQL protocol is backwards compatible but the Java
> > > API is
> > > > > > > not. For example "com.datastax.driver.core.Session" is now
> > > > > > > "com.datastax.oss.driver.api.core.session.Session" and there
> is no
> > > more
> > > > > > > "Cluster" class. Might be fairly trivial to fix though, if
> that's
> > >

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

Totally agree, that's what my branch does (see link in previous email). The
more I work with it, the more I think I can abstract it further from their
JDBC-like API but I started with a bunch of delegate classes then I figure
I'll see where I can consolidate to more abstract concepts. If I don't have
to support Cassandra 3 with the new API, so much the better.

Regards,
Matt

On Mon, Mar 18, 2024 at 4:14 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Matt et al,
>
> It is good to see the background effort on moving Cassandra
> capabilities in a supportable direction.
>
> I think new Cassandra components will require a significant departure
> from current Controller Service abstractions. Right now, the existing
> service interface does not provide a clean abstraction from the
> Cassandra library, which is part of the reason for the current
> coupling to the legacy driver version.
>
> Following up from Joe's comments, it seems like the cleanest way
> forward is to deprecate the current bundle on the 1.x branch, and
> remove the current bundle from the main branch. That will provide a
> clean slate for new Service and Processor implementations, without
> concern for uncertain compatibility questions.
>
> Regards,
> David Handermann
>
> On Mon, Mar 18, 2024 at 2:35 PM Matt Burgess  wrote:
> >
> > What do y'all think about removing the individual connection properties
> > from the Cassandra processors for NiFi 2.0 and requiring a
> > CassandraSessionProvider instead? I think we started doing that elsewhere
> > (Elasticsearch maybe?), I noticed duplicate code in the
> > CassandraSessionProvider and AbstractCassandraProcessor, if we keep those
> > properties I can refactor them into a utility class.
> >
> > Thanks,
> > Matt
> >
> >
> > On Fri, Mar 15, 2024 at 2:44 PM Steven Matison  >
> > wrote:
> >
> > > I got through quite a bit of work to enable 4.x…
> > >
> > > The 3.x pieces that were not backwards compatible is very edge use
> case and
> > > could have been done slightly differently but with work around.
> > >
> > > https://github.com/steven-matison/nifi/tree/nifi-10120-1
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Mar 15, 2024 at 2:30 PM Matt Burgess 
> wrote:
> > >
> > > > Oops used the wrong email address so if there have been responses to
> the
> > > > Cassandra thread since mine I missed them, my bad!
> > > >
> > > > On Fri, Mar 15, 2024 at 2:00 PM Matt Burgess 
> > > wrote:
> > > >
> > > > > I believe the CQL protocol is backwards compatible but the Java
> API is
> > > > > not. For example "com.datastax.driver.core.Session" is now
> > > > > "com.datastax.oss.driver.api.core.session.Session" and there is no
> more
> > > > > "Cluster" class. Might be fairly trivial to fix though, if that's
> the
> > > > path
> > > > > of least resistance.
> > > > >
> > > > > On Fri, Mar 15, 2024 at 1:40 PM Joe Witt 
> wrote:
> > > > >
> > > > >> Matt
> > > > >>
> > > > >> I dont know a ton about Cassandra but when I looked at
> client/driver
> > > > notes
> > > > >> for 4+ it said it was compatible all the way back to 3.x.   Not
> sure
> > > > what
> > > > >> that means but it surely seems worth exploring.  Also I dont know
> if
> > > the
> > > > >> 4.x drivers get rid of the vulnerable bits.
> > > > >>
> > > > >> Thanks
> > > > >>
> > > > >> On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess <
> mattyb...@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >> > At the very least we should upgrade to Cassandra 3.11.6:
> > > > >> >
> > > >
> https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt
> > > > >> >
> > > > >> > On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess <
> mattyb...@apache.org>
> > > > >> wrote:
> > > > >> >
> > > > >> > > If the community agrees to get rid of Cassandra 3 that'll
> save me
> > > > >> effort
> > > > >> > > on the refactor after I add Cassandra 4 :) Otherwise those
> > > > >> > > vulnerabilities would only be in a "new" Cas

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

What do y'all think about removing the individual connection properties
from the Cassandra processors for NiFi 2.0 and requiring a
CassandraSessionProvider instead? I think we started doing that elsewhere
(Elasticsearch maybe?), I noticed duplicate code in the
CassandraSessionProvider and AbstractCassandraProcessor, if we keep those
properties I can refactor them into a utility class.

Thanks,
Matt


On Fri, Mar 15, 2024 at 2:44 PM Steven Matison 
wrote:

> I got through quite a bit of work to enable 4.x…
>
> The 3.x pieces that were not backwards compatible is very edge use case and
> could have been done slightly differently but with work around.
>
> https://github.com/steven-matison/nifi/tree/nifi-10120-1
>
>
>
>
>
>
> On Fri, Mar 15, 2024 at 2:30 PM Matt Burgess  wrote:
>
> > Oops used the wrong email address so if there have been responses to the
> > Cassandra thread since mine I missed them, my bad!
> >
> > On Fri, Mar 15, 2024 at 2:00 PM Matt Burgess 
> wrote:
> >
> > > I believe the CQL protocol is backwards compatible but the Java API is
> > > not. For example "com.datastax.driver.core.Session" is now
> > > "com.datastax.oss.driver.api.core.session.Session" and there is no more
> > > "Cluster" class. Might be fairly trivial to fix though, if that's the
> > path
> > > of least resistance.
> > >
> > > On Fri, Mar 15, 2024 at 1:40 PM Joe Witt  wrote:
> > >
> > >> Matt
> > >>
> > >> I dont know a ton about Cassandra but when I looked at client/driver
> > notes
> > >> for 4+ it said it was compatible all the way back to 3.x.   Not sure
> > what
> > >> that means but it surely seems worth exploring.  Also I dont know if
> the
> > >> 4.x drivers get rid of the vulnerable bits.
> > >>
> > >> Thanks
> > >>
> > >> On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess 
> > >> wrote:
> > >>
> > >> > At the very least we should upgrade to Cassandra 3.11.6:
> > >> >
> > https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt
> > >> >
> > >> > On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess 
> > >> wrote:
> > >> >
> > >> > > If the community agrees to get rid of Cassandra 3 that'll save me
> > >> effort
> > >> > > on the refactor after I add Cassandra 4 :) Otherwise those
> > >> > > vulnerabilities would only be in a "new" Cassandra 3 services NAR
> > that
> > >> > > would not be included in the convenience binary.
> > >> > >
> > >> > > On Fri, Mar 15, 2024 at 1:28 PM Joe Witt 
> > wrote:
> > >> > >
> > >> > >> Mike, Matt,
> > >> > >>
> > >> > >> Happy to hear you both have active efforts or are interested in
> > doing
> > >> > so.
> > >> > >> Can you help me understand more specifically what that means for
> > the
> > >> > >> current set of components?
> > >> > >>
> > >> > >> The CVE hits are concerning and long standing.  Supporting
> > Cassandra
> > >> 3
> > >> > >> implies the current set of dependencies would remain too right?
> > >> > >>
> > >> > >> Is the current set of components we have ones we want to retain?
> > We
> > >> > >> certainly need Cassandra components - but are the ones we have
> now
> > >> the
> > >> > >> right ones?
> > >> > >>
> > >> > >> Thanks
> > >> > >> Joe
> > >> > >>
> > >> > >> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess <
> > mattyb...@apache.org>
> > >> > >> wrote:
> > >> > >>
> > >> > >> > I'm actively working this, I pushed my branch up in case anyone
> > >> wants
> > >> > to
> > >> > >> > take a look [1]. The idea is to abstract the Cassandra API "up
> a
> > >> > couple
> > >> > >> > levels" and provide implementations for Cassandra 3, 4, and
> > >> eventually
> > >> > >> 5.
> > >> > >> > For JDBC-like interfaces this is a PITA because of the API
> > >> (Statement,
> > >> > >> > PreparedStatement, B

Re: DB Dynamic Connection

[Matt Burgess]

True, but my concern is that you might see performance issues with a new
connection each time, especially if the same value(s) come in many times in
a row (i.e. choosing the same connection config). Having a small cache
might afford you some speedups.

Regards,
Matt

On Sun, Mar 10, 2024 at 9:17 AM Eduardo Fontes 
wrote:

> Hi Matt,
>
> I don't think I need a pool or a cache, since DB connection will be used
> once for an object (table/view). So I think that won't be a problem create
> a DB connection, read object and destroy connection, for each object.
>
> I'll try to implement this using DBCPService Controller Interface.
>
> Thanks for your consideration.
>
> Eduardo Fontes
>
> On Tue, Mar 5, 2024 at 11:10 PM Matt Burgess  wrote:
>
> > Eduardo,
> >
> > It doesn't sound like DBCPConnectionPoolLookup will work for you because
> of
> > all the different connection strings. I don't know if there's a good
> reason
> > why we couldn't create the BasicDataSource when getConnection() is
> called,
> > passing in a Map of FlowFile attributes (that's how the Lookup version
> > works). One issue I do see is with "churn" if we're recreating the data
> > source each time. At that point it's not pooling connections. I suppose
> you
> > could have an internal cache of data sources but it would have to be
> > bounded and/or configurable and have a least-recently-used (LRU) eviction
> > strategy.
> >
> > DBCPService is the name of the controller service interface that the
> > database processors use, but that's a misnomer since the API doesn't
> > mention pooling specifically. Instead you could have an implementation
> that
> > uses a cache vs a pooling approach. But Apache DBCP does handle a lot of
> > the management (validation, eviction, idle timeouts, etc.)  so unless
> > there's no way to avoid the potential memory/performance issues (like
> > having 50+ controller services in a PG) you could try to wrangle smaller
> > pools per data source and cache those if that's ok for your use case.
> >
> > My two cents,
> > Matt
> >
> > On Tue, Mar 5, 2024 at 7:25 PM Eduardo Fontes 
> > wrote:
> >
> > > Hi Everybody!
> > >
> > > I'm thinking about make a generic ingestor with Apache NiFi but I found
> > > some difficulties because of the DataBase Connection Pool controller.
> It
> > > doesn't accept flowfiles parameters for its properties, specially
> > > connection string, username and password (for security reasons, some
> > > sensitive parameter name instead password itself).
> > >
> > > This is important because, as a generic ingestor, I might have hundreds
> > of
> > > different connection strings, and I had a lot of problems when I tried
> to
> > > put 50 DBCP controllers in a Process Group.
> > >
> > > I wouldn't like to create a flow for each ingestion, but one flow for
> > each
> > > database vendor.
> > >
> > > Does anyone have any suggestions on how I can achieve this? Would it be
> > > easy to create a parameterized DBCP controller? (That I could do it
> > myself)
> > >
> > > Best regards.
> > >
> > > Eduardo Fontes
> > > Data Eng / System Analyst Sr.
> > >
> >
>

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

Oops used the wrong email address so if there have been responses to the
Cassandra thread since mine I missed them, my bad!

On Fri, Mar 15, 2024 at 2:00 PM Matt Burgess  wrote:

> I believe the CQL protocol is backwards compatible but the Java API is
> not. For example "com.datastax.driver.core.Session" is now
> "com.datastax.oss.driver.api.core.session.Session" and there is no more
> "Cluster" class. Might be fairly trivial to fix though, if that's the path
> of least resistance.
>
> On Fri, Mar 15, 2024 at 1:40 PM Joe Witt  wrote:
>
>> Matt
>>
>> I dont know a ton about Cassandra but when I looked at client/driver notes
>> for 4+ it said it was compatible all the way back to 3.x.   Not sure what
>> that means but it surely seems worth exploring.  Also I dont know if the
>> 4.x drivers get rid of the vulnerable bits.
>>
>> Thanks
>>
>> On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess 
>> wrote:
>>
>> > At the very least we should upgrade to Cassandra 3.11.6:
>> > https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt
>> >
>> > On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess 
>> wrote:
>> >
>> > > If the community agrees to get rid of Cassandra 3 that'll save me
>> effort
>> > > on the refactor after I add Cassandra 4 :) Otherwise those
>> > > vulnerabilities would only be in a "new" Cassandra 3 services NAR that
>> > > would not be included in the convenience binary.
>> > >
>> > > On Fri, Mar 15, 2024 at 1:28 PM Joe Witt  wrote:
>> > >
>> > >> Mike, Matt,
>> > >>
>> > >> Happy to hear you both have active efforts or are interested in doing
>> > so.
>> > >> Can you help me understand more specifically what that means for the
>> > >> current set of components?
>> > >>
>> > >> The CVE hits are concerning and long standing.  Supporting Cassandra
>> 3
>> > >> implies the current set of dependencies would remain too right?
>> > >>
>> > >> Is the current set of components we have ones we want to retain?  We
>> > >> certainly need Cassandra components - but are the ones we have now
>> the
>> > >> right ones?
>> > >>
>> > >> Thanks
>> > >> Joe
>> > >>
>> > >> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess 
>> > >> wrote:
>> > >>
>> > >> > I'm actively working this, I pushed my branch up in case anyone
>> wants
>> > to
>> > >> > take a look [1]. The idea is to abstract the Cassandra API "up a
>> > couple
>> > >> > levels" and provide implementations for Cassandra 3, 4, and
>> eventually
>> > >> 5.
>> > >> > For JDBC-like interfaces this is a PITA because of the API
>> (Statement,
>> > >> > PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping
>> we
>> > >> can
>> > >> > find a common pattern for abstracting the third-party library
>> > >> > implementation and API from the NiFi component (Processor,
>> > >> > ControllerService, etc.) API. I think we're doing something similar
>> > for
>> > >> > Kafka?
>> > >> >
>> > >> > Regards,
>> > >> > Matt
>> > >> >
>> > >> > [1] https://github.com/mattyb149/nifi/tree/cassy4
>> > >> >
>> > >> >
>> > >> > On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen <
>> mikerthom...@gmail.com>
>> > >> > wrote:
>> > >> >
>> > >> > > That’s been on my todo list for a little while but things kept
>> > coming
>> > >> up.
>> > >> > > I think I could get started on that now. Based on my initial
>> > research
>> > >> it
>> > >> > > appears that scylla uses the exact same api as datastax so
>> > supporting
>> > >> > both
>> > >> > > in a cql bundle should theoretically be fairly easy.
>> > >> > >
>> > >> > >
>> > >> > > Sent from my iPhone
>> > >> > >
>> > >> > > > On Mar 14, 2024, at 6:18 PM, Joe Witt 
>> wrote:
>> > >> > > >
>> > >> > > > Te

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

I believe the CQL protocol is backwards compatible but the Java API is not.
For example "com.datastax.driver.core.Session" is now
"com.datastax.oss.driver.api.core.session.Session" and there is no more
"Cluster" class. Might be fairly trivial to fix though, if that's the path
of least resistance.

On Fri, Mar 15, 2024 at 1:40 PM Joe Witt  wrote:

> Matt
>
> I dont know a ton about Cassandra but when I looked at client/driver notes
> for 4+ it said it was compatible all the way back to 3.x.   Not sure what
> that means but it surely seems worth exploring.  Also I dont know if the
> 4.x drivers get rid of the vulnerable bits.
>
> Thanks
>
> On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess 
> wrote:
>
> > At the very least we should upgrade to Cassandra 3.11.6:
> > https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt
> >
> > On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess 
> wrote:
> >
> > > If the community agrees to get rid of Cassandra 3 that'll save me
> effort
> > > on the refactor after I add Cassandra 4 :) Otherwise those
> > > vulnerabilities would only be in a "new" Cassandra 3 services NAR that
> > > would not be included in the convenience binary.
> > >
> > > On Fri, Mar 15, 2024 at 1:28 PM Joe Witt  wrote:
> > >
> > >> Mike, Matt,
> > >>
> > >> Happy to hear you both have active efforts or are interested in doing
> > so.
> > >> Can you help me understand more specifically what that means for the
> > >> current set of components?
> > >>
> > >> The CVE hits are concerning and long standing.  Supporting Cassandra 3
> > >> implies the current set of dependencies would remain too right?
> > >>
> > >> Is the current set of components we have ones we want to retain?  We
> > >> certainly need Cassandra components - but are the ones we have now the
> > >> right ones?
> > >>
> > >> Thanks
> > >> Joe
> > >>
> > >> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess 
> > >> wrote:
> > >>
> > >> > I'm actively working this, I pushed my branch up in case anyone
> wants
> > to
> > >> > take a look [1]. The idea is to abstract the Cassandra API "up a
> > couple
> > >> > levels" and provide implementations for Cassandra 3, 4, and
> eventually
> > >> 5.
> > >> > For JDBC-like interfaces this is a PITA because of the API
> (Statement,
> > >> > PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping
> we
> > >> can
> > >> > find a common pattern for abstracting the third-party library
> > >> > implementation and API from the NiFi component (Processor,
> > >> > ControllerService, etc.) API. I think we're doing something similar
> > for
> > >> > Kafka?
> > >> >
> > >> > Regards,
> > >> > Matt
> > >> >
> > >> > [1] https://github.com/mattyb149/nifi/tree/cassy4
> > >> >
> > >> >
> > >> > On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen <
> mikerthom...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > That’s been on my todo list for a little while but things kept
> > coming
> > >> up.
> > >> > > I think I could get started on that now. Based on my initial
> > research
> > >> it
> > >> > > appears that scylla uses the exact same api as datastax so
> > supporting
> > >> > both
> > >> > > in a cql bundle should theoretically be fairly easy.
> > >> > >
> > >> > >
> > >> > > Sent from my iPhone
> > >> > >
> > >> > > > On Mar 14, 2024, at 6:18 PM, Joe Witt 
> wrote:
> > >> > > >
> > >> > > > Team,
> > >> > > >
> > >> > > > Cassandra remains a really important system to be able to send
> > data
> > >> to.
> > >> > > > However, it seems like we've not maintained these well.  We have
> > >> what
> > >> > > > appears to be at least a full generation behind on client
> versions
> > >> (we
> > >> > > are
> > >> > > > on 3x vs 4x which is the latest stable with 5x apparently coming
> > >> > > shortly).
> > >> > > >
> > >> > > > We have components to send data, query data, and use Cassandra
> as
> > a
> > >> > cache
> > >> > > > store.  We have older mechanisms for json/avro and publish
> > >> mechanisms
> > >> > for
> > >> > > > records.
> > >> > > >
> > >> > > > The libraries we do have depend on outdated versions of Guava
> and
> > >> > result
> > >> > > in
> > >> > > > many CVE hits.
> > >> > > >
> > >> > > > I am inclined to think we should deprecate the 1.x components
> and
> > >> > remove
> > >> > > > them as-is from the 2.x line.  Then re-introduce them with
> record
> > >> only
> > >> > > > interfaces and built against the latest stable
> > >> > > Cassandra/Datastax/ScyllaDB
> > >> > > > interfaces.
> > >> > > >
> > >> > > > I'd love to hear thoughts from those closer to this space both
> as
> > a
> > >> > user
> > >> > > > and developer so we can make good next steps.
> > >> > > >
> > >> > > > Thanks
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

At the very least we should upgrade to Cassandra 3.11.6:
https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt

On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess  wrote:

> If the community agrees to get rid of Cassandra 3 that'll save me effort
> on the refactor after I add Cassandra 4 :) Otherwise those
> vulnerabilities would only be in a "new" Cassandra 3 services NAR that
> would not be included in the convenience binary.
>
> On Fri, Mar 15, 2024 at 1:28 PM Joe Witt  wrote:
>
>> Mike, Matt,
>>
>> Happy to hear you both have active efforts or are interested in doing so.
>> Can you help me understand more specifically what that means for the
>> current set of components?
>>
>> The CVE hits are concerning and long standing.  Supporting Cassandra 3
>> implies the current set of dependencies would remain too right?
>>
>> Is the current set of components we have ones we want to retain?  We
>> certainly need Cassandra components - but are the ones we have now the
>> right ones?
>>
>> Thanks
>> Joe
>>
>> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess 
>> wrote:
>>
>> > I'm actively working this, I pushed my branch up in case anyone wants to
>> > take a look [1]. The idea is to abstract the Cassandra API "up a couple
>> > levels" and provide implementations for Cassandra 3, 4, and eventually
>> 5.
>> > For JDBC-like interfaces this is a PITA because of the API (Statement,
>> > PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping we
>> can
>> > find a common pattern for abstracting the third-party library
>> > implementation and API from the NiFi component (Processor,
>> > ControllerService, etc.) API. I think we're doing something similar for
>> > Kafka?
>> >
>> > Regards,
>> > Matt
>> >
>> > [1] https://github.com/mattyb149/nifi/tree/cassy4
>> >
>> >
>> > On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen 
>> > wrote:
>> >
>> > > That’s been on my todo list for a little while but things kept coming
>> up.
>> > > I think I could get started on that now. Based on my initial research
>> it
>> > > appears that scylla uses the exact same api as datastax so supporting
>> > both
>> > > in a cql bundle should theoretically be fairly easy.
>> > >
>> > >
>> > > Sent from my iPhone
>> > >
>> > > > On Mar 14, 2024, at 6:18 PM, Joe Witt  wrote:
>> > > >
>> > > > Team,
>> > > >
>> > > > Cassandra remains a really important system to be able to send data
>> to.
>> > > > However, it seems like we've not maintained these well.  We have
>> what
>> > > > appears to be at least a full generation behind on client versions
>> (we
>> > > are
>> > > > on 3x vs 4x which is the latest stable with 5x apparently coming
>> > > shortly).
>> > > >
>> > > > We have components to send data, query data, and use Cassandra as a
>> > cache
>> > > > store.  We have older mechanisms for json/avro and publish
>> mechanisms
>> > for
>> > > > records.
>> > > >
>> > > > The libraries we do have depend on outdated versions of Guava and
>> > result
>> > > in
>> > > > many CVE hits.
>> > > >
>> > > > I am inclined to think we should deprecate the 1.x components and
>> > remove
>> > > > them as-is from the 2.x line.  Then re-introduce them with record
>> only
>> > > > interfaces and built against the latest stable
>> > > Cassandra/Datastax/ScyllaDB
>> > > > interfaces.
>> > > >
>> > > > I'd love to hear thoughts from those closer to this space both as a
>> > user
>> > > > and developer so we can make good next steps.
>> > > >
>> > > > Thanks
>> > >
>> >
>>
>

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

If the community agrees to get rid of Cassandra 3 that'll save me effort on
the refactor after I add Cassandra 4 :) Otherwise those
vulnerabilities would only be in a "new" Cassandra 3 services NAR that
would not be included in the convenience binary.

On Fri, Mar 15, 2024 at 1:28 PM Joe Witt  wrote:

> Mike, Matt,
>
> Happy to hear you both have active efforts or are interested in doing so.
> Can you help me understand more specifically what that means for the
> current set of components?
>
> The CVE hits are concerning and long standing.  Supporting Cassandra 3
> implies the current set of dependencies would remain too right?
>
> Is the current set of components we have ones we want to retain?  We
> certainly need Cassandra components - but are the ones we have now the
> right ones?
>
> Thanks
> Joe
>
> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess 
> wrote:
>
> > I'm actively working this, I pushed my branch up in case anyone wants to
> > take a look [1]. The idea is to abstract the Cassandra API "up a couple
> > levels" and provide implementations for Cassandra 3, 4, and eventually 5.
> > For JDBC-like interfaces this is a PITA because of the API (Statement,
> > PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping we can
> > find a common pattern for abstracting the third-party library
> > implementation and API from the NiFi component (Processor,
> > ControllerService, etc.) API. I think we're doing something similar for
> > Kafka?
> >
> > Regards,
> > Matt
> >
> > [1] https://github.com/mattyb149/nifi/tree/cassy4
> >
> >
> > On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen 
> > wrote:
> >
> > > That’s been on my todo list for a little while but things kept coming
> up.
> > > I think I could get started on that now. Based on my initial research
> it
> > > appears that scylla uses the exact same api as datastax so supporting
> > both
> > > in a cql bundle should theoretically be fairly easy.
> > >
> > >
> > > Sent from my iPhone
> > >
> > > > On Mar 14, 2024, at 6:18 PM, Joe Witt  wrote:
> > > >
> > > > Team,
> > > >
> > > > Cassandra remains a really important system to be able to send data
> to.
> > > > However, it seems like we've not maintained these well.  We have what
> > > > appears to be at least a full generation behind on client versions
> (we
> > > are
> > > > on 3x vs 4x which is the latest stable with 5x apparently coming
> > > shortly).
> > > >
> > > > We have components to send data, query data, and use Cassandra as a
> > cache
> > > > store.  We have older mechanisms for json/avro and publish mechanisms
> > for
> > > > records.
> > > >
> > > > The libraries we do have depend on outdated versions of Guava and
> > result
> > > in
> > > > many CVE hits.
> > > >
> > > > I am inclined to think we should deprecate the 1.x components and
> > remove
> > > > them as-is from the 2.x line.  Then re-introduce them with record
> only
> > > > interfaces and built against the latest stable
> > > Cassandra/Datastax/ScyllaDB
> > > > interfaces.
> > > >
> > > > I'd love to hear thoughts from those closer to this space both as a
> > user
> > > > and developer so we can make good next steps.
> > > >
> > > > Thanks
> > >
> >
>

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

I'm actively working this, I pushed my branch up in case anyone wants to
take a look [1]. The idea is to abstract the Cassandra API "up a couple
levels" and provide implementations for Cassandra 3, 4, and eventually 5.
For JDBC-like interfaces this is a PITA because of the API (Statement,
PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping we can
find a common pattern for abstracting the third-party library
implementation and API from the NiFi component (Processor,
ControllerService, etc.) API. I think we're doing something similar for
Kafka?

Regards,
Matt

[1] https://github.com/mattyb149/nifi/tree/cassy4


On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen  wrote:

> That’s been on my todo list for a little while but things kept coming up.
> I think I could get started on that now. Based on my initial research it
> appears that scylla uses the exact same api as datastax so supporting both
> in a cql bundle should theoretically be fairly easy.
>
>
> Sent from my iPhone
>
> > On Mar 14, 2024, at 6:18 PM, Joe Witt  wrote:
> >
> > Team,
> >
> > Cassandra remains a really important system to be able to send data to.
> > However, it seems like we've not maintained these well.  We have what
> > appears to be at least a full generation behind on client versions (we
> are
> > on 3x vs 4x which is the latest stable with 5x apparently coming
> shortly).
> >
> > We have components to send data, query data, and use Cassandra as a cache
> > store.  We have older mechanisms for json/avro and publish mechanisms for
> > records.
> >
> > The libraries we do have depend on outdated versions of Guava and result
> in
> > many CVE hits.
> >
> > I am inclined to think we should deprecate the 1.x components and remove
> > them as-is from the 2.x line.  Then re-introduce them with record only
> > interfaces and built against the latest stable
> Cassandra/Datastax/ScyllaDB
> > interfaces.
> >
> > I'd love to hear thoughts from those closer to this space both as a user
> > and developer so we can make good next steps.
> >
> > Thanks
>

Re: DB Dynamic Connection

[Matt Burgess]

Eduardo,

It doesn't sound like DBCPConnectionPoolLookup will work for you because of
all the different connection strings. I don't know if there's a good reason
why we couldn't create the BasicDataSource when getConnection() is called,
passing in a Map of FlowFile attributes (that's how the Lookup version
works). One issue I do see is with "churn" if we're recreating the data
source each time. At that point it's not pooling connections. I suppose you
could have an internal cache of data sources but it would have to be
bounded and/or configurable and have a least-recently-used (LRU) eviction
strategy.

DBCPService is the name of the controller service interface that the
database processors use, but that's a misnomer since the API doesn't
mention pooling specifically. Instead you could have an implementation that
uses a cache vs a pooling approach. But Apache DBCP does handle a lot of
the management (validation, eviction, idle timeouts, etc.)  so unless
there's no way to avoid the potential memory/performance issues (like
having 50+ controller services in a PG) you could try to wrangle smaller
pools per data source and cache those if that's ok for your use case.

My two cents,
Matt

On Tue, Mar 5, 2024 at 7:25 PM Eduardo Fontes 
wrote:

> Hi Everybody!
>
> I'm thinking about make a generic ingestor with Apache NiFi but I found
> some difficulties because of the DataBase Connection Pool controller. It
> doesn't accept flowfiles parameters for its properties, specially
> connection string, username and password (for security reasons, some
> sensitive parameter name instead password itself).
>
> This is important because, as a generic ingestor, I might have hundreds of
> different connection strings, and I had a lot of problems when I tried to
> put 50 DBCP controllers in a Process Group.
>
> I wouldn't like to create a flow for each ingestion, but one flow for each
> database vendor.
>
> Does anyone have any suggestions on how I can achieve this? Would it be
> easy to create a parameterized DBCP controller? (That I could do it myself)
>
> Best regards.
>
> Eduardo Fontes
> Data Eng / System Analyst Sr.
>

Re: UpdateAttribute Failure Relationship

[Matt Burgess]

Mike's option #2 seems solid but would take a lot of work and there will
always be inputs we don't account for. I support that work but in code
sometimes we just do a "catch(Throwable)" just so it doesn't blow up. What
about a subjectless "try" or "trycatch" function you can wrap around your
whole expression? If no exception is thrown, the evaluated value will be
returned but if one is thrown, you can provide some alternate value that
you can check downstream. As this is optional it would retain the current
behavior unless you use it, and then it takes the place of all those
ifElse(isXYZValid()) calls we'd need throughout the expression.

Regards,
Matt


On Wed, Feb 7, 2024 at 8:11 PM Phillip Lord  wrote:

> IMO... UpdateAttribute has been around since the beginning of time, I can't
> see adding a failure relationship. At the same time I understand the want
> for such exceptions to be handled more gracefully rather than rolling back
> indefinitely.
> I'd vote in favor of considering Moser's option #2... and being able to
> implement an "if this then that" logic within your flow.
>
> Also just thinking... for every UA failure you have to consider a good
> failure-management strategy, which MIGHT add a lot of noise to the flow.
> Something that might otherwise easily be identified in a downstream
> component and/or database/etc.
>
> My 2 cents **
> Phil
>
>
>
>
>
> On Wed, Feb 7, 2024 at 5:18 PM Adam Taft  wrote:
>
> > Or better, the failure relationship just doesn't even exist until the
> > property "Has Failure Relationship" is set to True.  This involves
> updating
> > UpdateAttribute to have dynamic relationships (the failure relationships
> > appearing on true), which isn't hard to do in processor code.
> >
> > This has the advantage of being backwards compatible for existing users
> and
> > allows the failure relationship to exist for new configurations.
> Obviously
> > the processor would need an update to catch Expression Language
> exceptions
> > and then route conditionally to failure.
> >
> > Just thinking out loud.
> > /Adam
> >
> >
> >
> > On Wed, Feb 7, 2024 at 1:48 PM u...@moosheimer.com 
> > wrote:
> >
> > > Hi Mike,
> > >
> > > How about the option of introducing a new property that decides whether
> > to
> > > route to the 'failure' relationship in the event of an error?
> > > If this property is set to false, then the 'failure' relationship is
> > > automatically set to 'terminate' (since nothing is routed there
> anyway).
> > >
> > > Then everyone can decide whether and where they want to use this new
> > > feature or not.
> > > All other options would still be possible with such a solution.
> > >
> > > -- Uwe
> > >
> > > > Am 07.02.2024 um 22:15 schrieb Michael Moser :
> > > >
> > > > Hi Dan,
> > > >
> > > > This has been discussed in the past, as you found with those two Jira
> > > > tickets.  Personally, I'm still not sure whether a new failure
> > > relationship
> > > > on UpdateAttribute in 2.0 is a good approach.  I have heard from some
> > > > dataflow managers that would not want to go through their entire
> graph
> > > when
> > > > upgrading to 2.0 and update every UpdateAttribute configuration.
> > > >
> > > > I have heard some alternatives to a 'failure' relationship that I
> would
> > > > like to share as options.
> > > >
> > > > 1) Add a new property to UpdateAttribute that controls whether a
> > flowfile
> > > > that causes an expression language exception either yields and rolls
> > > back,
> > > > or silently fails to update the attribute and sends the flowfile to
> > > > success.  I personally don't like this, because the use case for
> > "silent
> > > > failure" seems really like a rarely needed edge case.
> > > >
> > > > 2) Identify all expression language methods that can throw an
> exception
> > > and
> > > > document that fact in the Expression Language Guide (some methods
> > already
> > > > mention they can throw an "exception bulletin").  Then implement new
> > > > expression methods to check if an expression could fail, and use that
> > in
> > > > UpdateAttribute advanced rules.  For example, if the format() and
> > > > formatInstant() methods can fail on a negative number, we create a
> new
> > > > method such as isValidMilliseconds().  This already exists for some
> > > cases,
> > > > such as isJson() which can do a quick check of some value before
> > calling
> > > > jsonPathDelete() on it.
> > > >
> > > > I'm curious to hear more thoughts on this.
> > > >
> > > > -- Mike
> > > >
> > > >
> > > >
> > > >> On Wed, Jan 31, 2024 at 11:02 AM Dan S  wrote:
> > > >>
> > > >> My team is requesting a failure relationship for UpdateAttribute as
> > > seen in
> > > >> NIFI-5448  and
> > > NIFI-6344
> > > >>  as we are
> > > >> experiencing the same problem where a NIFI Expression Language is
> > > throwing
> > > >> an exception. In the PR for NIFI-5448 it was mentioned 

Re: [VOTE] Release Apache NiFi 2.0.0-M2 (RC4)

[Matt Burgess]

+1 (binding)

Ran full build

Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /Users/mburgess/.sdkman/candidates/maven/current
Java version: 21, vendor: Oracle Corporation
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "14.2.1", arch: "aarch64", family: "mac"

Verified various flows. Thanks for RM'ing David!

-Matt

On Thu, Jan 25, 2024 at 9:30 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0-M2.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on and the convenience binaries are available
> on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M2
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1264
>
> Git Tag: nifi-2.0.0-M2-RC4
> Git Commit ID: 640b7bdfbbb8842f057a9bf49dc2b9b5d092abda
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/640b7bdfbbb8842f057a9bf49dc2b9b5d092abda
>
> Checksums of nifi-2.0.0-M2-source-release.zip
>
> SHA256: 1946eceb3ae4f541545e93ddc6dd2cbe2a3302a6a46e6c584f3ffc1c1bd1e18b
> SHA512:
> e8e67e1e67359553479c0721a1c49ae6706cc6882eadf92e1f5ccc2619637ab87119a06221980d4c34d99b7b6d9a2138c77440b407074090e727b5d4447ab799
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 214
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353861
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M2
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-2.0.0-M2
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.25.0 (RC1)

[Matt Burgess]

+1 (binding)

Ran full clean install -Pcontrib-check

Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /Users/mburgess/.sdkman/candidates/maven/current
Java version: 1.8.0_372, vendor: Azul Systems, Inc.,
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "14.2.1", arch: "aarch64", family: "mac"

Verified CREATE TABLE for Oracle and byte[] generation in GenerateRecord,
ran various flows. Thanks for RM'ing Pierre!


On Thu, Jan 25, 2024 at 1:53 PM Pierre Villard 
wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.25.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.25.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1263
>
> Git Tag: nifi-1.25.0-RC1
> Git Commit ID: 6ecc398d3f92425447e43242af4992757e25b3c5
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/6ecc398d3f92425447e43242af4992757e25b3c5
>
> Checksums of nifi-1.25.0-source-release.zip
>
> SHA256: 6a14b35bf5beb4ae3fcf76df8d09676e61c6bc309a97dc6785eae84b7cbaef78
> SHA512:
>
> 1ffb2586ce9591ce2c5aa39fdec43a89ffd29081a31d51dc95dd953cb7f94584d0a0171bb1ba7c9495f431aec3770d324dbabb319b01bb6fdce5a0a00426fffa
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/pvillard.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 103
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353860
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.25.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [] +1 Release this package as nifi-1.25.0
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.24.0 (RC5)

[Matt Burgess]

+1 (binding)
Ran through release helper (verifying keys and hashes and such) and
tested with a handful of different flows. Thanks for seeing this
through Pierre!

- Matt

On Thu, Nov 23, 2023 at 10:13 AM Pierre Villard
 wrote:
>
> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.24.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.24.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1251
>
> Git Tag: nifi-1.24.0-RC5
> Git Commit ID: 5241f434829ca46a26a475600ef7c00e1e271e37
> GitHub Commit Link:
> https://github.com/apache/nifi/commit/5241f434829ca46a26a475600ef7c00e1e271e37
>
> Checksums of nifi-1.24.0-source-release.zip
>
> SHA256: 715eb61cdc017a5f7ba4d845ae962fdf83d389829db2a8948be14f99f2cc8272
> SHA512:
> 574147002b905ef64447edec0c7308f4fc3a97b3c7f01edca05464b2b418bcd3922f956d093736014443eec88ceba36af81df37398c5fe23a1288235b79d7306
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/pvillard.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 285
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353443
>
> Release note highlights can be found on the project wiki:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.24.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [] +1 Release this package as nifi-1.24.0
> [] +0 no opinion
> [] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.24.0 (RC3)

[Matt Burgess]

+1 (binding)

Ran through release helper, ran NiFi standalone with Java 8, tested
various flows and components.  Thanks for RM'ing Pierre!

On Thu, Nov 16, 2023 at 2:01 PM Pierre Villard
 wrote:
>
> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.24.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.24.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1241
>
> Git Tag: nifi-1.24.0-RC3
> Git Commit ID: 586a1a8789e9c39914f4a6088ac26e22d60eeb91
> GitHub Commit Link:
> https://github.com/apache/nifi/commit/586a1a8789e9c39914f4a6088ac26e22d60eeb91
>
> Checksums of nifi-1.24.0-source-release.zip
>
> SHA256: 590cf9b1219f9fd66c81cc1740b2e245d85e341cdc280c769d354084dc27ee8a
> SHA512:
> 8d3b9cb9c4686242620ad40ad83fadb972403ee70a101cbb6fa0116b54ad7793702da230871281c0de40322ddfdbfc89dacba7b690465e7b2329850dca5132e8
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/pvillard.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 280
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353443
>
> Release note highlights can be found on the project wiki:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.24.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [] +1 Release this package as nifi-1.24.0
> [] +0 no opinion
> [] -1 Do not release this package because...

Re: Removing JRuby?

[Matt Burgess]

I believe it is because in both ExecuteScript and ExecuteGroovyScript
you can do "regular" groovy, but EGS has extra built-ins such as easy
access to controller services, Groovy SQL stuff, etc, and we could
keep building it out. But IMO we'd have to port the rest of the
scripted components (ScriptedReader/Writer, etc.) over to the Groovy
bundle and make sure there's a drop-in replacement in the Python stuff
before we'd want to deprecate the scripted bundle.

On the JRuby front, is that something you use actively? This question
is for you and the entire community of course.

Regards,
Matt

On Mon, Nov 6, 2023 at 7:12 AM Mike Thomsen  wrote:
>
> If we deprecate ExecuteScript, I think we need to have groovyx be ready to
> function as a drop-in replacement if it's not there already.
>
> On Sun, Nov 5, 2023 at 9:21 PM Matt Burgess  wrote:
>
> > IIRC the removal of these engines was mostly due to lack of use or at
> > least the perception thereof. If JRuby is being used by the community
> > actively, I'm happy to revisit that discussion. Luaj's JSR-223
> > interface left something to be desired, but JRuby just needed a system
> > variable set or something like that.
> >
> > For the groovyx bundle, because it is Groovy-specific I tend to think
> > we could make better use of that than ExecuteScript, especially if we
> > do get rid of all the engines. We have a Groovy-specific processor, a
> > "real" Python SDK, and no more Nashorn. Perhaps we move all the
> > scripted components to the Groovy bundle, although I believe some
> > folks still make use of Jython for these. Of course if we reinstate
> > JRuby for ExecuteScript it's probably best to keep things the way they
> > are, or create a jruby bundle. The original scripting bundle was
> > aiming to support several engines, but if it turns out only one or two
> > will be useful, it may not be worth shoehorning all that JSR-223 logic
> > when engine-specific components could be simpler, more easily
> > maintained, and allow for the idioms of the language to be better used
> > (as is done in the groovyx bundle).
> >
> > Just my two cents, looking forward to everyone's thoughts!
> >
> > - Matt
> >
> > On Sun, Nov 5, 2023 at 8:31 PM Mike Thomsen 
> > wrote:
> > >
> > > https://issues.apache.org/jira/browse/NIFI-11646
> > >
> > > I get the removal of Lua, but not the removal of JRuby. It's a clean
> > > reimplementation of Ruby native to the JVM and AFAICT is pound for pound
> > as
> > > actively maintained as Groovy.
> > >
> > > Also, at this point, does it make sense to even keep the groovyx bundle
> > > rather than deprecate it for 2.X?
> >

Re: Removing JRuby?

[Matt Burgess]

IIRC the removal of these engines was mostly due to lack of use or at
least the perception thereof. If JRuby is being used by the community
actively, I'm happy to revisit that discussion. Luaj's JSR-223
interface left something to be desired, but JRuby just needed a system
variable set or something like that.

For the groovyx bundle, because it is Groovy-specific I tend to think
we could make better use of that than ExecuteScript, especially if we
do get rid of all the engines. We have a Groovy-specific processor, a
"real" Python SDK, and no more Nashorn. Perhaps we move all the
scripted components to the Groovy bundle, although I believe some
folks still make use of Jython for these. Of course if we reinstate
JRuby for ExecuteScript it's probably best to keep things the way they
are, or create a jruby bundle. The original scripting bundle was
aiming to support several engines, but if it turns out only one or two
will be useful, it may not be worth shoehorning all that JSR-223 logic
when engine-specific components could be simpler, more easily
maintained, and allow for the idioms of the language to be better used
(as is done in the groovyx bundle).

Just my two cents, looking forward to everyone's thoughts!

- Matt

On Sun, Nov 5, 2023 at 8:31 PM Mike Thomsen  wrote:
>
> https://issues.apache.org/jira/browse/NIFI-11646
>
> I get the removal of Lua, but not the removal of JRuby. It's a clean
> reimplementation of Ruby native to the JVM and AFAICT is pound for pound as
> actively maintained as Groovy.
>
> Also, at this point, does it make sense to even keep the groovyx bundle
> rather than deprecate it for 2.X?

Re: Provenance events without FlowFiles?

[Matt Burgess]

I won't speak for Mark but I took that to mean that you pass in a
generic Java (or custom) File object (no matter what or where it
points to) so the ProvenanceReporter can use the appropriate
information from that object (like size) to populate the provenance
event, not any real reference to the file itself. There may also be
security concerns with including information about the source
location: that needs to be determined as well.

On Thu, Oct 26, 2023 at 2:59 PM Pierre Villard
 wrote:
>
> >
> > Rather, I’d say it's an UPLOAD_FILE event. So I’d lean more toward an
> > uploadFile() method on ProvenanceReporter that takes as an argument a
> > `File` (as well as a FlowFile). The size would come from the File itself,
> > and the event would convey the information about the local file that was
> > uploaded - probably in the Event Details.
> >
>
> Would that mean that for the "bytes transferred" graph in Status History,
> we would combine SEND and UPLOAD_FILE events? Because, right now, it's not
> showing anything which is confusing.
>
> Also, I'm not sure about the 'File' object. While we have only the local
> file system as an option today for the File Resource Service, I'd expect
> additional implementations such as implementations for CSPs. So we could
> have the case where PutAzureBlobStorage is used with a FileResourceService
> for Google Cloud Storage for example (in order to improve efficiency of
> data movement between cloud providers) and, in this case, not sure we would
> have a 'File' object. Unless you're talking about a more generic File
> object here and not the object for local file system.
>
> Le jeu. 26 oct. 2023 à 09:16, Matt Burgess  a écrit :
>
> > AFAIK it is fine and appropriate to issue multiple provenance events
> > for a single FlowFile. In the case for PutAzureBlobStorage uploading a
> > file to Azure, it is the incoming FlowFile that triggers the upload.
> > Before reporting a provenance event, attributes are added to the
> > FlowFile, so that "version" of the FlowFile can be the one used to
> > report a SEND event. I have done this to said processor as part of a
> > large refactor/improvement of the provenance capability:
> >
> > session.getProvenanceReporter().send(flowFile,
> > blob.getSnapshotQualifiedUri().toString(), transferMillis,
> > REL_SUCCESS);
> >
> > Having said that, to Mark's point it's probably better to have a
> > separate UPLOAD_FILE event, I can change that in my code.
> >
> > I added a couple like this to similar processors, such as
> > TriggerHiveMetastoreEvent:
> >
> > session.getProvenanceReporter().invokeRemoteProcess(flowFile,
> > hiveMetastoreUrl, REL_SUCCESS);
> >
> > I am still working on this, I need to write up a Jira with a thorough
> > treatment of the material and eventually get a PR up for review.
> >
> > Regards,
> > Matt
> >
> > On Thu, Oct 26, 2023 at 12:02 PM Mark Payne  wrote:
> > >
> > > Lehel,
> > >
> > > I don’t believe we should be trying to create a “Mock FlowFile.” I am ok
> > with an update to the ProvenanceReporter interface. But I don’t think it
> > should accept a “size” parameter. Rather, I think this is a completely
> > different type of event that is occurring. This is not a “send” in that
> > it’s not sending the contents of the FlowFile to a remote system. Rather,
> > I’d say it's an UPLOAD_FILE event. So I’d lean more toward an uploadFile()
> > method on ProvenanceReporter that takes as an argument a `File` (as well as
> > a FlowFile). The size would come from the File itself, and the event would
> > convey the information about the local file that was uploaded - probably in
> > the Event Details.
> > >
> > > Thanks
> > > -Mark
> > >
> > >
> > > > On Oct 26, 2023, at 10:36 AM, Lehel Boér  wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I would like to address a particular scenario that has recently come
> > to my attention regarding the use of the PutAzureBlobStorage processor with
> > the FileResourceService.
> > > >
> > > > When the PutAzureBlobStorage processor is used with the
> > FileResourceService, it currently uploads a file from the user's local
> > filesystem to Azure, but it does not create a FlowFile. Instead, it
> > utilizes the incoming FlowFile solely to send a provenance event. In this
> > case the size of the provenance event is the incoming FlowFile's size
> > instead of the uploaded one.
> > > >
> > > > There are potential solutions to address thi

Re: Provenance events without FlowFiles?

[Matt Burgess]

AFAIK it is fine and appropriate to issue multiple provenance events
for a single FlowFile. In the case for PutAzureBlobStorage uploading a
file to Azure, it is the incoming FlowFile that triggers the upload.
Before reporting a provenance event, attributes are added to the
FlowFile, so that "version" of the FlowFile can be the one used to
report a SEND event. I have done this to said processor as part of a
large refactor/improvement of the provenance capability:

session.getProvenanceReporter().send(flowFile,
blob.getSnapshotQualifiedUri().toString(), transferMillis,
REL_SUCCESS);

Having said that, to Mark's point it's probably better to have a
separate UPLOAD_FILE event, I can change that in my code.

I added a couple like this to similar processors, such as
TriggerHiveMetastoreEvent:

session.getProvenanceReporter().invokeRemoteProcess(flowFile,
hiveMetastoreUrl, REL_SUCCESS);

I am still working on this, I need to write up a Jira with a thorough
treatment of the material and eventually get a PR up for review.

Regards,
Matt

On Thu, Oct 26, 2023 at 12:02 PM Mark Payne  wrote:
>
> Lehel,
>
> I don’t believe we should be trying to create a “Mock FlowFile.” I am ok with 
> an update to the ProvenanceReporter interface. But I don’t think it should 
> accept a “size” parameter. Rather, I think this is a completely different 
> type of event that is occurring. This is not a “send” in that it’s not 
> sending the contents of the FlowFile to a remote system. Rather, I’d say it's 
> an UPLOAD_FILE event. So I’d lean more toward an uploadFile() method on 
> ProvenanceReporter that takes as an argument a `File` (as well as a 
> FlowFile). The size would come from the File itself, and the event would 
> convey the information about the local file that was uploaded - probably in 
> the Event Details.
>
> Thanks
> -Mark
>
>
> > On Oct 26, 2023, at 10:36 AM, Lehel Boér  wrote:
> >
> > Hi everyone,
> >
> > I would like to address a particular scenario that has recently come to my 
> > attention regarding the use of the PutAzureBlobStorage processor with the 
> > FileResourceService.
> >
> > When the PutAzureBlobStorage processor is used with the 
> > FileResourceService, it currently uploads a file from the user's local 
> > filesystem to Azure, but it does not create a FlowFile. Instead, it 
> > utilizes the incoming FlowFile solely to send a provenance event. In this 
> > case the size of the provenance event is the incoming FlowFile's size 
> > instead of the uploaded one.
> >
> > There are potential solutions to address this issue and ensure that the 
> > provenance events are handled effectively. Two main options have been 
> > proposed:
> >
> >
> >  *   Create a Mock FlowFile: A mock FlowFile with a size matching that of 
> > the local file being uploaded could be generated. This mock FlowFile would 
> > serve as the basis for the provenance event, even though its size might not 
> > reflect the actual content.
> >
> >  *   Modify the ProvenanceReporter Interface: Alternatively, we could 
> > introduce a new method in the ProvenanceReporter interface that doesn't 
> > require a FlowFile but instead accepts a "size" parameter as an argument. 
> > This would eliminate the need for a mock FlowFile.
> >
> > The lack of a FlowFile operation in this situation creates a distinct 
> > challenge because provenance events are typically tied to FlowFiles. Still, 
> > it's important to indicate data transmission for monitoring and tracking.
> >
> > While the idea of a "size" parameter for the provenance event seems 
> > preferable, we need to carefully consider its feasibility, potential 
> > complexities, and community acceptance. The FileResourceService already 
> > deviates from NiFi's concept of using FlowFiles to hold payload data, and 
> > we must avoid further complicating the framework unless absolutely 
> > necessary.
> >
> > If you have any insights or suggestions, please feel free to reply to this 
> > email or join the discussion.
> >
> > Best Regards,
> > Lehel
>

Re: JOLTTransformRecord problem

[Matt Burgess]

For some reason I don't have the original thread, I must've
inadvertently deleted it. IIRC your example input was a single JSON
object and I said if that were the case you could use
JoltTransformJSON instead. However if that is NOT the case (which is
your point c above) then you have a couple of options:

1) To continue using JoltTransformJSON with a top-level array you need
to surround your spec with
"*": {  }
and will need to use "[&1]." in front of all the output fields. This
will output the transformation to the same index in the array as it
was in the input.

2) One major difference between JoltTransformJSON and
JoltTransformRecord is that the former reads the entire thing into
memory, where JoltTransformRecord reads one record at a time. So your
current spec should work with JoltTransformRecord, but if you are
still getting the original error, can you provide (or re-provide if
you already did, I can't find the original thread) sample input that
represents the "real" input (not just one JSON object if you'll be
getting multiple records or if the top-level is an array even with
only one object in it), desired output, and the error with full stack
trace? I'm guessing there is an inference error with complex fields,
if you know what the input and output schemas are you can provide them
to the Reader and Writer respectively instead of using "Infer schema".
That should work around any inference issues.
With NiFi 1.23.2 you also have the new ExtractRecordSchema processor,
you can try that before your JoltTransformRecord processor with the
same reader and see what it comes out with as a schema. Then you can
manually alter it to better match your data and use that in the Reader
specified in JoltTransformRecord.

Regards,
Matt

On Tue, Oct 10, 2023 at 4:47 PM Mark Woodcock  wrote:
>
> Chris,
>
> 1) I've upgraded to 1.23.2  (which appears to be the latest and greatest).
>
> 2) I've tested the JoltTransformRecord with
> a) JsonTreeReader w/ InferredSchema
> b) JsonRecordSetWriter w/ InheritsSchema
> c) a GetFile processor which grabs a text file with the various bits of
> test data
>
> It appears that your suspicions are correct:
> i) if I test with just that single record as the entire content of the
> file, the processor is successful.
> ii) if I test with multiple records, none of which have the complicated
> inner field, all is successful.
> c) if I test with multiple records, where at least one has the complicated
> inner field, I get the earlier noted error.
>
> IOW, yep, it only happens with *more* data.
>
> bummer,
>
> mew
>
> On Tue, Oct 10, 2023 at 2:43 PM Chris Sampson
>  wrote:
>
> > Using your example (single JSON Object and Jolt Spec) seems to work fine
> > in both JoltTransformJSON and JoltTransformRecord when run on the current
> > main branch (which is for the upcoming 2.0.0 release).
> >
> > To test, I setup a GenerateFlowFile processor to output the example JSON
> > you gave, then sent that through both of the Jolt processors using a
> > JsonTreeReader with “Inferred Schema”, and a JsonRecordSetWriter that
> > “Inherits Schema” for the Record processor.
> >
> > If you run *just* your example from this email chain through the Jolt
> > processors on the version of NiFi you’re using, do you see the errors you
> > mention, or does that only happen with more data?
> >
> >
> > Cheers,
> >
> > ---
> > Chris Sampson
> > IT Consultant
> > chris.samp...@naimuri.com
> >
> >
> > > On 10 Oct 2023, at 15:45, Mark Woodcock 
> > wrote:
> > >
> > > H,
> > >
> > > One small problem:  While JOLTTransformJSON is quite lovely (a) it has a
> > > great "advanced" interface that allows one to test their spec and json
> > > inputs and (b) it actually works for the cases that I noted...it treats
> > the
> > > input a single blob of JSON.  Unfortunately, my input files are
> > collections
> > > of JSON records (which--less the noted problem--JOLTTransformRecord does
> > > quite nicely with)--that's literally how they arrive, not the result of
> > me
> > > formatting them at all.
> > >
> > > Is there a way to get JTJ to treat the input as records?
> > > Does 1.22 or 1.23 have the fix for JTR?
> > >
> > > thx,
> > >
> > > mew
> > >
> > >
> > > On Mon, Oct 9, 2023 at 3:21 PM Mark Woodcock  wrote:
> > >
> > >> confirmed:  version 1.21.
> > >> How recent is the fix?
> > >>
> > >> thx,
> > >>
> > >> mew
> > >>
> > >>
> > >> On Sun, Oct 8, 2023 at 11:39 PM Mark Woodcock 
> > wrote:
> > >>
> > >>> Matt,
> > >>>
> > >>> Unfortunately (at home now) the details are all at work at the moment,
> > >>> but I know that I didn't start this work until April (at the
> > earliest), so
> > >>> I'm surely using at least 1.21; is the fix more recent than that?
> >  {If so,
> > >>> perhaps there is a bug.}
> > >>>
> > >>> Fortunately, yea, JSON out is the intent; I need the data to be in that
> > >>> format to set up a subsequent transform to AVRO, so it seems there are
> > two
> > >>> possible ways out (depending on which version I'm 

Re: JOLTTransformRecord problem

[Matt Burgess]

mew,

What version of NiFi are you using? We may have fixed the bug in newer
versions but if you are using the latest, this is bug. We shouldn't
infer CHOICE[STRING,RECORD] when the only entry is a RECORD.

Since you only have one top-level record in your FlowFile you can use
JoltTransformJSON instead as a workaround. If you want JSON output
then it's a solution and not a workaround :) The error here isn't
about JOLT but about JoltTransformRecord using NiFi Records to apply
the transformation to rather than the raw JSON (the latter is what
JoltTransformJSON does).

Regards,
Matt

Re: How to Connect Apache NiFi to PostgreSql and using nifi insert records from csv file to database

[Matt Burgess]

Gennady,

That is a great explanation by Lars, also note that by default GetFile
will remove the CSV from your system. If you want to keep it but only
fetch it once, use ListFile -> FetchFile rather than GetFile.

Regards,
Matt

On Fri, Oct 6, 2023 at 12:44 PM Lars Winderling
 wrote:
>
> Hi Gennady,
>
> you should set up a jdbc connection pool controller service and connect it to 
> your postgres instance. Then you will need a csv record reader controller 
> service, configured with the correct csv format. Eventually, put an 
> ExecuteSQLRecord processor onto the canvas. It should use the connection pool 
> and the csv reader. Check the remaining properties of the processor for more 
> config options. Cf. also 
> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html.
> Of course, you will need some processor to get the csv files in the first 
> place. For local files, use GetFile. So that depends on the source of your 
> files. When the correct processor is found, connect it to the sql processor 
> (success relationship).
> When using GetFile, you will need to put a GenerateFlowfile processor in 
> front of it, and connect them. When starting it, it will produce a flowfile. 
> The GetFile proc will obtain the data, the sql processor will insert it into 
> the db.
> Make sure to configure error relationships as well by e.g. terminating them 
> (just to get you started).
> I hope, you will manage to go on from there. The docs are pretty extensive, 
> as is the user mailing list archive.
>
> Best, Lars
>
> On 6 October 2023 18:06:56 CEST, Gennady Kondratov 
>  wrote:
> >How to Connect Apache NiFi to PostgreSql and using nifi insert records from 
> >csv file to database/
> >i'd like to ask you how to get it or where looking for about it
> >

Re: NIFI Jira ticket not updated with link to NIFI PR code in Github

[Matt Burgess]

Dan,

That happens sometimes, it can take a while for the link to be added
to the Jira, but usually it does happen eventually. Sometimes I'll add
the link manually then remove it when the system catches up and adds
the link to the Jira.

Regards,
Matt

On Thu, Sep 7, 2023 at 10:49 AM Dan S  wrote:
>
> I recently submitted a PR for
> https://issues.apache.org/jira/browse/NIFI-11197 at
> https://github.com/apache/nifi/pull/7665 but the Jira ticket has not been
> updated with the link to the PR as it usually is. Please advise. Thanks!

Re: PGVector and Database Driver

[Matt Burgess]

Maybe this [1]? Perhaps you have to call unwrap() yourself in this
case. IIRC you don't have access to the DataSource but you can check
it directly on the connection.

Regards,
Matt

[1] 
https://stackoverflow.com/questions/36986653/cast-java-sql-connection-to-pgconnection

On Thu, Aug 31, 2023 at 8:15 PM u...@moosheimer.com  wrote:
>
> Mark & Matt,
>
> Thanks for the quick help. I really appreciate it.
>
> PGvector.addVectorType(con) returns the following:
> *java.sql.SQLException: Cannot unwrap to org.postgresql.PGConnection*
>
> Could this be a connection pool issue?
>
> Interestingly, I didn't call addVectorType() at all in my test java code
> and it still works?!
> I'll have to check again ... maybe I'm not seeing it correctly anymore.
> It is already 2:05 a.m. here.
>
>
> Regards,
> Uwe
>
>
> java.sql.SQLException: Cannot unwrap to org.postgresql.PGConnection
>
> On 31.08.23 18:53, Matt Burgess wrote:
> > This means the JDBC driver you're using does not support the use of
> > the two-argument setObject() call when the object is a PGVector. Did
> > you register the Vector type by calling:
> >
> > PGvector.addVectorType(conn);
> >
> > The documentation [1] says that the two-argument setObject() should
> > work if you have registered the Vector type.
> >
> > Regards,
> > Matt
> >
> > [1]https://github.com/pgvector/pgvector-java
> >
> > On Thu, Aug 31, 2023 at 12:01 PM Mark Payne  wrote:
> >> Hey Uwe,
> >>
> >> The DBCPConnectionPool returns a java.sql.Connection. From that you’d 
> >> create a Statement. So I’m a little confused when you say that you’ve got 
> >> it working in Pure JDBC but not with NiFi, as the class returned IS pure 
> >> JDBC. Perhaps you can share a code snippet of what you’re doing in the 
> >> “Pure JDBC” route that is working versus what you’re doing in the NiFi 
> >> processor that’s not working?
> >>
> >> Thanks
> >> -Mark
> >>
> >>
> >>> On Aug 31, 2023, at 10:58 AM,u...@moosheimer.com  wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am currently writing a processor to write OpenAI embeddings to Postgres.
> >>> I am using DBCPConnectionPool for this.
> >>> I use Maven to integrate PGVector (https://github.com/pgvector/pgvector).
> >>>
> >>> With pure JDBC this works fine. With the database classes from NiFi I get 
> >>> the error:
> >>> *Cannot infer the SQL type to use for an instance of 
> >>> com.pgvector.PGvector. Use setObject() with an explicit Types value to 
> >>> specify the type to use.*
> >>>
> >>> I use -> setObject (5, new PGvector(embeddingArray)).
> >>> embeddingArray is defined as: float[] embeddingArray
> >>>
> >>> Of course I know why I get the error from NiFi and not from the JDBC 
> >>> driver, but unfortunately this knowledge does not help me.
> >>>
> >>> Can anyone tell me what SQLType I need to specify for this?
> >>> I have searched the internet and the NiFi sources on GitHub for several 
> >>> hours now and have found nothing.
> >>>
> >>> One option would be to use native JDBC and ignore the ConnectionPool. But 
> >>> that would be a very bad style in my opinion.
> >>> Perhaps there is a better solution?
> >>>
> >>> Any help, especially from Matt B., is appreciated as I'm at a loss.
> >>> Thanks guys.

Re: PGVector and Database Driver

[Matt Burgess]

This means the JDBC driver you're using does not support the use of
the two-argument setObject() call when the object is a PGVector. Did
you register the Vector type by calling:

PGvector.addVectorType(conn);

The documentation [1] says that the two-argument setObject() should
work if you have registered the Vector type.

Regards,
Matt

[1] https://github.com/pgvector/pgvector-java

On Thu, Aug 31, 2023 at 12:01 PM Mark Payne  wrote:
>
> Hey Uwe,
>
> The DBCPConnectionPool returns a java.sql.Connection. From that you’d create 
> a Statement. So I’m a little confused when you say that you’ve got it working 
> in Pure JDBC but not with NiFi, as the class returned IS pure JDBC. Perhaps 
> you can share a code snippet of what you’re doing in the “Pure JDBC” route 
> that is working versus what you’re doing in the NiFi processor that’s not 
> working?
>
> Thanks
> -Mark
>
>
> > On Aug 31, 2023, at 10:58 AM, u...@moosheimer.com wrote:
> >
> > Hi,
> >
> > I am currently writing a processor to write OpenAI embeddings to Postgres.
> > I am using DBCPConnectionPool for this.
> > I use Maven to integrate PGVector (https://github.com/pgvector/pgvector).
> >
> > With pure JDBC this works fine. With the database classes from NiFi I get 
> > the error:
> > *Cannot infer the SQL type to use for an instance of com.pgvector.PGvector. 
> > Use setObject() with an explicit Types value to specify the type to use.*
> >
> > I use -> setObject (5, new PGvector(embeddingArray)).
> > embeddingArray is defined as: float[] embeddingArray
> >
> > Of course I know why I get the error from NiFi and not from the JDBC 
> > driver, but unfortunately this knowledge does not help me.
> >
> > Can anyone tell me what SQLType I need to specify for this?
> > I have searched the internet and the NiFi sources on GitHub for several 
> > hours now and have found nothing.
> >
> > One option would be to use native JDBC and ignore the ConnectionPool. But 
> > that would be a very bad style in my opinion.
> > Perhaps there is a better solution?
> >
> > Any help, especially from Matt B., is appreciated as I'm at a loss.
> > Thanks guys.
>

Re: Jira contributor access

[Matt Burgess]

Umar,

Welcome to the NiFi project! Please use the Self-Service Portal at
https://selfserve.apache.org/jira-account.html to request access to
Jira per the current Apache rules. A PMC member will review and
approve as prudent.

Thank you,
Matt

On Sun, Aug 27, 2023 at 3:35 PM Umar Hussain  wrote:
>
> Hello NiFi Team,
> I need contributor access to NiFi Jira to assign myself to this ticket (
> https://issues.apache.org/jira/browse/NIFI-11995).
> Please provide me the access.
> Username: umar.hussain
> Email: umarhussain.w...@gmail.com
>
> Thank you
>
> --
> Regards,
>
> Umar Hussain
> Software Engineer
> https://umarhussain.xyz

Re: Apache NiFi compatibility with SQL Server 2008 R2

[Matt Burgess]

Frank,

Yes NiFi is compatible with MSSQL 2008, there's even a specific
database adapter in some processors to generate the correct SQL for it
(it's called "MSSQL2008DatabaseAdapter").

Regards,
Matt

On Thu, Aug 24, 2023 at 11:23 AM Frank Mansilla
 wrote:
>
> Dear Apache NiFi Community,
>
> I am interested in using Apache NiFi to manage data streams involving SQL
> Server 2008 R2.
> Could you confirm if NiFi is compatible with this database engine?
> I would greatly appreciate your guidance on this.
>
> Thanks and regards!
>
> Frank Mansilla
> frankmansilla1...@gmail.com
> Bolivar, Buenos Aires, Argentina

Re: FileMaker and Apache NiFi

[Matt Burgess]

Samuel,

Where do you see that DBCPConnectionPool uses JNDI? As far as I know
it just uses Apache DBCP and JDBC DataSources using the supplied
driver.

Regards,
Matt

On Wed, Aug 23, 2023 at 9:14 AM Namazi, Samuel  wrote:
>
> To whom it may concern,
>
> I am trying to connect my FileMaker database to NiFi as a data source. 
> However that ist not so easily done, as NiFi's DBCPConnectionPool (1.22.0) 
> uses JNDI, but FileMaker does not support the JNDI standard for JDBC. It 
> probably is possible to use a JDBC-ODBC Bridge or create a custom processor 
> and controller to make it work, but those options aren't really considerable 
> for me. Am I overlooking something and there is an easy solution to this ? If 
> there is something else that I can do, I would greatly appreciate your help.
>
> Kind regards
>
> Samuel Namazi

Re: [VOTE] Release Apache NiFi 1.23.2 (RC1)

[Matt Burgess]

+1 (binding)

Ran through release helper and my usual flows verifying no
regressions. Thanks for RM'ing David!

On Mon, Aug 21, 2023 at 4:16 PM David Handermann
 wrote:
>
> Team,
>
> NOTE: This is a shortened vote window given the narrow scope of changes.
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.23.2.
>
> Please review the following guide for how to verify a release candidate build:
>
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The source being voted on the and the convenience binaries are
> available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.2
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1235
>
> Git Tag: nifi-1.23.2-RC1
> Git Commit ID: dec043e590f26ba2f3594f4f297dcd2b7e565ab7
> GitHub Commit Link:
> https://github.com/apache/nifi/commit/dec043e590f26ba2f3594f4f297dcd2b7e565ab7
>
> Checksums of nifi-1.23.2-source-release.zip
>
> SHA256: cfaa374c383a316de6d9230a0786ebe7dd97c03f8903b36365faa27bf6424fac
> SHA512: 
> a63fa624532aa3f1769013d67499caf48b9771bc3515ffca4daba2dd379722c3b25bea4519b56bc647d4e05db122e7a41519a52a703a0179b431fa881dadb4cf
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved in this version: 1
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353568
>
> Release note highlights can be found on the project wiki:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.2
>
> The vote will be open for 24 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-1.23.2
> [] +0 no opinion
> [] -1 Do not release this package because...

Re: [DISCUSS] Release Apache NiFi 1.23.2 for NIFI-11971

[Matt Burgess]

+1

On Mon, Aug 21, 2023 at 9:58 AM David Handermann
 wrote:
>
> Team,
>
> Thanks for the work on release for NiFi 1.23.1 last week.
>
> As it happens, another important bug surfaced last week with Content
> Repository handling of empty files, described in NIFI-11971 [1]. This
> bug was introduced due to changes for NIFI-11670 [2] released in NiFi
> 1.23.0.
>
> This issue is serious enough that we should release a new patch
> version 1.23.2, limited to this particular fix.
>
> I plan to prepare a release candidate build and send out a
> short-duration vote thread later today.
>
> Regards,
> David Handermann
>
> [1] https://issues.apache.org/jira/browse/NIFI-11971
> [2] https://issues.apache.org/jira/browse/NIFI-11670

Re: JoltTransformJSON EL when using file

[Matt Burgess]

Matthew,

What was your second case trying to use ${now():toNumber()} ? My unit
test evaluated the expression into an integer and it worked (versus
putting the expression in quotes which would make it a string).

Regards,
Matt

On Tue, Aug 15, 2023 at 4:09 AM Matthew Hawkins  wrote:
>
> Hi Matt,
>
> The tag will be ${firstname}, the spec is correct.
> (well, I also get lastname, but it's unimportant)
>
> Weird thing is this failed similarly with a file based input on the Record
> based jolt processor as well on a completely different system.
>
> Both Linux (Ubuntu 22.04), both OpenJDK 17, both recompiled nifi from
> source with include-graph,include-media,include-rules,include-sql-reporting
>
> In the second case I was trying to get ${now():toNumber()} into a json
> record using either default or modify-overwrite-beta (tried both). The
> following success processor failed on reading schema as the literal
> ${now...} was a string not a long and the JVM refused the type cast. If I
> insert the transform directly then it still fails with a type cast problem
> from generic Object :/ (that was midnight last night, and I tossed my hands
> in the air and turned the server off)
>
> I'm pulling mqtt and opc-ua off a raspberry pi and dumping it into
> postgresql. Trying to be a good nifi citizen and use record based
> processing where possible. Since I don't run Hadoop at home, it's all
> manual schemas and using postgres as the perm data store. I'm using json as
> mqtt does it intrinsically and I can jolt transform other data into json
> form and then store it easily in postgres. Well, that was the idea 
>
> Kr,
>
> On Sun, 13 Aug 2023, 10:44 Matt Burgess,  wrote:
>
> > Just to follow up, I added a unit test to put EL in a JOLT spec and it
> > worked. I noticed you referred to "attrname" in your post but your
> > spec refers to "firstname", is that a typo?
> >
> > Regards,
> > Matt
> >
> > On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess  wrote:
> > >
> > > I added file support to JoltTransformJSON under NIFI-4957 [1], a first
> > > glance at the code seems like it should work, but I'll try to
> > > reproduce it and follow up, thanks for bringing this to our attention!
> > >
> > > Regards,
> > > Matt
> > >
> > > [1] https://issues.apache.org/jira/browse/NIFI-4957
> > >
> > > On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins 
> > wrote:
> > > >
> > > > Hi devs,
> > > >
> > > > Using 1.23.0 I have a simple transform adding some flowfile attributes
> > into
> > > > the content.
> > > > When putting the spec directly to the processor it works fine. If I
> > have
> > > > the spec in an external file however it seems to put in the literal
> > string
> > > > ${attrname} - ie the EL didn't process.
> > > >
> > > > Have I done something incorrectly or should I be now asking for an
> > account
> > > > to log a bug?
> > > >
> > > > Sample spec run using Chain DSL:
> > > >
> > > > [{
> > > >   "operation": "shift",
> > > >   "spec": {
> > > > "@": "values"
> > > >   },
> > > >   {
> > > > "operation": "default",
> > > > "spec": {
> > > >   "firstname": "${firstname}"
> > > > }
> > > >   }
> > > > ]
> > > >
> > > > Kind regards,
> > > > Matthew
> >

Re: [VOTE] Release Apache NiFi 1.23.1 (RC1)

[Matt Burgess]

+1 (binding)

Ran through release helper, verified NIFI-11922 [1] and stored flows
in Registry.

Apache Maven 3.9.1 (2e178502fcdbffc201671fb2537d0cb4b4cc58f8)
Java version: 17.0.7, vendor: Oracle Corporation, runtime:
/Users/mburgess/.sdkman/candidates/java/17.0.7-oracle
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.4.1", arch: "aarch64", family: "mac"

Good work team, and thanks for RM'ing David!

[1] https://issues.apache.org/jira/browse/NIFI-11922

On Tue, Aug 15, 2023 at 2:08 PM David Handermann
 wrote:
>
> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.23.1.
>
> Please review the following guide for how to verify a release candidate
> build:
>
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The source being voted upon and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.1/
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1234
>
> Git Tag: nifi-1.23.1-RC1
> Git Commit ID: f739e9dd2d476b3c0df5a806ff1dffd24be52916
> GitHub Commit Link:
> https://github.com/apache/nifi/commit/f739e9dd2d476b3c0df5a806ff1dffd24be52916
>
> Checksums of nifi-1.23.1-source-release.zip:
>
> SHA256: 01f5f67a9f9703232f6fe260f6c1da73f3e3a0764b8e8464f915cfad168278e6
> SHA512:
> f8a010ad5a5dd1f71fe04e5d2bf1c6637d2d0a8a7c580a0ff4dbd76c12c2e5ab4ac43e1f5314f9fca85cebe1606bd5e7ae0a8b62f577ddc68696ebd0155d
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file available here:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 51 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353541
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.1
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-1.23.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [DISCUSS] Prepare Apache NiFi 1.23.1 Release?

[Matt Burgess]

+1, thank you David!

On Sat, Aug 12, 2023 at 10:26 AM David Handermann
 wrote:
>
> Team,
>
> Following the release of Apache NiFi 1.23.0, several important bugs have
> been resolved in a few Processors [1], and there have been a handful of
> incremental dependency upgrades [2]. Preparing an incremental patch release
> would provide the opportunity to incorporate these bug fixes and pull in a
> selected number of dependency upgrades.
>
> I am willing to handle the Release Manager responsibilities. If there is a
> positive response to this approach, I can put together a selected list of
> issues next week.
>
> Regards,
> David Handermann
>
> [1]
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20issuetype%20%3D%20Bug%20AND%20fixVersion%20%3D%201.24.0
>
> [2]
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%201.24.0

Re: JoltTransformJSON EL when using file

[Matt Burgess]

Just to follow up, I added a unit test to put EL in a JOLT spec and it
worked. I noticed you referred to "attrname" in your post but your
spec refers to "firstname", is that a typo?

Regards,
Matt

On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess  wrote:
>
> I added file support to JoltTransformJSON under NIFI-4957 [1], a first
> glance at the code seems like it should work, but I'll try to
> reproduce it and follow up, thanks for bringing this to our attention!
>
> Regards,
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-4957
>
> On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins  wrote:
> >
> > Hi devs,
> >
> > Using 1.23.0 I have a simple transform adding some flowfile attributes into
> > the content.
> > When putting the spec directly to the processor it works fine. If I have
> > the spec in an external file however it seems to put in the literal string
> > ${attrname} - ie the EL didn't process.
> >
> > Have I done something incorrectly or should I be now asking for an account
> > to log a bug?
> >
> > Sample spec run using Chain DSL:
> >
> > [{
> >   "operation": "shift",
> >   "spec": {
> > "@": "values"
> >   },
> >   {
> > "operation": "default",
> > "spec": {
> >   "firstname": "${firstname}"
> > }
> >   }
> > ]
> >
> > Kind regards,
> > Matthew

Re: JoltTransformJSON EL when using file

[Matt Burgess]

I added file support to JoltTransformJSON under NIFI-4957 [1], a first
glance at the code seems like it should work, but I'll try to
reproduce it and follow up, thanks for bringing this to our attention!

Regards,
Matt

[1] https://issues.apache.org/jira/browse/NIFI-4957

On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins  wrote:
>
> Hi devs,
>
> Using 1.23.0 I have a simple transform adding some flowfile attributes into
> the content.
> When putting the spec directly to the processor it works fine. If I have
> the spec in an external file however it seems to put in the literal string
> ${attrname} - ie the EL didn't process.
>
> Have I done something incorrectly or should I be now asking for an account
> to log a bug?
>
> Sample spec run using Chain DSL:
>
> [{
>   "operation": "shift",
>   "spec": {
> "@": "values"
>   },
>   {
> "operation": "default",
> "spec": {
>   "firstname": "${firstname}"
> }
>   }
> ]
>
> Kind regards,
> Matthew

Re: Java 17 features in 2.x

[Matt Burgess]

That's a fair point and I think represents the way we want to go
forward. If I understand correctly, you're saying bug fixes meant for
both lines shouldn't need/use Java 17 features but new capabilities
for 2.0 should encourage the use of Java 17 features when prudent?

Thanks,
Matt

On Mon, Aug 7, 2023 at 8:33 PM Joe Witt  wrote:
>
> The views shared thus far are certainly reasonable but I do want to add
> a different take.
>
> The reason we want to do major releases from time to time is so that we can
> take advantage of leaps in the language and frameworks we rely on.  To that
> end it would seem unfortunate to not start aggressively taking advantage of
> that.  In particular we've been held to Java 8 standards for at least 7
> years.  I would advocate we allow and even encourage usage of Java 17
> features/syntax to help move forward.
>
> The point about backporting is important and I agree the 'easiest' way is
> if changes made to main are backportable.  Then again we don't really need
> everything to be backportable and for sure that will start happening less
> and less.  If we're talking about 'bug fixes' then it probably makes sense
> to prefer for now they avoid Java 17 assuming a given fix should target
> both 2.x and 1.x lines.  But if we're talking about new features or even
> improvements I think we should be free to move on to Java 17
> idioms/benefits.  If a contrib does this then it just won't go to the 1.x
> line.  This atrophy is natural/ok I think and lets us give the 2.x line the
> attention/growth it deserves.
>
> Thanks
>
> On Mon, Aug 7, 2023 at 2:19 PM David Handermann 
> wrote:
>
> > Mike,
> >
> > Thanks for raising the question. Following Matt's comments, I recommend
> > minimizing use of Java 17 features to make it easier to backport changes
> > for now.
> >
> > Incremental adjustments can be done when backporting, but it requires the
> > author and reviewer to pay careful attention since the GitHub automated
> > builds for the main branch run on Java 17.
> >
> > As Matt recommended, the alternative is to provide separate pull requests
> > for the main and support branches.
> >
> > We already have a few Java 11 and 17 references on the main branch for
> > things like List.of(), and most of these are easy to adjust when
> > backporting, but they do require careful attention.
> >
> > Regards,
> > David Handermann
> >
> > On Mon, Aug 7, 2023 at 4:04 PM Matt Burgess  wrote:
> >
> > > In my opinion that's ok, but I think it would be helpful if a PR is
> > > going to be backported to support/nifi-1.x that the PR author provides
> > > two PRs, one against main with Java 17 features and one against
> > > support/nifi-1.x with Java 8 features. That being said, allowing Java
> > > 17 features may make maintenance tougher while there's an active 1.x
> > > branch. Maybe we should wait until we only support NiFi 2.x?
> > >
> > > On Mon, Aug 7, 2023 at 4:59 PM Mike Thomsen 
> > > wrote:
> > > >
> > > > Since we're standardizing on 17, we're free and clear to use Java 17
> > > > features, right?
> > >
> >

Re: Java 17 features in 2.x

[Matt Burgess]

In my opinion that's ok, but I think it would be helpful if a PR is
going to be backported to support/nifi-1.x that the PR author provides
two PRs, one against main with Java 17 features and one against
support/nifi-1.x with Java 8 features. That being said, allowing Java
17 features may make maintenance tougher while there's an active 1.x
branch. Maybe we should wait until we only support NiFi 2.x?

On Mon, Aug 7, 2023 at 4:59 PM Mike Thomsen  wrote:
>
> Since we're standardizing on 17, we're free and clear to use Java 17
> features, right?

Re: [VOTE] Release Apache NiFi 1.23.0 (RC3)

[Matt Burgess]

+1 (binding)

Ran through release helper, verified H2 migration on Registry,
ExtractRecordSchema, various other flows to ensure no regressions.

Thanks for RM'ing David!

On Tue, Jul 25, 2023 at 5:33 PM David Handermann
 wrote:
>
> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.23.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The source being voted upon and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.0/
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1233
>
> Git Tag: nifi-1.23.0-RC3
> Git Commit ID: 27a690a30a6ae77c217a47ac0601e85970777ca2
> GitHub Commit Link:
> https://github.com/apache/nifi/commit/27a690a30a6ae77c217a47ac0601e85970777ca2
>
> Checksums of nifi-1.23.0-source-release.zip:
>
> SHA256: 39c97d89804b005cc995c56a87a3df6f68c44ee42114dffe756bbac90a3593cf
> SHA512:
> f256ca731a67435e9883626931abc58f28cda9deb6a7d0a84ed97b78104e43b3b638ee2297d79f92bf5a1e19f62cc78e0a886fa0094593ab34b21d658c59eadd
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file available here:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 132 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353320
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-1.23.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.23.0 (RC2)

[Matt Burgess]

+1 (binding), verified hashes, keys, and commit IDs, ran flows to
verify NIFI-11783 [1], NIFI-11807 [2], and verified no regression in
NIFI-11621 [3].

Apache Maven 3.9.1 (2e178502fcdbffc201671fb2537d0cb4b4cc58f8)
Maven home: /Users/mburgess/.sdkman/candidates/maven/current
Java version: 17.0.7, vendor: Oracle Corporation, runtime:
/Users/mburgess/.sdkman/candidates/java/17.0.7-oracle
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.4.1", arch: "aarch64", family: "mac"

Thanks for RM'ing David!

[1] https://issues.apache.org/jira/browse/NIFI-11783
[2] https://issues.apache.org/jira/browse/NIFI-11807
[3] https://issues.apache.org/jira/browse/NIFI-11621

On Thu, Jul 20, 2023 at 8:16 PM David Handermann
 wrote:
>
> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.23.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1231
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.0/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.23.0-RC2
> The Git commit ID is 2c707ad4e35fa7086024b966f279c8d4da486144
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=2c707ad4e35fa7086024b966f279c8d4da486144
> https://github.com/apache/nifi/commit/2c707ad4e35fa7086024b966f279c8d4da486144
>
> Checksums of nifi-1.23.0-source-release.zip:
> SHA256: 79d8a2cc6f3c2e2b6a31398d2347a0fa5e2849cd47058f6ee7798b231c114b94
> SHA512:
> d80ecab0eb5721ceca1f8840902841fd33ff173ca2c6aff8beefeac6dddc3fe865f2dca9d1e985b339ad75a19ebafffc95d637840957258ae80961bf8db5b0ac
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 115 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353320
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.23.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: Use of attribute uuid and other "native" attributes

[Matt Burgess]

In general I recommend only sending on those attributes that will be
used at some point downstream (unless you have an "original"
relationship that should maintain the original state with respect to
provenance). If you don't know that ahead of time you'll probably need
to send all/most of the attributes just in case.

Are you using session.create() or session.clone()? They both set a new
"uuid" attribute on the created FlowFile, with at least the latter
setting some other attributes as well (see the Developer Guide [1] for
more details).

Regards,
Matt

[1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html

On Tue, Jul 18, 2023 at 12:25 PM Russell Bateman  wrote:
>
> I have a custom processor, /SplitHl7v4Resources/, that splits out
> individual FHIR resources (Patients, Observations, Encounters, etc.)
> from great Bundle flowfiles. So, for a given flowfile, it's split into
> hundreds of smaller ones.
>
> When I do this, I leave the existing NiFi attributes as they were on the
> original flowfile.
>
> As I contemplate the uuid attribute, it occurs to me that I should find
> out what its *significance is for provenance and other potential
> debugging/tracing concerns*. I never really look at it, but, if there
> were some kind of melt-down in a production environment, would I care
> that it multiplied across hundreds of flowfiles besided the original one?
>
> Also these two other NiFi attributes remain unchanged:
>
> filename
> path
>
>
> I do garnish each flowfile with many pointed/significant new attributes
> like resource.type that are my own. In my processing, I don't care about
> NiFi's original attributes, but should I?
>
> Thanks,
> Russ

Re: board report for July 2023

[Matt Burgess]

Looks good to me, this project always has impressive things to report
to the board and they have consistently appreciated it as well, it
clearly indicates a healthy and active community. Thanks for this!

On Tue, Jul 4, 2023 at 12:52 PM Joe Witt  wrote:
>
> Team,
>
> Here is the submitted board report.  Thanks again to all for
> continuing to make this a great project and useful tool for people and
> organizations all over the world!
>
> ## Description:
> The mission of NiFi is the creation and maintenance of software related to
> providing an easy to use, powerful, and reliable system to process and
> distribute data.
>
> Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
> integrate with and leverage the command and control of NiFi. There are both
> Java and C++ implementations.
>
> Apache NiFi Registry is a centralized registry for key configuration items
> including flow versions, assets, and extensions for Apache NiFi and Apache
> MiNiFi.
>
> Apache NiFi Nar Maven Plugin is a release artifact used for supporting the
> NiFi classloader isolation model.
>
> Apache NiFi Flow Design System is a theme-able set of high quality UI
> components and utilities for use across the various Apache NiFi web
> applications in order to provide a more consistent user experience.
>
> ## Project Status:
> Current project status: Ongoing. High.
> Issues for the board: None.
>
> ## Membership Data:
> Apache NiFi was founded 2015-07-14 (9 years ago)
> There are currently 63 committers and 34 PMC members in this project.
> The Committer-to-PMC ratio is roughly 2:1.
>
> Community changes, past quarter:
> - Nándor Soma Abonyi was added to the PMC on 2023-06-25
> - No new committers. Last addition was Ferenc Kis on 2023-03-23.
>
> ## Project Activity:
> The NiFi community released NiFi 1.22.0 Jun 2023 and NiFi 1.21.0 in Apr 2023.
> Each release provided new features, stability, and addressed reported
> vulnerabilities within NiFi and dependencies.  Meanwhile the community is
> rapidly approaching a NiFi 2.0 major release already containing more than 466
> changes. The MiNiFiCPP project also released 0.14.0 in April and we've
> released the latest NiFi Nar Maven Plugin version 1.5.1 in June.
>
>
> ## Community Health:
> Community health remains strong and we see high activity across the various
> channels of contribution and engagement including JIRA, Github, Slack, Mailing
> Lists and so on.  Meetups are less common these days but we still see good
> activity in conferences, blog posts, and more. We also continue to enjoy
> excellent engagement for discussions, voting threads, and release candidate
> evaluation.  Slack remains a super popular point of engagement and we've grown
> again this quarter to more than 2700 people in our 'general' slack channel
> alone. The dev mailing list activity was 20% higher this quarter than last
> while the user list is less active.  Much of the user list activity has
> moved to slack as we've reported for a while now but still the user list is
> an effective tool for collaboration with more than 100 emails in this quarter.
> Via JIRA, commits, and our issues list we still see a lot of activity and even
> growth in activity this quarter.  Importantly our rate of PR activity remains
> high with more than 350+ PRs in the quarter again but importantly we have more
> closed than opened showing we're finally turning the corner in this regard.
> Some of the activity levels and commit levels reflect that we're managing
> actively both the 1.x and 2.x release lines now so we do anticipate many of
> these productivity metrics will drop in the coming quarters as we get the 2.0
> release out and stable.

Re: Jira contributor access

[Matt Burgess]

It appears you have an existing Jira account [1], I have added you as
a Contributor to the NiFi projects in Jira. If you cannot log into
your Jira account and cannot reset your password I believe you have to
create a new one via [2].

Regards,
Matt

[1] 
https://issues.apache.org/jira/secure/ViewProfile.jspa?name=ZmlCode=jira.user.profile.panels:user-profile-summary-panel
[2] https://selfserve.apache.org/jira-account.html

On Mon, Jul 3, 2023 at 10:02 AM 张美丽  wrote:
>
> Hi,
> I am ZmlCode.
> I report the problem of https://issues.apache.org/jira/browse/NIFI-10847.
> But no one helped me review it for a long time.
>
>
> Recently, after reading the official documents, I know that I need to verify 
> the current account by email first.
> So I came to the certification.
>
>
> My account is ZmlCode and email is zhangmeili_c...@163.com .Please.
> Thanks again to NIFI for this program, I learned a lot from it and helped me 
> grow up a lot.
> I hope I can make a small contribution in the future. May be.
>
>
> Have a nice day!
> Best regards.

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.5.1

[Matt Burgess]

+1 (binding)

Verified signatures and hashes, built with the system below, built
NiFi NARs and verified the issue I was encountering is fixed. Thanks
for RM'ing Nandor!

Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39)
Maven home: /Users/mburgess/.sdkman/candidates/maven/current
Java version: 11.0.19, vendor: Eclipse Adoptium, runtime:
/Users/mburgess/.sdkman/candidates/java/11.0.19-tem
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.4", arch: "aarch64", family: "mac"

On Wed, Jun 14, 2023 at 6:55 PM Nandor Soma Abonyi
 wrote:
>
> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache NiFi 
> NAR Maven Plugin 1.5.1.
>
> The source being voted upon, including signatures, digests, etc. can be found 
> at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.5.1/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+NAR+Maven+Plugin+release+candidate
>
> The Git tag is nifi-nar-maven-plugin-1.5.1-rc1
> The Git commit ID is 39fc959426ea405df6360969b55ae2adad47e1aa
> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=39fc959426ea405df6360969b55ae2adad47e1aa
>
> Checksums of nifi-nar-maven-plugin-1.5.1-source-release.zip:
> SHA256: 0ddc4efbfe504f9ed6477b8c572f2c6e5ba0c953e2e5b063bdfbd1f934eda6bf
> SHA512: 
> a7281c8a3769db37e3491f3a5e54533b5f26bdcad99f8adb1e5609f1de17309aefb3d49eb9231e75a814e42566525b9afe4a11bda2c4ab48e8bab5a298b72b24
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/nsabonyi.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 3 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353009
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.5.1
>
> The vote will be open for 72 hours. Please download the release
> candidate and evaluate the necessary items including checking hashes,
> signatures, build from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-nar-maven-plugin-1.5.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because …
>

Re: [DISCUSS] Release NiFi NAR Maven Plugin 1.5.1

[Matt Burgess]

+1 from me, I ran into the issue so would like to have that fix
available.  Thanks for volunteering to RM!

On Mon, Jun 12, 2023 at 5:36 AM Nandor Soma Abonyi
 wrote:
>
> Hello Apache NiFi Community,
>
> I'd like to initiate a discussion about the next release of 
> nifi-nar-maven-plugin.
>
> Currently, nifi-nar-maven-plugin doesn’t use remote repositories, which was
> fixed in [1]. Though it is the only change since the last release, I consider 
> it significant
> enough to suggest releasing nifi-nar-maven-plugin as 1.5.1.
>
> I would be happy to take on RM duties for this release.
>
> Do you agree it is time for a new release?
>
> Thanks,
> Nandor Soma Abonyi
>
> [1] NIFI-11324  NAR Plugin 
> extension docs not correctly resolving overridden versions from system 
> properties
>

Re: [VOTE] Release Apache NiFi 1.22.0 (RC1)

[Matt Burgess]

+1 (binding)

Ran through release helper and tested various flows including
verifying NIFI-11621 [1] for nested array inference and NIFI-11653 [2]
for removing H2 support for DBCPConnectionPool. Also tested some
version control capability for NiFi Registry.

Apache Maven 3.8.2 (ea98e05a04480131370aa0c110b8c54cf726c06f)
Maven home: /Users/mburgess/.sdkman/candidates/maven/current
Java version: 11.0.19, vendor: Eclipse Adoptium, runtime:
/Users/mburgess/.sdkman/candidates/java/11.0.19-tem
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.4", arch: "aarch64", family: "mac"

Thanks for RM'ing Joe!

[1] https://issues.apache.org/jira/browse/NIFI-11621
[2] https://issues.apache.org/jira/browse/NIFI-11653

On Tue, Jun 6, 2023 at 5:38 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.22.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1228
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.22.0/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.22.0-RC1
> The Git commit ID is 71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
> https://github.com/apache/nifi/commit/71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
>
> Checksums of nifi-1.22.0-source-release.zip:
> SHA256: d63a5f1db95160434670f112a0ee6e06fa141182bd0f12f0e58e3229991f3743
> SHA512:
> 5f6da64e75a2d5446f1f274fe3de7e97290f5b916eabbc0491a99c6db33f74fbdea001b403044e75bc5c2183fb0500dfc143d0f4cba19d3511f866fff774d7f5
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 186 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353069
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.22.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.22.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: User-defined ProvenanceReporter

[Matt Burgess]

All,

We've been discussing this in Slack [1], we can continue there and if
anyone is interested and not on our Apache Slack channel, I encourage
you to join [2] but I can also copy-paste into this thread if need be.

Regards,
Matt

[1] https://apachenifi.slack.com/archives/C0L9VCD47/p1682895885163669
[2] 
https://join.slack.com/t/apachenifi/shared_invite/zt-1u8wrsi2b-dE6Rf0lEzdYWgkBaPykceg

On Sun, Apr 30, 2023 at 7:09 PM Yu Chen  wrote:
>
> Hi,
>
> May I ask if there is a method I can use to implement my own
> ProvenanceReporter (
> https://github.com/apache/nifi/blob/main/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java)
> to replace the default StandardProvenanceReporter (
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/repository/StandardProvenanceReporter.java)
> ?
>
>
> Thanks
> Yu

Re: Issue upgrading Nifi(1.7.0) & Minifi(0.5.0) to 1.20.0

[Matt Burgess]

Ignasi,

A lot has changed between MiNiFi 0.5.0 and MiNiFi 1.20.0, including changes
to NARs, how they are loaded, etc. For custom NARs I would try to
rebuild them using the same version of NiFi and MiNiFi dependencies. The
latest Apache NiFi release is now 1.21.0 so I would change the NiFi and
MiNiFi dependencies in your custom NARs to 1.21.0. You may find that there
are compilation errors or some issue that wasn't there before, these would
be things introduced between NiFi 1.7.0 and NiFi 1.21.0. I can't think of
any offhand but wanted to mention it in case your NARs no longer
compile/build.

Regards,
Matt


On Fri, Apr 28, 2023 at 5:31 AM Garriga, Ignasi
 wrote:

> Hi all,
>
> We're trying to upgrade versions of Apache Nifi 1.7.0 with Minifi 0.5.0
> to the latest embedded version 1.20.0.
> We have some custom nars that when we add to the project there are
> problems with the Classloader and bundle not found:
>
> ERROR [main] o.a.n.n.StandardExtensionDiscoveringManager Could not
> instantiate class of type 
> org.apache.nifi.processors.standard.JoltTransformJSON
> using ClassLoader for bundle default:unknown:unversioned because the bundle
> could not be found
>
>
> In the following pictures I share with you the docker-compose that we use
> to create an image and all libraries that we use to compile. It can help to
> identify something or give a clue to find the solution.
>
> pm-data-pipeline:
> image: apache/nifi-minifi:1.20.0
> restart: always
> hostname: pm-data-pipeline
> container_name: pm-data-pipeline
> environment:
>   - NIFI_WEB_HTTPS_PORT=9443
> ports:
>   - "8080:8080"
>   - "9443:9443"
> volumes:
>   - 
> ../../../assets/templates/nMonFlow.yml:/opt/minifi/minifi-1.20.0/conf/config.yml
>   - 
> ../../../assets/nifi-invoke-igw-nar-1.6.0.nar:/opt/minifi/minifi-1.20.0/lib/nifi-invoke-igw-nar-1.6.0.nar
>   - 
> ../../../assets/pm-data-pipeline-ssl-context-service-nar-1.0.0.nar:/opt/minifi/minifi-1.20.0/lib/pm-data-pipeline-ssl-context-service-nar-1.0.0.nar
>   - 
> ../../../assets/nifi-amqp-nar-1.20.0.nar:/opt/minifi/minifi-1.20.0/lib/nifi-amqp-nar-1.20.0.nar
>
>
> [image: image.png]
>
>
> Thank you in advance for your support and don't hesitate to contact us if
> you need more information .
>
> Regards,
>
>
> --
> *--*
>
> *Ignasi Garriga Sala* Coagulation Software Developer
>
> Roche Diagnostics SL
> Patient Care POC
> Av. de la Generalitat, 171-173
> E-08174 Sant Cugat del Vallès
>
> M +34 619 048 267
> *ignasi.garr...@roche.com* 
>
> roche.es  | AularioRoche
>  | RocheDialog
>  | Twitter
>  | YouTube 
> | Facebook  | LinkedIn
> 
>
> *Confidencialidad* - Este mensaje ha sido dirigido sólo al destinatario o
> a los destinatarios mencionados y puede contener información confidencial
> y/o de uso restringido a profesionales de la salud, en cuyo caso su
> remisión a terceros que no sean profesionales de la salud no está
> permitida. Si usted tiene dudas respecto a quién puede remitir este mensaje
> y/o la información que contiene, por favor, consulte con su remitente. Si
> usted no debería haber recibido este correo o piensa que el destinatario es
> incorrecto, por favor contacte con el remitente y borre este mensaje. El
> uso no autorizado de la información contenida en este mensaje está
> prohibido.
>
> *Protección de datos* - Roche Diagnostics, S.L.U. (con domicilio en Avda.
> de la Generalitat, 171-173, 08174 Sant Cugat del Vallès, Barcelona) es el
> responsable del tratamiento de sus datos de contacto corporativos con el
> fin de poder mantener relaciones con usted o con la empresa, entidad y
> organización para la cual usted trabaja o colabora. Para obtener más
> información sobre cómo tratamos sus datos y cómo ejercer sus derechos,
> consulte nuestra política de privacidad
> .
>
>

Re: [VOTE] Release Apache NiFi 1.21.0 (RC2)

[Matt Burgess]

+1 (binding), ran through the release helper and a few flows
(including one with the CaptureChangeMySQL changes) on a Mac with Java
11.

Thanks for RM'ing Joe!

On Mon, Apr 3, 2023 at 8:11 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.21.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1226
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.21.0/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.21.0-RC2
> The Git commit ID is 892f822107da84ca0dcfefdb5c91c5bc11dc3dc1
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=892f822107da84ca0dcfefdb5c91c5bc11dc3dc1
>
> Checksums of nifi-1.21.0-source-release.zip:
> SHA256: 598bf8e90f871f4ca25709dd4ecf3da16c814c08c0d8b2c8744dbd34df34dea5
> SHA512: 
> 58c10976bc22fb5406d9d1d9b7ca7d90c2dbed99565d00f1172bb33b375e9e068da5003d9dbbd87d2b17626e4e310b999c8581718532934e855c2134be763f04
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 126 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352899
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.21.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.21.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [discuss] NiFi support for Hadoop ecosystem components

[Matt Burgess]

As one of the small number of people that fight the battle, I like the
idea of Option 1 (full disclosure: I work for a vendor). From a
community standpoint (I'm on the PMC) I'm not strongly opposed to
Option 2 although I wouldn't want to be the one managing and releasing
the artifacts :) Having said that, unless it remained maintained and
released, I feel like it would just be a component graveyard (or maybe
more like the Apache Attic), in which case it seems unnecessary and
that's why I'm behind Option 1. Interested to hear others' thoughts of
course.

Thanks,
Matt

On Fri, Mar 24, 2023 at 2:07 PM Joe Witt  wrote:
>
> Team,
>
> For the full time NiFi has been in Apache we've built with support for
> various Hadoop ecosystem components like HDFS, Hive, HBase, others,
> and more recently formats/serialization modes like necessary for
> Parquet, Orc, Iceberg, etc..
>
> All of these things however present endless challenges with
> compatibility across different versions (Hive being the most difficult
> by far), vendors (hadoop vendors, cloud vendors, etc..).  And also
> super notably the incredible number of dependencies, dependency
> conflicts, inclusions/exclusions, old log libs, vulnerability updates,
> etc..  And last but certainly not least a big reason why our build has
> grown so much.
>
> We have a couple options:
> 1. Deprecate these components in NiFi 1.x and drop them entirely in
> NiFi 2.x.  Leave this as a problem for vendors to deal with.  NiFi
> users interacting with such components are nearly exclusively doing so
> with vendors anyway.
>
> 2. Remove the components from NiFi main code line and create a
> separate repo for 'nifi-hadoop-extensions'.  We manage those
> independently and release them periodically.  They would be available
> for people to grab the nars if they want to use them.  We include none
> of them in the convenience binary going forward by default.
>
> 3. Change nothing.  Continue to battle with the above listed items.
> This is admittedly a bit of a non-option.  We can't keep spending the
> same time/energy on these we have.  It is a very small number of
> people that fight this battle.
>
> Look forward to hearing thoughts on these options or others we might consider.
>
> Thanks

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.5.0 - RC2

[Matt Burgess]

+1 (binding)

Ran through release helper, built NiFi bundles/NARs using this version
of the plugin, all looks good. Thanks for RM'ing Kevin!

On Tue, Mar 14, 2023 at 5:00 PM Kevin Doran  wrote:
>
> Hello Apache NiFi Community,
>
> Issues with RC1 have been addressed in RC2, and I am pleased to again
> be calling this vote for the source release of Apache NiFi NAR Maven
> Plugin 1.5.0.
>
> The source being voted upon, including signatures, digests, etc. can
> be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1222/org/apache/nifi/nifi-nar-maven-plugin/1.5.0/
>
> The Git tag is nifi-nar-maven-plugin-1.5.0-RC2
> The Git commit ID is 277f9e8998ca76a972c90d8ecf3f771414c86700
> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=277f9e8998ca76a972c90d8ecf3f771414c86700
>
> Checksum of nifi-nar-maven-plugin-1.5.0-source-release.zip:
>
> SHA512: 
> a265587f8bfb31359cae2335394d88d76d36ae9806030e38fc9846264f20a4e70d642c4f0c08425b843794aaabaeea416b1104ede26671ffcbe001e1976d4efd
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/kdoran.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 4 issues were closed/resolved for this release:
> https://issues.apache.org/jira/projects/NIFI/versions/12352895
>
> Release note highlights can be found here:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352895
>
> *The vote will be open for 24 hours.* Please download the release
> candidate and evaluate the necessary items including checking hashes,
> signatures, build from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-nar-maven-plugin-1.5.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because ...

Re: [VOTE] Release Apache NiFi 1.20.0 (RC1)

[Matt Burgess]

+1 (binding)

Ran through the release helper and a few flows. I did see this as a
race condition for InvokeScriptedProcessor but it fixes itself on the
first validation so I don't think we need to tank the RC as a result,
I wrote up [1] to investigate:

2023-02-07 16:26:39,825 ERROR [NiFi Web Server-32]
o.a.n.p.script.InvokeScriptedProcessor
InvokeScriptedProcessor[id=2bbc053d-8b08-3206-7c47-cc00a08beb64] Error
adding script engine Groovy
2023-02-07 16:26:39,826 ERROR [NiFi Web Server-32]
o.a.n.p.script.InvokeScriptedProcessor
InvokeScriptedProcessor[id=2bbc053d-8b08-3206-7c47-cc00a08beb64]
Unable to load script: No script runner available
org.apache.nifi.processor.exception.ProcessException: No script runner available
at 
org.apache.nifi.processors.script.InvokeScriptedProcessor.reloadScript(InvokeScriptedProcessor.java:371)
at 
org.apache.nifi.processors.script.InvokeScriptedProcessor.reloadScriptBody(InvokeScriptedProcessor.java:326)
at 
org.apache.nifi.processors.script.InvokeScriptedProcessor.setup(InvokeScriptedProcessor.java:230)
at 
org.apache.nifi.processors.script.InvokeScriptedProcessor.onConfigurationRestored(InvokeScriptedProcessor.java:222)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
at 
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
at 
org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotations(ReflectionUtils.java:316)
at 
org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotation(ReflectionUtils.java:93)
at 
org.apache.nifi.controller.StandardProcessorNode.onConfigurationRestored(StandardProcessorNode.java:2115)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.addProcessor(StandardVersionedComponentSynchronizer.java:2417)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronizeProcessors(StandardVersionedComponentSynchronizer.java:932)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:422)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.lambda$synchronize$0(StandardVersionedComponentSynchronizer.java:260)
at 
org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:550)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:255)
at 
org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3972)
at 
org.apache.nifi.groups.StandardProcessGroup.updateFlow(StandardProcessGroup.java:3952)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.updateProcessGroupFlow(StandardProcessGroupDAO.java:435)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at 
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
at 
org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:308)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
at 
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at 

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.4.0

[Matt Burgess]

+1 binding, ran through release helper (noticed the tag hadn't been
pushed but is now, thanks!) Built NiFi with 1.4.0-RC1 plugin, verified
all worked successfully. Thanks for RM'ing Kevin!

On Mon, Jan 30, 2023 at 6:19 PM Kevin Doran  wrote:
>
> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi NAR Maven Plugin 1.4.0.
>
> The source being voted upon, including signatures, digests, etc. can
> be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1219/org/apache/nifi/nifi-nar-maven-plugin/1.4.0/
>
> The Git tag is nifi-nar-maven-plugin-1.4.0-RC1
> The Git commit ID is 09d7bb9ff679d0eed9feaa066d2cbdd347a20204
> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=09d7bb9ff679d0eed9feaa066d2cbdd347a20204
>
> Checksum of nifi-nar-maven-plugin-1.4.0-source-release.zip
>
> SHA512: 
> d48bfde8a5bab8a17b320889297c09efa162a7017db2268e274db626926ff9b58173abab54db4e1a50b5664fc86a9501ce2ef267b3e3b768eb80ef5c929860c9
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/kdoran.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 6 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352124
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.4.0
>
> The vote will be open for 72 hours. Please download the release
> candidate and evaluate the necessary items including checking hashes,
> signatures, build from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-nar-maven-plugin-1.4.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because ...

Re: Line number to identify where CSVReader failed

[Matt Burgess]

I was thinking the same, CSVRecordReader could keep track of the
number of records read and if an exception is thrown during iteration
over reading the records, we can output the number of records read
successfully.

On Thu, Jan 19, 2023 at 3:47 PM Joe Witt  wrote:
>
> Dan,
>
> Seems like our record reader mechanism should offer the concept of tracking
> which record it is on such that this could be logged.  It looks from a
> quick check like we track record count on writing so something similar on
> the interface of the reader could be quite helpful.
>
> Perhaps best to file a JIRA.  Someone else might have a better idea of what
> you can do now.
>
> Thanks
>
> On Thu, Jan 19, 2023 at 1:39 PM Dan S  wrote:
>
> > For both QueryRecord and ValidateRecord when I use a CSVReader on a file
> > which has different delimiters than the rest of the file, the error message
> > logged does not include the line number where the parsing failed.  When
> > looking at the code, I did not see any hooks for getting that information.
> > Is there a way to get the line number so it would be easy to identify
> > which lines would need to be fixed?
> >

Re: Getting a NiFi Jira account

[Matt Burgess]

Kim,

According to the new Jira guidelines [1], please submit the following
information to priv...@nifi.apache.org and the PMC will create an
account for you and add you as a Contributor to the NiFi Jira
projects:

email address
preferred username (N.B. hyphens not allowed)
alternate username (in case the preferred one is already in use)
display name, if it is different from the username

Regards,
Matt

[1] https://infra.apache.org/jira-guidelines.html#who

On Mon, Jan 16, 2023 at 9:18 AM Kim Pedersen  wrote:
>
> Hello,
>
> I want to submit a PR on NiFi at Github. I have been reading the contributor 
> guide, concluding that I need to submit a Jira issue before creating the PR. 
> So I need a NiFi Jira acconut. Can you help me with that or point me in the 
> right direction?
>
> Regards,
> Kim Pedersen

Re: nifi proxy issue + jira account

[Matt Burgess]

Kyle,

Per the new Apache Jira instructions [1], please provide the following
information to priv...@nifi.apache.org  and I can
create an account for you and add you as a
contributor:

- email address
- preferred username (N.B. hyphens not allowed)
- alternate username (in case the preferred one is already in use)
- display name, if it is different from the username


Regards,
Matt

[1] https://infra.apache.org/jira-guidelines.html#who

On Thu, Dec 15, 2022 at 4:43 PM Nguyen, Kyle 
wrote:

> Hi.  Thanks for maintaining the project.
>
>
>
> 1.   Can I get a jira account please?  Would like to request *
> StandardProxyConfigurationService* integration into 
> *StandardOath2AccessTokenProvider
> *and into *QuerySalesforceObject*
>
> 2.   Or perhaps you can help me more directly on this stackoverflow:
> https://stackoverflow.com/questions/74482819/connecting-to-an-external-http-api-behind-a-proxy-from-nifi
>
>
>
> I also tried this ENV VAR in my docker nifi instance.  It doesn’t work.
>
>
>
> JAVA_FLAGS=-Dhttp.nonProxyHosts="localhost,.amazonaws.com"
> -Dhttp.proxyHost=proxy.foo.com -Dhttp.proxyPort=3128 -Dhttp.proxyUser=foo
> -Dhttp.proxyPassword=bar
>
>
>
>
>
> [image: cid:17261f6184a4cff311]
>
> *Kyle Nguyen*
> Corporate Technology, Software Engineer
>
>
>
> *Millennium Management LLC*
>
> 399 Park Avenue  |  New York, NY 10022
>
>  +1.212.708.1366  |  +1.929.837.1788
>
> mlp.com 
>
>
>
>
>
> ##
>
> The information contained in this communication is confidential and
>
> may contain information that is privileged or exempt from disclosure
>
> under applicable law. If you are not a named addressee, please notify
>
> the sender immediately and delete this email from your system.
>
> If you have received this communication, and are not a named
>
> recipient, you are hereby notified that any dissemination,
>
> distribution or copying of this communication is strictly prohibited.
> ##
>
>

Re: Log Cleansing

[Matt Burgess]

Bryan,

Do you have an example of the kind of cleansing you'd like to do? NiFi
has things like GrokReader [1], syslog parsers and such to parse logs,
and the corresponding writers to format the log text the way you want.

Regards,
Matt

[1] 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.grok.GrokReader/index.html

On Wed, Dec 14, 2022 at 2:16 PM Bryan Paul  wrote:
>
> Good Afternoon,
>
> Our company is using NIFI as a log data pipeline.  I know there are other 
> solutions on the market that offer log cleansing as a mechanism to remove 
> noise from logs that are sent to SIEM tools.  Does NIFI have this capability?

Re: [VOTE] Adopt NiFi 2.0 Proposed Release Goals

[Matt Burgess]

+1 (binding)

On Mon, Dec 12, 2022 at 12:02 PM David Handermann
 wrote:
>
> Team,
>
> Following positive feedback on NiFi 2.0 Proposed Release Goals [1] on the
> recent discussion thread [2], I am calling this vote to adopt the following
> as Release Goals for NiFi 2.0:
>
> 1. Remove Java 8 support and require Java 11
> 2. Remove deprecated components
> 3. Remove deprecated component properties
> 4. Remove components integrating with unmaintained services
> 5. Remove compatibility classes and methods
> 6. Remove flow.xml.gz in favor of flow.json.gz
> 7. Remove duplicative features
> 8. Upgrade internal Java API references
> 9. Reorganize standard components
> 10. Implement migration tools for upgrading flows
>
> A positive vote indicates agreement on these goals and the initiation of
> the following actions:
>
> 1. Rename NiFi 2.0 Proposed Release Goals to NiFi 2.0 Release Goals
> 2. Create version 1 branch in Git for subsequent support releases on the
> version 1 series
> 3. Update the current main branch in Git to version 2.0.0-SNAPSHOT
>
> The vote will be open for 72 hours and follow standard procedures for
> release votes.
>
> Please review the linked goals and discussions for background.
>
> [ ] +1 Adopt NiFi 2.0 Release Goals
> [ ] +0 No opinion
> [ ] -1 Do not adopt NiFi 2.0 Release Goals for the following reasons...
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Proposed+Release+Goals
> [2] https://lists.apache.org/thread/xo77p9t3xg4k70356xrqbdg4m9sg7sf8

Re: is cdc db2

[Matt Burgess]

Sagar,

There is an open Jira for SQL Server CDC [1] which may have been
abandoned waiting for review. I can take a look at the Pull Request
but it would clearly need changes to match the current version of NiFi
at the very least. I couldn't find a Jira for a CDC processor for DB2,
but feel free to file a New Feature Jira for such a processor; if you
need a Jira account please follow the new guidelines that should be at
the top of any Jira page [2].

Regards,
Matt

[1] https://issues.apache.org/jira/browse/NIFI-4521
[2] https://infra.apache.org/jira-guidelines.html#who

On Wed, Dec 7, 2022 at 6:25 AM Pallikonda, Vidyasagar
 wrote:
>
> Hi Team,
>
> Are there processors available to pull data using CDC (Change Data Center) 
> from DB2 and SQL Server?
>
>
> Thanks
> Sagar

Re: [VOTE] Release Apache NiFi 1.19.1 (RC2)

[Matt Burgess]

+1 (binding)

Ran through release helper, tested a handful of flows including one
with the new UpdateDatabaseTable processor. I verified the existence
of NIFI-10956 in this release; however it has been in multiple
releases and I put a PR up, IMO it is not a blocker and can wait until
the next release (dot or otherwise).

Thanks for RM'ing Joe!

On Mon, Dec 5, 2022 at 2:58 PM Joe Witt  wrote:
>
> Hello,
>
> Please note this is intended as a shorter than normal release cycle
> given the limited scope and the importance of getting some of these
> fixes to those already trying 1.19.  If we need to extend due to lack
> of vote participation we will.
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.19.1.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1218
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.19.1/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.19.1-RC2
> The Git commit ID is a7236ecc9123113ba5b9aaa3baab06354778116f
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=a7236ecc9123113ba5b9aaa3baab06354778116f
>
> Checksums of nifi-1.19.1-source-release.zip:
> SHA256: 1172b133096c9d88b185413afc1eb6220d3f0542df6cdf82ceb7b01ae6fa15ed
> SHA512: 
> 3b571c9151bb755653f5e5ca3c275813134572c0fc28d8730a0dc5c1390df118b2be1f013f9acc0e394a959ff5dc853402f2c03d8f504010d6b835b43d786784
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 36 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352626
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.19.1
>
> The vote will be open for 36 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.19.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.19.0 (RC1)

[Matt Burgess]

+1 (binding) Ran through release helper, tested a couple of flows
including one with the new UpdateDatabaseTable processor. Thanks for
RM'ing Joe!

On Wed, Nov 23, 2022 at 10:58 AM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.19.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1216
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.19.0/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.19.0-RC1
> The Git commit ID is ec87bf93add2f645d2ea426002e0c24db54614ff
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=ec87bf93add2f645d2ea426002e0c24db54614ff
>
> Checksums of nifi-1.19.0-source-release.zip:
> SHA256: 909c4fce81b305af955540f83115474d139c132e886847df7b76e94efd3d4641
> SHA512:
> b3ba95984841140aa2a9a2ea471dd446d6fbd71dba320f723839e0938a7873f847e415a88b7c38e59907ad051c896949f0c7093c6d5e6b5db549de929ec32c6e
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 221 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352150
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.19.0
>
> The vote will be open for at least 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.19.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [discuss] NiFi 1.19.0

[Matt Burgess]

Awesome, thanks all! I'd like to wait for Github actions to be happy
and then merge?

On Tue, Nov 15, 2022 at 7:11 PM Mark Bean  wrote:
>
> NIFI-10703 has been worked through. I think it's good to go now.
>
> On Tue, Nov 15, 2022 at 4:43 PM Matt Burgess  wrote:
>
> > I can take that review too, if your suggestions are incorporated. I
> > started looking the other day but didn't get to dig in. I'll try to
> > reproduce in the meantime.
> >
> > Regards,
> > Matt
> >
> > On Tue, Nov 15, 2022 at 4:42 PM Nathan Gough  wrote:
> > >
> > > We might also want https://issues.apache.org/jira/browse/NIFI-10787
> > >
> > > On Tue, Nov 15, 2022 at 4:05 PM Mark Bean  wrote:
> > >
> > > > I will be on it in about 2 hours, if not addressed sooner.
> > > >
> > > > On Tue, Nov 15, 2022 at 3:46 PM Joe Witt  wrote:
> > > >
> > > > > NIFI-10703 would be great to get in.  Just a matter of gettin' it
> > done
> > > > and
> > > > > merged I think.  You're on it?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Tue, Nov 15, 2022 at 1:38 PM Mark Bean 
> > wrote:
> > > > >
> > > > > > Sorry.. check that. There's a typo in the latest commit on that PR.
> > > > > >
> > > > > > On Tue, Nov 15, 2022 at 3:35 PM Mark Bean 
> > > > wrote:
> > > > > >
> > > > > > > Could we get NIFIDEVS-10703 (PR #6638) in there too? AFAIK, it's
> > good
> > > > > to
> > > > > > > go.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mark
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 15, 2022 at 2:56 PM Joe Witt 
> > wrote:
> > > > > > >
> > > > > > >> Hello All,
> > > > > > >>
> > > > > > >> I plan to kick off the RC as soon as
> > > > > > >> https://issues.apache.org/jira/browse/NIFI-10701 is merged.
> > > > > > >>
> > > > > > >> Thanks
> > > > > > >>
> > > > > > >> On Thu, Nov 10, 2022 at 12:52 PM David Handermann <
> > > > > > >> exceptionfact...@apache.org> wrote:
> > > > > > >>
> > > > > > >> > Joe,
> > > > > > >> >
> > > > > > >> > Thanks for initiating the discussion. I agree with moving
> > toward a
> > > > > > >> 1.19.0
> > > > > > >> > release instead of 1.18.1 given over 150 Jira issues already
> > > > tagged
> > > > > > with
> > > > > > >> > 1.19.0.
> > > > > > >> >
> > > > > > >> > We should start another thread soon regarding 2.0. One of the
> > key
> > > > > > >> things to
> > > > > > >> > do before that, however, would be to finish adding deprecation
> > > > > logging
> > > > > > >> to
> > > > > > >> > remaining features we plan to remove in 2.0. Having a minor
> > > > release
> > > > > in
> > > > > > >> > version 1 series deprecating targets for removal should be a
> > > > > > >> prerequisite
> > > > > > >> > before branching and starting on 2.0.
> > > > > > >> >
> > > > > > >> > In the meantime, if that means we plan on a 1.20.0 release, I
> > am
> > > > in
> > > > > > >> favor
> > > > > > >> > of moving forward with 1.19.0.
> > > > > > >> >
> > > > > > >> > Regards,
> > > > > > >> > David Handermann
> > > > > > >> >
> > > > > > >> > On Thu, Nov 10, 2022 at 1:44 PM Joe Witt 
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > Team,
> > > > > > >> > >
> > > > > > >> > > We keep getting a slew of emails on various lists, slack,
> > and
> > > > the
> > > > > > >> > security
> > > > > > >> > > list about commons text.  We need a 1.18.1 or a 1.19.0.
> > > > > > >> > >
> > > > > > >> > > Given all that has already happened on 1.19.0 I'm inclined
> > to
> > > > just
> > > > > > 'do
> > > > > > >> > > that' release and get it going.
> > > > > > >> > >
> > > > > > >> > > I know we need to start getting serious about 2.0 but
> > perhaps
> > > > the
> > > > > > most
> > > > > > >> > > realistic path for that is we branch and come back to main
> > after
> > > > > we
> > > > > > >> get
> > > > > > >> > > that figured out.  But we need to keep moving forward in the
> > > > > > meantime.
> > > > > > >> > >
> > > > > > >> > > Please share if there is anything we know about on the main
> > line
> > > > > > which
> > > > > > >> > > would make doing a 1.19.0 problematic.
> > > > > > >> > >
> > > > > > >> > >
> > https://issues.apache.org/jira/projects/NIFI/versions/12352345
> > > > > > >> > >
> > > > > > >> > > Thanks
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> >

Re: [discuss] NiFi 1.19.0

[Matt Burgess]

I can take that review too, if your suggestions are incorporated. I
started looking the other day but didn't get to dig in. I'll try to
reproduce in the meantime.

Regards,
Matt

On Tue, Nov 15, 2022 at 4:42 PM Nathan Gough  wrote:
>
> We might also want https://issues.apache.org/jira/browse/NIFI-10787
>
> On Tue, Nov 15, 2022 at 4:05 PM Mark Bean  wrote:
>
> > I will be on it in about 2 hours, if not addressed sooner.
> >
> > On Tue, Nov 15, 2022 at 3:46 PM Joe Witt  wrote:
> >
> > > NIFI-10703 would be great to get in.  Just a matter of gettin' it done
> > and
> > > merged I think.  You're on it?
> > >
> > > Thanks
> > >
> > > On Tue, Nov 15, 2022 at 1:38 PM Mark Bean  wrote:
> > >
> > > > Sorry.. check that. There's a typo in the latest commit on that PR.
> > > >
> > > > On Tue, Nov 15, 2022 at 3:35 PM Mark Bean 
> > wrote:
> > > >
> > > > > Could we get NIFIDEVS-10703 (PR #6638) in there too? AFAIK, it's good
> > > to
> > > > > go.
> > > > >
> > > > > Thanks,
> > > > > Mark
> > > > >
> > > > >
> > > > > On Tue, Nov 15, 2022 at 2:56 PM Joe Witt  wrote:
> > > > >
> > > > >> Hello All,
> > > > >>
> > > > >> I plan to kick off the RC as soon as
> > > > >> https://issues.apache.org/jira/browse/NIFI-10701 is merged.
> > > > >>
> > > > >> Thanks
> > > > >>
> > > > >> On Thu, Nov 10, 2022 at 12:52 PM David Handermann <
> > > > >> exceptionfact...@apache.org> wrote:
> > > > >>
> > > > >> > Joe,
> > > > >> >
> > > > >> > Thanks for initiating the discussion. I agree with moving toward a
> > > > >> 1.19.0
> > > > >> > release instead of 1.18.1 given over 150 Jira issues already
> > tagged
> > > > with
> > > > >> > 1.19.0.
> > > > >> >
> > > > >> > We should start another thread soon regarding 2.0. One of the key
> > > > >> things to
> > > > >> > do before that, however, would be to finish adding deprecation
> > > logging
> > > > >> to
> > > > >> > remaining features we plan to remove in 2.0. Having a minor
> > release
> > > in
> > > > >> > version 1 series deprecating targets for removal should be a
> > > > >> prerequisite
> > > > >> > before branching and starting on 2.0.
> > > > >> >
> > > > >> > In the meantime, if that means we plan on a 1.20.0 release, I am
> > in
> > > > >> favor
> > > > >> > of moving forward with 1.19.0.
> > > > >> >
> > > > >> > Regards,
> > > > >> > David Handermann
> > > > >> >
> > > > >> > On Thu, Nov 10, 2022 at 1:44 PM Joe Witt 
> > > wrote:
> > > > >> >
> > > > >> > > Team,
> > > > >> > >
> > > > >> > > We keep getting a slew of emails on various lists, slack, and
> > the
> > > > >> > security
> > > > >> > > list about commons text.  We need a 1.18.1 or a 1.19.0.
> > > > >> > >
> > > > >> > > Given all that has already happened on 1.19.0 I'm inclined to
> > just
> > > > 'do
> > > > >> > > that' release and get it going.
> > > > >> > >
> > > > >> > > I know we need to start getting serious about 2.0 but perhaps
> > the
> > > > most
> > > > >> > > realistic path for that is we branch and come back to main after
> > > we
> > > > >> get
> > > > >> > > that figured out.  But we need to keep moving forward in the
> > > > meantime.
> > > > >> > >
> > > > >> > > Please share if there is anything we know about on the main line
> > > > which
> > > > >> > > would make doing a 1.19.0 problematic.
> > > > >> > >
> > > > >> > > https://issues.apache.org/jira/projects/NIFI/versions/12352345
> > > > >> > >
> > > > >> > > Thanks
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >

Re: [discuss] NiFi 1.19.0

[Matt Burgess]

Mark,

I can review any incoming changes when they land, if that helps.

Regards,
Matt

On Tue, Nov 15, 2022 at 4:05 PM Mark Bean  wrote:
>
> I will be on it in about 2 hours, if not addressed sooner.
>
> On Tue, Nov 15, 2022 at 3:46 PM Joe Witt  wrote:
>
> > NIFI-10703 would be great to get in.  Just a matter of gettin' it done and
> > merged I think.  You're on it?
> >
> > Thanks
> >
> > On Tue, Nov 15, 2022 at 1:38 PM Mark Bean  wrote:
> >
> > > Sorry.. check that. There's a typo in the latest commit on that PR.
> > >
> > > On Tue, Nov 15, 2022 at 3:35 PM Mark Bean  wrote:
> > >
> > > > Could we get NIFIDEVS-10703 (PR #6638) in there too? AFAIK, it's good
> > to
> > > > go.
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > > >
> > > > On Tue, Nov 15, 2022 at 2:56 PM Joe Witt  wrote:
> > > >
> > > >> Hello All,
> > > >>
> > > >> I plan to kick off the RC as soon as
> > > >> https://issues.apache.org/jira/browse/NIFI-10701 is merged.
> > > >>
> > > >> Thanks
> > > >>
> > > >> On Thu, Nov 10, 2022 at 12:52 PM David Handermann <
> > > >> exceptionfact...@apache.org> wrote:
> > > >>
> > > >> > Joe,
> > > >> >
> > > >> > Thanks for initiating the discussion. I agree with moving toward a
> > > >> 1.19.0
> > > >> > release instead of 1.18.1 given over 150 Jira issues already tagged
> > > with
> > > >> > 1.19.0.
> > > >> >
> > > >> > We should start another thread soon regarding 2.0. One of the key
> > > >> things to
> > > >> > do before that, however, would be to finish adding deprecation
> > logging
> > > >> to
> > > >> > remaining features we plan to remove in 2.0. Having a minor release
> > in
> > > >> > version 1 series deprecating targets for removal should be a
> > > >> prerequisite
> > > >> > before branching and starting on 2.0.
> > > >> >
> > > >> > In the meantime, if that means we plan on a 1.20.0 release, I am in
> > > >> favor
> > > >> > of moving forward with 1.19.0.
> > > >> >
> > > >> > Regards,
> > > >> > David Handermann
> > > >> >
> > > >> > On Thu, Nov 10, 2022 at 1:44 PM Joe Witt 
> > wrote:
> > > >> >
> > > >> > > Team,
> > > >> > >
> > > >> > > We keep getting a slew of emails on various lists, slack, and the
> > > >> > security
> > > >> > > list about commons text.  We need a 1.18.1 or a 1.19.0.
> > > >> > >
> > > >> > > Given all that has already happened on 1.19.0 I'm inclined to just
> > > 'do
> > > >> > > that' release and get it going.
> > > >> > >
> > > >> > > I know we need to start getting serious about 2.0 but perhaps the
> > > most
> > > >> > > realistic path for that is we branch and come back to main after
> > we
> > > >> get
> > > >> > > that figured out.  But we need to keep moving forward in the
> > > meantime.
> > > >> > >
> > > >> > > Please share if there is anything we know about on the main line
> > > which
> > > >> > > would make doing a 1.19.0 problematic.
> > > >> > >
> > > >> > > https://issues.apache.org/jira/projects/NIFI/versions/12352345
> > > >> > >
> > > >> > > Thanks
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >

Re: Adding Kotlin support to the scripting bundle

[Matt Burgess]

Mike,

As an alternative it might be worth looking into a processor
specifically for Kotlin like we have with ExecuteGroovyScript (EGS).
ExecuteGroovyScript doesn't use JSR-223 and instead goes right for the
GroovyShell. This allows for cool things like Groovy-specific idioms
and helpful "meta-capabilities" such as redirect I/O to/from FlowFile
objects. Depending on how they've done their entry point(s), an
ExecuteKotlinScript processor could leverage compiled scripts without
needing the JSR-223 impl, as well as adding some Kotlin-ness to the
scripting capabilities. I hesitate to suggest this for any of the
other scripting languages but as Kotlin becomes more popular and given
its tight integration with the JVM (vs Jython for example), that might
be the way to go.

Regards,
Matt


On Sun, Nov 6, 2022 at 6:16 PM Mike Thomsen  wrote:
>
> I finally chased down the behavior differences that I found and
> discussed with Matt in the previous PR. Kotlin's got very promising
> (long term) JSR223 support but with a few caveats in the short term:
>
> 1. Its ScriptEngine does not support ScriptEngine#get properly, so any
> component that uses get(String) to fetch an object defined in a script
> won't work with Kotlin (ex InvokeScriptedProcessor).
> 2. Invocable vs Compilable have very different behaviors WRT bindings.
> 3. Until Kotlin's JSR223 engine matures, we'd probably need to limit
> it to ExecuteScript with extra documentation explaining that it
> doesn't behave quite as expected.
>
> What I figured out, and it makes sense given how Kotlin works, is that
> bindings cannot be used to provide dynamic injection of variables. So
> just calling "session" as an implied ProcessSession won't work. You
> have to do something like this:
>
> val session = bindings["session"] as ProcessSession
>
> Either on your own or in a setup block for every script.
>
> I feel like there's value here, but wanted to throw it out there to
> other devs before going down the process of finalizing the
> ExecuteScript support and starting to exclude it from the other
> components.
>
> On a side note, we might want to add an InvokeCompiledScript processor
> that's like InvokeScriptedProcessor but is designed for Kotlin, JRuby
> and Groovy which work really well with taking scripts down to Java
> byte code and working with streamlined fat jars. Could be another
> angle on simplifying deployments of scripts and script-like
> components.
>
> Thanks,
>
> Mike

Re: Jira contributor access

[Matt Burgess]

Mathew,

I have added you as a Contributor to the NiFi projects in Jira.
Looking forward to your contributions!

Regards,
Matt

On Mon, Oct 3, 2022 at 2:25 AM Mathew Kiprop  wrote:
>
> My Jira username is '*mkapkiai*'
>
> On Mon, Oct 3, 2022 at 9:21 AM Mathew Kiprop 
> wrote:
>
> > Hello Apache NiFi team,
> >
> > I hope this email finds you well.
> > I am writing to request 'Jira contributor access'.
> >
> > Regards,
> > Mathew Kiprop
> >

Re: [VOTE] Release Apache NiFi 1.18.0 (RC4)

[Matt Burgess]

+1 binding

Ran through release helper, verified NIFI-10568 (statefulness added to
runtime component manifests) and NIFI-9042 (DatabaseParameterProvider)
and ran a couple of flows exercising various components such as
record-based processors and controller services.

Thanks for RM'ing Joe!


On Mon, Oct 3, 2022 at 4:45 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.18.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1214
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.18.0/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.18.0-RC4
> The Git commit ID is 109e54cd585902a981d1b370b3dc4d1620be438c
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=109e54cd585902a981d1b370b3dc4d1620be438c
>
> Checksums of nifi-1.18.0-source-release.zip:
> SHA256: 925cbb92c107d0fa3194a349d985cff4933a61b2555eff57b1b81433fe37c139
> SHA512: 
> f143215b1746342e7584f5ad65b546fcc378cd78aa17628fb605dfdbbaf11e897a0173dd67807fc90cb18c17124a4227d5fe07e7ed609d9ed1904503b757c604
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 184 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352150
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.18.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.18.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.18.0 (RC3)

[Matt Burgess]

+1 (binding)

Ran through release helper, tested DatabaseParameterProvider and
various flows. Thanks for RM'ing Joe!

On Thu, Sep 29, 2022 at 3:22 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.18.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1213
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.18.0/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.18.0-RC3
> The Git commit ID is 5bc64c812b2c76ee2879d8081ceadf62d5e3c702
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=5bc64c812b2c76ee2879d8081ceadf62d5e3c702
>
> Checksums of nifi-1.18.0-source-release.zip:
> SHA256: bd1b675f17dbf712089a79f7bc043eae2df63bcc2e08b2012a6431641037679f
> SHA512: 
> cea43af57089128ff65bb53e76b2fdfa8dec7397e2bf45d41e35b758b731355075839b9c018ee6284cb15e293b105e248d88748148960ad80ae387824139f52b
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 171 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352150
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.18.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.18.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: New NiFi PMC Member Nathan Gough

[Matt Burgess]

Congratulations Nathan! Well deserved.

On Wed, Sep 7, 2022 at 12:10 PM David Handermann
 wrote:
>
> Apache NiFi Community,
>
> On behalf of the Apache NiFi Project Management Committee, I am very
> pleased to announce that Nathan Gough has accepted the invitation to join
> the PMC.
>
> Nathan has been a consistent contributor to the project since 2018. In
> addition to developing features and fixes, he regularly reviews pull
> requests and assists others through mailing lists and Slack conversations.
> He also plays an active role in reviewing and updating security reports for
> the project.
>
> Please join us in congratulating and welcoming Nathan to the Apache NiFi
> PMC. Congratulations Nathan!

Re: [VOTE] Release Apache NiFi 1.17.0 (RC2)

[Matt Burgess]

+1 (binding)

Ran through release helper and tested various flows. Thanks for RM'ing Joe!

On Thu, Jul 28, 2022 at 10:29 AM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.17.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1210
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.17.0/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.17.0-RC2
> The Git commit ID is 8d256784d84cc28bf5642e1bf38dec3eba0c5f23
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=8d256784d84cc28bf5642e1bf38dec3eba0c5f23
>
> Checksums of nifi-1.17.0-source-release.zip:
> SHA256: 8b9b2088ad966329248cfae7792f576f4f30fea4b4e50f055f84724dba4fe8a3
> SHA512:
> 2429348ad514ca7ab9df86aaa57207f1434044c6f7e947d0950ca9826b4f1aa51061617a17444c086eed19b1f26a5ebbe3b455cafed9d219727adf26ecb5f8d2
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 310 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351438
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.17.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-1.17.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.5

[Matt Burgess]

+1 (binding)

Built on OSX 10.15.7 with OpenJDK Runtime Environment Zulu11.54+23-CA
(build 11.0.14+9-LTS), ran NiFi build and verified manifest changes.
Thanks for RM'ing Bryan!

On Wed, Jul 20, 2022 at 4:37 PM Bryan Bende  wrote:
>
> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi NAR Maven Plugin 1.3.5.
>
> The source being voted upon, including signatures, digests, etc. can
> be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.5/
>
> The Git tag is nifi-nar-maven-plugin-1.3.5-RC1
> The Git commit ID is 77f516a65d796b715278bedd06407d4a8f98f771
> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=77f516a65d796b715278bedd06407d4a8f98f771
>
> Checksums of nifi-nar-maven-plugin-1.3.5-source-release.zip:
> SHA256: 073271bbd28b3d0b9b74c42ab889b4e7301e52b9a0579a3484e7ad0b7f8d3532
> SHA512: 
> 9fcc487a197a8d3c5cd297696eb303015c9cde32f89541682dfac921e862c1f1ee8ba5e41573eeb7a5b5190b95c3b49443fca9ed5d5a3f7768a4ea1b031829af
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/bbende.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 2 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351862
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.5
>
> The vote will be open for 24 hours. Please download the release
> candidate and evaluate the necessary items including checking hashes,
> signatures, build from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-nar-maven-plugin-1.3.5
> [ ] +0 no opinion
> [ ] -1 Do not release this package because ...

Re: Jira contributor access

[Matt Burgess]

Kirill,

I have added you as a contributor to the Apache NiFi Jira projects.
Looking forward to your contributions!

Regards,
Matt

On Mon, Jul 11, 2022 at 2:11 AM Kirill Morozov
 wrote:
>
> Hi,
> could anyone help me with this ?
>
> Thank you!
>
> чт, 7 июл. 2022 г. в 16:05, Kirill Morozov :
>
> > Hello,
> > please give me Jira contributor access
> >
> > My jira username: kmor
> >
> > Thank you!
> >

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.4

[Matt Burgess]

+1 (binding) Checked commit and SHAs, built with Java 1.8 on Mac,
built NiFi from main with the RC Maven plugin. Thanks for RM'ing
Kevin!

On Thu, Jun 16, 2022 at 5:11 PM Kevin Doran  wrote:
>
> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi NAR Maven Plugin 1.3.4.
>
> The source being voted upon, including signatures, digests, etc. can
> be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.4/
>
> The Git tag is nifi-nar-maven-plugin-1.3.4-RC1
> The Git commit ID is c4d9563bff2b5c2120e7f1e181343e2c01cc0422
> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=c4d9563bff2b5c2120e7f1e181343e2c01cc0422
>
> Checksums of nifi-nar-maven-plugin-1.3.4-source-release.zip:
> SHA256: acee55a767b9c90c8884e8c6e5fe936088243f778dc7e1de6f281cd4c8a85cab
> SHA512: 
> 267cc157117d179517257f47c8d92f3dc5e594b230bb019ccc6e2cb2884b450af56603214fa651bc33aceb3bf0bdb9369aa779672cb3f6f0efd2bacb166f9051
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/kdoran.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 3 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350909
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.4
>
> The vote will be open for 72 hours. Please download the release
> candidate and evaluate the necessary items including checking hashes,
> signatures, build from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-nar-maven-plugin-1.3.4
> [ ] +0 no opinion
> [ ] -1 Do not release this package because ...

Re: Redshift driver class is not working dbcpconnectionpool for 1.16.1 version

[Matt Burgess]

Sanjeet,

I'm not really sure why that configuration worked before with no
Driver Location property set, did you put the driver in NiFi's lib/
directory? In general we discourage that and instead recommend you put
the driver JAR(s) somewhere else and use the Driver Location property
to point at it.

Regards,
Matt

On Wed, Jun 8, 2022 at 9:04 AM sanjeet rath  wrote:
>
> Hi ,
>
> Any updates on  the trailed  mail is really helpful.
>
> Thanks & Regards
> Sanjeet
>
> On Tue, 7 Jun 2022 at 10:18 PM, sanjeet rath  wrote:
>
> > Hi ,
> >
> > I am seeing one different behaviour of dbcpconnectionpool service in
> > 1.16.1 version of nifi.
> >
> > I am passing the Database driver class name  : Jdbc.redshift.RedshiftDriver
> > And DB connection URL. I am passing the required name & pwd and other
> > things.
> > Driver path i am not passing anything.
> >
> > With these values I am seeing no error in 1.12.1  version of nofi ,
> > however in 1.16.1 i am seeing “Driver class jdbc.redshift.redshift.redshift
> > not found.” Screenshot is attached for your reference.
> >
> > Could you please let me know this behaviour is due to some change in
> > 1.16.1 as the same controller with the same config 1.12.1 no issue .
> >
> > Also please let me know how to solve this issue.
> >
> > Thanks & Regards
> > Sanjeet
> >
> >
> >
> > --
> Sanjeet Kumar Rath,
> mob- +91 8777577470

Re: [VOTE] Release Apache NiFi 1.16.2 (RC3)

[Matt Burgess]

+1 (binding)

Ran through release helper, connected with a secure RPG from MiNiFi to
NiFi (part of verifying NIFI-9967), started with an existing
flow.xml.gz (no json at first) in secure NiFi, uploaded a flow
definition and exercised various controller services and processors,
and saved the flow to a bucket in an unsecure Registry.

Thanks for RM'ing Joe!

On Sun, May 22, 2022 at 11:44 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.16.2.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1203
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.2/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.16.2-RC3
> The Git commit ID is 06f04958272dafc30ce357c4c4edcaf470050b52
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=06f04958272dafc30ce357c4c4edcaf470050b52
>
> Checksums of nifi-1.16.2-source-release.zip:
> SHA256: 1fecf7d9f6001cc8e58d4a46ece08e141de705bcd227338ba79e9cb574267415
> SHA512:
> 1f4fd4e5e9f24949830a75949b302a67b8826049406ab8296c4b8c99a5a0aa1d211f84f98699b3af6fb41efa305f35a3f85b21d7958dc09c027cc1ed836c169f
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 34 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351721
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.2
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-1.16.2
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: Requesting PYTHON documentation for the Python side of a NiFi ExecuteScript processor execution

[Matt Burgess]

Paul,

The first thing I would mention (which helps explain the rest) is that
NiFi uses the Jython [1] engine, it is not pure Python. So although
you cannot use native (CPython) libraries like Pandas, you can import
and use Java classes, which is what the NiFi API is written in. So
when you say "Python functions available for the ExecuteScript API",
what that really means is that your Jython code can call Java methods
on Java objects, such as outputstream.write() or session.transfer().

A good place to start to understand the NiFi API is the Developer
Guide [2]. This covers the concepts you may run into in your script,
in fact way more than you may need. To narrow it down, start with the
ProcessSession API [3]. This shows you the methods you can call on the
session, such as writing to a flow file using a StreamCallback [4],
and within that you can see the process method [5].  For the
ProcessSession, you can also see the transfer() and putAttribute()
methods which are very often used in scripts. There are other places
where you might interact with the API, such as FlowFile [6].

I have a few examples on my blog [7] of using Jython in ExecuteScript,
such as converting JSON into a different JSON structure [8]. I tried
to explain the usage of the NiFi API there though I admit it could be
more in-depth. In addition I plan to write a good amount of
documentation (to include language-specific examples) under NIFI-1954
[9]. I am the author of the ExecuteScript Cookbook series you
mentioned as well. Please feel free to respond with any additional
questions you may have, happy to help.

Regards,
Matt

[1] https://www.jython.org/
[2] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
[3] 
https://javadoc.io/static/org.apache.nifi/nifi-api/1.16.1/org/apache/nifi/processor/ProcessSession.html
[4] 
https://javadoc.io/static/org.apache.nifi/nifi-api/1.16.1/org/apache/nifi/processor/ProcessSession.html#write-org.apache.nifi.flowfile.FlowFile-org.apache.nifi.processor.io.StreamCallback-
[5] 
https://javadoc.io/static/org.apache.nifi/nifi-api/1.16.1/org/apache/nifi/processor/io/StreamCallback.html
[6] 
https://javadoc.io/static/org.apache.nifi/nifi-api/1.16.1/org/apache/nifi/flowfile/FlowFile.html
[7] https://funnifi.blogspot.com/
[8] 
https://funnifi.blogspot.com/2016/03/executescript-json-to-json-revisited_14.html
[9] https://issues.apache.org/jira/browse/NIFI-1954

On Wed, May 4, 2022 at 1:07 PM Markiewicz, Paul  wrote:
>
> Hello, my name is PaulM.
>
> I am very new at both NiFi and relatively new at Python.
>
>
> I have successfully cobbled together a NiFi ExecuteScript processor
> (complete with embeded Python Code).
> That python code CAN/DOES read the original Flow-File content, sets/updates
> a NiFi attribute of my choosing,
> repopulates the Flow-File content with some junk (io.environ) data (and some
> other Hello-World lines..).
> Then successfully, returns (from the Python execution), the new attribute
> and new Flow-File content to
> the NiFi ExecuteScript.
>
>
> I had this success by reading a PART-1/PART-2 example (of Python) found
> within a cloudera link..
> (((
> https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-
> part-1/ta-p/248922 ))).
>
>
> What I am hoping to get from this developer mailing list eMail request (((
> dev@nifi.apache.org )))
> is where/how to locate DOCUMENTATION for what is available within the
> **PYTHON SIDE** of these library calls.
>
>
> For example, my Python uses these IMPORT STATEMENTS:
>
>
> import os
> import datetime
>
> from org.python.core.util.FileUtil import wrap
> from org.apache.nifi.processors.script import ExecuteScript
> from org.apache.commons.io import IOUtils
> from java.nio.charset import StandardCharsets
> from org.apache.nifi.processor.io import StreamCallback
>
>
> ...and from the example code (which does work...)  I am especially
> interested in
> the PYTHON documentation for "ExecuteScript" so that I can also know things
> like the complement
> FAILURE variable name, to this SUCCESS:
>
> session.transfer(flow_file, ExecuteScript.REL_SUCCESS)
>
>
> ... and learn all the other Python functions available for the ExecuteScript
> API.
>
>
> Hopefully that documentation would ALSO point out important concepts like
> inside my "class PyStreamCallback(StreamCallback):" the execution flow will
> automagically call "process"
> within that class... and all the various (correct) data types that are
> expected within the Python execution.
>
> ... and that the
> "outputStream.write(bytearray(newFlowContentStr.encode('utf-8')))"
> expects a parameter of type bytearray, to the write() function.
>
>
>
> I thank you in advance.
>
> Sincerely,
> PaulM
>
>

Re: [VOTE] Release Apache NiFi 1.16.1

[Matt Burgess]

+1 (binding)

Ran through release helper, tested various flows including version
control with the Registry, InvokeScriptedProcessor, Relationship
retry, etc.  I did create [1] for MiNiFi but since there's a
workaround it's not a dealbreaker IMO, I'm already working on it for
1.17.0.

Thanks for RM'ing Joe!

[1] https://issues.apache.org/jira/browse/NIFI-9967

On Mon, Apr 25, 2022 at 2:29 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.16.1.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1200
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.1/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.16.1-RC1
> The Git commit ID is 81166797e552b9d14b482807632f2f04321b2018
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=81166797e552b9d14b482807632f2f04321b2018
>
> Checksums of nifi-1.16.1-source-release.zip:
> SHA256: 2c39b45ba8eec42d601d5c9facad623d14660dd209c011b4a13b2b559b9f3e32
> SHA512:
> cd670ab558937cac709dea0b4be3351559f57c9e9aedf54d5153706eee386a021262ef199e2bf9485763cf955931edfd6a24ca1c5a0748a77e3eeb91c490cbda
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 83 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351504
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.1
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-1.16.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: Need some feedback on how upgrading Avro might cause problems

[Matt Burgess]

Agreed, I thought I saw that in there but we should change that if it
isn't in the PR, I'll update.

On Thu, Apr 7, 2022 at 2:07 PM Mike Thomsen  wrote:
>
> In addition to that point about schema inference, the correct thing to
> do with inference would be to do the type,null ordering and not
> null,type.
>
> On Thu, Apr 7, 2022 at 12:51 PM Matt Burgess  wrote:
> >
> > I agree that moving forward with the upgrade is the right thing to do,
> > the example Mike pointed out is invalid according to the 1.8.1 spec
> > [1] even though it was mistakenly allowed by Avro itself. Since we
> > don't infer default values I think we can correctly require that
> > hand-written schemas conform to the spec, especially with respect to
> > default values, and our inferred schemas will adhere to the spec.
> >
> > I also agree we should mention this in the Migration Guidance doc as
> > it is a requirement that is now being enforced when it wasn't before.
> >
> > There might be an issue with Hive 1 but I don't think we're on the
> > hook for that, if Hive has issues with nullable types and/or default
> > values of null, it should obey the spec the same as we will.
> >
> > Regards,
> > Matt
> >
> > [1] https://avro.apache.org/docs/1.8.1/spec.html#Unions
> >
> > On Thu, Apr 7, 2022 at 9:20 AM David Handermann
> >  wrote:
> > >
> > > Mike,
> > >
> > > Thanks for raising this issue for discussion, and thanks Isha and Edward
> > > for the comments. It is helpful to get an understanding of the potential
> > > impact.
> > >
> > > In general, it seems like moving forward with the upgrade is the best 
> > > idea,
> > > given the issues with older versions. For integration with other products
> > > outside of NiFi, using the latest version of Avro should provide better
> > > compatibility.
> > >
> > > That being said, it is important to consider the impact on existing
> > > deployments. At minimum, this change is something that should be described
> > > in the NiFi Migration Guidance associated with the released version. As 
> > > far
> > > as supporting different versions, it is possible to run older versions of
> > > the various components, although this is more complicated given the scope
> > > of the upgrade. This seems like a case where providing more detailed
> > > migration steps would be helpful, to include issues related to cached
> > > schemas or inferred schemas used as the basis for customization.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Thu, Apr 7, 2022 at 6:08 AM Edward Armes  
> > > wrote:
> > >
> > > > I've had a quick look in JIRA and it looks like this might have 
> > > > happened as
> > > > a side effect of AVRO-1544.
> > > >
> > > > I think it is worth upgrading especially given that it looks like few of
> > > > the changes refer to updating against newer bundled dependencies some of
> > > > which seem to  have CVE's against them.
> > > >
> > > > Depending on peoples feelings would it be wroth creating 2 versions one
> > > > using Avro 1.8 and one using Avro 1.11.0 and then removing the 1.8 
> > > > version
> > > > in a later release?
> > > >
> > > > On an additional note in cases where people are writing schemas 
> > > > manually I
> > > > suspect they are probably going to be validating against the later 
> > > > versions
> > > > of Avro using the projects tooling and that may create issues further 
> > > > down
> > > > the line.
> > > >
> > > > Edward
> > > >
> > > > On Thu, Apr 7, 2022 at 11:52 AM Isha Lamboo <
> > > > isha.lam...@virtualsciences.nl>
> > > > wrote:
> > > >
> > > > > Hi Mike,
> > > > >
> > > > > The "Infer schema" functionality in NiFi currently generates schemas 
> > > > > with
> > > > > the order that will be invalid under Avro 1.9+. I noticed because I've
> > > > been
> > > > > using that to copy-paste schemas that were "almost right" so I could
> > > > > manually fix them.
> > > > >
> > > > > I guess that inferred schemas should be fine if the inferring logic is
> > > > > also upgraded by the dependency, but for any cached schemas and my own
> > > > > 

Re: Need some feedback on how upgrading Avro might cause problems

[Matt Burgess]

I agree that moving forward with the upgrade is the right thing to do,
the example Mike pointed out is invalid according to the 1.8.1 spec
[1] even though it was mistakenly allowed by Avro itself. Since we
don't infer default values I think we can correctly require that
hand-written schemas conform to the spec, especially with respect to
default values, and our inferred schemas will adhere to the spec.

I also agree we should mention this in the Migration Guidance doc as
it is a requirement that is now being enforced when it wasn't before.

There might be an issue with Hive 1 but I don't think we're on the
hook for that, if Hive has issues with nullable types and/or default
values of null, it should obey the spec the same as we will.

Regards,
Matt

[1] https://avro.apache.org/docs/1.8.1/spec.html#Unions

On Thu, Apr 7, 2022 at 9:20 AM David Handermann
 wrote:
>
> Mike,
>
> Thanks for raising this issue for discussion, and thanks Isha and Edward
> for the comments. It is helpful to get an understanding of the potential
> impact.
>
> In general, it seems like moving forward with the upgrade is the best idea,
> given the issues with older versions. For integration with other products
> outside of NiFi, using the latest version of Avro should provide better
> compatibility.
>
> That being said, it is important to consider the impact on existing
> deployments. At minimum, this change is something that should be described
> in the NiFi Migration Guidance associated with the released version. As far
> as supporting different versions, it is possible to run older versions of
> the various components, although this is more complicated given the scope
> of the upgrade. This seems like a case where providing more detailed
> migration steps would be helpful, to include issues related to cached
> schemas or inferred schemas used as the basis for customization.
>
> Regards,
> David Handermann
>
> On Thu, Apr 7, 2022 at 6:08 AM Edward Armes  wrote:
>
> > I've had a quick look in JIRA and it looks like this might have happened as
> > a side effect of AVRO-1544.
> >
> > I think it is worth upgrading especially given that it looks like few of
> > the changes refer to updating against newer bundled dependencies some of
> > which seem to  have CVE's against them.
> >
> > Depending on peoples feelings would it be wroth creating 2 versions one
> > using Avro 1.8 and one using Avro 1.11.0 and then removing the 1.8 version
> > in a later release?
> >
> > On an additional note in cases where people are writing schemas manually I
> > suspect they are probably going to be validating against the later versions
> > of Avro using the projects tooling and that may create issues further down
> > the line.
> >
> > Edward
> >
> > On Thu, Apr 7, 2022 at 11:52 AM Isha Lamboo <
> > isha.lam...@virtualsciences.nl>
> > wrote:
> >
> > > Hi Mike,
> > >
> > > The "Infer schema" functionality in NiFi currently generates schemas with
> > > the order that will be invalid under Avro 1.9+. I noticed because I've
> > been
> > > using that to copy-paste schemas that were "almost right" so I could
> > > manually fix them.
> > >
> > > I guess that inferred schemas should be fine if the inferring logic is
> > > also upgraded by the dependency, but for any cached schemas and my own
> > > manually saved schemas this will be somewhat painful.
> > > My use case for manually saving inferred schemas is mostly database
> > > migration where some inferred "choice" fields are not supported for the
> > > target database.
> > >
> > > Hope this helps,
> > >
> > > Isha
> > >
> > > -Oorspronkelijk bericht-
> > > Van: Mike Thomsen 
> > > Verzonden: donderdag 7 april 2022 12:11
> > > Aan: dev@nifi.apache.org
> > > Onderwerp: Need some feedback on how upgrading Avro might cause problems
> > >
> > > Thread is here for full details:
> > >
> > >
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F5900%23pullrequestreview-922490039data=04%7C01%7Cisha.lamboo%40virtualsciences.nl%7C8c89ae3e621c4a255c3308da187eea09%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C637849231546931433%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=ztcxOWXBkpZFqEh%2Bu%2BG0du5BLUPyZ3WaMxqpeqn%2FexI%3Dreserved=0
> > >
> > > It looks like Avro 1.8's schema parser may have been more lenient (or
> > > buggy) in enforcing the specification with respect to the ordering of a
> > > union for a nullable type. 1.9.X and higher are definitely more
> > opinionated
> > > and throw exceptions on schemas that used to work on 1.8.X. For example,
> > > this used to be valid:
> > >
> > > {
> > >   "name": "message",
> > >   "type": [ "null", "string" ],
> > >   "default": "Hello, world"
> > > }
> > >
> > > Now Avro **correctly** throws an exception per the specification.
> > > Under 1.8 it did not, and as you can see here, I had to change numerous
> > > test schemas in order to make them work under 1.9 to 

Re: Jira contributor access

[Matt Burgess]

Marc,

I have added you as a Contributor to NiFi Jira projects. Welcome, and
looking forward to your contributions!

Regards,
Matt

On Sun, Mar 20, 2022 at 8:21 AM Marc  wrote:
>
> Hello,
>
> I would like to request Jira contributor access.
> My Username is nerdfunk
>
> Many thanks
> Marc

Re: [VOTE] Release Apache NiFi 1.16.0 (rc3)

[Matt Burgess]

+1 Release this package as nifi-1.16.0

Ran through release helper, ran a couple flows through NiFi including
one for PutDatabaseRecord (to validate the changes), verified MiNiFi
started and autogenerated the sensitive propert(ies), verified H2
upgrade on NiFi Registry.

Thanks for RM'ing Joe!

On Mon, Mar 21, 2022 at 5:42 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.16.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1198
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.0/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.16.0-RC3
> The Git commit ID is b019a9191f1c83bc7f547dc02c1b679b8936acee
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=b019a9191f1c83bc7f547dc02c1b679b8936acee
>
> Checksums of nifi-1.16.0-source-release.zip:
> SHA256: 2f16cb94df404d1bcc9c32835ba98da8940005a5d7ea5504c155ee42021a221e
> SHA512:
> cbd95f15cec5ffe506fef204526267b18b77d7266f6dc3e1bbc3c7600aac12e119977f7d8cf93dbbbc86fbb0739ba88aaa11a5381d29a463ec9a0c9a18f4e9e6
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 401 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350741
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-1.16.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [discuss] pulling together a NiFi 1.16

[Matt Burgess]

I'm all for bringing into the next RC, if something goes sideways
during the RC validation we can always take it back out.

-Matt

On Wed, Mar 16, 2022 at 5:39 PM Joe Witt  wrote:
>
> Team
>
> We will have RC3 now because of a regression.  But we also now have an
> interesting opportunity.
>
> With https://github.com/apache/nifi/pull/5870/files we could both build and
> run on Java 17.  The changes are largely about tests and such.  I'm
> inclined to pull this in during the rc process.  I encourage others to
> chime in/review/etc.. in case there are some strong reasons not to pull it
> in.
>
> Adding 'Java 17 now supported' would be quite a nice thing...
>
> Thanks
>
> On Thu, Mar 10, 2022 at 9:10 AM Joe Witt  wrote:
>
> > Team,
> >
> > Mike's PR got merged it looks like.
> >
> > I'm initiating 1.16 RC.  For all commits going forward please tag as fix
> > version 1.17
> > https://issues.apache.org/jira/projects/NIFI/versions/12351438
> >
> > If we have failed RC's i'll pull things into 1.16 as needed.
> >
> > Thanks
> >
> > On Wed, Mar 9, 2022 at 9:25 AM Joe Witt  wrote:
> >
> >> Mike
> >>
> >> I left a comment on the PR.  But as usual with these releases there are
> >> always things that are close/nearly there/just need a review/etc..  If that
> >> or anything else lands by the time the RC is generated then we're good.
> >>
> >> Thanks
> >>
> >> On Wed, Mar 9, 2022 at 9:21 AM Mike Thomsen 
> >> wrote:
> >>
> >>> Joe,
> >>>
> >>> I would like to see this review closed out before a 1.16 RC if
> >>> possible: https://github.com/apache/nifi/pull/4646 I think it's mainly
> >>> waiting on someone to verify that all of the changes have been made.
> >>>
> >>> Thanks,
> >>>
> >>> Mike
> >>>
> >>>
> >>> On Wed, Mar 9, 2022 at 10:54 AM Joe Witt  wrote:
> >>> >
> >>> > Mark
> >>> >
> >>> > The single user authorizer and default setup install is just to avoid
> >>> > having wide open systems by default.  So if you want to make changes to
> >>> > security settings and do it right you dont' use that mode.  Happy to
> >>> have
> >>> > improvements within that scope of intent but does not sound like
> >>> anything
> >>> > we'd wait for.  When it lands it lands.
> >>> >
> >>> > Thanks
> >>> >
> >>> > On Wed, Mar 9, 2022 at 8:49 AM Mark Bean 
> >>> wrote:
> >>> >
> >>> > > Joe,
> >>> > >
> >>> > > I just discovered an issue yesterday that might need attention
> >>> first. I
> >>> > > haven't investigated fully yet nor created a ticket because I don't
> >>> yet
> >>> > > fully understand it. However, it appears as though the
> >>> > > single-user-authorizer may not be behaving as intended. When I
> >>> updated
> >>> > > nifi.properties to swap the self-signed, auto-generated keystore and
> >>> > > truststore with "real" ones, single-user became _every_ user. My
> >>> suspicion
> >>> > > is that any user whose browser presents a cert that was signed by a
> >>> CA in
> >>> > > the truststore is allowed in - without even prompting for
> >>> > > username/password.
> >>> > >
> >>> > > It may be considered a configuration error to allow this to happen.
> >>> Still,
> >>> > > this seems like extremely dangerous behavior.
> >>> > >
> >>> > > -Mark
> >>> > >
> >>> > >
> >>> > > On Wed, Mar 9, 2022 at 10:42 AM Joe Witt  wrote:
> >>> > >
> >>> > > > Team
> >>> > > >
> >>> > > > We appear to be at a good point to start pulling together the
> >>> release
> >>> > > > candidate for 1.16.
> >>> > > >
> >>> > > > https://issues.apache.org/jira/projects/NIFI/versions/12350741
> >>> > > >
> >>> > > > I'm basically waiting for
> >>> > > https://issues.apache.org/jira/browse/NIFI-9761
> >>> > > > to land then will start pulling together the release.
> >>> > > >
> >>> > > > Thanks
> >>> > > >
> >>> > > > On Mon, Feb 14, 2022 at 11:18 AM Joe Witt 
> >>> wrote:
> >>> > > >
> >>> > > > > Eduardo
> >>> > > > >
> >>> > > > > Getting reviewers on the UI/rest/front-end are among the
> >>> toughest as
> >>> > > > > there just aren't as many of those folks.
> >>> > > > >
> >>> > > > > The reply from Pierre was probably most telling. It looks fine
> >>> but
> >>> > > > > many of us would pause to merge without knowing precisely what
> >>> the
> >>> > > > > implications are.  What happens on a taxed system with many
> >>> > > > > CSs...I''ll comment on the PR.
> >>> > > > >
> >>> > > > > Thanks
> >>> > > > > Joe
> >>> > > > >
> >>> > > > > On Mon, Feb 14, 2022 at 11:13 AM Eduardo Fontes
> >>> > > > >  wrote:
> >>> > > > > >
> >>> > > > > > Hi All,
> >>> > > > > >
> >>> > > > > > Is it possible to include
> >>> > > > > https://issues.apache.org/jira/browse/NIFI-8927
> >>> > > > > > in release 1.16?
> >>> > > > > > I've been asking for a review
> >>> > > https://github.com/apache/nifi/pull/5247
> >>> > > > > > since AUG/2021 and I don't understand why nobody did it. It's a
> >>> > > simple
> >>> > > > > and
> >>> > > > > > useful UI feature.
> >>> > > > > >
> >>> > > > > > Peace out.
> >>> > > > > > Eduardo Fontes
> >>> > > > >
> >>> > > >
> >>> > >
> >>>
> 

Re: Trying to add Kotlin support for scripting and hitting what looks like a weird classloader issue

[Matt Burgess]

Mike,

I haven't had a chance to repro the problem but when I've seen this
with scripting engines it's usually because they do some stuff under
the hood with the system classloader and not the thread's context
classloader...

What classpath issue are you seeing?

Regards,
Matt

On Wed, Mar 2, 2022 at 2:35 PM Mike Thomsen  wrote:
>
> Here's the branch:
> https://github.com/MikeThomsen/nifi/commit/588da17a42f9ad34192be31bd5252cad518147fd
>
> Here's a gist that works just fine when you run it locally:
> https://gist.github.com/MikeThomsen/90d81c584bfd21d43cfe143963f716a9
>
> It appears to be something specific to how we handle classpaths. Any thoughts?
>
> Thanks,
>
> Mike

Re: Nifi 1.14.0 not launching after copying nifi-splunk-nar-1.15.3.nar

[Matt Burgess]

Deepak,

As a general rule you can't mix and match versions of a NAR due to possible
changes in the way they're loaded, packaged, etc. If you're using NiFi
1.14.0 I recommend using the 1.14.0 version of that NAR [1]. If one "just
works" in a different version that's more about serendipity than support :)

Regards,
Matt

[1]
https://repository.apache.org/content/repositories/releases/org/apache/nifi/nifi-splunk-nar/1.14.0/nifi-splunk-nar-1.14.0.nar

On Mon, Feb 28, 2022 at 12:26 PM Chirthani, Deepak Reddy <
c-deepakreddy.chirth...@charter.com> wrote:

> Hello,
>
> We are currently using GetSplunk processor to pull data from Splunk
> Enterprise on a scheduled basis in our Nifi – 1.12.1 instance. We have
> observed that new properties added to the GetSplunk processor in the latest
> version of Nifi 1.15.3 and we would like to bring that processor to our
> current instance of Nifi to meet our requirements.
>
> For accomplishing this, we brought the nifi-splunk-nar-1.15.3.nar file and
> tested that by uploaded it to the lib folder of a separate
> Nifi-1.14.0(local instance) and restarted it. Nifi – 1.14.0 was unable to
> launch however it was able to launch if I removed that particular nar file.
>
> Any suggestions why this is happening? In the past we were able to bring
> processors from lower versions of Nifi successfully but not higher versions.
>
>
>
> Thanks
>
>
>
> *[image: image005]*
>
> *Deepak Reddy* | Data Engineer
> ​IT Centers of Excellence
> 13736 Riverport Dr., Maryland Heights, MO 63043
>
>
> The contents of this e-mail message and
> any attachments are intended solely for the
> addressee(s) and may contain confidential
> and/or legally privileged information. If you
> are not the intended recipient of this message
> or if this message has been addressed to you
> in error, please immediately alert the sender
> by reply e-mail and then delete this message
> and any attachments. If you are not the
> intended recipient, you are notified that
> any use, dissemination, distribution, copying,
> or storage of this message or any attachment
> is strictly prohibited.
>

Re: Questions about how to use multiple Controller Services in a Processor

[Matt Burgess]

ZhangXinchen,

Rather than having one NAR for each API, you'd want a single API NAR
that includes both API JARs. We do this for
nifi-standard-services-api-nar as an example.

Regards,
Matt

On Fri, Feb 25, 2022 at 8:55 PM 413155245 <413155...@qq.com.invalid> wrote:
>
> Hi all：
>
>
> When I'm developing my customized Processor A, which needs to integrate 
> customized Controller Service B and customized Controller Service C. These 
> projects are all generated by NiFi maven archetype.
>
>
> I add service-api of Controller Service B and service-api of Controller 
> Service C in POM of Processor A project.
>
>
> However, only one NAR dependency can be relied on, because each NAR 
> represents a separated ClassLoader.
>
>
> So, how to integrate multiple Controller Services in one Processor, and how 
> to organize NAR dependency in POM of the Processor?
>
>
> Thanks!
>
>
> ZhangXinchen

Re: [DISCUSS] release nifi-fds 0.3.0

[Matt Burgess]

+1 Thanks Scott!

On Mon, Jan 24, 2022 at 11:26 AM Scott Aslan  wrote:
>
> All,
>
> I would like to kick out a release of nifi-fds. There are multiple npm
> audit issues and the version of Angular is old. Puls, the nifi-fds v0.2.0
> requires node v8.10.0 and npm v5.6.0 which are very old and already past
> their respective end of life. These older versions of node and npm are also
> blocking [1].
>
> I can take care of putting together the release.
>
> Thanks,
>
> Scott
>
> [1] https://issues.apache.org/jira/browse/NIFI-9554

Re: [VOTE] Release Apache NiFi 1.15.3 (rc1)

[Matt Burgess]

+! (binding), ran thru the release helper and tried some scripting
flows. Thanks for RM'ing Joe!

On Thu, Jan 13, 2022 at 3:44 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.15.3.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1194
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.3/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.15.3-RC1
> The Git commit ID is 753c311382005acadc16c64952d7b1eaaf2550d5
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=753c311382005acadc16c64952d7b1eaaf2550d5
>
> Checksums of nifi-1.15.3-source-release.zip:
> SHA256: 29fcc35c81de80e0fe3f59044e6fbf21bcf523e656aa64914e7546e1d7705e6b
> SHA512: 
> cabd1f1ad4942a61df0400488d35521202598c217ad8da97dc2d5abe21136604d1f1bb3de9ceb63bb441943de2e29e3515f5cf63607080094e1418d79eb5216b
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 11 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351203
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.3
>
> The vote will be open
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [ ] +1 Release this package as nifi-1.15.3
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.15.2

[Matt Burgess]

+1 (binding) ran through release helper and a couple of flows. Thanks
for RM'ing Joe!


On Tue, Dec 21, 2021 at 5:51 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.15.2.
>
> This vote is purely bug fix and security focused. This is a
> continuation of our efforts to promptly and thoroughly respond to
> log4shell and related concerns.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1193
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.2/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.15.2-RC1
> The Git commit ID is 1ea460b8556b07057366abb74a5552ace8946e87
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=1ea460b8556b07057366abb74a5552ace8946e87
>
> Checksums of nifi-1.15.2-source-release.zip:
> SHA256: 29fcc35c81de80e0fe3f59044e6fbf21bcf523e656aa64914e7546e1d7705e6b
> SHA512: 
> cabd1f1ad4942a61df0400488d35521202598c217ad8da97dc2d5abe21136604d1f1bb3de9ceb63bb441943de2e29e3515f5cf63607080094e1418d79eb5216b
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 8 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351132
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.2
>
> Given the nature of the vote and its limited scope
> the vote will be open for 24 hours or until we have sufficient
> votes (instead of the normal 72 hours).
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [ ] +1 Release this package as nifi-1.15.2
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [ANNOUNCE] New Apache NiFi Committer Margot Tien

[Matt Burgess]

Congratulations Margot!!

On Wed, Dec 15, 2021 at 1:46 PM Matt Gilman  wrote:
>
> Apache NiFi community,
>
> On behalf of the Apache NiFi PMC, I am very pleased to announce that Margot
> has accepted the PMC's invitation to become a committer on the Apache NiFi
> project. We greatly appreciate all of Margot's hard work and generous
> contributions to the project. We look forward to continued involvement in
> the project.
>
> Margot has been contributing to NiFi and NiFi Registry for years. Her
> contributions have covered both back-end and front-end improvements in both
> projects in addition to release verification and thoughtful PR reviews.
>
> Welcome and congratulations!

Re: [VOTE] Release Apache NiFi 1.15.1 (rc1)

[Matt Burgess]

+1 (binding)

Ran through release helper, tested various flows and components
including LogAttribute, InvokeScriptedProcessor, etc. Everything looks
good. Thanks for RM'ing and the quick turnaround Joe!

On Tue, Dec 14, 2021 at 10:35 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.15.1.
>
> This vote, unlike most, is purely stability and security focused.
> This vote is rooted
> in a prompt response to the 'log4shell' vulnerability and related
> logging announcements.
> It also includes other easy to incorporate bugs and improvements.  It
> should be easy to
> upgrade from any 1.15 install to this and just as easy as it was to go
> from pre 1.15 to
> this 1.15.1.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1192
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.1/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.15.1-RC1
> The Git commit ID is 2a756372fc7097ece6258c2af47b9a5f26384b02
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=2a756372fc7097ece6258c2af47b9a5f26384b02
>
> Checksums of nifi-1.15.1-source-release.zip:
> SHA256: 83d06011f0d2608d2d9cf951deae04d7b0921f2a7c8b1052ca9d058cf46b7d52
> SHA512: 
> 009161e81e207a16060d9efd37e9b9abd1c1d5b5d57024a2b4c0d0ea17050f65b3a025632718161cba41948fe51d93aed65a4daba2542fce4da51d0184872039
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 45 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351055
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.1
>
> Given the nature of the vote being about a prompt release to remove vulnerable
> logging related libraries the vote will be open for 24 hours (instead
> of the normal 72 hours).
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [ ] +1 Release this package as nifi-1.15.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.3

[Matt Burgess]

+1 (binding), ran through release helper and verified NARs were loaded
successfully into a NiFi instance.  Thanks for RM'ing Bryan!

On Tue, Nov 30, 2021 at 3:56 PM Bryan Bende  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi NAR Maven Plugin 1.3.3.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1188
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.3/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-nar-maven-plugin-1.3.3-RC1
> The Git commit ID is 99f1ff7184e00cecc4763b7bcbdf0d12dfeffef2
> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=99f1ff7184e00cecc4763b7bcbdf0d12dfeffef2
>
> Checksums of nifi-nar-maven-plugin-1.3.3-source-release.zip:
> SHA256: 36d6579dfdbc4e1450a13457196ff399dbd344624348a62ee5e247cd5962dc77
> SHA512: 
> 0e90100d90e9dd142283aab729957e12cc6abbfdf2f86447f488bcbe5890d55564c19a0efc473307744f813bcdac2636b027ad4e5beebf1ce1ed6ee44deaccbe
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/bbende.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 1 issue was closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348815
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.3
>
> The vote will be open for 72 hours. Please download the release
> candidate and evaluate the necessary items including checking hashes,
> signatures, build from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-nar-maven-plugin-1.3.3
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [DISCUSS] The future of (Mi)NiFi Command and Control (C2)

[Matt Burgess]

I'm the same, historically I've viewed and implemented C2 as process
control as well. These days with sophisticated containerization and
deployment tools, I'm thinking we leave the process orchestration to
them and concentrate on controlling what the processes are doing.

All, any other thoughts, questions, or concerns? If not, I will update
the C2 page(s) on the Wiki as discussed and start writing up some
Jiras to cover the work, and we can continue individual feature
discussions there.

Thanks,
Matt

On Sat, Nov 20, 2021 at 6:19 AM Marc Parisi  wrote:
>
> " This would align it more to the way NiFi works as
> well, allowing C2 to be used for flow control and metadata exchange
> rather than process control."
>
> This particularly caught my eye and is something that makes sense from an
> evolutionary perspective.
>
> I've previously viewed command and control as a mechanism to control the
> agent processes and their flows.
>
> From an operational perspective, I think that agent process control would
> be nice to have (if not a necessity for some ); however, if C2 is scoped to
> metadata and flow control at least -- through the MVP or permanently --
> then the conversation is simpler and I think your proposal is great.
>
> Thanks for the additional insight, Matt.
>
> Thanks,
> Marc
>
>
>
>
>
> On Thu, Nov 18, 2021 at 4:37 PM Matt Burgess  wrote:
>
> > Marc,
> >
> > That's a great point and reminds me that I missed two whole sections
> > of the proposal: 1) The refactor of minifi-bootstrap, and 2) the
> > standardization of serialized flow control to VersionedFlowSnapshot
> > (i.e. JSON not XML or YAML).
> >
> > 1) Currently it's the bootstrap process that reaches out to the C2
> > server and handles the commands such as UPDATE and RESTART. In fact,
> > UPDATE updates the flow on-disk and RESTART then kills and restarts
> > the MiNiFi process.  I think with the time that's passed since its
> > inception, it has kind of gotten bloated and has taken on capabilities
> > probably best left to a separate module, since those things make it
> > inconsistent with the way NiFi bootstrap works . In that vein I'd
> > propose (at some point, not necessarily MVP but that'd be great) that
> > we standardize the bootstraps as much as possible, and put all the C2
> > stuff into the C2 modules (i.e. the C2 Client in the case of
> > minifi-bootstrap). I vaguely alluded to that when I said the C2 client
> > would need access to the internal objects, but that glossed WAY over
> > the actual implication, which is that the C2 client module will be the
> > thing talking to the C2 server to handle all the C2 messages including
> > UPDATE, and I think RESTART might be replaced with RELOAD or absorbed
> > into UPDATE, but happy to discuss that as well.
> >
> > If there are issues such as updating an agent/instance with
> > configuration properties, I anticipate a new proposal and design to
> > cover that, but I invite current C2 Server users to jump in here
> > please. As it stands I believe we haven't implemented enough of the C2
> > capabilities to be affected, unless the C2 server's flow update file
> > includes configurable property updates. Otherwise I would think just a
> > flow update would suffice, and could be done without restarting the
> > process itself. This would align it more to the way NiFi works as
> > well, allowing C2 to be used for flow control and metadata exchange
> > rather than process control.
> >
> > 2) IIRC, the "config.yml' approach came from two use cases, to be able
> > to update the agent process/configuration itself and to make it easier
> > to "design" a flow by having more user-friendly names for the various
> > configuration and component properties without a UI designer. These
> > days, the minifi-toolkit allows the user to convert a flow from
> > flow.xml.gz to a config.yml, and I'm aware of at least one vendor
> > product for flow design & publish via a GUI. Between the
> > aforementioned lack of a need to update the agent process, and ongoing
> > work [1] to standardize the flow definition for NiFi, Registry,
> > Stateless, etc. to VersionedFlowSnapshot, I think MiNiFi should follow
> > suit. I can't speak to the C++ side of things but perhaps those more
> > familiar with that project can weigh in.
> >
> > My original email turned out to be more focused on the MVP products,
> > but these additional aspects (as well as any others the community
> > would like to offer) should be discussed too. If these are not
> > necessary for MVP, perhaps we can get the work going on the API an

Re: [DISCUSS] The future of (Mi)NiFi Command and Control (C2)

[Matt Burgess]

Marc,

That's a great point and reminds me that I missed two whole sections
of the proposal: 1) The refactor of minifi-bootstrap, and 2) the
standardization of serialized flow control to VersionedFlowSnapshot
(i.e. JSON not XML or YAML).

1) Currently it's the bootstrap process that reaches out to the C2
server and handles the commands such as UPDATE and RESTART. In fact,
UPDATE updates the flow on-disk and RESTART then kills and restarts
the MiNiFi process.  I think with the time that's passed since its
inception, it has kind of gotten bloated and has taken on capabilities
probably best left to a separate module, since those things make it
inconsistent with the way NiFi bootstrap works . In that vein I'd
propose (at some point, not necessarily MVP but that'd be great) that
we standardize the bootstraps as much as possible, and put all the C2
stuff into the C2 modules (i.e. the C2 Client in the case of
minifi-bootstrap). I vaguely alluded to that when I said the C2 client
would need access to the internal objects, but that glossed WAY over
the actual implication, which is that the C2 client module will be the
thing talking to the C2 server to handle all the C2 messages including
UPDATE, and I think RESTART might be replaced with RELOAD or absorbed
into UPDATE, but happy to discuss that as well.

If there are issues such as updating an agent/instance with
configuration properties, I anticipate a new proposal and design to
cover that, but I invite current C2 Server users to jump in here
please. As it stands I believe we haven't implemented enough of the C2
capabilities to be affected, unless the C2 server's flow update file
includes configurable property updates. Otherwise I would think just a
flow update would suffice, and could be done without restarting the
process itself. This would align it more to the way NiFi works as
well, allowing C2 to be used for flow control and metadata exchange
rather than process control.

2) IIRC, the "config.yml' approach came from two use cases, to be able
to update the agent process/configuration itself and to make it easier
to "design" a flow by having more user-friendly names for the various
configuration and component properties without a UI designer. These
days, the minifi-toolkit allows the user to convert a flow from
flow.xml.gz to a config.yml, and I'm aware of at least one vendor
product for flow design & publish via a GUI. Between the
aforementioned lack of a need to update the agent process, and ongoing
work [1] to standardize the flow definition for NiFi, Registry,
Stateless, etc. to VersionedFlowSnapshot, I think MiNiFi should follow
suit. I can't speak to the C++ side of things but perhaps those more
familiar with that project can weigh in.

My original email turned out to be more focused on the MVP products,
but these additional aspects (as well as any others the community
would like to offer) should be discussed too. If these are not
necessary for MVP, perhaps we can get the work going on the API and
MVP and continue the discussions on the Wiki and/or mailing list as
folks want to contribute the extra goodies :)

Thanks,
Matt

[1] https://issues.apache.org/jira/browse/NIFI-9069

On Thu, Nov 18, 2021 at 4:07 PM Marc Parisi  wrote:
>
> Hi Matt,
>
> I think this is great. I like the MVP set of features.
>
> Considering use cases and potential maintenance needs, are there thoughts
> on including RESTART in the MVP for cases where agent UPDATES may be
> blocked for whatever reason ( I/O, agent problems, etc )?
>
> Does the risk of the RESTART verb being somewhat nebulous outweigh any
> benefits?
>
> Again, great stuff!
>
> Thanks,
> Marc
>
>
> On Thu, Nov 18, 2021 at 2:20 PM Matt Burgess  wrote:
>
> > Hello all,
> >
> > I'd like to start a discussion about the current and future Command
> > and Control (C2) capabilities in terms of the NiFi/MiNiFi Java
> > codebase. The current implementation[1] is IMO not a pure reference
> > implementation of the C2 spec [2]. It has fairly limited capability
> > and uses the PullHttpChangeIngestor for the agent to ask for flow
> > updates, which makes any bidirectional communication difficult. Rather
> > than try to build out from what we currently have, I propose a new
> > implementation (deprecating and eventually removing the old one) that
> > is more generic, extensible, etc. This would include the following
> > options:
> >
> > - Renaming of the minifi-c2 modules and perhaps the release artifacts,
> > updating the documentation accordingly
> > - Creation of a top-level "c2" or "nifi-c2" module that includes an
> > API module, assembly, NAR, implementation, services bundle, etc. as
> > needed
> > - Adding/deleting/changing/using properties related to C2 configuration
> > - Changing the common runtime/loadi

[DISCUSS] The future of (Mi)NiFi Command and Control (C2)

[Matt Burgess]

Hello all,

I'd like to start a discussion about the current and future Command
and Control (C2) capabilities in terms of the NiFi/MiNiFi Java
codebase. The current implementation[1] is IMO not a pure reference
implementation of the C2 spec [2]. It has fairly limited capability
and uses the PullHttpChangeIngestor for the agent to ask for flow
updates, which makes any bidirectional communication difficult. Rather
than try to build out from what we currently have, I propose a new
implementation (deprecating and eventually removing the old one) that
is more generic, extensible, etc. This would include the following
options:

- Renaming of the minifi-c2 modules and perhaps the release artifacts,
updating the documentation accordingly
- Creation of a top-level "c2" or "nifi-c2" module that includes an
API module, assembly, NAR, implementation, services bundle, etc. as
needed
- Adding/deleting/changing/using properties related to C2 configuration
- Changing the common runtime/loading code to look for a C2 client
implementation and instantiate that at startup, configuring it with
the appropriate properties and injecting the objects needed for C2
capabilities
- Implementing a new C2 server that supports more of the C2 spec, and
making adding more of the commands easier going forward

I propose the Minimum Viable Product (MVP) C2 Client would have the
following capabilities:
- A "Manifest Reporter" that can either push a manifest (a document
containing all the agent/instance capabilities such as available
processors and their metadata, e.g.) and/or respond to a DESCRIBE
MANIFEST request (see [2])
- A "Heartbeat Reporter" to send HEARTBEAT messages and also to reply
with ACKNOWLEDGEMENT messages and dispatching any commands such as
UPDATE
- The Heartbeat Reporter is analogous to the PullHttpChangeIngestor in
the old system, but is much more capable
- Supports HTTPS

I propose the MVP C2 Server would have the following capabilities:
- Handles manifest messages (push and/or pull) aka DESCRIBE MANIFEST
- Handles the HEARTBEAT / UPDATE / ACK exchange, perhaps initially
using the same approach as the old C2 system (loading the flow from a
file for the UPDATE command) but would be extensible/configurable
- Supports HTTPS

This should give us not only feature parity with the old C2 system,
but also presents a framework to build out the rest of the C2 design
and add other capabilities. I should note that this proposal is not
specific to MiNiFi Java. It should aim to align and use the same
protocol(s) as MiNiFi C++ [3], and should allow us to perhaps add C2
capabilities to NiFi itself in the future. We could track the proposal
and progress at [4] as I believe that's what the page is intended for.

I'd like to hear your thoughts on these proposals, and I'm happy to
answer any questions or clarify as necessary. Also as this proceeds
I'm happy to write up any Jira epics/cases/etc. and do the initial
project structure and API setup work so others can dig in.

Regards,
Matt

[1] https://github.com/apache/nifi/blob/main/minifi/minifi-c2/README.md
[2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design
[3] https://github.com/apache/nifi-minifi-cpp/blob/main/C2.md
[4] 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048430=624428108

Re: [DISCUSS] Docs from non packaged NARs

[Matt Burgess]

100% agree, didn't give any thought on how to do it but definitely on
board :) We show users where to find these NARs when they ask (or
there are some links around), and if you add the NAR(s) the docs show
up in your NiFi instance, but to really showcase the power of NiFi we
should publicly display all the docs for all the components published
as a result of a release.

Regards,
Matt

On Tue, Nov 9, 2021 at 6:17 PM Pierre Villard
 wrote:
>
> Hi team,
>
> I'd like to discuss the possibility to change the release process we have
> for the next releases so that the documentation of all components we have
> in the repository are published as part of the release process.
>
> All of the documentation for components that are in specific profiles are
> not only skipped for the final convenience binary but also for the
> documentation. We have a tremendous amount of great components in this "by
> default excluded" NARs and it would be helpful to have the corresponding
> documentation to at least redirect users when we get some questions on the
> users mailing list.
>
> I feel like this should only be a release process thing but I could be
> wrong.
>
> Something that could be even better would be to enable all the profiles by
> default but only for nifi-docs module? Not sure if that is even a
> possibility.
>
> Thoughts?
>
> Thanks,
> Pierre

Re: [VOTE] Release Apache NiFi 1.15.0 (rc3)

[Matt Burgess]

+1 Release this package as nifi-1.15.0

Ran through the release helper and some tests to verify various Jiras
were included. Thanks again for RM'ing Joe!

On Wed, Nov 3, 2021 at 3:29 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.15.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1186
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.15.0-RC3
> The Git commit ID is 7fdc07cccdc0e23d4986557a9314e36859704a3b
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=7fdc07cccdc0e23d4986557a9314e36859704a3b
>
> Checksums of nifi-1.15.0-source-release.zip:
> SHA256: 56fe317248941fdbc6216ec973e6ce898d0f682a54e2d063edbabf8542f5288a
> SHA512: 
> 63f10d9127cf1613c29bf2306be3d7a5b931b31304920706e0df5ea2fe87b58c410efed6ac37afc38d12c65e69f14aec7afb926000fc90cc13f15c738cd15c7f
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 234 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350382
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [ ] +1 Release this package as nifi-1.15.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...