Re: Is there a need for the ConvertAvroToJson processor in 2.x?

[Mike Thomsen]

IIRC, the plan was to remove those ConvertToX processors in 2.X

On Wed, Apr 10, 2024 at 11:36 AM Dan S  wrote:

> I noticed that the ConvertAvroToJson processor is not marked as deprecated
> in 2.x. Is there something it offers which cannot be done with the
> ConvertRecord processor configured with an AvroReader and
> JsonRecordSetWriter?
>

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

Fair enough, Joe.

Matt,

I poked around your branch a little this evening. I agree with you and
David 100% now on the need for some sort of abstraction. I think the Graph
Bundle's model could serve as a good starting point for how to approach the
problem. The client drivers in that bundle do the heavy lifting of doing
the querying and passing the results back via a callback system to the
processors that call them to ensure that the processors don't know anything
other than they're getting back a Map of result data each iteration.
Thoughts?

On Fri, Mar 22, 2024 at 2:48 PM Joe Witt  wrote:

> Mike,
>
> The bundles we include cannot have libs with know vulns and that last a
> very long time.  That is a more pressing blocker.
>
> As noted top of thread we all recognize the importance of being able to
> integrate with Cassandra but including that must come with active mx
> especially as it relates to vulns.
>
> Thanks
> Joe
>
> On Fri, Mar 22, 2024 at 11:42 AM Mike Thomsen 
> wrote:
>
> > The scope tag was probably copy pasta. You raise a valid point about the
> > processor dependencies that completely slipped my mind. That said, I
> think
> > it's premature to remove the cassandra bundle until we have a working
> > replacement. I would consider that support a blocker for 2.X.
> >
> > On Fri, Mar 22, 2024 at 12:00 PM Matt Burgess 
> wrote:
> >
> > > David beat me to it :) IMO the only NAR that should have any
> dependencies
> > > on Cassandra is the services NAR, not the processors or services API.
> > >
> > > On Fri, Mar 22, 2024 at 11:10 AM David Handermann <
> > > exceptionfact...@apache.org> wrote:
> > >
> > > > Mike,
> > > >
> > > > Thanks for sharing the branch, it is helpful to have that as a
> > > > reference example. Have you been able to exercise any of that
> approach
> > > > at runtime?
> > > >
> > > > Based on what is there right now, attempting to mark the DataStax
> > > > java-driver-core as provided does not look like it will work. It may
> > > > pass unit tests, but runtime NAR class loading requires that classes
> > > > be available in the same NAR, or in a parent NAR. That means when
> NiFi
> > > > tries to load the Controller Service interface, it must have access
> to
> > > > a version of the relevant Cassandra driver classes. By marking the
> > > > dependency as provided, it will not be available in the API NAR, and
> > > > thus not available when loading the service interface. Including it
> in
> > > > the API NAR won't work either, because it conflicts with the ScyllaDB
> > > > java-driver-core in the implementation NAR.
> > > >
> > > > This is the reason Matt and I highlighted for providing a layer of
> > > > abstraction at the Controller Service API level.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Fri, Mar 22, 2024 at 8:13 AM Mike Thomsen  >
> > > > wrote:
> > > > >
> > > > > Work so far: https://github.com/MikeThomsen/nifi/tree/cql-changes
> > > > >
> > > > > On Thu, Mar 21, 2024 at 9:52 AM Mike Thomsen <
> mikerthom...@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > > > Matt/David,
> > > > > >
> > > > > > By this evening, I should be at a point where I can share my
> > branch.
> > > It
> > > > > > should be far enough along that y'all can see what I mean about
> how
> > > > most of
> > > > > > the changes really weren't that complicated. My sense is that if
> we
> > > > > > collaborate on it, we can probably get it ready for a PR within a
> > > week
> > > > or
> > > > > > two.
> > > > > >
> > > > > > It would probably be a good idea to plan to revisit the Cassandra
> > > DMC's
> > > > > > design and make it more flexible.
> > > > > >
> > > > > > One nice thing about the new DataStax driver is that it supports
> > > > > > configuration by a very detailed configuration file format, so we
> > can
> > > > give
> > > > > > users that option + combine it with EL/parameters (I envision an
> > > option
> > > > > > where the user puts EL in the file, we load the file, preprocess
> > the
> > > > EL and
> > > > > &

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

The scope tag was probably copy pasta. You raise a valid point about the
processor dependencies that completely slipped my mind. That said, I think
it's premature to remove the cassandra bundle until we have a working
replacement. I would consider that support a blocker for 2.X.

On Fri, Mar 22, 2024 at 12:00 PM Matt Burgess  wrote:

> David beat me to it :) IMO the only NAR that should have any dependencies
> on Cassandra is the services NAR, not the processors or services API.
>
> On Fri, Mar 22, 2024 at 11:10 AM David Handermann <
> exceptionfact...@apache.org> wrote:
>
> > Mike,
> >
> > Thanks for sharing the branch, it is helpful to have that as a
> > reference example. Have you been able to exercise any of that approach
> > at runtime?
> >
> > Based on what is there right now, attempting to mark the DataStax
> > java-driver-core as provided does not look like it will work. It may
> > pass unit tests, but runtime NAR class loading requires that classes
> > be available in the same NAR, or in a parent NAR. That means when NiFi
> > tries to load the Controller Service interface, it must have access to
> > a version of the relevant Cassandra driver classes. By marking the
> > dependency as provided, it will not be available in the API NAR, and
> > thus not available when loading the service interface. Including it in
> > the API NAR won't work either, because it conflicts with the ScyllaDB
> > java-driver-core in the implementation NAR.
> >
> > This is the reason Matt and I highlighted for providing a layer of
> > abstraction at the Controller Service API level.
> >
> > Regards,
> > David Handermann
> >
> > On Fri, Mar 22, 2024 at 8:13 AM Mike Thomsen 
> > wrote:
> > >
> > > Work so far: https://github.com/MikeThomsen/nifi/tree/cql-changes
> > >
> > > On Thu, Mar 21, 2024 at 9:52 AM Mike Thomsen 
> > wrote:
> > >
> > > > Matt/David,
> > > >
> > > > By this evening, I should be at a point where I can share my branch.
> It
> > > > should be far enough along that y'all can see what I mean about how
> > most of
> > > > the changes really weren't that complicated. My sense is that if we
> > > > collaborate on it, we can probably get it ready for a PR within a
> week
> > or
> > > > two.
> > > >
> > > > It would probably be a good idea to plan to revisit the Cassandra
> DMC's
> > > > design and make it more flexible.
> > > >
> > > > One nice thing about the new DataStax driver is that it supports
> > > > configuration by a very detailed configuration file format, so we can
> > give
> > > > users that option + combine it with EL/parameters (I envision an
> option
> > > > where the user puts EL in the file, we load the file, preprocess the
> > EL and
> > > > load that into the driver)
> > > >
> > > > On Wed, Mar 20, 2024 at 4:01 PM Mike Thomsen  >
> > > > wrote:
> > > >
> > > >> If it were that simple, they would probably have just gone with that
> > > >> solution. That said, the API is functionally vendor agnostic at this
> > point
> > > >> at the Java API level. So I see no need to add abstraction above
> > that. I've
> > > >> got probably 2/3 of nifi-cassandra-bundle converted. Hitting a few
> > pain
> > > >> points where I'm having to dig deep into the docs to make progress,
> > but so
> > > >> far, so good.
> > > >>
> > > >> On Wed, Mar 20, 2024 at 2:38 PM Matt Burgess 
> > > >> wrote:
> > > >>
> > > >>> It would be interesting to see if you exclude the Scylla API JAR
> > from the
> > > >>> Scylla implementation and instead include DataStax's, if that
> works.
> > > >>> However I'm still leaning towards a vendor-agnostic API.
> > > >>>
> > > >>> On Wed, Mar 20, 2024 at 11:26 AM Mike Thomsen <
> > mikerthom...@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>> > At first glance, the package names look identical to me:
> > > >>> >
> > > >>> >
> > https://java-driver.docs.scylladb.com/scylla-4.15.0.x/api/index.html
> > > >>> >
> > > >>> > So I see no reason to not take them at their word that it's
> drop-in
> > > >>> >
> > > >>> > On 

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

Work so far: https://github.com/MikeThomsen/nifi/tree/cql-changes

On Thu, Mar 21, 2024 at 9:52 AM Mike Thomsen  wrote:

> Matt/David,
>
> By this evening, I should be at a point where I can share my branch. It
> should be far enough along that y'all can see what I mean about how most of
> the changes really weren't that complicated. My sense is that if we
> collaborate on it, we can probably get it ready for a PR within a week or
> two.
>
> It would probably be a good idea to plan to revisit the Cassandra DMC's
> design and make it more flexible.
>
> One nice thing about the new DataStax driver is that it supports
> configuration by a very detailed configuration file format, so we can give
> users that option + combine it with EL/parameters (I envision an option
> where the user puts EL in the file, we load the file, preprocess the EL and
> load that into the driver)
>
> On Wed, Mar 20, 2024 at 4:01 PM Mike Thomsen 
> wrote:
>
>> If it were that simple, they would probably have just gone with that
>> solution. That said, the API is functionally vendor agnostic at this point
>> at the Java API level. So I see no need to add abstraction above that. I've
>> got probably 2/3 of nifi-cassandra-bundle converted. Hitting a few pain
>> points where I'm having to dig deep into the docs to make progress, but so
>> far, so good.
>>
>> On Wed, Mar 20, 2024 at 2:38 PM Matt Burgess 
>> wrote:
>>
>>> It would be interesting to see if you exclude the Scylla API JAR from the
>>> Scylla implementation and instead include DataStax's, if that works.
>>> However I'm still leaning towards a vendor-agnostic API.
>>>
>>> On Wed, Mar 20, 2024 at 11:26 AM Mike Thomsen 
>>> wrote:
>>>
>>> > At first glance, the package names look identical to me:
>>> >
>>> > https://java-driver.docs.scylladb.com/scylla-4.15.0.x/api/index.html
>>> >
>>> > So I see no reason to not take them at their word that it's drop-in
>>> >
>>> > On Wed, Mar 20, 2024 at 11:04 AM David Handermann <
>>> > exceptionfact...@apache.org> wrote:
>>> >
>>> > > Mike,
>>> > >
>>> > > One important thing to mention about the DataStax vs ScyllaDB driver
>>> > > is that the Maven coordinates are different, and managing the
>>> > > dependencies correctly will make or break the implementation.
>>> > >
>>> > > In other words, if it is possible to use the DataStax 4 core JAR in
>>> > > the Controller Service API, but use the ScyllaDB 3 query JAR in the
>>> > > ScyllaDB implementation, then that could avoid the need for
>>> additional
>>> > > abstraction. Without taking a closer look, however, I would be
>>> > > surprised if this worked.
>>> > >
>>> > > Although ScyllaDB highlights their forked driver as a drop-in
>>> > > replacement for the DataStax version, and maintains the same Java
>>> > > package names, there is a difference between a complete replacement
>>> > > and a shared API JAR. Without a common API JAR, that both
>>> > > implementations can use, it will be necessary to provide an
>>> > > abstraction in NiFi that avoids depending on either library at the
>>> > > Controller Service API level.
>>> > >
>>> > > Regards,
>>> > > David Handermann
>>> > >
>>> > > On Wed, Mar 20, 2024 at 8:25 AM Mike Thomsen >> >
>>> > > wrote:
>>> > > >
>>> > > > Matt/David,
>>> > > >
>>> > > > I got some time this morning to take a crack at directly migrating
>>> it
>>> > > over
>>> > > > to the DataStax 4.17 driver. Definitely got a lot of work to do,
>>> but so
>>> > > far
>>> > > > I haven't hit any real snags. This is a branch that reverts the
>>> commit
>>> > to
>>> > > > remove the cassandra bundle and reuses the existing features as a
>>> > > > foundation. From what I'm seeing so far (feels like I'm about 25%
>>> done)
>>> > > it
>>> > > > should be doable to reuse the existing bundle, but rename it to the
>>> > "CQL
>>> > > > Bundle" and just add a second controller service for Scylla that is
>>> > > > otherwise 100% the same codewise.
>>> > > >
>>> > > > On Tu

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

Matt/David,

By this evening, I should be at a point where I can share my branch. It
should be far enough along that y'all can see what I mean about how most of
the changes really weren't that complicated. My sense is that if we
collaborate on it, we can probably get it ready for a PR within a week or
two.

It would probably be a good idea to plan to revisit the Cassandra DMC's
design and make it more flexible.

One nice thing about the new DataStax driver is that it supports
configuration by a very detailed configuration file format, so we can give
users that option + combine it with EL/parameters (I envision an option
where the user puts EL in the file, we load the file, preprocess the EL and
load that into the driver)

On Wed, Mar 20, 2024 at 4:01 PM Mike Thomsen  wrote:

> If it were that simple, they would probably have just gone with that
> solution. That said, the API is functionally vendor agnostic at this point
> at the Java API level. So I see no need to add abstraction above that. I've
> got probably 2/3 of nifi-cassandra-bundle converted. Hitting a few pain
> points where I'm having to dig deep into the docs to make progress, but so
> far, so good.
>
> On Wed, Mar 20, 2024 at 2:38 PM Matt Burgess  wrote:
>
>> It would be interesting to see if you exclude the Scylla API JAR from the
>> Scylla implementation and instead include DataStax's, if that works.
>> However I'm still leaning towards a vendor-agnostic API.
>>
>> On Wed, Mar 20, 2024 at 11:26 AM Mike Thomsen 
>> wrote:
>>
>> > At first glance, the package names look identical to me:
>> >
>> > https://java-driver.docs.scylladb.com/scylla-4.15.0.x/api/index.html
>> >
>> > So I see no reason to not take them at their word that it's drop-in
>> >
>> > On Wed, Mar 20, 2024 at 11:04 AM David Handermann <
>> > exceptionfact...@apache.org> wrote:
>> >
>> > > Mike,
>> > >
>> > > One important thing to mention about the DataStax vs ScyllaDB driver
>> > > is that the Maven coordinates are different, and managing the
>> > > dependencies correctly will make or break the implementation.
>> > >
>> > > In other words, if it is possible to use the DataStax 4 core JAR in
>> > > the Controller Service API, but use the ScyllaDB 3 query JAR in the
>> > > ScyllaDB implementation, then that could avoid the need for additional
>> > > abstraction. Without taking a closer look, however, I would be
>> > > surprised if this worked.
>> > >
>> > > Although ScyllaDB highlights their forked driver as a drop-in
>> > > replacement for the DataStax version, and maintains the same Java
>> > > package names, there is a difference between a complete replacement
>> > > and a shared API JAR. Without a common API JAR, that both
>> > > implementations can use, it will be necessary to provide an
>> > > abstraction in NiFi that avoids depending on either library at the
>> > > Controller Service API level.
>> > >
>> > > Regards,
>> > > David Handermann
>> > >
>> > > On Wed, Mar 20, 2024 at 8:25 AM Mike Thomsen 
>> > > wrote:
>> > > >
>> > > > Matt/David,
>> > > >
>> > > > I got some time this morning to take a crack at directly migrating
>> it
>> > > over
>> > > > to the DataStax 4.17 driver. Definitely got a lot of work to do,
>> but so
>> > > far
>> > > > I haven't hit any real snags. This is a branch that reverts the
>> commit
>> > to
>> > > > remove the cassandra bundle and reuses the existing features as a
>> > > > foundation. From what I'm seeing so far (feels like I'm about 25%
>> done)
>> > > it
>> > > > should be doable to reuse the existing bundle, but rename it to the
>> > "CQL
>> > > > Bundle" and just add a second controller service for Scylla that is
>> > > > otherwise 100% the same codewise.
>> > > >
>> > > > On Tue, Mar 19, 2024 at 6:41 PM Mike Thomsen <
>> mikerthom...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > A cursory look at the Cassandra 5 stuff didn’t indicate any
>> > > > > incompatibility. So yeah, I think we are likely pretty safe to use
>> > the
>> > > 4.17
>> > > > > driver
>> > > > > Sent from my iPhone
>> > > > >
>> > > > > > On Mar 19, 2024, at 3

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

If it were that simple, they would probably have just gone with that
solution. That said, the API is functionally vendor agnostic at this point
at the Java API level. So I see no need to add abstraction above that. I've
got probably 2/3 of nifi-cassandra-bundle converted. Hitting a few pain
points where I'm having to dig deep into the docs to make progress, but so
far, so good.

On Wed, Mar 20, 2024 at 2:38 PM Matt Burgess  wrote:

> It would be interesting to see if you exclude the Scylla API JAR from the
> Scylla implementation and instead include DataStax's, if that works.
> However I'm still leaning towards a vendor-agnostic API.
>
> On Wed, Mar 20, 2024 at 11:26 AM Mike Thomsen 
> wrote:
>
> > At first glance, the package names look identical to me:
> >
> > https://java-driver.docs.scylladb.com/scylla-4.15.0.x/api/index.html
> >
> > So I see no reason to not take them at their word that it's drop-in
> >
> > On Wed, Mar 20, 2024 at 11:04 AM David Handermann <
> > exceptionfact...@apache.org> wrote:
> >
> > > Mike,
> > >
> > > One important thing to mention about the DataStax vs ScyllaDB driver
> > > is that the Maven coordinates are different, and managing the
> > > dependencies correctly will make or break the implementation.
> > >
> > > In other words, if it is possible to use the DataStax 4 core JAR in
> > > the Controller Service API, but use the ScyllaDB 3 query JAR in the
> > > ScyllaDB implementation, then that could avoid the need for additional
> > > abstraction. Without taking a closer look, however, I would be
> > > surprised if this worked.
> > >
> > > Although ScyllaDB highlights their forked driver as a drop-in
> > > replacement for the DataStax version, and maintains the same Java
> > > package names, there is a difference between a complete replacement
> > > and a shared API JAR. Without a common API JAR, that both
> > > implementations can use, it will be necessary to provide an
> > > abstraction in NiFi that avoids depending on either library at the
> > > Controller Service API level.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Wed, Mar 20, 2024 at 8:25 AM Mike Thomsen 
> > > wrote:
> > > >
> > > > Matt/David,
> > > >
> > > > I got some time this morning to take a crack at directly migrating it
> > > over
> > > > to the DataStax 4.17 driver. Definitely got a lot of work to do, but
> so
> > > far
> > > > I haven't hit any real snags. This is a branch that reverts the
> commit
> > to
> > > > remove the cassandra bundle and reuses the existing features as a
> > > > foundation. From what I'm seeing so far (feels like I'm about 25%
> done)
> > > it
> > > > should be doable to reuse the existing bundle, but rename it to the
> > "CQL
> > > > Bundle" and just add a second controller service for Scylla that is
> > > > otherwise 100% the same codewise.
> > > >
> > > > On Tue, Mar 19, 2024 at 6:41 PM Mike Thomsen  >
> > > wrote:
> > > >
> > > > > A cursory look at the Cassandra 5 stuff didn’t indicate any
> > > > > incompatibility. So yeah, I think we are likely pretty safe to use
> > the
> > > 4.17
> > > > > driver
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Mar 19, 2024, at 3:35 PM, Matt Burgess 
> > > wrote:
> > > > > >
> > > > > > Is it likely now (due to the refactor) that we will simply be
> able
> > > to
> > > > > > upgrade the driver when Cassandra 5 is GA? Also does anyone use
> > > Netflix's
> > > > > > Astyanax [1]?
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > >
> >
> https://cassandra.apache.org/doc/stable/cassandra/getting_started/drivers.html#java
> > > > > >
> > > > > >> On Tue, Mar 19, 2024 at 3:10 PM Mike Thomsen <
> > > mikerthom...@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> Realistically, I think we are only likely to see two drivers:
> > > > > >>
> > > > > >> * DataStax
> > > > > >> * ScyllaDB
> > > > > >>
> > > > > >> The latter makes a selling point of being a binary compatible,
>

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

At first glance, the package names look identical to me:

https://java-driver.docs.scylladb.com/scylla-4.15.0.x/api/index.html

So I see no reason to not take them at their word that it's drop-in

On Wed, Mar 20, 2024 at 11:04 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Mike,
>
> One important thing to mention about the DataStax vs ScyllaDB driver
> is that the Maven coordinates are different, and managing the
> dependencies correctly will make or break the implementation.
>
> In other words, if it is possible to use the DataStax 4 core JAR in
> the Controller Service API, but use the ScyllaDB 3 query JAR in the
> ScyllaDB implementation, then that could avoid the need for additional
> abstraction. Without taking a closer look, however, I would be
> surprised if this worked.
>
> Although ScyllaDB highlights their forked driver as a drop-in
> replacement for the DataStax version, and maintains the same Java
> package names, there is a difference between a complete replacement
> and a shared API JAR. Without a common API JAR, that both
> implementations can use, it will be necessary to provide an
> abstraction in NiFi that avoids depending on either library at the
> Controller Service API level.
>
> Regards,
> David Handermann
>
> On Wed, Mar 20, 2024 at 8:25 AM Mike Thomsen 
> wrote:
> >
> > Matt/David,
> >
> > I got some time this morning to take a crack at directly migrating it
> over
> > to the DataStax 4.17 driver. Definitely got a lot of work to do, but so
> far
> > I haven't hit any real snags. This is a branch that reverts the commit to
> > remove the cassandra bundle and reuses the existing features as a
> > foundation. From what I'm seeing so far (feels like I'm about 25% done)
> it
> > should be doable to reuse the existing bundle, but rename it to the "CQL
> > Bundle" and just add a second controller service for Scylla that is
> > otherwise 100% the same codewise.
> >
> > On Tue, Mar 19, 2024 at 6:41 PM Mike Thomsen 
> wrote:
> >
> > > A cursory look at the Cassandra 5 stuff didn’t indicate any
> > > incompatibility. So yeah, I think we are likely pretty safe to use the
> 4.17
> > > driver
> > > Sent from my iPhone
> > >
> > > > On Mar 19, 2024, at 3:35 PM, Matt Burgess 
> wrote:
> > > >
> > > > Is it likely now (due to the refactor) that we will simply be able
> to
> > > > upgrade the driver when Cassandra 5 is GA? Also does anyone use
> Netflix's
> > > > Astyanax [1]?
> > > >
> > > > [1]
> > > >
> > >
> https://cassandra.apache.org/doc/stable/cassandra/getting_started/drivers.html#java
> > > >
> > > >> On Tue, Mar 19, 2024 at 3:10 PM Mike Thomsen <
> mikerthom...@gmail.com>
> > > wrote:
> > > >>
> > > >> Realistically, I think we are only likely to see two drivers:
> > > >>
> > > >> * DataStax
> > > >> * ScyllaDB
> > > >>
> > > >> The latter makes a selling point of being a binary compatible,
> drop-in
> > > >> replacement for the former.
> > > >>
> > > >> That's why I don't see a need to have an abstraction layer per se. I
> > > think
> > > >> we only need "DataStaxConnectionProviderImpl" and
> > > >> "ScyllaDBConnectionProviderImpl" with the difference being which
> jar is
> > > >> imported by maven.
> > > >>
> > > >> On Tue, Mar 19, 2024 at 2:59 PM David Handermann <
> > > >> exceptionfact...@apache.org> wrote:
> > > >>
> > > >>> Mike,
> > > >>>
> > > >>> Thanks for the reply and clarification.
> > > >>>
> > > >>> I agree there is no need to maintain support for the DataStax 3
> driver
> > > >>> and Java API, any new components should be built on the latest
> version
> > > >>> of the driver.
> > > >>>
> > > >>> What we do need going forward is to avoid, if at all possible,
> having
> > > >>> a DataStax 4 dependency in the Controller Service API.
> > > >>>
> > > >>> One example of this is the WebClientServiceProvider interface. That
> > > >>> Controller Service API does not have any third-party dependencies.
> The
> > > >>> Controller Service implementation,
> StandardWebClientServiceProvider,
> > > >>> 

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

Matt/David,

I got some time this morning to take a crack at directly migrating it over
to the DataStax 4.17 driver. Definitely got a lot of work to do, but so far
I haven't hit any real snags. This is a branch that reverts the commit to
remove the cassandra bundle and reuses the existing features as a
foundation. From what I'm seeing so far (feels like I'm about 25% done) it
should be doable to reuse the existing bundle, but rename it to the "CQL
Bundle" and just add a second controller service for Scylla that is
otherwise 100% the same codewise.

On Tue, Mar 19, 2024 at 6:41 PM Mike Thomsen  wrote:

> A cursory look at the Cassandra 5 stuff didn’t indicate any
> incompatibility. So yeah, I think we are likely pretty safe to use the 4.17
> driver
> Sent from my iPhone
>
> > On Mar 19, 2024, at 3:35 PM, Matt Burgess  wrote:
> >
> > Is it likely now (due to the refactor) that we will simply be able to
> > upgrade the driver when Cassandra 5 is GA? Also does anyone use Netflix's
> > Astyanax [1]?
> >
> > [1]
> >
> https://cassandra.apache.org/doc/stable/cassandra/getting_started/drivers.html#java
> >
> >> On Tue, Mar 19, 2024 at 3:10 PM Mike Thomsen 
> wrote:
> >>
> >> Realistically, I think we are only likely to see two drivers:
> >>
> >> * DataStax
> >> * ScyllaDB
> >>
> >> The latter makes a selling point of being a binary compatible, drop-in
> >> replacement for the former.
> >>
> >> That's why I don't see a need to have an abstraction layer per se. I
> think
> >> we only need "DataStaxConnectionProviderImpl" and
> >> "ScyllaDBConnectionProviderImpl" with the difference being which jar is
> >> imported by maven.
> >>
> >> On Tue, Mar 19, 2024 at 2:59 PM David Handermann <
> >> exceptionfact...@apache.org> wrote:
> >>
> >>> Mike,
> >>>
> >>> Thanks for the reply and clarification.
> >>>
> >>> I agree there is no need to maintain support for the DataStax 3 driver
> >>> and Java API, any new components should be built on the latest version
> >>> of the driver.
> >>>
> >>> What we do need going forward is to avoid, if at all possible, having
> >>> a DataStax 4 dependency in the Controller Service API.
> >>>
> >>> One example of this is the WebClientServiceProvider interface. That
> >>> Controller Service API does not have any third-party dependencies. The
> >>> Controller Service implementation, StandardWebClientServiceProvider,
> >>> has a dependency on OkHttp to implement HTTP communication. That is
> >>> the kind of abstraction that would be ideal, and I believe that also
> >>> aligns with what Matt has described.
> >>>
> >>> Regards,
> >>> David Handermann
> >>>
> >>> On Tue, Mar 19, 2024 at 1:45 PM Mike Thomsen 
> >>> wrote:
> >>>>
> >>>> ** we can dump v3 **DRIVER** compatibility, since later 4.X Java
> >> drivers
> >>>> are backward compatible with Cassandra 3
> >>>>
> >>>> On Tue, Mar 19, 2024 at 2:43 PM Mike Thomsen 
> >>> wrote:
> >>>>
> >>>>> David,
> >>>>>
> >>>>> Before we proceed, I think we should make sure we're all
> >> understanding
> >>> the
> >>>>> same problem here. Starting with this:
> >>>>>
> >>>>>> I believe the CQL protocol is backwards compatible but the Java API
> >>> is
> >>>>> not.
> >>>>>> For example "com.datastax.driver.core.Session" is now
> >>>>>> "com.datastax.oss.driver.api.core.session.Session" and there is no
> >>> more
> >>>>>> "Cluster" class. Might be fairly trivial to fix though, if that's
> >> the
> >>>>> path
> >>>>>> of least resistance.
> >>>>>
> >>>>> From what I've learned using Cassandra 3 and 4 in my day job and
> >>> reading
> >>>>> up on this stuff for the sake of discussion, that all tracks. We used
> >>> the
> >>>>> ~4.11 driver in Spring Boot on both v3 and v4 clusters without issue
> >>> during
> >>>>> an upgrade. So I don't see any reason to factor in the "changes from
> >>>>> DataStax 3 to 4" since the changes were likely a one-off decis

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

A cursory look at the Cassandra 5 stuff didn’t indicate any incompatibility. So 
yeah, I think we are likely pretty safe to use the 4.17 driver
Sent from my iPhone

> On Mar 19, 2024, at 3:35 PM, Matt Burgess  wrote:
> 
> Is it likely now (due to the refactor) that we will simply be able to
> upgrade the driver when Cassandra 5 is GA? Also does anyone use Netflix's
> Astyanax [1]?
> 
> [1]
> https://cassandra.apache.org/doc/stable/cassandra/getting_started/drivers.html#java
> 
>> On Tue, Mar 19, 2024 at 3:10 PM Mike Thomsen  wrote:
>> 
>> Realistically, I think we are only likely to see two drivers:
>> 
>> * DataStax
>> * ScyllaDB
>> 
>> The latter makes a selling point of being a binary compatible, drop-in
>> replacement for the former.
>> 
>> That's why I don't see a need to have an abstraction layer per se. I think
>> we only need "DataStaxConnectionProviderImpl" and
>> "ScyllaDBConnectionProviderImpl" with the difference being which jar is
>> imported by maven.
>> 
>> On Tue, Mar 19, 2024 at 2:59 PM David Handermann <
>> exceptionfact...@apache.org> wrote:
>> 
>>> Mike,
>>> 
>>> Thanks for the reply and clarification.
>>> 
>>> I agree there is no need to maintain support for the DataStax 3 driver
>>> and Java API, any new components should be built on the latest version
>>> of the driver.
>>> 
>>> What we do need going forward is to avoid, if at all possible, having
>>> a DataStax 4 dependency in the Controller Service API.
>>> 
>>> One example of this is the WebClientServiceProvider interface. That
>>> Controller Service API does not have any third-party dependencies. The
>>> Controller Service implementation, StandardWebClientServiceProvider,
>>> has a dependency on OkHttp to implement HTTP communication. That is
>>> the kind of abstraction that would be ideal, and I believe that also
>>> aligns with what Matt has described.
>>> 
>>> Regards,
>>> David Handermann
>>> 
>>> On Tue, Mar 19, 2024 at 1:45 PM Mike Thomsen 
>>> wrote:
>>>> 
>>>> ** we can dump v3 **DRIVER** compatibility, since later 4.X Java
>> drivers
>>>> are backward compatible with Cassandra 3
>>>> 
>>>> On Tue, Mar 19, 2024 at 2:43 PM Mike Thomsen 
>>> wrote:
>>>> 
>>>>> David,
>>>>> 
>>>>> Before we proceed, I think we should make sure we're all
>> understanding
>>> the
>>>>> same problem here. Starting with this:
>>>>> 
>>>>>> I believe the CQL protocol is backwards compatible but the Java API
>>> is
>>>>> not.
>>>>>> For example "com.datastax.driver.core.Session" is now
>>>>>> "com.datastax.oss.driver.api.core.session.Session" and there is no
>>> more
>>>>>> "Cluster" class. Might be fairly trivial to fix though, if that's
>> the
>>>>> path
>>>>>> of least resistance.
>>>>> 
>>>>> From what I've learned using Cassandra 3 and 4 in my day job and
>>> reading
>>>>> up on this stuff for the sake of discussion, that all tracks. We used
>>> the
>>>>> ~4.11 driver in Spring Boot on both v3 and v4 clusters without issue
>>> during
>>>>> an upgrade. So I don't see any reason to factor in the "changes from
>>>>> DataStax 3 to 4" since the changes were likely a one-off decision
>>> meant to
>>>>> position the driver for better future support and stability.
>>>>> 
>>>>> TL;DR, we can dump v3 compatibility and the only thing our users will
>>>>> notice is if we make the controller service totally incompatible with
>>> the
>>>>> one they're already using which is something we can actively avoid.
>>>>> 
>>>>> On Tue, Mar 19, 2024 at 2:00 PM David Handermann <
>>>>> exceptionfact...@apache.org> wrote:
>>>>> 
>>>>>> All,
>>>>>> 
>>>>>> I support a Controller Service API abstraction around the Cassandra
>>>>>> Driver. The changes from DataStax 3 to 4 already highlight the need
>>>>>> for that abstraction. The donation of the DataStax Java driver to
>>>>>> Apache [1] also shows the value of providing some level of
>> isolation,
>>>

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

Realistically, I think we are only likely to see two drivers:

* DataStax
* ScyllaDB

The latter makes a selling point of being a binary compatible, drop-in
replacement for the former.

That's why I don't see a need to have an abstraction layer per se. I think
we only need "DataStaxConnectionProviderImpl" and
"ScyllaDBConnectionProviderImpl" with the difference being which jar is
imported by maven.

On Tue, Mar 19, 2024 at 2:59 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Mike,
>
> Thanks for the reply and clarification.
>
> I agree there is no need to maintain support for the DataStax 3 driver
> and Java API, any new components should be built on the latest version
> of the driver.
>
> What we do need going forward is to avoid, if at all possible, having
> a DataStax 4 dependency in the Controller Service API.
>
> One example of this is the WebClientServiceProvider interface. That
> Controller Service API does not have any third-party dependencies. The
> Controller Service implementation, StandardWebClientServiceProvider,
> has a dependency on OkHttp to implement HTTP communication. That is
> the kind of abstraction that would be ideal, and I believe that also
> aligns with what Matt has described.
>
> Regards,
> David Handermann
>
> On Tue, Mar 19, 2024 at 1:45 PM Mike Thomsen 
> wrote:
> >
> > ** we can dump v3 **DRIVER** compatibility, since later 4.X Java drivers
> > are backward compatible with Cassandra 3
> >
> > On Tue, Mar 19, 2024 at 2:43 PM Mike Thomsen 
> wrote:
> >
> > > David,
> > >
> > > Before we proceed, I think we should make sure we're all understanding
> the
> > > same problem here. Starting with this:
> > >
> > > > I believe the CQL protocol is backwards compatible but the Java API
> is
> > > not.
> > > > For example "com.datastax.driver.core.Session" is now
> > > > "com.datastax.oss.driver.api.core.session.Session" and there is no
> more
> > > > "Cluster" class. Might be fairly trivial to fix though, if that's the
> > > path
> > > > of least resistance.
> > >
> > > From what I've learned using Cassandra 3 and 4 in my day job and
> reading
> > > up on this stuff for the sake of discussion, that all tracks. We used
> the
> > > ~4.11 driver in Spring Boot on both v3 and v4 clusters without issue
> during
> > > an upgrade. So I don't see any reason to factor in the "changes from
> > > DataStax 3 to 4" since the changes were likely a one-off decision
> meant to
> > > position the driver for better future support and stability.
> > >
> > > TL;DR, we can dump v3 compatibility and the only thing our users will
> > > notice is if we make the controller service totally incompatible with
> the
> > > one they're already using which is something we can actively avoid.
> > >
> > > On Tue, Mar 19, 2024 at 2:00 PM David Handermann <
> > > exceptionfact...@apache.org> wrote:
> > >
> > >> All,
> > >>
> > >> I support a Controller Service API abstraction around the Cassandra
> > >> Driver. The changes from DataStax 3 to 4 already highlight the need
> > >> for that abstraction. The donation of the DataStax Java driver to
> > >> Apache [1] also shows the value of providing some level of isolation,
> > >> if at all possible.
> > >>
> > >> I have not taken a close look at the Matt's branch, and the details of
> > >> the abstraction are important, but having the abstraction can be
> > >> useful to avoid getting back to this same situation.
> > >>
> > >> Regards,
> > >> David Handermann
> > >>
> > >> [1] https://github.com/apache/cassandra-java-driver/
> > >>
> > >> On Tue, Mar 19, 2024 at 12:37 PM Mike Thomsen  >
> > >> wrote:
> > >> >
> > >> > Matt,
> > >> >
> > >> > I got that. My point was that the Java changes appear to be a one
> time
> > >> > thing that DataStax did to make a better driver with a much more
> > >> > future-proof API. Since Scylla tracks them as closely as possible, I
> > >> > suspect that we don't need to plan for a bunch of abstraction to
> isolate
> > >> > Java changes.
> > >> >
> > >> > On Tue, Mar 19, 2024 at 11:07 AM Steven Matison <
> > >> steven.mati...@gmail.com>
> > >> > wrote:
> > >&g

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

** we can dump v3 **DRIVER** compatibility, since later 4.X Java drivers
are backward compatible with Cassandra 3

On Tue, Mar 19, 2024 at 2:43 PM Mike Thomsen  wrote:

> David,
>
> Before we proceed, I think we should make sure we're all understanding the
> same problem here. Starting with this:
>
> > I believe the CQL protocol is backwards compatible but the Java API is
> not.
> > For example "com.datastax.driver.core.Session" is now
> > "com.datastax.oss.driver.api.core.session.Session" and there is no more
> > "Cluster" class. Might be fairly trivial to fix though, if that's the
> path
> > of least resistance.
>
> From what I've learned using Cassandra 3 and 4 in my day job and reading
> up on this stuff for the sake of discussion, that all tracks. We used the
> ~4.11 driver in Spring Boot on both v3 and v4 clusters without issue during
> an upgrade. So I don't see any reason to factor in the "changes from
> DataStax 3 to 4" since the changes were likely a one-off decision meant to
> position the driver for better future support and stability.
>
> TL;DR, we can dump v3 compatibility and the only thing our users will
> notice is if we make the controller service totally incompatible with the
> one they're already using which is something we can actively avoid.
>
> On Tue, Mar 19, 2024 at 2:00 PM David Handermann <
> exceptionfact...@apache.org> wrote:
>
>> All,
>>
>> I support a Controller Service API abstraction around the Cassandra
>> Driver. The changes from DataStax 3 to 4 already highlight the need
>> for that abstraction. The donation of the DataStax Java driver to
>> Apache [1] also shows the value of providing some level of isolation,
>> if at all possible.
>>
>> I have not taken a close look at the Matt's branch, and the details of
>> the abstraction are important, but having the abstraction can be
>> useful to avoid getting back to this same situation.
>>
>> Regards,
>> David Handermann
>>
>> [1] https://github.com/apache/cassandra-java-driver/
>>
>> On Tue, Mar 19, 2024 at 12:37 PM Mike Thomsen 
>> wrote:
>> >
>> > Matt,
>> >
>> > I got that. My point was that the Java changes appear to be a one time
>> > thing that DataStax did to make a better driver with a much more
>> > future-proof API. Since Scylla tracks them as closely as possible, I
>> > suspect that we don't need to plan for a bunch of abstraction to isolate
>> > Java changes.
>> >
>> > On Tue, Mar 19, 2024 at 11:07 AM Steven Matison <
>> steven.mati...@gmail.com>
>> > wrote:
>> >
>> > > That was kinda where i got stuck and fell out on my branch/jira.
>> Mike and
>> > > I wanted to make a new controller service , without backward
>> compatibility;
>> > > and remove the duplicate driver/connection properties found in some
>> of the
>> > > processors.
>> > >
>> > > I agree taking out all old stuff and making new controller service
>> makes
>> > > most sense.  4.x and 5.x should be mostly backwards compatible to
>> 2&3.x
>> > > with how it’s used within current processors.
>> > >
>> > >
>> > >
>> > > On Tue, Mar 19, 2024 at 10:49 AM Matt Burgess 
>> > > wrote:
>> > >
>> > > > The abstraction is to isolate Java API changes, not protocol
>> > > compatibility
>> > > > Changing to the java-driver comes with a number of changes to the
>> code
>> > > (see
>> > > > Steven's and my branches), if we can abstract that API it should
>> lead to
>> > > > more maintainable code in the future by not having to change any
>> > > > processors, just the controller service implementation.
>> > > >
>> > > >
>> > > > On Tue, Mar 19, 2024 at 10:14 AM Mike Thomsen <
>> mikerthom...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> https://opensource.docs.scylladb.com/stable/using-scylla/drivers/cql-drivers/scylla-java-driver.html
>> > > > >
>> > > > > Directly quoting Scylla docs here:
>> > > > >
>> > > > > > The Scylla Java Driver is a drop-in replacement for the
>> DataStax Java
>> > > > > Driver. As such, no code changes are needed to use this driver.
>> > > > >
>> > > >

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

David,

Before we proceed, I think we should make sure we're all understanding the
same problem here. Starting with this:

> I believe the CQL protocol is backwards compatible but the Java API is
not.
> For example "com.datastax.driver.core.Session" is now
> "com.datastax.oss.driver.api.core.session.Session" and there is no more
> "Cluster" class. Might be fairly trivial to fix though, if that's the path
> of least resistance.

>From what I've learned using Cassandra 3 and 4 in my day job and reading up
on this stuff for the sake of discussion, that all tracks. We used the
~4.11 driver in Spring Boot on both v3 and v4 clusters without issue during
an upgrade. So I don't see any reason to factor in the "changes from
DataStax 3 to 4" since the changes were likely a one-off decision meant to
position the driver for better future support and stability.

TL;DR, we can dump v3 compatibility and the only thing our users will
notice is if we make the controller service totally incompatible with the
one they're already using which is something we can actively avoid.

On Tue, Mar 19, 2024 at 2:00 PM David Handermann <
exceptionfact...@apache.org> wrote:

> All,
>
> I support a Controller Service API abstraction around the Cassandra
> Driver. The changes from DataStax 3 to 4 already highlight the need
> for that abstraction. The donation of the DataStax Java driver to
> Apache [1] also shows the value of providing some level of isolation,
> if at all possible.
>
> I have not taken a close look at the Matt's branch, and the details of
> the abstraction are important, but having the abstraction can be
> useful to avoid getting back to this same situation.
>
> Regards,
> David Handermann
>
> [1] https://github.com/apache/cassandra-java-driver/
>
> On Tue, Mar 19, 2024 at 12:37 PM Mike Thomsen 
> wrote:
> >
> > Matt,
> >
> > I got that. My point was that the Java changes appear to be a one time
> > thing that DataStax did to make a better driver with a much more
> > future-proof API. Since Scylla tracks them as closely as possible, I
> > suspect that we don't need to plan for a bunch of abstraction to isolate
> > Java changes.
> >
> > On Tue, Mar 19, 2024 at 11:07 AM Steven Matison <
> steven.mati...@gmail.com>
> > wrote:
> >
> > > That was kinda where i got stuck and fell out on my branch/jira.  Mike
> and
> > > I wanted to make a new controller service , without backward
> compatibility;
> > > and remove the duplicate driver/connection properties found in some of
> the
> > > processors.
> > >
> > > I agree taking out all old stuff and making new controller service
> makes
> > > most sense.  4.x and 5.x should be mostly backwards compatible to 2&3.x
> > > with how it’s used within current processors.
> > >
> > >
> > >
> > > On Tue, Mar 19, 2024 at 10:49 AM Matt Burgess 
> > > wrote:
> > >
> > > > The abstraction is to isolate Java API changes, not protocol
> > > compatibility
> > > > Changing to the java-driver comes with a number of changes to the
> code
> > > (see
> > > > Steven's and my branches), if we can abstract that API it should
> lead to
> > > > more maintainable code in the future by not having to change any
> > > > processors, just the controller service implementation.
> > > >
> > > >
> > > > On Tue, Mar 19, 2024 at 10:14 AM Mike Thomsen <
> mikerthom...@gmail.com>
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > >
> > >
> https://opensource.docs.scylladb.com/stable/using-scylla/drivers/cql-drivers/scylla-java-driver.html
> > > > >
> > > > > Directly quoting Scylla docs here:
> > > > >
> > > > > > The Scylla Java Driver is a drop-in replacement for the DataStax
> Java
> > > > > Driver. As such, no code changes are needed to use this driver.
> > > > >
> > > > > On Tue, Mar 19, 2024 at 10:13 AM Mike Thomsen <
> mikerthom...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Matt,
> > > > > >
> > > > > > I don't think we need to really "abstract above" the drivers
> because
> > > > the
> > > > > > Java DataStax driver appears to support 4.X all the way back to
> 2.X,
> > > as
> > > > > > well as the enterprise versions from DataStax
> > > > > >
> > > > > >
>

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

Matt,

I got that. My point was that the Java changes appear to be a one time
thing that DataStax did to make a better driver with a much more
future-proof API. Since Scylla tracks them as closely as possible, I
suspect that we don't need to plan for a bunch of abstraction to isolate
Java changes.

On Tue, Mar 19, 2024 at 11:07 AM Steven Matison 
wrote:

> That was kinda where i got stuck and fell out on my branch/jira.  Mike and
> I wanted to make a new controller service , without backward compatibility;
> and remove the duplicate driver/connection properties found in some of the
> processors.
>
> I agree taking out all old stuff and making new controller service makes
> most sense.  4.x and 5.x should be mostly backwards compatible to 2&3.x
> with how it’s used within current processors.
>
>
>
> On Tue, Mar 19, 2024 at 10:49 AM Matt Burgess 
> wrote:
>
> > The abstraction is to isolate Java API changes, not protocol
> compatibility
> > Changing to the java-driver comes with a number of changes to the code
> (see
> > Steven's and my branches), if we can abstract that API it should lead to
> > more maintainable code in the future by not having to change any
> > processors, just the controller service implementation.
> >
> >
> > On Tue, Mar 19, 2024 at 10:14 AM Mike Thomsen 
> > wrote:
> >
> > >
> > >
> >
> https://opensource.docs.scylladb.com/stable/using-scylla/drivers/cql-drivers/scylla-java-driver.html
> > >
> > > Directly quoting Scylla docs here:
> > >
> > > > The Scylla Java Driver is a drop-in replacement for the DataStax Java
> > > Driver. As such, no code changes are needed to use this driver.
> > >
> > > On Tue, Mar 19, 2024 at 10:13 AM Mike Thomsen 
> > > wrote:
> > >
> > > > Matt,
> > > >
> > > > I don't think we need to really "abstract above" the drivers because
> > the
> > > > Java DataStax driver appears to support 4.X all the way back to 2.X,
> as
> > > > well as the enterprise versions from DataStax
> > > >
> > > > https://docs.datastax.com/en/driver-matrix/docs/java-drivers.html
> > > >
> > > > Similar situation with Scylla. When I looked at the driver, it
> appeared
> > > to
> > > > copy verbatim the entire public API of that driver. So I think before
> > we
> > > > dive into abstractions, it's worth doing a bit more validation of
> these
> > > > details. IMHO, this might be a much lighter lift than anticipated.
> > > >
> > > >
> > > > On Mon, Mar 18, 2024 at 4:30 PM Matt Burgess 
> > > wrote:
> > > >
> > > >> Totally agree, that's what my branch does (see link in previous
> > email).
> > > >> The
> > > >> more I work with it, the more I think I can abstract it further from
> > > their
> > > >> JDBC-like API but I started with a bunch of delegate classes then I
> > > figure
> > > >> I'll see where I can consolidate to more abstract concepts. If I
> don't
> > > >> have
> > > >> to support Cassandra 3 with the new API, so much the better.
> > > >>
> > > >> Regards,
> > > >> Matt
> > > >>
> > > >> On Mon, Mar 18, 2024 at 4:14 PM David Handermann <
> > > >> exceptionfact...@apache.org> wrote:
> > > >>
> > > >> > Matt et al,
> > > >> >
> > > >> > It is good to see the background effort on moving Cassandra
> > > >> > capabilities in a supportable direction.
> > > >> >
> > > >> > I think new Cassandra components will require a significant
> > departure
> > > >> > from current Controller Service abstractions. Right now, the
> > existing
> > > >> > service interface does not provide a clean abstraction from the
> > > >> > Cassandra library, which is part of the reason for the current
> > > >> > coupling to the legacy driver version.
> > > >> >
> > > >> > Following up from Joe's comments, it seems like the cleanest way
> > > >> > forward is to deprecate the current bundle on the 1.x branch, and
> > > >> > remove the current bundle from the main branch. That will provide
> a
> > > >> > clean slate for new Service and Processor implementations, without
> > > >> > concern for uncertain compatibility questio

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

https://opensource.docs.scylladb.com/stable/using-scylla/drivers/cql-drivers/scylla-java-driver.html

Directly quoting Scylla docs here:

> The Scylla Java Driver is a drop-in replacement for the DataStax Java
Driver. As such, no code changes are needed to use this driver.

On Tue, Mar 19, 2024 at 10:13 AM Mike Thomsen 
wrote:

> Matt,
>
> I don't think we need to really "abstract above" the drivers because the
> Java DataStax driver appears to support 4.X all the way back to 2.X, as
> well as the enterprise versions from DataStax
>
> https://docs.datastax.com/en/driver-matrix/docs/java-drivers.html
>
> Similar situation with Scylla. When I looked at the driver, it appeared to
> copy verbatim the entire public API of that driver. So I think before we
> dive into abstractions, it's worth doing a bit more validation of these
> details. IMHO, this might be a much lighter lift than anticipated.
>
>
> On Mon, Mar 18, 2024 at 4:30 PM Matt Burgess  wrote:
>
>> Totally agree, that's what my branch does (see link in previous email).
>> The
>> more I work with it, the more I think I can abstract it further from their
>> JDBC-like API but I started with a bunch of delegate classes then I figure
>> I'll see where I can consolidate to more abstract concepts. If I don't
>> have
>> to support Cassandra 3 with the new API, so much the better.
>>
>> Regards,
>> Matt
>>
>> On Mon, Mar 18, 2024 at 4:14 PM David Handermann <
>> exceptionfact...@apache.org> wrote:
>>
>> > Matt et al,
>> >
>> > It is good to see the background effort on moving Cassandra
>> > capabilities in a supportable direction.
>> >
>> > I think new Cassandra components will require a significant departure
>> > from current Controller Service abstractions. Right now, the existing
>> > service interface does not provide a clean abstraction from the
>> > Cassandra library, which is part of the reason for the current
>> > coupling to the legacy driver version.
>> >
>> > Following up from Joe's comments, it seems like the cleanest way
>> > forward is to deprecate the current bundle on the 1.x branch, and
>> > remove the current bundle from the main branch. That will provide a
>> > clean slate for new Service and Processor implementations, without
>> > concern for uncertain compatibility questions.
>> >
>> > Regards,
>> > David Handermann
>> >
>> > On Mon, Mar 18, 2024 at 2:35 PM Matt Burgess 
>> wrote:
>> > >
>> > > What do y'all think about removing the individual connection
>> properties
>> > > from the Cassandra processors for NiFi 2.0 and requiring a
>> > > CassandraSessionProvider instead? I think we started doing that
>> elsewhere
>> > > (Elasticsearch maybe?), I noticed duplicate code in the
>> > > CassandraSessionProvider and AbstractCassandraProcessor, if we keep
>> those
>> > > properties I can refactor them into a utility class.
>> > >
>> > > Thanks,
>> > > Matt
>> > >
>> > >
>> > > On Fri, Mar 15, 2024 at 2:44 PM Steven Matison <
>> steven.mati...@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > I got through quite a bit of work to enable 4.x…
>> > > >
>> > > > The 3.x pieces that were not backwards compatible is very edge use
>> > case and
>> > > > could have been done slightly differently but with work around.
>> > > >
>> > > > https://github.com/steven-matison/nifi/tree/nifi-10120-1
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Mar 15, 2024 at 2:30 PM Matt Burgess 
>> > wrote:
>> > > >
>> > > > > Oops used the wrong email address so if there have been responses
>> to
>> > the
>> > > > > Cassandra thread since mine I missed them, my bad!
>> > > > >
>> > > > > On Fri, Mar 15, 2024 at 2:00 PM Matt Burgess > >
>> > > > wrote:
>> > > > >
>> > > > > > I believe the CQL protocol is backwards compatible but the Java
>> > API is
>> > > > > > not. For example "com.datastax.driver.core.Session" is now
>> > > > > > "com.datastax.oss.driver.api.core.session.Session" and there is
>> no
>> > more
>> > > > > 

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

know
> > if
> > > > the
> > > > > >> 4.x drivers get rid of the vulnerable bits.
> > > > > >>
> > > > > >> Thanks
> > > > > >>
> > > > > >> On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess <
> > mattyb...@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > At the very least we should upgrade to Cassandra 3.11.6:
> > > > > >> >
> > > > >
> > https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt
> > > > > >> >
> > > > > >> > On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess <
> > mattyb...@apache.org>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > If the community agrees to get rid of Cassandra 3 that'll
> > save me
> > > > > >> effort
> > > > > >> > > on the refactor after I add Cassandra 4 :) Otherwise those
> > > > > >> > > vulnerabilities would only be in a "new" Cassandra 3
> services
> > NAR
> > > > > that
> > > > > >> > > would not be included in the convenience binary.
> > > > > >> > >
> > > > > >> > > On Fri, Mar 15, 2024 at 1:28 PM Joe Witt <
> joe.w...@gmail.com>
> > > > > wrote:
> > > > > >> > >
> > > > > >> > >> Mike, Matt,
> > > > > >> > >>
> > > > > >> > >> Happy to hear you both have active efforts or are
> interested
> > in
> > > > > doing
> > > > > >> > so.
> > > > > >> > >> Can you help me understand more specifically what that
> means
> > for
> > > > > the
> > > > > >> > >> current set of components?
> > > > > >> > >>
> > > > > >> > >> The CVE hits are concerning and long standing.  Supporting
> > > > > Cassandra
> > > > > >> 3
> > > > > >> > >> implies the current set of dependencies would remain too
> > right?
> > > > > >> > >>
> > > > > >> > >> Is the current set of components we have ones we want to
> > retain?
> > > > > We
> > > > > >> > >> certainly need Cassandra components - but are the ones we
> > have
> > > > now
> > > > > >> the
> > > > > >> > >> right ones?
> > > > > >> > >>
> > > > > >> > >> Thanks
> > > > > >> > >> Joe
> > > > > >> > >>
> > > > > >> > >> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess <
> > > > > mattyb...@apache.org>
> > > > > >> > >> wrote:
> > > > > >> > >>
> > > > > >> > >> > I'm actively working this, I pushed my branch up in case
> > anyone
> > > > > >> wants
> > > > > >> > to
> > > > > >> > >> > take a look [1]. The idea is to abstract the Cassandra
> API
> > "up
> > > > a
> > > > > >> > couple
> > > > > >> > >> > levels" and provide implementations for Cassandra 3, 4,
> and
> > > > > >> eventually
> > > > > >> > >> 5.
> > > > > >> > >> > For JDBC-like interfaces this is a PITA because of the
> API
> > > > > >> (Statement,
> > > > > >> > >> > PreparedStatement, BoundStatement, ResultSet, etc.) but
> I'm
> > > > > hoping
> > > > > >> we
> > > > > >> > >> can
> > > > > >> > >> > find a common pattern for abstracting the third-party
> > library
> > > > > >> > >> > implementation and API from the NiFi component
> (Processor,
> > > > > >> > >> > ControllerService, etc.) API. I think we're doing
> something
> > > > > similar
> > > > > >> > for
> > > > > >> > >> > Kafka?
> > &

Re: [discuss] What to do with the Cassandra components

[Mike Thomsen]

That’s been on my todo list for a little while but things kept coming up. I 
think I could get started on that now. Based on my initial research it appears 
that scylla uses the exact same api as datastax so supporting both in a cql 
bundle should theoretically be fairly easy.


Sent from my iPhone

> On Mar 14, 2024, at 6:18 PM, Joe Witt  wrote:
> 
> Team,
> 
> Cassandra remains a really important system to be able to send data to.
> However, it seems like we've not maintained these well.  We have what
> appears to be at least a full generation behind on client versions (we are
> on 3x vs 4x which is the latest stable with 5x apparently coming shortly).
> 
> We have components to send data, query data, and use Cassandra as a cache
> store.  We have older mechanisms for json/avro and publish mechanisms for
> records.
> 
> The libraries we do have depend on outdated versions of Guava and result in
> many CVE hits.
> 
> I am inclined to think we should deprecate the 1.x components and remove
> them as-is from the 2.x line.  Then re-introduce them with record only
> interfaces and built against the latest stable Cassandra/Datastax/ScyllaDB
> interfaces.
> 
> I'd love to hear thoughts from those closer to this space both as a user
> and developer so we can make good next steps.
> 
> Thanks

Re: [DISCUSS] New schema repository idea (with proof of concept)

[Mike Thomsen]

Sure. The JAR is not actually scanned. You have to use dynamic properties
to map a schema name to a particular fully qualified class. I would assume
that most people using it are just taking a JAR that was generated or
written by hand to be a client library with only (or mainly) POJOs
representing the model. You can see a simple example here:

https://github.com/MikeThomsen/nifi-pojo-schema-repository-bundle/tree/main/test-pojos/src/main/java/org/apache/nifi/pojo/complex

Nothing fancy per se. It's just POJOs with some standard annotations from
the Avro lib.

I would imagine this repo would be a big help for teams that can't/won't
commit to a contract first design but have to work with NiFi and other big
data systems.

On Thu, Mar 7, 2024 at 5:39 PM David Handermann 
wrote:

> Mike,
>
> Thanks for the reply. I agree that file and property-based registries
> are useful, so the main question seems to be a compiled-code-derived
> registry as you have described.
>
> It seems that the general use case could still be supported through
> file-backed registry, but without requiring the dynamic class loading
> associated with a custom JAR.
>
> Loading code from a JAR also presents greater security risks than
> loading schema files, so if this were to be supported, it would
> require additional permission restrictions.
>
> To help think through this a bit more, can you describe the use case a
> bit more? How would someone prepare a JAR for referencing in this
> proposed registry?
>
> Regards,
> David Handermann
>
> On Thu, Mar 7, 2024 at 4:30 PM Mike Thomsen 
> wrote:
> >
> > You raise some good points, but I think there's still ample room for
> > file-based schema registries within NiFi. With regard to the the edge
> cases
> > with schema generation, I think an argument can also be made for "not
> > letting the perfect be the enemy of the good."
> >
> > On Wed, Mar 6, 2024 at 9:34 AM David Handermann <
> exceptionfact...@apache.org>
> > wrote:
> >
> > > Mike,
> > >
> > > Thanks for raising this question, and providing the example repository.
> > >
> > > Although it sounds like a POJO-based repository could be useful in
> > > some cases, it does not seem like something that should be included
> > > for community support.
> > >
> > > Part of the value of a Schema Registry is a shared location for data
> > > description. Although supporting property or file-based Schema
> > > Registries is useful in NiFi itself, the general pattern is
> > > externalized storage and maintenance of schema definitions.
> > >
> > > From another angle, this could be similar to code-first versus
> > > contract-first API development. Each approach has its positives and
> > > negatives. When it comes to a Schema Registry, however, it seems like
> > > the contract needs to be defined outside of code.
> > >
> > > Introspecting JAR files also raises questions about what to include or
> > > exclude, and how to handle edge cases for certain class definitions.
> > > This seems like the more significant problem. For this reason, it
> > > seems better to rely on external operations to produce Avro schema
> > > definitions, rather than supporting that in NiFi itself.
> > >
> > > Those are my initial thoughts, perhaps others can provide additional
> > > perspective.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Sat, Mar 2, 2024 at 9:18 AM Mike Thomsen 
> > > wrote:
> > > >
> > > > I've had this project on the back burner for a while and wanted to
> share
> > > it
> > > > with the team. It's a schema repository implementation that is
> designed
> > > to
> > > > take a JAR file with POJOs and use Jackson's schema generation API to
> > > > generate Avro schemas from those on startup. It also uses (via
> Jackson)
> > > > Avro annotations to help specify particular implementation details
> where
> > > > necessary. The code can be found here. Haven't worked on it lately,
> but
> > > it
> > > > should easily run on 1.25:
> > > >
> > > > https://github.com/MikeThomsen/nifi-pojo-schema-repository-bundle
> > > >
> > > > I am planning to get the repo ready for a PR unless someone raises
> > > reasons
> > > > why including it might be a poor fit. I think for a lot of teams this
> > > might
> > > > be a killer feature because it would allow them to use Avro with
> existing
> > > > enterprise POJOs and stuff like that without having to write them by
> > > hand.
> > > >
> > > > Thoughts?
> > >
>

Re: [DISCUSS] New schema repository idea (with proof of concept)

[Mike Thomsen]

You raise some good points, but I think there's still ample room for
file-based schema registries within NiFi. With regard to the the edge cases
with schema generation, I think an argument can also be made for "not
letting the perfect be the enemy of the good."

On Wed, Mar 6, 2024 at 9:34 AM David Handermann 
wrote:

> Mike,
>
> Thanks for raising this question, and providing the example repository.
>
> Although it sounds like a POJO-based repository could be useful in
> some cases, it does not seem like something that should be included
> for community support.
>
> Part of the value of a Schema Registry is a shared location for data
> description. Although supporting property or file-based Schema
> Registries is useful in NiFi itself, the general pattern is
> externalized storage and maintenance of schema definitions.
>
> From another angle, this could be similar to code-first versus
> contract-first API development. Each approach has its positives and
> negatives. When it comes to a Schema Registry, however, it seems like
> the contract needs to be defined outside of code.
>
> Introspecting JAR files also raises questions about what to include or
> exclude, and how to handle edge cases for certain class definitions.
> This seems like the more significant problem. For this reason, it
> seems better to rely on external operations to produce Avro schema
> definitions, rather than supporting that in NiFi itself.
>
> Those are my initial thoughts, perhaps others can provide additional
> perspective.
>
> Regards,
> David Handermann
>
> On Sat, Mar 2, 2024 at 9:18 AM Mike Thomsen 
> wrote:
> >
> > I've had this project on the back burner for a while and wanted to share
> it
> > with the team. It's a schema repository implementation that is designed
> to
> > take a JAR file with POJOs and use Jackson's schema generation API to
> > generate Avro schemas from those on startup. It also uses (via Jackson)
> > Avro annotations to help specify particular implementation details where
> > necessary. The code can be found here. Haven't worked on it lately, but
> it
> > should easily run on 1.25:
> >
> > https://github.com/MikeThomsen/nifi-pojo-schema-repository-bundle
> >
> > I am planning to get the repo ready for a PR unless someone raises
> reasons
> > why including it might be a poor fit. I think for a lot of teams this
> might
> > be a killer feature because it would allow them to use Avro with existing
> > enterprise POJOs and stuff like that without having to write them by
> hand.
> >
> > Thoughts?
>

[DISCUSS] New schema repository idea (with proof of concept)

[Mike Thomsen]

I've had this project on the back burner for a while and wanted to share it
with the team. It's a schema repository implementation that is designed to
take a JAR file with POJOs and use Jackson's schema generation API to
generate Avro schemas from those on startup. It also uses (via Jackson)
Avro annotations to help specify particular implementation details where
necessary. The code can be found here. Haven't worked on it lately, but it
should easily run on 1.25:

https://github.com/MikeThomsen/nifi-pojo-schema-repository-bundle

I am planning to get the repo ready for a PR unless someone raises reasons
why including it might be a poor fit. I think for a lot of teams this might
be a killer feature because it would allow them to use Avro with existing
enterprise POJOs and stuff like that without having to write them by hand.

Thoughts?

Re: UpdateAttribute Failure Relationship

[Mike Thomsen]

How about a third option which is to provide three options:

1) Default - status quo, exceptions cause it to yield
2) Exception = moves forward to success w/ an error attribute, an error log
statement that triggers a bulletin, etc to let data manages know what's
happening.
3) Exception = moves to a failure relationship that is otherwise
autoterminated

On Thu, Feb 8, 2024 at 7:12 PM Matt Burgess  wrote:

> Mike's option #2 seems solid but would take a lot of work and there will
> always be inputs we don't account for. I support that work but in code
> sometimes we just do a "catch(Throwable)" just so it doesn't blow up. What
> about a subjectless "try" or "trycatch" function you can wrap around your
> whole expression? If no exception is thrown, the evaluated value will be
> returned but if one is thrown, you can provide some alternate value that
> you can check downstream. As this is optional it would retain the current
> behavior unless you use it, and then it takes the place of all those
> ifElse(isXYZValid()) calls we'd need throughout the expression.
>
> Regards,
> Matt
>
>
> On Wed, Feb 7, 2024 at 8:11 PM Phillip Lord 
> wrote:
>
> > IMO... UpdateAttribute has been around since the beginning of time, I
> can't
> > see adding a failure relationship. At the same time I understand the want
> > for such exceptions to be handled more gracefully rather than rolling
> back
> > indefinitely.
> > I'd vote in favor of considering Moser's option #2... and being able to
> > implement an "if this then that" logic within your flow.
> >
> > Also just thinking... for every UA failure you have to consider a good
> > failure-management strategy, which MIGHT add a lot of noise to the flow.
> > Something that might otherwise easily be identified in a downstream
> > component and/or database/etc.
> >
> > My 2 cents **
> > Phil
> >
> >
> >
> >
> >
> > On Wed, Feb 7, 2024 at 5:18 PM Adam Taft  wrote:
> >
> > > Or better, the failure relationship just doesn't even exist until the
> > > property "Has Failure Relationship" is set to True.  This involves
> > updating
> > > UpdateAttribute to have dynamic relationships (the failure
> relationships
> > > appearing on true), which isn't hard to do in processor code.
> > >
> > > This has the advantage of being backwards compatible for existing users
> > and
> > > allows the failure relationship to exist for new configurations.
> > Obviously
> > > the processor would need an update to catch Expression Language
> > exceptions
> > > and then route conditionally to failure.
> > >
> > > Just thinking out loud.
> > > /Adam
> > >
> > >
> > >
> > > On Wed, Feb 7, 2024 at 1:48 PM u...@moosheimer.com 
> > > wrote:
> > >
> > > > Hi Mike,
> > > >
> > > > How about the option of introducing a new property that decides
> whether
> > > to
> > > > route to the 'failure' relationship in the event of an error?
> > > > If this property is set to false, then the 'failure' relationship is
> > > > automatically set to 'terminate' (since nothing is routed there
> > anyway).
> > > >
> > > > Then everyone can decide whether and where they want to use this new
> > > > feature or not.
> > > > All other options would still be possible with such a solution.
> > > >
> > > > -- Uwe
> > > >
> > > > > Am 07.02.2024 um 22:15 schrieb Michael Moser :
> > > > >
> > > > > Hi Dan,
> > > > >
> > > > > This has been discussed in the past, as you found with those two
> Jira
> > > > > tickets.  Personally, I'm still not sure whether a new failure
> > > > relationship
> > > > > on UpdateAttribute in 2.0 is a good approach.  I have heard from
> some
> > > > > dataflow managers that would not want to go through their entire
> > graph
> > > > when
> > > > > upgrading to 2.0 and update every UpdateAttribute configuration.
> > > > >
> > > > > I have heard some alternatives to a 'failure' relationship that I
> > would
> > > > > like to share as options.
> > > > >
> > > > > 1) Add a new property to UpdateAttribute that controls whether a
> > > flowfile
> > > > > that causes an expression language exception either yields and
> rolls
> > > > back,
> > > > > or silently fails to update the attribute and sends the flowfile to
> > > > > success.  I personally don't like this, because the use case for
> > > "silent
> > > > > failure" seems really like a rarely needed edge case.
> > > > >
> > > > > 2) Identify all expression language methods that can throw an
> > exception
> > > > and
> > > > > document that fact in the Expression Language Guide (some methods
> > > already
> > > > > mention they can throw an "exception bulletin").  Then implement
> new
> > > > > expression methods to check if an expression could fail, and use
> that
> > > in
> > > > > UpdateAttribute advanced rules.  For example, if the format() and
> > > > > formatInstant() methods can fail on a negative number, we create a
> > new
> > > > > method such as isValidMilliseconds().  This already exists for some
> > > > cases,
> > > > > such as isJson() which can do a 

Re: UpdateAttribute Failure Relationship

[Mike Thomsen]

Obviously, three options in a drop down :-D

On Fri, Feb 9, 2024 at 7:49 AM Mike Thomsen  wrote:

> How about a third option which is to provide three options:
>
> 1) Default - status quo, exceptions cause it to yield
> 2) Exception = moves forward to success w/ an error attribute, an error
> log statement that triggers a bulletin, etc to let data manages know what's
> happening.
> 3) Exception = moves to a failure relationship that is otherwise
> autoterminated
>
> On Thu, Feb 8, 2024 at 7:12 PM Matt Burgess  wrote:
>
>> Mike's option #2 seems solid but would take a lot of work and there will
>> always be inputs we don't account for. I support that work but in code
>> sometimes we just do a "catch(Throwable)" just so it doesn't blow up. What
>> about a subjectless "try" or "trycatch" function you can wrap around your
>> whole expression? If no exception is thrown, the evaluated value will be
>> returned but if one is thrown, you can provide some alternate value that
>> you can check downstream. As this is optional it would retain the current
>> behavior unless you use it, and then it takes the place of all those
>> ifElse(isXYZValid()) calls we'd need throughout the expression.
>>
>> Regards,
>> Matt
>>
>>
>> On Wed, Feb 7, 2024 at 8:11 PM Phillip Lord 
>> wrote:
>>
>> > IMO... UpdateAttribute has been around since the beginning of time, I
>> can't
>> > see adding a failure relationship. At the same time I understand the
>> want
>> > for such exceptions to be handled more gracefully rather than rolling
>> back
>> > indefinitely.
>> > I'd vote in favor of considering Moser's option #2... and being able to
>> > implement an "if this then that" logic within your flow.
>> >
>> > Also just thinking... for every UA failure you have to consider a good
>> > failure-management strategy, which MIGHT add a lot of noise to the flow.
>> > Something that might otherwise easily be identified in a downstream
>> > component and/or database/etc.
>> >
>> > My 2 cents **
>> > Phil
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Feb 7, 2024 at 5:18 PM Adam Taft  wrote:
>> >
>> > > Or better, the failure relationship just doesn't even exist until the
>> > > property "Has Failure Relationship" is set to True.  This involves
>> > updating
>> > > UpdateAttribute to have dynamic relationships (the failure
>> relationships
>> > > appearing on true), which isn't hard to do in processor code.
>> > >
>> > > This has the advantage of being backwards compatible for existing
>> users
>> > and
>> > > allows the failure relationship to exist for new configurations.
>> > Obviously
>> > > the processor would need an update to catch Expression Language
>> > exceptions
>> > > and then route conditionally to failure.
>> > >
>> > > Just thinking out loud.
>> > > /Adam
>> > >
>> > >
>> > >
>> > > On Wed, Feb 7, 2024 at 1:48 PM u...@moosheimer.com > >
>> > > wrote:
>> > >
>> > > > Hi Mike,
>> > > >
>> > > > How about the option of introducing a new property that decides
>> whether
>> > > to
>> > > > route to the 'failure' relationship in the event of an error?
>> > > > If this property is set to false, then the 'failure' relationship is
>> > > > automatically set to 'terminate' (since nothing is routed there
>> > anyway).
>> > > >
>> > > > Then everyone can decide whether and where they want to use this new
>> > > > feature or not.
>> > > > All other options would still be possible with such a solution.
>> > > >
>> > > > -- Uwe
>> > > >
>> > > > > Am 07.02.2024 um 22:15 schrieb Michael Moser > >:
>> > > > >
>> > > > > Hi Dan,
>> > > > >
>> > > > > This has been discussed in the past, as you found with those two
>> Jira
>> > > > > tickets.  Personally, I'm still not sure whether a new failure
>> > > > relationship
>> > > > > on UpdateAttribute in 2.0 is a good approach.  I have heard from
>> some
>> > > > > dataflow managers that would not want to go through their entire
>> > graph
>> > >

Re: How to get files after a condition has been met

[Mike Thomsen]

I think the easiest way to do the conditional fetch would be to use
ConsumeKafka (or something equivalent for this use case like ListenHttp)
and FetchFile.

On Mon, Feb 5, 2024 at 4:23 PM Dan S  wrote:

> I have a requirement to load Json Schemas into the
> StandardJsonSchemaRegistry from the file system for use in validating JSON
> with the ValidateJson processor. I am trying to use the NIFI Rest API to
> update the StandardJsonSchemaRegistry. I am able to stop the ValidateJson
> processor and disable the StandardJsonSchemaRegistry using the NIFI Rest
> API. I do not know though how I can indicate either to GetFile or ListFile
> (to allow for retrieval of the JSON schemas) that ValidateJson has been
> stopped and StandardJsonSchemaRegistry has been disabled since both of
> these processors do not allow for an incoming relationship. How can I do
> this and if this is not possible how can I accomplish this?
>

Re: Removing JRuby?

[Mike Thomsen]

Matt,

I don't use JRuby, but the reason it raised my eyebrow a little was that
AFAIK JRuby is the complete opposite of Jython in terms of enterprise
readiness and parity with its C-based counterpart.

Edward,

The problem with Lua here is that the Java versions of Lua aren't being
maintained. I think the most recent release on any branch of them was 3
years ago. They're basically dead in the water as projects.

On Mon, Nov 6, 2023 at 2:57 PM Edward Armes  wrote:

> While I do like the ExecuteScript processors as they are great at digging
> you out of a hole. The performance of them isn't that great.
>
> That being said I would ve careful about dropping Lua support as there is a
> growing list of software that supports end user/administrator extensions
> via Lua for those that don't want to have to re-compile software
> themselves. On the other hand given that Jython doesn't yet have a Python 3
> implementation it could be argued dropping Jython support is a must given
> that the Python 2.x line is basically end of life.
>
> Now I wonder if its worth re-factoring the ExecuteScript processors to be
> per language implementations inheriting from a common base like a few
> processors do already.
>
> Edward
>
> On Mon, 6 Nov 2023, 16:24 Matt Burgess,  wrote:
>
> > I believe it is because in both ExecuteScript and ExecuteGroovyScript
> > you can do "regular" groovy, but EGS has extra built-ins such as easy
> > access to controller services, Groovy SQL stuff, etc, and we could
> > keep building it out. But IMO we'd have to port the rest of the
> > scripted components (ScriptedReader/Writer, etc.) over to the Groovy
> > bundle and make sure there's a drop-in replacement in the Python stuff
> > before we'd want to deprecate the scripted bundle.
> >
> > On the JRuby front, is that something you use actively? This question
> > is for you and the entire community of course.
> >
> > Regards,
> > Matt
> >
> > On Mon, Nov 6, 2023 at 7:12 AM Mike Thomsen 
> > wrote:
> > >
> > > If we deprecate ExecuteScript, I think we need to have groovyx be ready
> > to
> > > function as a drop-in replacement if it's not there already.
> > >
> > > On Sun, Nov 5, 2023 at 9:21 PM Matt Burgess 
> > wrote:
> > >
> > > > IIRC the removal of these engines was mostly due to lack of use or at
> > > > least the perception thereof. If JRuby is being used by the community
> > > > actively, I'm happy to revisit that discussion. Luaj's JSR-223
> > > > interface left something to be desired, but JRuby just needed a
> system
> > > > variable set or something like that.
> > > >
> > > > For the groovyx bundle, because it is Groovy-specific I tend to think
> > > > we could make better use of that than ExecuteScript, especially if we
> > > > do get rid of all the engines. We have a Groovy-specific processor, a
> > > > "real" Python SDK, and no more Nashorn. Perhaps we move all the
> > > > scripted components to the Groovy bundle, although I believe some
> > > > folks still make use of Jython for these. Of course if we reinstate
> > > > JRuby for ExecuteScript it's probably best to keep things the way
> they
> > > > are, or create a jruby bundle. The original scripting bundle was
> > > > aiming to support several engines, but if it turns out only one or
> two
> > > > will be useful, it may not be worth shoehorning all that JSR-223
> logic
> > > > when engine-specific components could be simpler, more easily
> > > > maintained, and allow for the idioms of the language to be better
> used
> > > > (as is done in the groovyx bundle).
> > > >
> > > > Just my two cents, looking forward to everyone's thoughts!
> > > >
> > > > - Matt
> > > >
> > > > On Sun, Nov 5, 2023 at 8:31 PM Mike Thomsen 
> > > > wrote:
> > > > >
> > > > > https://issues.apache.org/jira/browse/NIFI-11646
> > > > >
> > > > > I get the removal of Lua, but not the removal of JRuby. It's a
> clean
> > > > > reimplementation of Ruby native to the JVM and AFAICT is pound for
> > pound
> > > > as
> > > > > actively maintained as Groovy.
> > > > >
> > > > > Also, at this point, does it make sense to even keep the groovyx
> > bundle
> > > > > rather than deprecate it for 2.X?
> > > >
> >
>

Re: Removing JRuby?

[Mike Thomsen]

If we deprecate ExecuteScript, I think we need to have groovyx be ready to
function as a drop-in replacement if it's not there already.

On Sun, Nov 5, 2023 at 9:21 PM Matt Burgess  wrote:

> IIRC the removal of these engines was mostly due to lack of use or at
> least the perception thereof. If JRuby is being used by the community
> actively, I'm happy to revisit that discussion. Luaj's JSR-223
> interface left something to be desired, but JRuby just needed a system
> variable set or something like that.
>
> For the groovyx bundle, because it is Groovy-specific I tend to think
> we could make better use of that than ExecuteScript, especially if we
> do get rid of all the engines. We have a Groovy-specific processor, a
> "real" Python SDK, and no more Nashorn. Perhaps we move all the
> scripted components to the Groovy bundle, although I believe some
> folks still make use of Jython for these. Of course if we reinstate
> JRuby for ExecuteScript it's probably best to keep things the way they
> are, or create a jruby bundle. The original scripting bundle was
> aiming to support several engines, but if it turns out only one or two
> will be useful, it may not be worth shoehorning all that JSR-223 logic
> when engine-specific components could be simpler, more easily
> maintained, and allow for the idioms of the language to be better used
> (as is done in the groovyx bundle).
>
> Just my two cents, looking forward to everyone's thoughts!
>
> - Matt
>
> On Sun, Nov 5, 2023 at 8:31 PM Mike Thomsen 
> wrote:
> >
> > https://issues.apache.org/jira/browse/NIFI-11646
> >
> > I get the removal of Lua, but not the removal of JRuby. It's a clean
> > reimplementation of Ruby native to the JVM and AFAICT is pound for pound
> as
> > actively maintained as Groovy.
> >
> > Also, at this point, does it make sense to even keep the groovyx bundle
> > rather than deprecate it for 2.X?
>

Removing JRuby?

[Mike Thomsen]

https://issues.apache.org/jira/browse/NIFI-11646

I get the removal of Lua, but not the removal of JRuby. It's a clean
reimplementation of Ruby native to the JVM and AFAICT is pound for pound as
actively maintained as Groovy.

Also, at this point, does it make sense to even keep the groovyx bundle
rather than deprecate it for 2.X?

Default parameter storage

[Mike Thomsen]

I'm trying to track down the parameter sync issue I posted earlier.

My understanding is that the parameters are stored in the Registry unless
you configure your own parameter provider. Is that right? We're just using
a Registry instance without any custom providers. Trying to figure out why
we did not get the new parameters on the base context to sync.

Thanks,

Mike

Re: discuss NiFi 1.24 release

[Mike Thomsen]

This Xodus?

https://github.com/JetBrains/xodus

On Thu, Oct 12, 2023 at 1:30 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Mike,
>
> Thanks for raising this question. I am working on an automated
> migration for the support branch from H2 to Xodus. We previously
> handled automated migration from H2 1.4 to 2.1, and 2.1 to 2.2, so the
> basic mechanics are in place to extract content from H2. The general
> upgrade path will be to migrate to the latest version 1 release, and
> then upgrade to version 2.
>
> Regards,
> David Handermann
>
> On Thu, Oct 12, 2023 at 12:24 PM Mike Thomsen 
> wrote:
> >
> > When H2 goes, what will the upgrade path look like?
> >
> > On Tue, Oct 10, 2023 at 7:57 PM David Handermann <
> > exceptionfact...@apache.org> wrote:
> >
> > > Joe,
> > >
> > > Thanks for the reply, that sounds good.
> > >
> > > For reference, here is the Jira issue for tracking the initial
> > > implementation:
> > >
> > > https://issues.apache.org/jira/browse/NIFI-12206
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Tue, Oct 10, 2023 at 6:53 PM Joe Witt  wrote:
> > > >
> > > > David
> > > >
> > > > I think we can hold off for a few weeks - I'll respond to the Slack
> > > message
> > > > on that.
> > > >
> > > > Will be sad to see H2 go.  The original nifi flowfile repository ran
> on
> > > > H2.  Surprisingly fast and stable actually.  But happy to hear there
> is
> > > > such progress underway.
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Oct 10, 2023 at 4:27 PM David Handermann <
> > > > exceptionfact...@apache.org> wrote:
> > > >
> > > > > Joe,
> > > > >
> > > > > Thanks for initiating the discussion on a 1.24 release. There are a
> > > > > handful of new features, plus additional deprecations and
> dependency
> > > > > upgrades that would be very useful to release sooner rather than
> > > > > later.
> > > > >
> > > > > If we are considering a 1.24 release within the next week, then I
> > > > > would expect needing a 1.25 release to incorporate some additional
> > > > > deprecations around the same time we are ready for a 2.0 milestone
> > > > > release version.
> > > > >
> > > > > I am in the process of implementing a replacement for H2 to store
> flow
> > > > > configuration history, and I plan to have a pull request ready in
> the
> > > > > next few days. There will be some additional work to support
> migration
> > > > > on version 1 support branch.
> > > > >
> > > > > If we want to move forward with a 1.24 release within a week or so,
> > > > > then these changes could be targeted for 1.25. If we want to wait a
> > > > > few weeks, then I could see this being incorporated in 1.24.
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > >
> > > > > On Tue, Oct 10, 2023 at 6:14 PM Mark Payne 
> > > wrote:
> > > > > >
> > > > > > Thanks Joe. I wouldn’t argue against doing a 1.24 release.
> > > > > >
> > > > > > Thanks
> > > > > > -Mark
> > > > > >
> > > > > > > On Oct 10, 2023, at 7:01 PM, Joe Witt 
> wrote:
> > > > > > >
> > > > > > > Team,
> > > > > > >
> > > > > > > We have plenty of bits out there to kick a release.  I'm happy
> to
> > > RM
> > > > > this
> > > > > > > if nobody else wants to.
> > > > > > >
> > > > > > > Had a user on Slack ask for it.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Joe
> > > > > >
> > > > >
> > >
>

Re: discuss NiFi 1.24 release

[Mike Thomsen]

When H2 goes, what will the upgrade path look like?

On Tue, Oct 10, 2023 at 7:57 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Joe,
>
> Thanks for the reply, that sounds good.
>
> For reference, here is the Jira issue for tracking the initial
> implementation:
>
> https://issues.apache.org/jira/browse/NIFI-12206
>
> Regards,
> David Handermann
>
> On Tue, Oct 10, 2023 at 6:53 PM Joe Witt  wrote:
> >
> > David
> >
> > I think we can hold off for a few weeks - I'll respond to the Slack
> message
> > on that.
> >
> > Will be sad to see H2 go.  The original nifi flowfile repository ran on
> > H2.  Surprisingly fast and stable actually.  But happy to hear there is
> > such progress underway.
> >
> > Thanks
> >
> > On Tue, Oct 10, 2023 at 4:27 PM David Handermann <
> > exceptionfact...@apache.org> wrote:
> >
> > > Joe,
> > >
> > > Thanks for initiating the discussion on a 1.24 release. There are a
> > > handful of new features, plus additional deprecations and dependency
> > > upgrades that would be very useful to release sooner rather than
> > > later.
> > >
> > > If we are considering a 1.24 release within the next week, then I
> > > would expect needing a 1.25 release to incorporate some additional
> > > deprecations around the same time we are ready for a 2.0 milestone
> > > release version.
> > >
> > > I am in the process of implementing a replacement for H2 to store flow
> > > configuration history, and I plan to have a pull request ready in the
> > > next few days. There will be some additional work to support migration
> > > on version 1 support branch.
> > >
> > > If we want to move forward with a 1.24 release within a week or so,
> > > then these changes could be targeted for 1.25. If we want to wait a
> > > few weeks, then I could see this being incorporated in 1.24.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Tue, Oct 10, 2023 at 6:14 PM Mark Payne 
> wrote:
> > > >
> > > > Thanks Joe. I wouldn’t argue against doing a 1.24 release.
> > > >
> > > > Thanks
> > > > -Mark
> > > >
> > > > > On Oct 10, 2023, at 7:01 PM, Joe Witt  wrote:
> > > > >
> > > > > Team,
> > > > >
> > > > > We have plenty of bits out there to kick a release.  I'm happy to
> RM
> > > this
> > > > > if nobody else wants to.
> > > > >
> > > > > Had a user on Slack ask for it.
> > > > >
> > > > > Thanks
> > > > > Joe
> > > >
> > >
>

Re: [discuss] nifi 2.0 and Java 21…

[Mike Thomsen]

+1 100%

On Wed, Sep 6, 2023 at 11:48 AM Adam Taft  wrote:

> Yes, please. +1 Exactly what Mark said. Virtual threads have potential to
> be extremely impactful to applications like NiFi.
>
> /Adam
>
> On Wed, Sep 6, 2023 at 7:26 AM Mark Payne  wrote:
>
> > Thanks for bringing his up, Joe.
> >
> > I would definitely be a +1. I think the new virtual thread concept would
> > have great impact on us.
> > It would allow us to significantly simplify our scheduling logic, which
> > would help with code maintainability
> > but would also make configuration simpler. This is one of the most
> > difficult things for users to configure,
> > and I would very much welcome the ability to simplify this. It would
> > likely also yield better off-heap memory
> > utilization by reducing the number of native threads necessary.
> >
> > Thanks
> > -Mark
> >
> >
> > > On Sep 6, 2023, at 10:20 AM, Joe Witt  wrote:
> > >
> > > Team
> > >
> > > Thought it might be worth relighting this thread with Java 21 GA
> > imminent.
> > > Given the timing we should give consideration to having Java 21 as the
> > > basis for nifi 2.x to buy maximum time with LTS alignment.  There are
> > also
> > > a couple interesting language features we can likely take advantage of.
> > >
> > > What do you think?
> > >
> > > Thanks
> > > Joe
> > >
> > > On Wed, Jun 21, 2023 at 6:21 AM David Handermann <
> > > exceptionfact...@apache.org> wrote:
> > >
> > >> Hi Dirk,
> > >>
> > >> Thanks for summarizing your findings in the referenced Jira issues. It
> > >> sounds like subsequent discussion of Nashorn support may be better on
> a
> > new
> > >> thread.
> > >>
> > >> The Spring 6 and Jetty 11 upgrades are going to require significant
> > work.
> > >> One incremental step in that direction was making Java 17 the minimum
> > >> version, and upgrading to Jetty 10 should also help move things
> forward.
> > >>
> > >> Although compiling NiFi modules with a reference to the standalone
> > Nashorn
> > >> library may introduce issues, there should be other options for
> > referencing
> > >> the library at runtime. That requires custom class loading, which some
> > >> Processors support, so that seems like the general direction to go.
> > >>
> > >> If you have additional findings, feel free to start a new developer
> list
> > >> thread and that may gather additional feedback.
> > >>
> > >> Regards,
> > >> David Handermann
> > >>
> > >> On Wed, Jun 21, 2023 at 12:17 AM Dirk Arends <
> dirk.are...@fontis.com.au
> > >
> > >> wrote:
> > >>
> > >>> Since initially raising concerns about the move to Java 17 losing
> > >> Nashorn,
> > >>> I have been investigating the suggestion to use Nashorn as a
> standalone
> > >>> package as potential easier alternative to GraalVM. [1]
> > >>>
> > >>> While making some progress, a number of issues have been encountered
> > >> which
> > >>> I haven't been able to resolve as yet. More details are included in
> > >>> relevant JIRA tickets, but summarising:
> > >>>
> > >>> - Building NiFi with a recent Nashorn dependency leads to errors
> > >>> "Unsupported class file major version 61" [2]
> > >>> - Building NiFi using Java 17 highlights issues with the current
> Jetty
> > >>> version, which I believe would require an upgrade from 9.4.51 to
> > 11.0.15
> > >>> [3]
> > >>> - Jetty 11 then requires an upgrade of the Spring Framework version 5
> > to
> > >> 6.
> > >>> [4]
> > >>>
> > >>> The current steps to remove references to "Javascript" as a
> > preinstalled
> > >>> scripting language [5] are understandable, but it does seem there is
> > >> still
> > >>> quite a bit to do before Nashorn or another external scripting engine
> > >> could
> > >>> be used.
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/NIFI-11700: Java 17
> Nashorn
> > >>> standalone support
> > >>> [2] https://issues.apache.org/jira/browse/NIFI-11701: Support
> building
> > >>> with
> > >>> version 61 class files
> > >>> [3] https://issues.apache.org/jira/browse/NIFI-11702: Upgrade Jetty
> to
> > >>> version 11
> > >>> [4] https://issues.apache.org/jira/browse/NIFI-11703: Upgrade Spring
> > >>> Framework to version 6
> > >>> [5] https://issues.apache.org/jira/browse/NIFI-11713: Remove
> > Deprecated
> > >>> ECMAScript Support
> > >>>
> > >>> Regards,
> > >>> Dirk Arends
> > >>>
> > >>
> >
> >
>

Validation on white spaces in record api

[Mike Thomsen]

I lost a lot of time this morning debugging an issue that turned out to
just be a trailing whitespace char at the end of an alias value. That seems
like a good place to put extra validation in the record api; I can't
imagine a legit use case where someone would intentionally put trailing or
leading whitespaces in their aliases or name fields. Anyone see counter
points on this? IMHO even if it's "legal Avro," I don't see a reason for us
to be bound to that.

Re: Consultation on Learning and Development in Apache NiFi

[Mike Thomsen]

When you say create connectors, do you mean create custom NiFi components?

On Thu, Aug 17, 2023 at 11:46 AM Frank Mansilla 
wrote:

> Dear members of the Apache NiFi community,
>
> I hope this message finds you well. My name is Frank Mansilla and I am a
> data analyst based in Bolívar, Buenos Aires, Argentina. I would like to
> express my enthusiasm to learn and enter the world of Apache NiFi. I have
> been exploring the capabilities of NiFi and have found it to be an
> incredibly powerful tool for data processing and flow automation.
>
> So far, I have managed to understand the basics of NiFi and have developed
> some simple flows. However, my goal is to deepen my knowledge and skills so
> that I can create more complex flows and take advantage of all the advanced
> features that NiFi offers.
>
> I have a great interest in learning how to create and use connectors to
> integrate different systems and data sources into my flows. Also, I would
> like to fully understand the concept of Schema Registry and how I can apply
> it in my projects to ensure data consistency.
>
> I would greatly appreciate it if you could provide me with guidance,
> resources, and recommendations to learn and further my learning journey
> with Apache NiFi. If there are any tutorials, detailed documentation
> (besides the official documentation), examples of complex flows, or other
> resources that you find useful to someone in my position, I'd be happy to
> hear about them.
>
> Also, if there are events, conferences, webinars, or other spaces where I
> can interact with the community and learn from NiFi experts, I would be
> very interested in participating.
>
> I thank you in advance for your time and support. I am excited for the
> opportunity to grow and develop in this exciting field. I look forward to
> your response and the chance to learn alongside this amazing community.
>
> Thanks and best regards!
>
> Frank Mansilla
> frankmansilla1...@gmail.com
> Bolivar, Buenos Aires, Argentina
>

Re: Processing JSON Files Embedded with Base64 Encoded Files

[Mike Thomsen]

Can you be more specific on what you're trying to accomplish since the
RecordPath API has base64 encode/decode functions for stuff like decoding
inline base64 content?

On Tue, Aug 15, 2023 at 9:45 AM Reid, Chris 
wrote:

> Hello.
>
> I understand that there's currently no Apache NiFi processor that can
> process JSON files embedded with Base64 encoded files.
>
> I'd like to know if it's possible to develop a custom Apache NiFi
> processor to process JSON files embedded with Base64 files?
>
> Thank you.
>
> Chris
>

Re: Java 17 features in 2.x

[Mike Thomsen]

In a similar vein, at some point we might want to start experimenting with
trying to get NiFi to run Graal (without AOT compilation).

On Mon, Aug 7, 2023 at 11:31 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Thanks for the additional replies, I think the distinction between new
> features for the next version and bug fixes for both versions is a helpful
> approach.
>
> Having contributed and backported a number of changes, I have been more
> focused on compatibility, but I completely agree that we should take
> advantage of new features on the main branch. We have already started doing
> this in terms of certain libraries, and more work is in progress.
>
> Regards,
> David Handermann
>
> On Mon, Aug 7, 2023 at 7:54 PM Mark Bean  wrote:
>
> > I was working on a very similar response. Thanks Joe for the clear
> > articulation. I agree with this approach 100%.
> >
> > Adding one additional point, the differentiation of Java 17 in the NiFi
> 2.0
> > line may encourage and expedite the migration to a new major release. I
> > think we want to get there as soon as possible rather than continue the
> > burdensome effort of supporting new main lines of code.
> >
> >
> > On Mon, Aug 7, 2023 at 8:33 PM Joe Witt  wrote:
> >
> > > The views shared thus far are certainly reasonable but I do want to add
> > > a different take.
> > >
> > > The reason we want to do major releases from time to time is so that we
> > can
> > > take advantage of leaps in the language and frameworks we rely on.  To
> > that
> > > end it would seem unfortunate to not start aggressively taking
> advantage
> > of
> > > that.  In particular we've been held to Java 8 standards for at least 7
> > > years.  I would advocate we allow and even encourage usage of Java 17
> > > features/syntax to help move forward.
> > >
> > > The point about backporting is important and I agree the 'easiest' way
> is
> > > if changes made to main are backportable.  Then again we don't really
> > need
> > > everything to be backportable and for sure that will start happening
> less
> > > and less.  If we're talking about 'bug fixes' then it probably makes
> > sense
> > > to prefer for now they avoid Java 17 assuming a given fix should target
> > > both 2.x and 1.x lines.  But if we're talking about new features or
> even
> > > improvements I think we should be free to move on to Java 17
> > > idioms/benefits.  If a contrib does this then it just won't go to the
> 1.x
> > > line.  This atrophy is natural/ok I think and lets us give the 2.x line
> > the
> > > attention/growth it deserves.
> > >
> > > Thanks
> > >
> > > On Mon, Aug 7, 2023 at 2:19 PM David Handermann <
> > > exceptionfact...@apache.org>
> > > wrote:
> > >
> > > > Mike,
> > > >
> > > > Thanks for raising the question. Following Matt's comments, I
> recommend
> > > > minimizing use of Java 17 features to make it easier to backport
> > changes
> > > > for now.
> > > >
> > > > Incremental adjustments can be done when backporting, but it requires
> > the
> > > > author and reviewer to pay careful attention since the GitHub
> automated
> > > > builds for the main branch run on Java 17.
> > > >
> > > > As Matt recommended, the alternative is to provide separate pull
> > requests
> > > > for the main and support branches.
> > > >
> > > > We already have a few Java 11 and 17 references on the main branch
> for
> > > > things like List.of(), and most of these are easy to adjust when
> > > > backporting, but they do require careful attention.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Mon, Aug 7, 2023 at 4:04 PM Matt Burgess 
> > > wrote:
> > > >
> > > > > In my opinion that's ok, but I think it would be helpful if a PR is
> > > > > going to be backported to support/nifi-1.x that the PR author
> > provides
> > > > > two PRs, one against main with Java 17 features and one against
> > > > > support/nifi-1.x with Java 8 features. That being said, allowing
> Java
> > > > > 17 features may make maintenance tougher while there's an active
> 1.x
> > > > > branch. Maybe we should wait until we only support NiFi 2.x?
> > > > >
> > > > > On Mon, Aug 7, 2023 at 4:59 PM Mike Thomsen <
> mikerthom...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Since we're standardizing on 17, we're free and clear to use Java
> > 17
> > > > > > features, right?
> > > > >
> > > >
> > >
> >
>

Java 17 features in 2.x

[Mike Thomsen]

Since we're standardizing on 17, we're free and clear to use Java 17
features, right?

H2 CVE

[Mike Thomsen]

A colleague found this "CVE" report for H2. I agree with the H2 devs that
it's a big joke of a CVE, but it's something we might want to add something
to the documentation to discuss because it could cause grief for our users.

https://github.com/h2database/h2database/issues/3686

Re: Possible Docker bug in 1.13.2

[Mike Thomsen]

> Because I have custom processors that cannot be rebuilt (source-code
gone)

IntelliJ can probably give you a close version of the source code if you
expand the JAR and drag the class files for your custom code into the IDE.

On Wed, Jun 7, 2023 at 5:32 PM Russell Bateman 
wrote:

> I downloaded sources to 1.13.2 in order to hand-spin my own container
> image. When I got down to
> /nifi-1.13.2/nifi-docker/dockerhub/Dockerfile/, I found:
>
> ...
> ARG NIFI_VERSION=1.13.1
> ...
>
> and the version is also wrong in /DockerImage.txt/ which
> /DockerBuild.sh/ consumes.
>
> Indeed, the image that is built appears to be versioned 1.13.1 and not
> 1.13.2 when listing local Docker repository images:
>
> *REPOSITORY TAG IMAGE ID   CREATED  SIZE*
> apache/nifi 1.13.1  8c18038f152a   30 minutes ago   2.06GB
>
> Why am I juggling so ancient a version? Because I have custom processors
> that cannot be rebuilt (source-code gone) and will not run on 1.14.0 and
> later.
>
> Russ

Re: [discuss] nifi 2.0 and Java 17...

[Mike Thomsen]

I'm all in favor of making 17 the requirement for 2.0

On a related note, I was asked by one of our devops guys if 17 was a good
idea for 1.X. I know we support it, but is it considered just as good as 11
for 1.21+?

On Wed, May 31, 2023 at 6:15 PM Kevin Doran  wrote:

> Yeah I agree it makes sense to move to 17… as long as we are introducing a
> potential disruptive migration, we should move to the version we can
> support longer rather than 11 which will land us back in the same situation
> we were in with 8 EOL
> 
> From: Pierre Villard 
> Sent: Wednesday, May 31, 2023 4:10:47 PM
> To: dev@nifi.apache.org 
> Subject: Re: [discuss] nifi 2.0 and Java 17...
>
> Hey Joe,
>
> I'd recommend doing the move right now and say that NiFi 2.0 requires Java
> 17. And we would focus on Java 21 with NiFi 3.0.
> I honestly don't see any value in absolutely keeping support for Java 11
> right now.
>
> Pierre
>
> Le mer. 31 mai 2023 à 19:22, Joe Witt  a écrit :
>
> > Team,
> >
> > We've discussed in the past having NiFi 2.0 move from Java 8 to Java 11
> as
> > the required minimum while also working on Java 17.
> >
> > As we move on in time though we are now 4 months (Sept) from. Java 11
> > openJDK going end of support.  Meanwhile, the Spring 5.x line goes end of
> > support as of next year and Spring 6.x requires Java 17.  Also Java 21
> > comes out in Sept as well and is already the next LTS release.
> >
> > I am increasingly of the view that we should seriously discuss/consider
> > moving to Java 17 as our basis for NiFi 2.0 as otherwise it basically
> means
> > we'll be forced to move to NiFi 3.0 quite quickly.
> >
> > We already know we can build and run on Java 17 so we're good there.
> We'll
> > soon want to do the same for Java 21 ... and the more 'old stuff' we hold
> > on to the harder it is.
> >
> > Who will be seriously impacted by the removal of Java 11 and what was
> your
> > plan for upgrading to Java 17?
> >
> > thoughts?
> >
> > Thanks
> >
>

Re: Help in creating a custom record reader

[Mike Thomsen]

Relevant reading:

https://github.com/apache/nifi/tree/main/nifi-commons/nifi-record
https://github.com/apache/nifi/tree/main/nifi-commons/nifi-record-path
https://github.com/apache/nifi/tree/main/nifi-commons/nifi-schema-utils
https://github.com/apache/nifi/tree/main/nifi-nar-bundles/nifi-standard-services
https://github.com/apache/nifi/tree/main/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-service-api
https://github.com/apache/nifi/tree/main/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle

Those paths are where you will find the implementation details for
everything from the Record components to the core reader implementations.

On Mon, May 15, 2023 at 8:54 AM Mike Thomsen  wrote:

> You would need to create a new Maven project that has a root POM and two
> modules (one for the JAR and one for the NAR). You can find examples of
> that project structure in our repository or in a lot of community repos on
> GitHub.
>
> From there, you'd implement (going on memory here without the JavaDocs) a
> RecordReader and a RecordReaderFactory. For the Record objects themselves,
> the basic implementation you can use for that is a MapRecord which is just
> a wrapper around a Java Map with a Schema object associated with it.
>
> On Thu, May 11, 2023 at 10:29 AM Abhik Lodh 
> wrote:
>
>> Hi,
>> I am writing this mail because I wanted to create a custom Record Reader
>> in NiFi that parses ASCII files (or any other currently unsupported file
>> format) but I’m having a very hard time creating a custom controller
>> service that does this while integrating with ConvertRecord, UpdateRecord
>> and ValidateRecord just like other RecordReaders.
>>
>> Could anyone help me out with this?
>>
>> Thanks and regards,
>> Abhik Lodh
>>
>

Re: Help in creating a custom record reader

[Mike Thomsen]

You would need to create a new Maven project that has a root POM and two
modules (one for the JAR and one for the NAR). You can find examples of
that project structure in our repository or in a lot of community repos on
GitHub.

>From there, you'd implement (going on memory here without the JavaDocs) a
RecordReader and a RecordReaderFactory. For the Record objects themselves,
the basic implementation you can use for that is a MapRecord which is just
a wrapper around a Java Map with a Schema object associated with it.

On Thu, May 11, 2023 at 10:29 AM Abhik Lodh 
wrote:

> Hi,
> I am writing this mail because I wanted to create a custom Record Reader
> in NiFi that parses ASCII files (or any other currently unsupported file
> format) but I’m having a very hard time creating a custom controller
> service that does this while integrating with ConvertRecord, UpdateRecord
> and ValidateRecord just like other RecordReaders.
>
> Could anyone help me out with this?
>
> Thanks and regards,
> Abhik Lodh
>

Write access to Confluence

[Mike Thomsen]

Could someone with permissions to do it, grant me (mthomsen) write access
to our Confluence pages?

Thanks,

Mike

Deprecating the use of older Java date formatting code

[Mike Thomsen]

A new contributor raised some good questions on a PR:

https://github.com/apache/nifi/pull/7169

I rebased this branch as we discussed in the original PR, but I see that
meantime NiFi code become much more dependent on DateFormat. I see many
public methods in DataTypeUtils taking or returning DateFormat. I see some
transition methods like DateTypeUtils.getDateTimeFormatter. It makes me
feel that:


   - I don't see the wider picture of where you already use DateFormat now
  and how you planning to use it in the future
  - I don't know what is the plan to get rid of DateFormat because it
  looks like there is some
  - I don't know if you want to use a lenient, case insensitive format
  in all these places or some other one

If you want to draw me a wider picture I can try to continue this change
but otherwise, I don't want to introduce some ambiguous behaviour in date
format handling so I'll close this PR.


For NiFi 2.0, I think I would favor eliminating all pre-Java 8 Date/Time
API code and making it a blocker for new PRs to introduce functionality
that uses the old APIs. Thoughts?


Mike

Re: Are all 1.21 NARs uploaded as Maven artifacts?

[Mike Thomsen]

By it, I mean this:
https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-azure-nar/1.21.0/nifi-azure-nar-1.21.0.nar

On Mon, Apr 10, 2023 at 1:07 PM Mike Thomsen  wrote:

> When I clicked on it, I got a 404. Weird.
>
> On Mon, Apr 10, 2023 at 12:53 PM Bryan Bende  wrote:
>
>> The Azure NAR looks like it is there:
>>
>>
>> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-azure-nar/1.21.0/
>>
>> On Mon, Apr 10, 2023 at 12:40 PM Mike Thomsen 
>> wrote:
>> >
>> > It’s failing on the azure one which is why I am raising the issue. I
>> know we are still pushing that because it’s part of the convenient binary.
>> >
>> > Sent from my iPhone
>> >
>> > > On Apr 10, 2023, at 12:26 PM, Joe Witt  wrote:
>> > >
>> > > Hello
>> > >
>> > > Pretty sure all the normal stuff/paths followed and uploaded.  It
>> might be
>> > > that we're not pushing Atlas nars anymore?
>> > >
>> > > I dont recall but that is possible.
>> > >
>> > > Thanks
>> > >
>> > >> On Mon, Apr 10, 2023 at 9:21 AM Mike Thomsen 
>> wrote:
>> > >>
>> > >> Getting this error on two different machines on two different
>> networks:
>> > >>
>> > >> ownloaded from central:
>> > >>
>> > >>
>> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-sql-reporting-nar/1.21.0/nifi-sql-reporting-nar-1.21.0.nar
>> > >> (26 MB at 183 kB/s)
>> > >>
>> > >> Downloaded from central:
>> > >>
>> > >>
>> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-other-graph-services-nar/1.21.0/nifi-other-graph-services-nar-1.21.0.nar
>> > >> (49 MB at 332 kB/s)
>> > >>
>> > >> Downloaded from central:
>> > >>
>> > >>
>> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-atlas-nar/1.21.0/nifi-atlas-nar-1.21.0.nar
>> > >> (74 MB at 491 kB/s)
>> > >>
>> > >> [*INFO*]
>> > >>
>> **
>> > >>
>> > >> [*INFO*] *BUILD FAILURE*
>> > >>
>> > >> [*INFO*]
>> > >>
>> **
>> > >>
>> > >> [*INFO*] Total time:  02:54 min
>> > >>
>> > >> [*INFO*] Finished at: 2023-04-10T12:20:08-04:00
>> > >>
>> > >> [*INFO*]
>> > >>
>> **
>> > >>
>> > >> [*ERROR*] Failed to execute goal on project nifi-assembly: *Could not
>> > >> resolve dependencies for project
>> org.apache.nifi:nifi-assembly:pom:1.21.0:
>> > >> Could not find artifact org.apache.nifi:nifi-azure-nar:nar:1.21.0 in
>> > >> central (https://repo.maven.apache.org/maven2
>> > >> <https://repo.maven.apache.org/maven2>)* -> *[Help 1]*
>> > >>
>> > >> [*ERROR*]
>> > >>
>> > >> [*ERROR*] To see the full stack trace of the errors, re-run Maven
>> with the
>> > >> *-e* switch.
>> > >>
>> > >> [*ERROR*] Re-run Maven using the *-X* switch to enable full debug
>> logging.
>> > >>
>> > >> [*ERROR*]
>> > >>
>> > >> [*ERROR*] For more information about the errors and possible
>> solutions,
>> > >> please read the following articles:
>> > >>
>> > >> [*ERROR*] *[Help 1]*
>> > >>
>> > >>
>> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
>> > >>
>>
>

Re: Are all 1.21 NARs uploaded as Maven artifacts?

[Mike Thomsen]

When I clicked on it, I got a 404. Weird.

On Mon, Apr 10, 2023 at 12:53 PM Bryan Bende  wrote:

> The Azure NAR looks like it is there:
>
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-azure-nar/1.21.0/
>
> On Mon, Apr 10, 2023 at 12:40 PM Mike Thomsen 
> wrote:
> >
> > It’s failing on the azure one which is why I am raising the issue. I
> know we are still pushing that because it’s part of the convenient binary.
> >
> > Sent from my iPhone
> >
> > > On Apr 10, 2023, at 12:26 PM, Joe Witt  wrote:
> > >
> > > Hello
> > >
> > > Pretty sure all the normal stuff/paths followed and uploaded.  It
> might be
> > > that we're not pushing Atlas nars anymore?
> > >
> > > I dont recall but that is possible.
> > >
> > > Thanks
> > >
> > >> On Mon, Apr 10, 2023 at 9:21 AM Mike Thomsen 
> wrote:
> > >>
> > >> Getting this error on two different machines on two different
> networks:
> > >>
> > >> ownloaded from central:
> > >>
> > >>
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-sql-reporting-nar/1.21.0/nifi-sql-reporting-nar-1.21.0.nar
> > >> (26 MB at 183 kB/s)
> > >>
> > >> Downloaded from central:
> > >>
> > >>
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-other-graph-services-nar/1.21.0/nifi-other-graph-services-nar-1.21.0.nar
> > >> (49 MB at 332 kB/s)
> > >>
> > >> Downloaded from central:
> > >>
> > >>
> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-atlas-nar/1.21.0/nifi-atlas-nar-1.21.0.nar
> > >> (74 MB at 491 kB/s)
> > >>
> > >> [*INFO*]
> > >>
> **
> > >>
> > >> [*INFO*] *BUILD FAILURE*
> > >>
> > >> [*INFO*]
> > >>
> **
> > >>
> > >> [*INFO*] Total time:  02:54 min
> > >>
> > >> [*INFO*] Finished at: 2023-04-10T12:20:08-04:00
> > >>
> > >> [*INFO*]
> > >>
> **
> > >>
> > >> [*ERROR*] Failed to execute goal on project nifi-assembly: *Could not
> > >> resolve dependencies for project
> org.apache.nifi:nifi-assembly:pom:1.21.0:
> > >> Could not find artifact org.apache.nifi:nifi-azure-nar:nar:1.21.0 in
> > >> central (https://repo.maven.apache.org/maven2
> > >> <https://repo.maven.apache.org/maven2>)* -> *[Help 1]*
> > >>
> > >> [*ERROR*]
> > >>
> > >> [*ERROR*] To see the full stack trace of the errors, re-run Maven
> with the
> > >> *-e* switch.
> > >>
> > >> [*ERROR*] Re-run Maven using the *-X* switch to enable full debug
> logging.
> > >>
> > >> [*ERROR*]
> > >>
> > >> [*ERROR*] For more information about the errors and possible
> solutions,
> > >> please read the following articles:
> > >>
> > >> [*ERROR*] *[Help 1]*
> > >>
> > >>
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> > >>
>

Re: Are all 1.21 NARs uploaded as Maven artifacts?

[Mike Thomsen]

It’s failing on the azure one which is why I am raising the issue. I know we 
are still pushing that because it’s part of the convenient binary.

Sent from my iPhone

> On Apr 10, 2023, at 12:26 PM, Joe Witt  wrote:
> 
> Hello
> 
> Pretty sure all the normal stuff/paths followed and uploaded.  It might be
> that we're not pushing Atlas nars anymore?
> 
> I dont recall but that is possible.
> 
> Thanks
> 
>> On Mon, Apr 10, 2023 at 9:21 AM Mike Thomsen  wrote:
>> 
>> Getting this error on two different machines on two different networks:
>> 
>> ownloaded from central:
>> 
>> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-sql-reporting-nar/1.21.0/nifi-sql-reporting-nar-1.21.0.nar
>> (26 MB at 183 kB/s)
>> 
>> Downloaded from central:
>> 
>> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-other-graph-services-nar/1.21.0/nifi-other-graph-services-nar-1.21.0.nar
>> (49 MB at 332 kB/s)
>> 
>> Downloaded from central:
>> 
>> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-atlas-nar/1.21.0/nifi-atlas-nar-1.21.0.nar
>> (74 MB at 491 kB/s)
>> 
>> [*INFO*]
>> **
>> 
>> [*INFO*] *BUILD FAILURE*
>> 
>> [*INFO*]
>> **
>> 
>> [*INFO*] Total time:  02:54 min
>> 
>> [*INFO*] Finished at: 2023-04-10T12:20:08-04:00
>> 
>> [*INFO*]
>> **
>> 
>> [*ERROR*] Failed to execute goal on project nifi-assembly: *Could not
>> resolve dependencies for project org.apache.nifi:nifi-assembly:pom:1.21.0:
>> Could not find artifact org.apache.nifi:nifi-azure-nar:nar:1.21.0 in
>> central (https://repo.maven.apache.org/maven2
>> <https://repo.maven.apache.org/maven2>)* -> *[Help 1]*
>> 
>> [*ERROR*]
>> 
>> [*ERROR*] To see the full stack trace of the errors, re-run Maven with the
>> *-e* switch.
>> 
>> [*ERROR*] Re-run Maven using the *-X* switch to enable full debug logging.
>> 
>> [*ERROR*]
>> 
>> [*ERROR*] For more information about the errors and possible solutions,
>> please read the following articles:
>> 
>> [*ERROR*] *[Help 1]*
>> 
>> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
>> 

Are all 1.21 NARs uploaded as Maven artifacts?

[Mike Thomsen]

Getting this error on two different machines on two different networks:

ownloaded from central:
https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-sql-reporting-nar/1.21.0/nifi-sql-reporting-nar-1.21.0.nar
(26 MB at 183 kB/s)

Downloaded from central:
https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-other-graph-services-nar/1.21.0/nifi-other-graph-services-nar-1.21.0.nar
(49 MB at 332 kB/s)

Downloaded from central:
https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-atlas-nar/1.21.0/nifi-atlas-nar-1.21.0.nar
(74 MB at 491 kB/s)

[*INFO*]
**

[*INFO*] *BUILD FAILURE*

[*INFO*]
**

[*INFO*] Total time:  02:54 min

[*INFO*] Finished at: 2023-04-10T12:20:08-04:00

[*INFO*]
**

[*ERROR*] Failed to execute goal on project nifi-assembly: *Could not
resolve dependencies for project org.apache.nifi:nifi-assembly:pom:1.21.0:
Could not find artifact org.apache.nifi:nifi-azure-nar:nar:1.21.0 in
central (https://repo.maven.apache.org/maven2
)* -> *[Help 1]*

[*ERROR*]

[*ERROR*] To see the full stack trace of the errors, re-run Maven with the
*-e* switch.

[*ERROR*] Re-run Maven using the *-X* switch to enable full debug logging.

[*ERROR*]

[*ERROR*] For more information about the errors and possible solutions,
please read the following articles:

[*ERROR*] *[Help 1]*
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

Re: [VOTE] Release Apache NiFi 1.21.0 (RC2)

[Mike Thomsen]

+1 binding.

On Wed, Apr 5, 2023 at 5:53 AM Csaba Bejan  wrote:

> +1 (non-binding)
>
> - Went through the helper guide and did a clean build
> - Verified signatures and hashes
> - Built on OSX 11.6.6
> - AdoptOpenJDK (build 1.8.0_282-b08)
> - Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
>
> - Started NiFi and created a simple flow
> - NIFI-11332:
> Verified that MiNiFi starts up and configuration can be updated via C2
> protocol (UPDATE/CONFIGURATION)
> Verified that no unnecessary files remain open for each new heartbeat
> Thanks for RM’ing, Joe!
>
> Csaba
>
> > On 2023. Apr 4., at 23:33, David Handermann 
> wrote:
> >
> > +1 (binding)
> >
> > - Verified signatures and hashes
> > - Ran build using Maven 3.9.0
> > - Ran build on macOS 13.3 with Azul Zulu JDK 11.0.18 AArch64
> > - Ran build on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-362 x86_64
> > - Ran stateless and system tests on macOS 13.3 with Azul Zulu JDK 11.0.18
> > AArch64
> > - Ran stateless and system tests on Ubuntu 22.04 with Azul Zulu JDK
> > 1.8.0-362 x86_64
> >
> > - Ran NiFi on macOS 13.3 with Azul Zulu JDK 11.0.18 AArch64
> > - Ran NiFi on macOS 13.3 with Azul Zulu JDK 11.0.16 x86_64
> > - Ran NiFi on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-362 x86_64
> > - Ran NiFi Toolkit Encrypt Config command with AES-GCM
> > - Configured OpenID Connect authentication with Okta
> > - Configured cluster with external ZooKeeper and multiple NiFi nodes
> >
> > - Ran NiFi Registry on Ubuntu 22.04 with Azul Zulu JDK 11.0.18 x86_64
> > - Created Buckets
> > - Verified Flow Version Control
> >
> > NIFI-4890 Verified OpenID Connect integration with Okta with Refresh
> Tokens
> > enabled
> > NIFI-10896 Verified deprecation warnings logged for Ambari Reporting Task
> > NIFI-11134 Verified Label tracked in Flow Configuration History
> > NIFI-11154 Verified default schedule removed from ConsumeMQTT
> > NIFI-11156 Verified ValidateXml error contains schema failure details
> > NIFI-11176 Verified FlowSynchronizationIT runs on different platforms
> > NIFI-11181 Verified deprecation warnings logged for Kafka 1_0 Processors
> > NIFI-11184 Verified deprecation warnings logged for HashAttribute and
> > CryptographicHashAttribute
> > NIFI-11191 Verified multiple builds without HikariCPConnectionPoolTest
> > failures
> > NIFI-11205 Verified deprecation warnings logged for Kerberos Principal
> > properties in Kafka 2_6
> > NIFI-11245 Verified reduction of Maximum Timer Driven Thread Count
> without
> > restarting
> > NIFI-11246 Verified documentation updates to GetAzureEventHub and
> > ConsumeAzureEventHub
> > NIFI-11249 Verified deprecation warnings logged for Kafka 2_0 Processors
> > NIFI-11251 Verified versioning of nested Process Groups with Registry
> > NIFI-11256 Verified property display name updates for Azure Event Hub
> > Processors
> > NIFI-11261 Verified GetAzureEventHub disconnects on Primary Node State
> > changes
> > NIFI-11277 Verified deprecation warnings logged for bcrypt and scrypt
> > Properties Algorithms
> > NIFI-11280 Verified system tests run with updated Sensitive Properties
> > Algorithm
> > NIFI-11302 Verified Registry version displayed in About dialog on Java 8
> > and 11
> > NIFI-11333 Verified removing components disallowed while cluster node is
> > disconnected
> > NIFI-11337 Verified PutDatabaseRecord Max Batch Size is no longer a
> > dependent property
> > NIFI-11341 Verified ListenUDPRecord flow with malformed records produces
> > FlowFiles as expected
> > NIFI-11351 Verified Docker image build on macOS AArch64
> > NIFI-11363 Verified Flow Encryption Command does not truncate floating
> > point numbers
> > NIFI-11365 Verified OpenID Connect integration with reverse proxy and
> > context path changed
> > NIFI-11372 Verified Registry documentation is rendered as expected
> >
> > Thanks Joe!
> >
> > Regards,
> > David Handermann
> >
> > On Mon, Apr 3, 2023 at 7:11 PM Joe Witt  wrote:
> >
> >> Hello,
> >>
> >> I am pleased to be calling this vote for the source release of Apache
> >> NiFi 1.21.0.
> >>
> >> The source zip, including signatures, digests, etc. can be found at:
> >> https://repository.apache.org/content/repositories/orgapachenifi-1226
> >>
> >> The source being voted upon and the convenience binaries can be found
> at:
> >> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.21.0/
> >>
> >> A helpful reminder on how the release candidate verification process
> works:
> >>
> >>
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >>
> >> The Git tag is nifi-1.21.0-RC2
> >> The Git commit ID is 892f822107da84ca0dcfefdb5c91c5bc11dc3dc1
> >>
> >>
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=892f822107da84ca0dcfefdb5c91c5bc11dc3dc1
> >>
> >> Checksums of nifi-1.21.0-source-release.zip:
> >> SHA256: 598bf8e90f871f4ca25709dd4ecf3da16c814c08c0d8b2c8744dbd34df34dea5
> >> SHA512:
> >>
> 

Re: [VOTE] Release Apache NiFi 1.21.0 (RC1)

[Mike Thomsen]

+1 Binding

On Wed, Mar 29, 2023 at 6:01 PM Joe Witt  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.21.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1224
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.21.0/
>
> A helpful reminder on how the release candidate verification process works:
>
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.21.0-RC1
> The Git commit ID is eef5ca1afe8532521e5dd8f49029f8d5a3ac6028
>
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=eef5ca1afe8532521e5dd8f49029f8d5a3ac6028
>
> Checksums of nifi-1.21.0-source-release.zip:
> SHA256: 744bb0931d0965cfb882741300ee7fe2fa6ef896eb952a31929bec7f4f5ce46f
> SHA512:
> cbca6cf5f9a970efff37a2ddaadc92d1c7b6c1ed1abc4c0113d1413198c0798586b4587bb8beb3382c8831b4890c42154a34ba8f1354eba414c59cb4ede4e64a
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 110 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352899
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.21.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.21.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: 1.X branch

[Mike Thomsen]

I just put the apache/ part in there to make it extra clear which
branch I was referring to.

On Tue, Feb 21, 2023 at 9:38 AM Joe Witt  wrote:
>
> Mike
>
> Yes it is.  Just to avoid confusion the 'apache' part of that is just
> what you have called the repo in your git config.  The
> 'support/nifi-1.x' line is the correct one though for what is
> currently 1.21.0-SNAPSHOT.
>
> Just please be sure nothing goes there that doesn't also go to main
> (2.0.0-SNAPSHOT) unless it is a bug fix for something only present on
> 1.x line.
>
> Thanks
> Joe
>
> On Tue, Feb 21, 2023 at 7:24 AM Mike Thomsen  wrote:
> >
> > Is apache/support/nifi-1.x the correct branch to merge 1.21+ changes to?
> >
> > Thanks,
> >
> > Mike

1.X branch

[Mike Thomsen]

Is apache/support/nifi-1.x the correct branch to merge 1.21+ changes to?

Thanks,

Mike

Re: ValidateCsv vs ValidateRecord

[Mike Thomsen]

Also, I think there will be a high likelihood of format-specific
processors like ValidateCSV going away in NiFi 2.0. I wouldn't start
any new data engineering work using these processors.

On Wed, Jan 18, 2023 at 4:02 PM Pierre Villard
 wrote:
>
> Hi,
>
> ValidateCSV was released before and is using a specific library that is
> allowing you to define specific rules on what the CSV should contain (like
> for a given column that should contain numerical values, confirm the values
> are in a specific range, etc) [1]. ValidateRecord would only check things
> against a schema (type of fields, column names). I do believe that
> ValidateRecord + QueryRecord can achieve all of the features provided by
> ValidateCSV.
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.19.1/org.apache.nifi.processors.standard.ValidateCsv/additionalDetails.html
>
> HTH,
> Pierre
>
> Le mer. 18 janv. 2023 à 21:50, Dan S  a écrit :
>
> > I am trying to wrap my head around why there is a ValidateCsv processor
> > when there is a ValidateRecord processor which can do I believe everything
> > ValidateCsv can do and more. What is the community stance on this? Should
> > only ValidateRecord be used or is there still a use case for using
> > ValidateCsv?
> >

Re: [VOTE] Adopt NiFi 2.0 Proposed Release Goals

[Mike Thomsen]

+1 binding

On Tue, Dec 13, 2022 at 9:33 AM Mark Bean  wrote:
>
> +1 (non-binding)
>
> On Mon, Dec 12, 2022 at 12:02 PM David Handermann <
> exceptionfact...@apache.org> wrote:
>
> > Team,
> >
> > Following positive feedback on NiFi 2.0 Proposed Release Goals [1] on the
> > recent discussion thread [2], I am calling this vote to adopt the following
> > as Release Goals for NiFi 2.0:
> >
> > 1. Remove Java 8 support and require Java 11
> > 2. Remove deprecated components
> > 3. Remove deprecated component properties
> > 4. Remove components integrating with unmaintained services
> > 5. Remove compatibility classes and methods
> > 6. Remove flow.xml.gz in favor of flow.json.gz
> > 7. Remove duplicative features
> > 8. Upgrade internal Java API references
> > 9. Reorganize standard components
> > 10. Implement migration tools for upgrading flows
> >
> > A positive vote indicates agreement on these goals and the initiation of
> > the following actions:
> >
> > 1. Rename NiFi 2.0 Proposed Release Goals to NiFi 2.0 Release Goals
> > 2. Create version 1 branch in Git for subsequent support releases on the
> > version 1 series
> > 3. Update the current main branch in Git to version 2.0.0-SNAPSHOT
> >
> > The vote will be open for 72 hours and follow standard procedures for
> > release votes.
> >
> > Please review the linked goals and discussions for background.
> >
> > [ ] +1 Adopt NiFi 2.0 Release Goals
> > [ ] +0 No opinion
> > [ ] -1 Do not adopt NiFi 2.0 Release Goals for the following reasons...
> >
> > [1]
> >
> > https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Proposed+Release+Goals
> > [2] https://lists.apache.org/thread/xo77p9t3xg4k70356xrqbdg4m9sg7sf8
> >

Re: [DISCUSS] Finalizing Release Goals for NiFi 2.0

[Mike Thomsen]

> Mike - what do you mean by "controller service-based configuration for

Using controller services for configuring bundles that connect an
external service such as Cassandra, Elasticsearch, etc. and removing
the option to configure connections on the processor.

> I don't think you were suggesting the minimum version be Java 17, were you?

I was. Partly as devil's advocate, partly because I actually want to
use Java 17 as a daily driver.

On Wed, Dec 7, 2022 at 2:20 PM Mark Bean  wrote:
>
> I agree this is a great start to a discussion with pointers to important
> docs for the 2.0 transition. Thanks David!
>
> Mike - what do you mean by "controller service-based configuration for
> connection details"?
>
> Also, the transition from Java 11 to 17 is not without potential issues.
> I've discovered one already. [1] I support stepping up on Java version
> requirements. Perhaps rather than the currently stated "Requires Java 8 or
> Java 11", the requirement can be "Requires Java 11 or Java 17". I don't
> think you were suggesting the minimum version be Java 17, were you? Either
> way, the issue with Java 17 needs to be identified and fixed as well as
> more thorough testing to find other possible edge cases before we move
> forward too aggressively.
>
> [1] https://issues.apache.org/jira/browse/NIFI-10958
>
> On Wed, Dec 7, 2022 at 1:33 PM Mike Thomsen  wrote:
>
> > Really good start on the discussion. One thing I'm curious about is
> > Java 11 vs 17. Java 8 -> 11 is major jump that I can understand why
> > businesses scoffed at that for a long time, but the lift from 11 to 17
> > was about like 7 -> 8. A 2.0 release seems like a good time to jump
> > straight to the latest official LTS for Java and start greenlighting
> > new language features that might simplify things.
> >
> > I would also add (since I didn't see it) a design goal of forcing a
> > complete shift in all bundles to using controller service-based
> > configurations for connection details. 2.0 feels like a really good
> > time for us to establish a community-wide best practice of
> > centralizing configurations in dedicated components.
> >
> > On Wed, Dec 7, 2022 at 9:13 AM Mark Payne  wrote:
> > >
> > > Yeah, agreed. I am very supportive, as well.
> > >
> > > Thanks for taking the time to put this together, David.
> > >
> > > -Mark
> > >
> > >
> > > > On Dec 7, 2022, at 4:07 AM, Pierre Villard <
> > pierre.villard...@gmail.com> wrote:
> > > >
> > > > Thanks for putting this together David. This is an excellent writeup
> > and
> > > > it's great to have a release where we focus on tech debt as well as
> > making
> > > > sure we stay up to date with our dependencies and what we support.
> > This is
> > > > a great opportunity for us to clean a lot of things in our code and I
> > can't
> > > > wait for us to get started with this. I'm definitely a +1 to have a
> > formal
> > > > vote on this proposal.
> > > >
> > > > Thanks,
> > > > Pierre
> > > >
> > > > Le mar. 6 déc. 2022 à 23:50, Joe Witt  a écrit :
> > > >
> > > >> David, All,
> > > >>
> > > >> This is an excellent writeup/good framing.  I am supportive of this
> > > >> as-is since it is achievable and lays out a clear path.  We can make
> > > >> milestone releases of NiFi 2.0.0 along the way until we achieve all
> > > >> the stated goals. I assume migration bits will be the long pole and
> > > >> once we have them sorted we can kick out a 2.0.0.   We already have a
> > > >> version guide that governs how long we'd keep 1.x maintained though
> > > >> the phase out is pretty natural as we move main to a 2.0.0 basis
> > > >> anyway.
> > > >>
> > > >> Not to confuse this thread but it makes me think we could do a similar
> > > >> framing for a NiFi 3.0 which lays out a potentially new approach to
> > > >> NiFi decoupling the web/interface from the runtime/operations and one
> > > >> which is more fundamentally K8S based.  But we can cross that bridge a
> > > >> bit later.  Does seem more and more like folks in the community would
> > > >> like to know more about the potential directions we can go.
> > > >>
> > > >> Thanks!
> > > >> Joe
> > > >>
> > > >> On Tue, Dec 6, 2022 at 1:53 PM David Handermann
> 

Re: [DISCUSS] Finalizing Release Goals for NiFi 2.0

[Mike Thomsen]

Really good start on the discussion. One thing I'm curious about is
Java 11 vs 17. Java 8 -> 11 is major jump that I can understand why
businesses scoffed at that for a long time, but the lift from 11 to 17
was about like 7 -> 8. A 2.0 release seems like a good time to jump
straight to the latest official LTS for Java and start greenlighting
new language features that might simplify things.

I would also add (since I didn't see it) a design goal of forcing a
complete shift in all bundles to using controller service-based
configurations for connection details. 2.0 feels like a really good
time for us to establish a community-wide best practice of
centralizing configurations in dedicated components.

On Wed, Dec 7, 2022 at 9:13 AM Mark Payne  wrote:
>
> Yeah, agreed. I am very supportive, as well.
>
> Thanks for taking the time to put this together, David.
>
> -Mark
>
>
> > On Dec 7, 2022, at 4:07 AM, Pierre Villard  
> > wrote:
> >
> > Thanks for putting this together David. This is an excellent writeup and
> > it's great to have a release where we focus on tech debt as well as making
> > sure we stay up to date with our dependencies and what we support. This is
> > a great opportunity for us to clean a lot of things in our code and I can't
> > wait for us to get started with this. I'm definitely a +1 to have a formal
> > vote on this proposal.
> >
> > Thanks,
> > Pierre
> >
> > Le mar. 6 déc. 2022 à 23:50, Joe Witt  a écrit :
> >
> >> David, All,
> >>
> >> This is an excellent writeup/good framing.  I am supportive of this
> >> as-is since it is achievable and lays out a clear path.  We can make
> >> milestone releases of NiFi 2.0.0 along the way until we achieve all
> >> the stated goals. I assume migration bits will be the long pole and
> >> once we have them sorted we can kick out a 2.0.0.   We already have a
> >> version guide that governs how long we'd keep 1.x maintained though
> >> the phase out is pretty natural as we move main to a 2.0.0 basis
> >> anyway.
> >>
> >> Not to confuse this thread but it makes me think we could do a similar
> >> framing for a NiFi 3.0 which lays out a potentially new approach to
> >> NiFi decoupling the web/interface from the runtime/operations and one
> >> which is more fundamentally K8S based.  But we can cross that bridge a
> >> bit later.  Does seem more and more like folks in the community would
> >> like to know more about the potential directions we can go.
> >>
> >> Thanks!
> >> Joe
> >>
> >> On Tue, Dec 6, 2022 at 1:53 PM David Handermann
> >>  wrote:
> >>>
> >>> Team,
> >>>
> >>> With the release of NiFi 1.19.0 deprecating support for Java 8, the end
> >> of
> >>> the year provides a good opportunity for finalizing general release goals
> >>> for NiFi 2.0.
> >>>
> >>> Based on previous discussions from July 2021 [1] and June 2022 [2], there
> >>> seems to be general agreement with focusing a NiFi 2.0 release on
> >> reducing
> >>> technical debt while providing a straightforward upgrade path for current
> >>> deployments.
> >>>
> >>> I have updated the NiFi 2.0 Proposed Release Goals [3] to reflect more
> >>> recent progress in several areas. I also linked the Deprecated Components
> >>> and Features [4] page outlining the current state of deprecated
> >>> capabilities.
> >>>
> >>> The most recent update to the Proposed Release Goals outlines
> >> implementing
> >>> migration tooling to make the upgrade process as easy as possible. The
> >>> addition of dedicated deprecation logging in NiFi 1.18.0 makes it easier
> >> to
> >>> warn of breaking changes, but the goal of migration tooling is to make it
> >>> easier to upgrade configurations.
> >>>
> >>> The Proposed Release Goals does not include any release timelines right
> >>> now, and we should anticipate supporting version 1 for a reasonable
> >> period
> >>> of time. As more and more libraries deprecate and drop support for Java
> >> 8,
> >>> it will become increasingly difficult to maintain a support branch, which
> >>> is one of the main drivers behind a NiFi 2.0 release that drops support
> >> for
> >>> Java 8.
> >>>
> >>> The general development strategy should involve transitioning the main
> >>> branch to a 2.0.0-SNAPSHOT version so new features and fixes will be
> >>> targeted to the new version. Migration tooling will need to be
> >> implemented
> >>> on a version 1 support branch, and fixes can be backported where
> >> possible,
> >>> in preparation for subsequent version 1 releases.
> >>>
> >>> With that background, I would like to move to a formal vote soon,
> >> changing
> >>> the Proposed Release Goals document to Planned Release Goals. Please
> >> weigh
> >>> the general goals highlighted, and if there are no major roadblocks
> >>> identified, I will follow up soon with a vote thread.
> >>>
> >>> Regards,
> >>> David Handermann
> >>>
> >>> [1] https://lists.apache.org/thread/yj8scrdbx3pdo7990123mc03q24rn1m7
> >>> [2] https://lists.apache.org/thread/mm1xf3b9nvrcgytb92oy3swvvc45fl34
> >>> [3]

Optional build profiles with Category X licenses

[Mike Thomsen]

I know the official binaries cannot contain Category X license code,
but is it allowable to have optional build profiles that integrate
Category X licensed code? An example scenario that lead me to wonder
if this is doable is I would like to try to add optional support for
Elastic's Painless as script engine, but it's covered under a Category
X license.

Thanks,

Mike

Re: merge PR 6335

[Mike Thomsen]

"doodzio," as far as I can tell, is not a committer or PMC member so
it's not been approved by someone with authority to finish a review.

On Sun, Nov 20, 2022 at 11:59 PM Akshara Uke  wrote:
>
> Hello,
>
> Requesting merge of https://github.com/apache/nifi/pull/6335 , the review
> is already completed.
>
> Thanks much,
> akshara

Re: [discuss] NiFi 1.19.0

[Mike Thomsen]

Can someone do a quick review on this? I think it would be a nice
little bullet point on our release to say we added preliminary support
for Neo4J 5 since it's so new.

https://github.com/apache/nifi/pull/6684

(Enabling the V5 profile has a hard dependency on Java 17)

On Fri, Nov 18, 2022 at 11:00 AM Joe Witt  wrote:
>
> Goodness.  Some pretty cool collaboration on things like the docker updates
> and NiFi 1.19 is looking healthy.  Time permitting, I'd like to start the
> RC today.
>
> Thanks
>
> On Tue, Nov 15, 2022 at 6:16 PM Kevin Doran  wrote:
>
> >  NIFI-10701 is merged. If folks are ok holding off a day or two more for
> > 1.19, I'll look into following the work Ferenc Kis did for that to knock
> > out NIFI-10820.
> >
> > On Nov 15, 2022 at 19:43:57, Matt Burgess  wrote:
> >
> > > Awesome, thanks all! I'd like to wait for Github actions to be happy
> > > and then merge?
> > >
> > > On Tue, Nov 15, 2022 at 7:11 PM Mark Bean  wrote:
> > >
> > >
> > > NIFI-10703 has been worked through. I think it's good to go now.
> > >
> > >
> > > On Tue, Nov 15, 2022 at 4:43 PM Matt Burgess 
> > wrote:
> > >
> > >
> > > > I can take that review too, if your suggestions are incorporated. I
> > >
> > > > started looking the other day but didn't get to dig in. I'll try to
> > >
> > > > reproduce in the meantime.
> > >
> > > >
> > >
> > > > Regards,
> > >
> > > > Matt
> > >
> > > >
> > >
> > > > On Tue, Nov 15, 2022 at 4:42 PM Nathan Gough 
> > > wrote:
> > >
> > > > >
> > >
> > > > > We might also want https://issues.apache.org/jira/browse/NIFI-10787
> > >
> > > > >
> > >
> > > > > On Tue, Nov 15, 2022 at 4:05 PM Mark Bean 
> > > wrote:
> > >
> > > > >
> > >
> > > > > > I will be on it in about 2 hours, if not addressed sooner.
> > >
> > > > > >
> > >
> > > > > > On Tue, Nov 15, 2022 at 3:46 PM Joe Witt 
> > wrote:
> > >
> > > > > >
> > >
> > > > > > > NIFI-10703 would be great to get in.  Just a matter of gettin' it
> > >
> > > > done
> > >
> > > > > > and
> > >
> > > > > > > merged I think.  You're on it?
> > >
> > > > > > >
> > >
> > > > > > > Thanks
> > >
> > > > > > >
> > >
> > > > > > > On Tue, Nov 15, 2022 at 1:38 PM Mark Bean  > >
> > >
> > > > wrote:
> > >
> > > > > > >
> > >
> > > > > > > > Sorry.. check that. There's a typo in the latest commit on that
> > > PR.
> > >
> > > > > > > >
> > >
> > > > > > > > On Tue, Nov 15, 2022 at 3:35 PM Mark Bean <
> > mark.o.b...@gmail.com
> > > >
> > >
> > > > > > wrote:
> > >
> > > > > > > >
> > >
> > > > > > > > > Could we get NIFIDEVS-10703 (PR #6638) in there too? AFAIK,
> > > it's
> > >
> > > > good
> > >
> > > > > > > to
> > >
> > > > > > > > > go.
> > >
> > > > > > > > >
> > >
> > > > > > > > > Thanks,
> > >
> > > > > > > > > Mark
> > >
> > > > > > > > >
> > >
> > > > > > > > >
> > >
> > > > > > > > > On Tue, Nov 15, 2022 at 2:56 PM Joe Witt  > >
> > >
> > > > wrote:
> > >
> > > > > > > > >
> > >
> > > > > > > > >> Hello All,
> > >
> > > > > > > > >>
> > >
> > > > > > > > >> I plan to kick off the RC as soon as
> > >
> > > > > > > > >> https://issues.apache.org/jira/browse/NIFI-10701 is merged.
> > >
> > > > > > > > >>
> > >
> > > > > > > > >> Thanks
> > >
> > > > > > > > >>
> > >
> > > > > > > > >> On Thu, Nov 10, 2022 at 12:52 PM David Handermann <
> > >
> > > > > > > > >> exceptionfact...@apache.org> wrote:
> > >
> > > > > > > > >>
> > >
> > > > > > > > >> > Joe,
> > >
> > > > > > > > >> >
> > >
> > > > > > > > >> > Thanks for initiating the discussion. I agree with moving
> > >
> > > > toward a
> > >
> > > > > > > > >> 1.19.0
> > >
> > > > > > > > >> > release instead of 1.18.1 given over 150 Jira issues
> > already
> > >
> > > > > > tagged
> > >
> > > > > > > > with
> > >
> > > > > > > > >> > 1.19.0.
> > >
> > > > > > > > >> >
> > >
> > > > > > > > >> > We should start another thread soon regarding 2.0. One of
> > > the
> > >
> > > > key
> > >
> > > > > > > > >> things to
> > >
> > > > > > > > >> > do before that, however, would be to finish adding
> > > deprecation
> > >
> > > > > > > logging
> > >
> > > > > > > > >> to
> > >
> > > > > > > > >> > remaining features we plan to remove in 2.0. Having a
> > minor
> > >
> > > > > > release
> > >
> > > > > > > in
> > >
> > > > > > > > >> > version 1 series deprecating targets for removal should
> > be a
> > >
> > > > > > > > >> prerequisite
> > >
> > > > > > > > >> > before branching and starting on 2.0.
> > >
> > > > > > > > >> >
> > >
> > > > > > > > >> > In the meantime, if that means we plan on a 1.20.0
> > release,
> > > I
> > >
> > > > am
> > >
> > > > > > in
> > >
> > > > > > > > >> favor
> > >
> > > > > > > > >> > of moving forward with 1.19.0.
> > >
> > > > > > > > >> >
> > >
> > > > > > > > >> > Regards,
> > >
> > > > > > > > >> > David Handermann
> > >
> > > > > > > > >> >
> > >
> > > > > > > > >> > On Thu, Nov 10, 2022 at 1:44 PM Joe Witt <
> > > joe.w...@gmail.com>
> > >
> > > > > > > wrote:
> > >
> > > > > > > > >> >
> > >
> > > > > > > > >> > > Team,
> > >
> > > > > > > > >> > >
> > >
> > > > > > > 

Re: Adding Kotlin support to the scripting bundle

[Mike Thomsen]

Full disclosure: I can barely write hello world in Kotlin right now.

Yeah, that seems like the right way to do it. What I found was that if
I did a bunch of preloads like setting up an implicit session,
context, REL_SUCCESS, REL_FAILURE by simply doing val session =
bindings[... for each import it worked very smoothly and gave the
ExecuteScript experience with Kotlin. I'll repurpose NIFI-4307 to
focus on this feature. I already have all of the test coverage ready
(I think) with my branch of ExecuteScript.

On Mon, Nov 7, 2022 at 2:37 PM David Handermann
 wrote:
>
> Concurring with Matt, this sounds like a potential opportunity for a
> scripted Processor implementation specific to Kotlin.
>
> Regards,
> David Handermann
>
> On Mon, Nov 7, 2022 at 1:11 PM Matt Burgess  wrote:
>
> > Mike,
> >
> > As an alternative it might be worth looking into a processor
> > specifically for Kotlin like we have with ExecuteGroovyScript (EGS).
> > ExecuteGroovyScript doesn't use JSR-223 and instead goes right for the
> > GroovyShell. This allows for cool things like Groovy-specific idioms
> > and helpful "meta-capabilities" such as redirect I/O to/from FlowFile
> > objects. Depending on how they've done their entry point(s), an
> > ExecuteKotlinScript processor could leverage compiled scripts without
> > needing the JSR-223 impl, as well as adding some Kotlin-ness to the
> > scripting capabilities. I hesitate to suggest this for any of the
> > other scripting languages but as Kotlin becomes more popular and given
> > its tight integration with the JVM (vs Jython for example), that might
> > be the way to go.
> >
> > Regards,
> > Matt
> >
> >
> > On Sun, Nov 6, 2022 at 6:16 PM Mike Thomsen 
> > wrote:
> > >
> > > I finally chased down the behavior differences that I found and
> > > discussed with Matt in the previous PR. Kotlin's got very promising
> > > (long term) JSR223 support but with a few caveats in the short term:
> > >
> > > 1. Its ScriptEngine does not support ScriptEngine#get properly, so any
> > > component that uses get(String) to fetch an object defined in a script
> > > won't work with Kotlin (ex InvokeScriptedProcessor).
> > > 2. Invocable vs Compilable have very different behaviors WRT bindings.
> > > 3. Until Kotlin's JSR223 engine matures, we'd probably need to limit
> > > it to ExecuteScript with extra documentation explaining that it
> > > doesn't behave quite as expected.
> > >
> > > What I figured out, and it makes sense given how Kotlin works, is that
> > > bindings cannot be used to provide dynamic injection of variables. So
> > > just calling "session" as an implied ProcessSession won't work. You
> > > have to do something like this:
> > >
> > > val session = bindings["session"] as ProcessSession
> > >
> > > Either on your own or in a setup block for every script.
> > >
> > > I feel like there's value here, but wanted to throw it out there to
> > > other devs before going down the process of finalizing the
> > > ExecuteScript support and starting to exclude it from the other
> > > components.
> > >
> > > On a side note, we might want to add an InvokeCompiledScript processor
> > > that's like InvokeScriptedProcessor but is designed for Kotlin, JRuby
> > > and Groovy which work really well with taking scripts down to Java
> > > byte code and working with streamlined fat jars. Could be another
> > > angle on simplifying deployments of scripts and script-like
> > > components.
> > >
> > > Thanks,
> > >
> > > Mike
> >

Re: Import Talend's .jar to Nifi

[Mike Thomsen]

If you mean "can you derive a NiFi flow automatically from a Talend
jar" then no. With any jar, if the jar is a fat jar that doesn't rely
on an external framework (ex Spark, Flink or MapReduce) then you can
conceivably load the logic with ExecuteScript using the jar as a
module.

On Mon, Nov 7, 2022 at 12:18 PM D SchlewerBecker
 wrote:
>
> Hi!
>
> I'm actually working on Talend and Nifi, and as I am fairly new on those, i
> have a question: is it possible to transfer the jobs created in Talend (and
> exported in .jar) to Nifi, so Nifi can do the same work as Talend,
> therefore replace it + being open source.
> Sorry if the question looks strange, I'm really discovering all of this
> Thanks for your time !

Adding Kotlin support to the scripting bundle

[Mike Thomsen]

I finally chased down the behavior differences that I found and
discussed with Matt in the previous PR. Kotlin's got very promising
(long term) JSR223 support but with a few caveats in the short term:

1. Its ScriptEngine does not support ScriptEngine#get properly, so any
component that uses get(String) to fetch an object defined in a script
won't work with Kotlin (ex InvokeScriptedProcessor).
2. Invocable vs Compilable have very different behaviors WRT bindings.
3. Until Kotlin's JSR223 engine matures, we'd probably need to limit
it to ExecuteScript with extra documentation explaining that it
doesn't behave quite as expected.

What I figured out, and it makes sense given how Kotlin works, is that
bindings cannot be used to provide dynamic injection of variables. So
just calling "session" as an implied ProcessSession won't work. You
have to do something like this:

val session = bindings["session"] as ProcessSession

Either on your own or in a setup block for every script.

I feel like there's value here, but wanted to throw it out there to
other devs before going down the process of finalizing the
ExecuteScript support and starting to exclude it from the other
components.

On a side note, we might want to add an InvokeCompiledScript processor
that's like InvokeScriptedProcessor but is designed for Kotlin, JRuby
and Groovy which work really well with taking scripts down to Java
byte code and working with streamlined fat jars. Could be another
angle on simplifying deployments of scripts and script-like
components.

Thanks,

Mike

Re: How to download CSV from web or git into Nifi server

[Mike Thomsen]

I think InvokeHttp would meet your needs.

On Sun, Oct 30, 2022 at 1:33 AM Anand Bheemireddy
 wrote:
>
> Any response
>
> On Thu, Oct 27, 2022 at 2:12 PM Anand Bheemireddy <
> bheemireddyan...@gmail.com> wrote:
>
> > Hi team,
> >
> > I got a requirement to download  files(CSV)  from web and save into NIFI
> > server. I want to integrate web/git hub to my  NIFI server so that I can
> > download files directly from web/git. Please help me is there any nifi
> > processor/ template/useful blog/url which best suits with my requirements
> > --
> > *Thanks:*
> > *B.Anand Reddy*
> > *9492744804*
> >
> > --
> *Thanks:*
> *B.Anand Reddy*
> *9492744804*

Re: Need to deprecate the deduplication functionality in the MongoDB GridFS processors

[Mike Thomsen]

I might have to look into it more then because I wasn't able to get it
to work on MongoDB 5.X. It might just be something like the Docker
container is set up with better defaults since "sane defaults" or the
lack thereof was historically something Mongo got a lot of flack for
not having.

On Thu, Oct 27, 2022 at 9:36 PM David Handermann
 wrote:
>
> Mike,
>
> Thanks for raising this issue for additional discussion. According to the
> MongoDB document referenced, the md5 option is deprecated, but not yet
> removed:
>
> > The MD5 algorithm is prohibited by FIPS 140-2. MongoDB drivers deprecate
> MD5 support and will remove MD5 generation in future releases. Applications
> that require a file digest should implement it outside of GridFS and store
> in files.metadata
> <https://www.mongodb.com/docs/manual/core/gridfs/#mongodb-data-files.metadata>
>
> There is a configuration option called disableMD5, but it still appears to
> be part of the GridFS specification. Were you able to confirm that it
> breaks in MongoDB 5 or 6?
>
> I agree that we should be able to address this behavior in the current
> version of NiFi, and it seems like having a transitional way forward would
> be helpful. If the Testcontainers change can verify the current MD5
> functionality, that should provide a good baseline for a subsequent PR to
> implement a new hashing strategy.
>
> Regards,
> David Handermann
>
> On Tue, Oct 25, 2022 at 1:36 PM Mike Thomsen  wrote:
>
> > As-is, the deduplication-by-hash functionality appears to now be
> > broken w/ Mongo 5 and higher. We can address that by doing some
> > updates to the code base and recommending users add a HashContent
> > processor before PutGridFS, but flows are going to break either way
> > thanks to changes in Mongo itself. That's why I'm not sure we should
> > be dogmatic about waiting.
> >
> > On Tue, Oct 25, 2022 at 2:15 PM Pierre Villard
> >  wrote:
> > >
> > > IMO we should start working on NiFi 2.0 going forward and it sounds like
> > a
> > > good opportunity to make such changes in our components.
> > >
> > >
> > > Le mar. 25 oct. 2022 à 19:33, Mike Thomsen  a
> > > écrit :
> > >
> > > > The hash-based deduplication strategy used the built-in "md5"
> > > > attribute to offload the work to the database. That functionality was
> > > > deprecated and AFAICT gone as of Mongo 5:
> > > >
> > > > https://www.mongodb.com/docs/manual/core/gridfs/#files.md5
> > > >
> > > > I am proposing two changes:
> > > >
> > > > * Remove deduplication
> > > > * Create a MongoDB DistributedMapCache client that can query on the
> > > > file metadata since GridFS stores metadata separately from chunks
> > > > making lookups that way cheap and flexible.
> > > >
> > > > I could easily add that to this PR which already covers Testcontainers
> > > > integration, making it super easy to test the changed behavior:
> > > >
> > > > https://github.com/apache/nifi/pull/6460
> > > >
> > > > Thoughts?
> > > >
> >

Re: Need to deprecate the deduplication functionality in the MongoDB GridFS processors

[Mike Thomsen]

As-is, the deduplication-by-hash functionality appears to now be
broken w/ Mongo 5 and higher. We can address that by doing some
updates to the code base and recommending users add a HashContent
processor before PutGridFS, but flows are going to break either way
thanks to changes in Mongo itself. That's why I'm not sure we should
be dogmatic about waiting.

On Tue, Oct 25, 2022 at 2:15 PM Pierre Villard
 wrote:
>
> IMO we should start working on NiFi 2.0 going forward and it sounds like a
> good opportunity to make such changes in our components.
>
>
> Le mar. 25 oct. 2022 à 19:33, Mike Thomsen  a
> écrit :
>
> > The hash-based deduplication strategy used the built-in "md5"
> > attribute to offload the work to the database. That functionality was
> > deprecated and AFAICT gone as of Mongo 5:
> >
> > https://www.mongodb.com/docs/manual/core/gridfs/#files.md5
> >
> > I am proposing two changes:
> >
> > * Remove deduplication
> > * Create a MongoDB DistributedMapCache client that can query on the
> > file metadata since GridFS stores metadata separately from chunks
> > making lookups that way cheap and flexible.
> >
> > I could easily add that to this PR which already covers Testcontainers
> > integration, making it super easy to test the changed behavior:
> >
> > https://github.com/apache/nifi/pull/6460
> >
> > Thoughts?
> >

Need to deprecate the deduplication functionality in the MongoDB GridFS processors

[Mike Thomsen]

The hash-based deduplication strategy used the built-in "md5"
attribute to offload the work to the database. That functionality was
deprecated and AFAICT gone as of Mongo 5:

https://www.mongodb.com/docs/manual/core/gridfs/#files.md5

I am proposing two changes:

* Remove deduplication
* Create a MongoDB DistributedMapCache client that can query on the
file metadata since GridFS stores metadata separately from chunks
making lookups that way cheap and flexible.

I could easily add that to this PR which already covers Testcontainers
integration, making it super easy to test the changed behavior:

https://github.com/apache/nifi/pull/6460

Thoughts?

Re: Suggestion for a new Expression Language method

[Mike Thomsen]

It would be something like this:

${ff_attr:isJson()}

So it should be a boolean.

On Fri, Oct 14, 2022 at 10:27 AM Dan S  wrote:
>
> I am looking to add isJson to the Apache NiFi Expression Language Guide
> <https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html>
> but
> I am not sure what category of functions this would fall under.
> I was thinking perhaps under Boolean Logic or Searching.  Does that make
> sense or is another category necessary?
>
> On Fri, Oct 7, 2022 at 4:12 PM Mike Thomsen  wrote:
>
> > NP. I'd recommend riffing on something like isNull. Should be a pretty
> > easy ticket. Tag me in the review.
> >
> > On Fri, Oct 7, 2022 at 12:19 PM Dan S  wrote:
> > >
> > > Mike,
> > >  I know you could raise a PR quickly but if you do not mind, I am still
> > > getting my feet wet with NIFI development so I would like to take a crack
> > > at implementing this. Thanks!
> > >
> > > On Fri, Oct 7, 2022 at 12:07 PM David Handermann <
> > > exceptionfact...@apache.org> wrote:
> > >
> > > > Dan,
> > > >
> > > > That comment appears to be historical. The current ObjectMapper
> > > > documentation indicates that instances are thread safe as long as all
> > > > configuration is completed before read or write operations:
> > > >
> > > >
> > > >
> > https://github.com/FasterXML/jackson-databind/blob/2.14/src/main/java/com/fasterxml/jackson/databind/ObjectMapper.java#L81
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Fri, Oct 7, 2022 at 11:05 AM Dan S  wrote:
> > > >
> > > > > Mike,
> > > > >  I noticed you did similar work in NIFI-5271
> > > > > <https://issues.apache.org/jira/browse/NIFI-5271>.  Joe Witt
> > commented
> > > > > there
> > > > >
> > > > > > We dont have guaranteed thread safety until the jackson 3.0
> > release.
> > > > >
> > > > >
> > > > > Has that changed?
> > > > >
> > > > > On Fri, Oct 7, 2022 at 11:55 AM Mike Thomsen  > >
> > > > > wrote:
> > > > >
> > > > > > To do it right, you want to use a tool like Jackson or Gson to do a
> > > > > > test parse after you've detected open and close statements that
> > seem
> > > > > > to be JSON open and close statements.
> > > > > >
> > > > > > On Fri, Oct 7, 2022 at 11:38 AM Dan S  wrote:
> > > > > > >
> > > > > > > Mike,
> > > > > > >  What do you mean by "this might be a somewhat heavy method so be
> > > > > > careful."
> > > > > > > ?
> > > > > >
> > > > >
> > > >
> >

Re: SSLContext from PEM files

[Mike Thomsen]

Here's the example from the Neo4J client driver javadoc:

https://neo4j.com/docs/api/java-driver/current/org/neo4j/driver/Config.TrustStrategy.html#trustCustomCertificateSignedBy-java.io.File...-

I think creating a custom component for Neo4J is the right move. It's
the only thing I've seen so far that requires a PEM and is
incompatible with a JKS or P12 file.

Besides, graph is a fast evolving field so I would assume most graph
users take it for granted that we might need to break things to keep
up with their stacks.

On Tue, Oct 11, 2022 at 3:12 PM David Handermann
 wrote:
>
> Mike,
>
> Thanks for raising this issue, can you provide some links to the
> documentation and source code for Neo4j?
>
> Although the SSL Context Service supports direct access to the Keystore and
> Trust Store properties, most use cases involve having the service
> instantiate an SSLContext. In this particular case, it may be better to
> specify those properties directly in a Neo4j component, as opposed to
> having an SSL Context Service that is essentially passing through property
> values.
>
> Those are a couple initial thoughts, having some additional background
> would help evaluate the best approach.
>
> Regards,
> David Handermann
>
> On Tue, Oct 11, 2022 at 12:36 PM Mike Thomsen 
> wrote:
>
> > Neo4J for some reason doesn't support the standard Java keystore types
> > or P12 files for its client SSL configuration. It requires the use of
> > PEM files. Would it be better to extend the SSLContext service types
> > to include support for PEM files or create an all new SSL Provider
> > type that is geared toward only reading from PEM files?
> >
> > Thanks,
> >
> > Mike
> >

SSLContext from PEM files

[Mike Thomsen]

Neo4J for some reason doesn't support the standard Java keystore types
or P12 files for its client SSL configuration. It requires the use of
PEM files. Would it be better to extend the SSLContext service types
to include support for PEM files or create an all new SSL Provider
type that is geared toward only reading from PEM files?

Thanks,

Mike

Re: Suggestion for a new Expression Language method

[Mike Thomsen]

NP. I'd recommend riffing on something like isNull. Should be a pretty
easy ticket. Tag me in the review.

On Fri, Oct 7, 2022 at 12:19 PM Dan S  wrote:
>
> Mike,
>  I know you could raise a PR quickly but if you do not mind, I am still
> getting my feet wet with NIFI development so I would like to take a crack
> at implementing this. Thanks!
>
> On Fri, Oct 7, 2022 at 12:07 PM David Handermann <
> exceptionfact...@apache.org> wrote:
>
> > Dan,
> >
> > That comment appears to be historical. The current ObjectMapper
> > documentation indicates that instances are thread safe as long as all
> > configuration is completed before read or write operations:
> >
> >
> > https://github.com/FasterXML/jackson-databind/blob/2.14/src/main/java/com/fasterxml/jackson/databind/ObjectMapper.java#L81
> >
> > Regards,
> > David Handermann
> >
> > On Fri, Oct 7, 2022 at 11:05 AM Dan S  wrote:
> >
> > > Mike,
> > >  I noticed you did similar work in NIFI-5271
> > > <https://issues.apache.org/jira/browse/NIFI-5271>.  Joe Witt commented
> > > there
> > >
> > > > We dont have guaranteed thread safety until the jackson 3.0 release.
> > >
> > >
> > > Has that changed?
> > >
> > > On Fri, Oct 7, 2022 at 11:55 AM Mike Thomsen 
> > > wrote:
> > >
> > > > To do it right, you want to use a tool like Jackson or Gson to do a
> > > > test parse after you've detected open and close statements that seem
> > > > to be JSON open and close statements.
> > > >
> > > > On Fri, Oct 7, 2022 at 11:38 AM Dan S  wrote:
> > > > >
> > > > > Mike,
> > > > >  What do you mean by "this might be a somewhat heavy method so be
> > > > careful."
> > > > > ?
> > > >
> > >
> >

Re: Suggestion for a new Expression Language method

[Mike Thomsen]

To do it right, you want to use a tool like Jackson or Gson to do a
test parse after you've detected open and close statements that seem
to be JSON open and close statements.

On Fri, Oct 7, 2022 at 11:38 AM Dan S  wrote:
>
> Mike,
>  What do you mean by "this might be a somewhat heavy method so be careful."
> ?

Re: Suggestion for a new Expression Language method

[Mike Thomsen]

I think this would be a fairly easy lift because Jackson's
ObjectMapper in 2.X is thread safe. We'd just need to put a warning in
the documentation saying "this might be a somewhat heavy method so be
careful."

I can make a PR for this pretty quickly.

On Fri, Oct 7, 2022 at 11:11 AM David Handermann
 wrote:
>
> Hi Dan,
>
> It seems like this could be helpful, although accurate detection might be
> difficult. It could be as simple as checking whether a string starts with a
> curly bracket or square bracket character. Another option would be
> attempting to parse the string using a JSON library, since that could cover
> additional bases. Either way, having sufficient unit tests and
> documentation would be helpful depending on the implementation approach.
>
> Regards,
> David Handermann
>
> On Fri, Oct 7, 2022 at 10:06 AM Pierre Villard 
> wrote:
>
> > Hi Dan,
> >
> > I personally think this is a fair addition to the expression language
> > options we have. Feel free to file a JIRA and submit a pull request if
> > you'd like to do so.
> >
> > Thanks,
> > Pierre
> >
> > Le ven. 7 oct. 2022 à 08:34, Dan S  a écrit :
> >
> > > My team has flow file attributes which we do not always know whether they
> > > are JSON or not. We would like to run JsonPath on them but when they are
> > > not  JSON we get flow files yielding as detailed in NIFI-10396
> > > . I understand the
> > > reasons given there not to support the patch but I was thinking of
> > possibly
> > > another solution which could short circuit the issue. I wanted to
> > propose a
> > > new Expression Language method isJson which would determine whether an
> > > attribute is JSON or not. This would allow for determining whether an
> > > attribute is JSON and pass it along to JsonPath or route accordingly when
> > > it is not JSON. Does that sound reasonable?
> > >
> >

Re: Queries regarding Nifi custom processor for Azure Data Explorer

[Mike Thomsen]

I think that is at least partly doable within NiFi (ex yes, you can
restrict processors to the primary node in a cluster), but I would
recommend you consider a different approach for NiFi. Unlike Spark,
NiFi is perfectly content to stream in huge amounts of data
continuously without any temporary storage besides its repositories
(flowfile, content, etc). Therefore, I think a potentially easier
solution would be for your team to explore creating a connector
between NiFi and Azure Data Explorer that allows the latter to
firehose the former with data as it comes in and let the chips fall
where they may.

You might find some useful concepts in the Kafka processors for things
like managing a continuous flow of record data from a stream and
converting it to blocks of NiFi record data (see the Record API in our
documentation for details).

FWIW, I manage a data set in AWS w/ NiFi that is over 50TB compressed,
and a fairly generic 5 node NiFi cluster crushes that data on cheap
EC2 instances without issue. So handling TBs of data is something
fairly out of the box for NiFi if you're worried about that.

On Sat, Oct 1, 2022 at 12:05 AM Tanmaya Panda
 wrote:
>
> Hi Team,
>
> We at Microsoft opensource, are developing a custom for Azure Data Explorer 
> sink connector for Apache Nifi. What we want to achieve is transactionality 
> with data ingestion. The source of the processor can be TBs of telemetry data 
> as well as CDC logs. What that means is in case of any failure while 
> ingesting data of particular partition to ADX, will delete/cleanup any other 
> ingested data of other partitions. Since Azure Data Explorer is an append 
> only database, unfortunately we cant perform delete on the ingested data of 
> same partition or other partition. So to achieve this kind of 
> transactionality of large ingested data, we are thinking to implement 
> something similar we have done for Apache Spark ADX connector. We will be 
> creating temporary tables inside Azure Data Explorer before ingesting to the 
> actual tables. The worker nodes in apache spark will create these temporary 
> tables and report the ingestion status to the driver node. The driver node on 
> receiving the success status of all the worker nodes performs ingestion on 
> the actual table or else the ingestion is aborted along with cleanup of 
> temporary table. So basically we aggregate the worker node task status in the 
> driver node in spark to take further decision on whether to ingest data into 
> ADX table or not.
> Question 1: Is this possible in Apache Nifi follows a zero master cluster 
> strategy as opposed to master-slave architecture of apache spark?
> Question 2: In our custom processor in Nifi, is it possible to run custom 
> code of a particular partition say the cluster coordinator node. Also is it 
> possible to get the details of the partition inside the processor?
> Question 3: Is is to possible to get the details of tasks executed at the 
> various partitions and take decisions based on the task status. Can all these 
> be done inside the same processor.
>
> Thanks,
> Tanmaya

Re: Timeouts on Windows builds

[Mike Thomsen]

It might be worth testing Windows against Java 11 or even 17. My guess
is that Java 8 is not getting much support on Windows since OpenJDK 8
seems to be mainly supported by RH/IBM.

On Thu, Sep 29, 2022 at 11:28 AM Joe Witt  wrote:
>
> Yeah definitely not - the CI/CD struggle is real and will always be...
> We just need to keep improving it as things change in those
> environments.
>
> On Thu, Sep 29, 2022 at 8:26 AM Mike Thomsen  wrote:
> >
> > Noticing a lot of timeouts on the Windows builds I've been restarting
> > lately. We don't consider those blockers if the Linux/macOS builds
> > succeed, right?

Timeouts on Windows builds

[Mike Thomsen]

Noticing a lot of timeouts on the Windows builds I've been restarting
lately. We don't consider those blockers if the Linux/macOS builds
succeed, right?

Re: Testcontainers for managing integration tests

[Mike Thomsen]

Indeed it is. Thanks for pointing that out.

On Wed, Sep 28, 2022 at 1:28 PM Pierre Villard
 wrote:
>
> I believe this is what we're already doing for NiFi Registry and its
> supported databases:
> https://github.com/apache/nifi/tree/main/nifi-registry/nifi-registry-core/nifi-registry-test
>
> Pierre
>
> Le mer. 28 sept. 2022 à 19:22, Mike Thomsen  a
> écrit :
>
> > We've been successfully using testcontainers (the FOSS project) where
> > I work for Elastic, Cassandra and Postgres. Was wondering if anyone
> > had any opinions about starting to integrate this project where we can
> > (again, IT only)
> >
> > URL for those interested: https://www.testcontainers.org/
> >
> > Not sure if this has already come up.
> >

Testcontainers for managing integration tests

[Mike Thomsen]

We've been successfully using testcontainers (the FOSS project) where
I work for Elastic, Cassandra and Postgres. Was wondering if anyone
had any opinions about starting to integrate this project where we can
(again, IT only)

URL for those interested: https://www.testcontainers.org/

Not sure if this has already come up.

Re: Nifi 1.17 does exit after start

[Mike Thomsen]

Probably didn't apt install openjdk on the machine. It's not installed
by default on Ubuntu.

On Wed, Sep 21, 2022 at 10:44 AM Joe Witt  wrote:
>
> Hello
>
> You will want to check the /logs/nifi-app.log for clues on what 
> happened.
>
> Thanks
>
> On Wed, Sep 21, 2022 at 7:43 AM Froim Teschner
>  wrote:
> >
> > Hello,
> >
> > I downloaded nifi-1.17.0-bin.zip on a ubuntu 20.04 system and extracted the 
> > file as suggested. After I launch nifi, it just exits again after a few 
> > seconds without any notice. I have set the Java home path! What else could 
> > I check?
> >
> > user@user:~/snap/nifi-1.17.0-bin/nifi-1.17.0$ bin/nifi.sh start
> >
> > Java home: /usr/lib/jvm/java-11-openjdk-amd64/
> > NiFi home: /home/froimteschner/snap/nifi-1.17.0-bin/nifi-1.17.0
> >
> > Bootstrap Config File: 
> > /home/froimteschner/snap/nifi-1.17.0-bin/nifi-1.17.0/conf/bootstrap.conf
> >
> >
> > user@user:~/snap/nifi-1.17.0-bin/nifi-1.17.0$
> >
> >
> >
> > Kind regards,
> > Froim

Re: Nifi python usage

[Mike Thomsen]

In general, I would recommend moving the Python code to Flask-based
rest services running on something like Kubernetes. That way you can
access them with InvokeHttp which has a much lower overhead than
ExecuteStreamCommand (no CPython startup per run). Creating
containerized Python services is not hard, so I would recommend that
route as a way of really dropping the resource needs of your NiFi
servers.

On Wed, Sep 14, 2022 at 11:13 AM Jeremy Dyer  wrote:
>
> Hi,
> As James mentioned in your example the "system" memory would be used by the
> process/script in this case.
>
> I wanted to add that if you use the "ExecuteScript" processor that the NiFi
> memory would be used. This is because NiFi uses Jython for processing
> Python files. So where a native Cython process would typically create a
> heap for a Python objects in a native process, Jython will instead use the
> Java heap
>
> Jeremy Dyer
>
> On Wed, Sep 14, 2022 at 8:36 AM James Halfpenny  wrote:
>
> > Hi,
> > The memory you allocate to NiFi will typically be the maximum heap size
> > for the Java process. If you spawn a separate Python process then it will
> > not be bound by the JVM heap size, so in your example it would come from
> > the 8GB allocated for the “system”.
> >
> > I would recommend if you see high resource usage on the server again you
> > take note of which processes are consuming the resources.
> >
> > Kind regards,
> > Jim
> >
> > > On 14 Sep 2022, at 14:18, never more  wrote:
> > >
> > > Hello!
> > > Can you help me with one question - can't find any information :(
> > >
> > > We have a nifi on my server (16 CPU, 32  RAM), in config our system adm.
> > > set ram usage for nifi = 24 Gb, rest 8 Gb - for the system.
> > > In Nifi we execute many Python scripts by using ExecuteStreamCommand
> > (just
> > > set path of script on server in properties).
> > > So the question is - which resources  python script uses - Nifi (24 gb)
> > or
> > > system (8 gb)?
> > >
> > > Maybe its a little stupid question, but I am afraid that improper use
> > of
> > > resources may lead to a service failure,
> > > because we  already  have several cases of service failure when we
> > noticed
> > > high utilization of resources (RAM 90-95% and CPU 80-90%)
> >

Re: H12 database lock issue in NIFI

[Mike Thomsen]

> is there any professional services help provided for nifi ?

Cloudera provides great support (I'm not a Cloudera employee). They would
be a good option. Another option would be to try updating to 1.16.X or
1.17. I think somewhere in there, the H2 version was updated so you might
get a better version of H2 in there.

On Fri, Aug 19, 2022 at 6:01 PM pradeep tharthius 
wrote:

> Hi Team,
> We are using NIFI for our ETL transformation needs in our application .
> The version we have is Nifi 1.15.1 . we have been hit with a roadblock
> with an issue where nifi crashes with a write lock on the H2 database .
>
> Please see the error below ,
>
> [image: image.png]
>
> We have tried using H2 in server mode but that did not resolve the issue .
> Any suggestions or guidance you can share based on similar experiences ?
>
> any help is appreciated , and is there any professional services help
> provided for nifi ? Please advise
>
> Regards,
> Tharthiusp
>
>
>

Re: [discuss] nifi 1.17

[Mike Thomsen]

I would put this one on the list of "potential blockers" for 1.17:
https://github.com/apache/nifi/pull/6226

On Fri, Jul 22, 2022 at 4:52 PM Joe Witt  wrote:
>
> Hello Everyone,
>
> Somehow Apache NiFi 1.16 is already four months old and we've landed just
> under 300 JIRAs [1] on the 1.17 line.  So certainly time and more than
> enough oomph in place to warrant a 1.17 release.  I have seen quite a few
> asks lately about when we'll do it.  So looking at things in JIRA it seems
> pretty well buttoned up.  A few bugs being worked out but with PRs.
>
> Following this cool discussion [2] we now have the ability to build NiFi on
> ARM/latest Macbooks which is super helpful for many in the community as
> well.
>
> I'm happy to do the RM task.  Will take a look at these few bug PRs on
> Monday and if we're looking good will kick off RC1.
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351438
> [2] https://lists.apache.org/thread/mm1xf3b9nvrcgytb92oy3swvvc45fl34
>
> Thanks
> Joe

Re: How to manage security artifacts from a custom processor

[Mike Thomsen]

> However, where/how do I install the certificates I'll be given for use?

The JDK keytool can be used to create the JKS artifacts you need.

On Tue, Jul 5, 2022 at 11:33 PM Russell Bateman  wrote:
>
> I appreciate the responses. I will try out the canonical
> /StandardSSLContextService/ first (since that's what I am using with
> Kafka), then imitate the other sample depending.
>
> However, where/how do I install the certificates I'll be given for use?
> I would expect something for certain representing the third-party
> service in a truststore and maybe another (a private key) in a keystore.
>
>
> On 7/5/22 16:30, Russell Bateman wrote:
> > From a custom processor, I intend to interface with a third-party
> > service (via simple HTTP client), however, I would need as I
> > understand it to
> >
> > a) maintain a private key by which I can identify myself to that
> > third-party service and
> > b) maintain a trusted-store certificate by which I can guarantee
> > the identity of the service.
> >
> > This is pretty far outside my own experience. I have been reading on
> > how this is achieved in Java, but in my mind a complication arises
> > from the fact that a custom NiFi processor lives within NiFi's JVM. My
> > question is therefore, how can I control the certificates and
> > authorities for my use in or associated with NiFi's JVM. Clearly, I
> > don't grok this well enough even to ask the question; I'm hoping
> > someone can see through what I'm asking and point me in a good
> > direction to study.
> >
> > I've written a pile of successful and useful custom NiFi processors to
> > cover proprietary needs, so custom-processor writing isn't a mystery.
> > Certificates, keys, trusts and security in general still is.
> >
> > Profuse thanks,
> >
> > Russ

Re: How to manage security artifacts from a custom processor

[Mike Thomsen]

Example of how to do this:

https://github.com/apache/nifi/blob/aa61494fc3a68b4806784f67ad837ee821d26da4/nifi-nar-bundles/nifi-standard-services/nifi-oauth2-provider-bundle/nifi-oauth2-provider-service/src/main/java/org/apache/nifi/oauth2/OAuth2TokenProviderImpl.java

On Tue, Jul 5, 2022 at 8:31 PM Otto Fowler  wrote:
>
>  Usually, you would write you custom processor to support the
> StandardSSLSocketService:
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-ssl-context-service-nar/1.16.3/org.apache.nifi.ssl.StandardSSLContextService/index.html
>
>
>
>
> From: Russell Bateman  
> Reply: dev@nifi.apache.org  
> Date: July 5, 2022 at 18:30:46
> To: NiFi Developers List  
> Subject:  How to manage security artifacts from a custom processor
>
> From a custom processor, I intend to interface with a third-party
> br/>servicee (via simple HTTP client), however, I would need as I
> understand br//>it to
>
> a) maintain a private key by which I can identify myself to that
> third-party service and
> b) maintain a trusted-store certificate by which I can guarantee the
> identity of the service.
>
> This is pretty far outside my own experience. I have been reading on how
> br/>this is achieved in Java, but in my mind a complication aarises from
> the br/>fact that a custom NiFFi processor lives within NiFi's JVM. My
> question br/>is therefore, how can I control the ceertificates and
> authorities for my br/>use in or associated with NiFFi's JVM. Clearly, I
> don't grok this well br/>enough even to ask the qquestion; I'm hoping
> someone can see through what br/>I'm asking and point me in a good
> direction to study.
>
> I've written a pile of successful and useful custom NiFi processors to
> br/>cover proprietary needs, so custom-processor writing isn''t a mystery.
> br/>Certificates, keys, trusts and security in general still is.
>
> Profuse thanks,
>
> Russ

Re: FW: Apache NIFI service SRG

[Mike Thomsen]

Patrick,

I sent you an email off the mailing list that might help you get what
you're looking for. I assume the right email for you is your email
listed here minus ".invalid." If not, sync up with me off list and I
think my company has some folks who can help you out.

Thanks,

Mike

On Fri, Jun 24, 2022 at 4:01 PM G. Patrick McDaniel
 wrote:
>
> Hello,
>
> Please see the email I sent to DISA RMS below. I am looking for the Apache 
> NIFI SRG, specifically that which talks about configuring the NIFI service. 
> Is there an SRG or is it pretty much covered in the Apache STIG?
>
> Thank you,
> VR
> Pat
>
>
>
> G Patrick McDaniel
> Ph: 202.203.1773
>
> From: McDaniel, G. Patrick (OCIO) (CON)
> Sent: Friday, June 24, 2022 2:37 PM
> To: disa.stig_...@mail.mil
> Subject: Apache NIFI service SRG
>
> Hi,
>
> I don't have a cac (any longer) and am having a hell of a time locating the 
> SRG that tells me how to configure the NIFI service that comes with the 
> Apache SRG/STIG. It's my understanding that it is not a STIG but an SRG. I 
> think the information is on the Apache website but I haven't been able to 
> locate anything on your website on the pages that don't require a cac.
>
> Any help is appreciated. Thank you.
>
> VR
> Pat
>
>
>
> G Patrick McDaniel
> Information Systems Security Engineer (ISSE)
> Information Security Risk Management Unit (ISRMU)
> Office of the Chief Information Officer (OCIO)
> FBIHQ/PP-2, RM. W5-78
> 375 E Street SW, Washington, DC 20024
> Ph: 202.203.1773
>

Re: Upgrading Cassandra driver and bringing in breaking changes with it

[Mike Thomsen]

Steve,

Do you want to break up the ticket into a controller service ticket
and an update ticket for the processors? You could do the former while
I do the latter if you want to get your hands dirty making a
contribution here.

Thanks,

Mike

On Fri, Jun 10, 2022 at 4:32 PM Steven Matison  wrote:
>
> I'd like to get in on this one if you guys need anything.
>
>
>
> On Fri, Jun 10, 2022 at 4:12 PM David Handermann <
> exceptionfact...@apache.org> wrote:
>
> > Mike,
> >
> > Thanks for digging into this issue! I agree that simplifying the
> > implementation makes going forward. The newer driver version involves
> > changes in the Controller Service method signatures either way, so
> > streamlining the connection handling approach would be helpful.
> >
> > Based on feedback, perhaps summarizing the planned implementation in the
> > Jira issue would be helpful.
> >
> > Regards,
> > David Handermann
> >
> > On Fri, Jun 10, 2022 at 3:06 PM Mike Thomsen 
> > wrote:
> >
> > > https://issues.apache.org/jira/browse/NIFI-9770
> > >
> > > Based on David's description saying it will require some deep changes
> > > to how various properties are configured, I think this ticket would be
> > > a good time to simplify the Cassandra bundle in general to just use a
> > > controller service to configure the connection. (Current approach is
> > > controller service or manual configuration on a per-processor basis).
> > >
> > > Thoughts?
> > >
> >

Upgrading Cassandra driver and bringing in breaking changes with it

[Mike Thomsen]

https://issues.apache.org/jira/browse/NIFI-9770

Based on David's description saying it will require some deep changes
to how various properties are configured, I think this ticket would be
a good time to simplify the Cassandra bundle in general to just use a
controller service to configure the connection. (Current approach is
controller service or manual configuration on a per-processor basis).

Thoughts?

Re: Regarding Creating Email Alerts and Monitoring Tasks

[Mike Thomsen]

Instead of creating an api/webhooks, you can just use a message queue
like Kafka, ActiveMQ, etc. Have NiFi subscribe to a queue and act on
inputs it gets from other systems. Does that answer your question?

On Thu, Jun 9, 2022 at 1:38 PM Shreyas Srinivasa
 wrote:
>
> Hi,
>
> I am Shreyas working as a Data Engineer, I am currently using NIFI as an ETL 
> tool. I was working on building email alerts in the ETL flow using PUTEMAIL 
> task however was interested to know if there was a way to create webhooks or 
> use api to create alerts in nifi [Alerts for example If there are no rows in 
> a table]
>
> In addition, Please feel free to let me know if there is any other 
> alternatives other than PUTEMAIL task for creating alerts and monitoring ETL 
> flows
>
> Really looking forward to hearing from you soon
>
> Thanks,
> Shreyas
>
>
> [Logo] 
> Shreyas Srinivasa
> Data Engineer
> P:
> www.hhaexchange.com 
> [Facebook icon]   [LinkedIn icon]  
>   [Twitter icon] 
>     [Vimeo icon]  
> 
>
> [Logo]
>
> 
> This email and its attachments may contain privileged and confidential 
> information and/or protected health information (PHI) and may be subject to 
> protection under the law, including the Health Insurance Portability and 
> Accountability Act of 1996, as amended (HIPAA). This transmission is intended 
> for the sole use of the individual or entity to whom it is addressed. If you 
> are not the intended recipient, you are notified that any use, dissemination, 
> distribution, printing or copying of this transmission is strictly prohibited 
> and may subject you to criminal or civil penalties. If you have received this 
> message in error or are not the named recipient, please notify us immediately 
> by contacting the sender at the electronic mail address noted above, and 
> delete and destroy all copies of this message. Thank you.
>
> Disclaimer
>
> The information contained in this communication from the sender is 
> confidential. It is intended solely for use by the recipient and others 
> authorized to receive it. If you are not the recipient, you are hereby 
> notified that any disclosure, copying, distribution or taking action in 
> relation of the contents of this information is strictly prohibited and may 
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been 
> automatically archived by Mimecast Ltd, an innovator in Software as a Service 
> (SaaS) for business. Providing a safer and more useful place for your human 
> generated data. Specializing in; Security, archiving and compliance. To find 
> out more visit the Mimecast website.

Re: [discuss] nifi 1.16.2

[Mike Thomsen]

Joe/Mark,

Correct me if I'm wrong, but 9993 is not an issue if you just turn off
content archiving, right?

Thanks,

Mike

On Mon, May 9, 2022 at 12:49 PM David Handermann
 wrote:
>
> Joe,
>
> Thanks for initiating this discussion and moving toward 1.16.2.
>
> The following two issues should be good candidates to include in 1.16.2,
> among other potentials:
>
> https://issues.apache.org/jira/browse/NIFI-9988 - Corrected handling of
> property decryption in authorizers.xml and login-identity-providers.xml
> https://issues.apache.org/jira/browse/NIFI-9980 - Corrected Google
> dependencies for GCP components
>
> Regards,
> David Handermann
>
> On Mon, May 9, 2022 at 11:39 AM Joe Witt  wrote:
>
> > Team,
> >
> > Just saw this one land https://issues.apache.org/jira/browse/NIFI-9993.
> > And while the underlying issue has been around a while this also seems
> > likely something that could create a lot of disruption.  I'm planning to
> > spin up a 1.16.2 to address it since I'm not sure how long until 1.17.0.
> >
> > I'll look for other things that have landed which fit and spin up an RC in
> > the next day or two but will keep an eye here for anything else flagged
> > that might make sense.
> >
> > Thanks
> >

Re: [VOTE] Release Apache NiFi 1.16.1

[Mike Thomsen]

+1 (binding) Ran a simple tika extraction flow on PDFs and DOCX files.
Worked fine. Saved that flow into the NiFi Registry.

On Tue, Apr 26, 2022 at 4:47 PM Matt Burgess  wrote:
>
> +1 (binding)
>
> Ran through release helper, tested various flows including version
> control with the Registry, InvokeScriptedProcessor, Relationship
> retry, etc.  I did create [1] for MiNiFi but since there's a
> workaround it's not a dealbreaker IMO, I'm already working on it for
> 1.17.0.
>
> Thanks for RM'ing Joe!
>
> [1] https://issues.apache.org/jira/browse/NIFI-9967
>
> On Mon, Apr 25, 2022 at 2:29 PM Joe Witt  wrote:
> >
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache NiFi
> > 1.16.1.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1200
> >
> > The source being voted upon and the convenience binaries can be found at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.1/
> >
> > A helpful reminder on how the release candidate verification process works:
> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >
> > The Git tag is nifi-1.16.1-RC1
> > The Git commit ID is 81166797e552b9d14b482807632f2f04321b2018
> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=81166797e552b9d14b482807632f2f04321b2018
> >
> > Checksums of nifi-1.16.1-source-release.zip:
> > SHA256: 2c39b45ba8eec42d601d5c9facad623d14660dd209c011b4a13b2b559b9f3e32
> > SHA512:
> > cd670ab558937cac709dea0b4be3351559f57c9e9aedf54d5153706eee386a021262ef199e2bf9485763cf955931edfd6a24ca1c5a0748a77e3eeb91c490cbda
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 83 issues were closed/resolved for this release:
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351504
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.1
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test. Then
> > please vote:
> >
> > [ ] +1 Release this package as nifi-1.16.1
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...

Anyone else seeing this on macOS with the RC1 build?

[Mike Thomsen]

This laptop can be a little wonky at times because of the corporate
lockdowns, so I wanted to see if any other committers/PMC members have
seen this in nifi-record-path:

[INFO] Running org.apache.nifi.record.path.util.TestFieldValueWalker

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.083 s - in org.apache.nifi.record.path.util.TestFieldValueWalker

[INFO] Running org.apache.nifi.record.path.util.TestFieldValueLogicalPathBuilder

[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.007 s - in org.apache.nifi.record.path.util.TestFieldValueLogicalPathBuilder

[INFO] Running org.apache.nifi.record.path.TestRecordPath

[ERROR] Tests run: 76, Failures: 0, Errors: 1, Skipped: 0, Time
elapsed: 0.336 s <<< FAILURE! - in
org.apache.nifi.record.path.TestRecordPath

[ERROR] org.apache.nifi.record.path.TestRecordPath  Time elapsed:
0.336 s  <<< ERROR!

java.lang.NullPointerException

at 
java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)

at 
java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)

at java.base/java.util.Properties.put(Properties.java:1301)

at java.base/java.util.Properties.setProperty(Properties.java:229)

at java.base/java.lang.System.setProperty(System.java:999)

at 
org.apache.nifi.record.path.TestRecordPath.setSystemTimezone(TestRecordPath.java:82)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)

at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

Re: Missing ECMAScript option with ExecuteScript Processor

[Mike Thomsen]

The Nashorn Javascript engine used to be bundled with the JRE, but it
was removed with JRE 15.

https://en.wikipedia.org/wiki/Nashorn_(JavaScript_engine)

On Sat, Apr 23, 2022 at 6:02 PM Bryan Doherty  wrote:
>
>  Hi,
>
> When I add an ExecuteScript block to the canvas, ECMAScript is NOT a Script
> Engine option.  All the other options are there.  Is there some underlying
> version of something on the OS that affects which Engine options are
> visible?
>
> *Using:*
> Amazon Linux EC2
> Nifi 1.16.0
> Java 17 & 18, tried both
> Node 16+
>
>
> Thanks
> Bry

1.16.1 RC?

[Mike Thomsen]

Do we have any particular tasks left under review that are blocking
the 1.16.1 RC?

Re: Compiling / Testing a single processor

[Mike Thomsen]

mvn test -Dtest= from the module where the
test resides.

On Sat, Apr 9, 2022 at 7:21 AM Phil H  wrote:
>
> Hi there,
>
> Now that my environment is working nicely, I am working on SplitContent 
> (Nifi-1517).
>
> Is there a way to tell maven to only compile and/or run the unit tests for 
> the SplitContent processor, rather than testing the entire 
> nifi-standard-processors bundle?
>
> Thanks,
> Phil

Re: Need some feedback on how upgrading Avro might cause problems

[Mike Thomsen]

In addition to that point about schema inference, the correct thing to
do with inference would be to do the type,null ordering and not
null,type.

On Thu, Apr 7, 2022 at 12:51 PM Matt Burgess  wrote:
>
> I agree that moving forward with the upgrade is the right thing to do,
> the example Mike pointed out is invalid according to the 1.8.1 spec
> [1] even though it was mistakenly allowed by Avro itself. Since we
> don't infer default values I think we can correctly require that
> hand-written schemas conform to the spec, especially with respect to
> default values, and our inferred schemas will adhere to the spec.
>
> I also agree we should mention this in the Migration Guidance doc as
> it is a requirement that is now being enforced when it wasn't before.
>
> There might be an issue with Hive 1 but I don't think we're on the
> hook for that, if Hive has issues with nullable types and/or default
> values of null, it should obey the spec the same as we will.
>
> Regards,
> Matt
>
> [1] https://avro.apache.org/docs/1.8.1/spec.html#Unions
>
> On Thu, Apr 7, 2022 at 9:20 AM David Handermann
>  wrote:
> >
> > Mike,
> >
> > Thanks for raising this issue for discussion, and thanks Isha and Edward
> > for the comments. It is helpful to get an understanding of the potential
> > impact.
> >
> > In general, it seems like moving forward with the upgrade is the best idea,
> > given the issues with older versions. For integration with other products
> > outside of NiFi, using the latest version of Avro should provide better
> > compatibility.
> >
> > That being said, it is important to consider the impact on existing
> > deployments. At minimum, this change is something that should be described
> > in the NiFi Migration Guidance associated with the released version. As far
> > as supporting different versions, it is possible to run older versions of
> > the various components, although this is more complicated given the scope
> > of the upgrade. This seems like a case where providing more detailed
> > migration steps would be helpful, to include issues related to cached
> > schemas or inferred schemas used as the basis for customization.
> >
> > Regards,
> > David Handermann
> >
> > On Thu, Apr 7, 2022 at 6:08 AM Edward Armes  wrote:
> >
> > > I've had a quick look in JIRA and it looks like this might have happened 
> > > as
> > > a side effect of AVRO-1544.
> > >
> > > I think it is worth upgrading especially given that it looks like few of
> > > the changes refer to updating against newer bundled dependencies some of
> > > which seem to  have CVE's against them.
> > >
> > > Depending on peoples feelings would it be wroth creating 2 versions one
> > > using Avro 1.8 and one using Avro 1.11.0 and then removing the 1.8 version
> > > in a later release?
> > >
> > > On an additional note in cases where people are writing schemas manually I
> > > suspect they are probably going to be validating against the later 
> > > versions
> > > of Avro using the projects tooling and that may create issues further down
> > > the line.
> > >
> > > Edward
> > >
> > > On Thu, Apr 7, 2022 at 11:52 AM Isha Lamboo <
> > > isha.lam...@virtualsciences.nl>
> > > wrote:
> > >
> > > > Hi Mike,
> > > >
> > > > The "Infer schema" functionality in NiFi currently generates schemas 
> > > > with
> > > > the order that will be invalid under Avro 1.9+. I noticed because I've
> > > been
> > > > using that to copy-paste schemas that were "almost right" so I could
> > > > manually fix them.
> > > >
> > > > I guess that inferred schemas should be fine if the inferring logic is
> > > > also upgraded by the dependency, but for any cached schemas and my own
> > > > manually saved schemas this will be somewhat painful.
> > > > My use case for manually saving inferred schemas is mostly database
> > > > migration where some inferred "choice" fields are not supported for the
> > > > target database.
> > > >
> > > > Hope this helps,
> > > >
> > > > Isha
> > > >
> > > > -Oorspronkelijk bericht-
> > > > Van: Mike Thomsen 
> > > > Verzonden: donderdag 7 april 2022 12:11
> > > > Aan: dev@nifi.apache.org
> > > > Onderwerp: Need some feedback on how upgrading Avro might cause problems
> > > >
&g

Deprecate Livy support?

[Mike Thomsen]

https://github.com/apache/incubator-livy

Looks like Livy hasn't had any updates in about two years. Maybe it's
time to deprecate for it since it looks like it hasn't made any
progress in moving out of the incubator.

Need some feedback on how upgrading Avro might cause problems

[Mike Thomsen]

Thread is here for full details:
https://github.com/apache/nifi/pull/5900#pullrequestreview-922490039

It looks like Avro 1.8's schema parser may have been more lenient (or
buggy) in enforcing the specification with respect to the ordering of
a union for a nullable type. 1.9.X and higher are definitely more
opinionated and throw exceptions on schemas that used to work on
1.8.X. For example, this used to be valid:

{
  "name": "message",
  "type": [ "null", "string" ],
  "default": "Hello, world"
}

Now Avro **correctly** throws an exception per the specification.
Under 1.8 it did not, and as you can see here, I had to change
numerous test schemas in order to make them work under 1.9 to 1.11.0:

https://github.com/apache/nifi/pull/5900/files#r835954170

As I said to Matt, I think we're in a "damned if you do, damned if you
don't" position here.

Thoughts?

Re: New build problem, Java 17

[Mike Thomsen]

Related. You said you're used to building in offline environments. You
were also using a really old build Java 8 early on in the previous
thread. If you're building on a commercial or government network, you
might want to throw the JDK and Maven on a personal laptop and do a
build there. Commercial and government setups are often filled with
security stuff that messes with Maven.

On Wed, Apr 6, 2022 at 8:02 PM Mike Thomsen  wrote:
>
> > However it’s not a network issue because…
>
> Could be something funky with how Java 17's TLS APIs are working on your 
> setup.
>
> Java 8 and Java 11 are your best bets for now anyway. Java 17 support
> is still in the early stages and isn't a preferred JDK/JRE.
>
> On Wed, Apr 6, 2022 at 7:00 PM Phil H  wrote:
> >
> > So I upgraded to the Azul JDK 17.  New repo, following the same 
> > instructions as before.
> >
> >  phil@Phils-MacBook-Pro nifi % mvn -T C2.0 clean install -Pinclude-grpc
> >  [INFO] Scanning for projects...
> >  Downloading from central: 
> > https://repo1.maven.org/maven2/org/apache/apache/25/apache-25.pom
> >  Downloading from apache-repo: 
> > https://repository.apache.org/content/repositories/releases/org/apache/apache/25/apache-25.pom
> >  Downloading from Shibboleth: 
> > https://build.shibboleth.net/nexus/content/repositories/releases/org/apache/apache/25/apache-25.pom
> >  [ERROR] [ERROR] Some problems were encountered while processing the 
> > POMs:
> >  [FATAL] Non-resolvable parent POM for 
> > org.apache.nifi:nifi:1.17.0-SNAPSHOT: Could not transfer artifact 
> > org.apache:apache:pom:25 from/to central (https://repo1.maven.org/maven2): 
> > transfer failed for 
> > https://repo1.maven.org/maven2/org/apache/apache/25/apache-25.pom and 
> > 'parent.relativePath' points at no local POM @ line 14, column 13
> >  @
> >  [ERROR] The build could not read 1 project -> [Help 1]
> >  [ERROR]
> >  [ERROR]   The project org.apache.nifi:nifi:1.17.0-SNAPSHOT 
> > (/Users/phil/nifi/pom.xml) has 1 error
> >  [ERROR] Non-resolvable parent POM for 
> > org.apache.nifi:nifi:1.17.0-SNAPSHOT: Could not transfer artifact 
> > org.apache:apache:pom:25 from/to central (https://repo1.maven.org/maven2): 
> > transfer failed for 
> > https://repo1.maven.org/maven2/org/apache/apache/25/apache-25.pom and 
> > 'parent.relativePath' points at no local POM @ line 14, column 13: Network 
> > is unreachable -> [Help 2]
> >  [ERROR]
> >  [ERROR] To see the full stack trace of the errors, re-run Maven with 
> > the -e switch.
> >  [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> >  [ERROR]
> >  [ERROR] For more information about the errors and possible solutions, 
> > please read the following articles:
> >  [ERROR] [Help 1] 
> > http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
> >  [ERROR] [Help 2] 
> > http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
> >
> > However it’s not a network issue because…
> >
> >  phil@Phils-MacBook-Pro nifi % curl 
> > https://repo1.maven.org/maven2/org/apache/apache/25/apache-25.pom
> >  
> >
> >  

Re: New build problem, Java 17

[Mike Thomsen]

> However it’s not a network issue because…

Could be something funky with how Java 17's TLS APIs are working on your setup.

Java 8 and Java 11 are your best bets for now anyway. Java 17 support
is still in the early stages and isn't a preferred JDK/JRE.

On Wed, Apr 6, 2022 at 7:00 PM Phil H  wrote:
>
> So I upgraded to the Azul JDK 17.  New repo, following the same instructions 
> as before.
>
>  phil@Phils-MacBook-Pro nifi % mvn -T C2.0 clean install -Pinclude-grpc
>  [INFO] Scanning for projects...
>  Downloading from central: 
> https://repo1.maven.org/maven2/org/apache/apache/25/apache-25.pom
>  Downloading from apache-repo: 
> https://repository.apache.org/content/repositories/releases/org/apache/apache/25/apache-25.pom
>  Downloading from Shibboleth: 
> https://build.shibboleth.net/nexus/content/repositories/releases/org/apache/apache/25/apache-25.pom
>  [ERROR] [ERROR] Some problems were encountered while processing the POMs:
>  [FATAL] Non-resolvable parent POM for 
> org.apache.nifi:nifi:1.17.0-SNAPSHOT: Could not transfer artifact 
> org.apache:apache:pom:25 from/to central (https://repo1.maven.org/maven2): 
> transfer failed for 
> https://repo1.maven.org/maven2/org/apache/apache/25/apache-25.pom and 
> 'parent.relativePath' points at no local POM @ line 14, column 13
>  @
>  [ERROR] The build could not read 1 project -> [Help 1]
>  [ERROR]
>  [ERROR]   The project org.apache.nifi:nifi:1.17.0-SNAPSHOT 
> (/Users/phil/nifi/pom.xml) has 1 error
>  [ERROR] Non-resolvable parent POM for 
> org.apache.nifi:nifi:1.17.0-SNAPSHOT: Could not transfer artifact 
> org.apache:apache:pom:25 from/to central (https://repo1.maven.org/maven2): 
> transfer failed for 
> https://repo1.maven.org/maven2/org/apache/apache/25/apache-25.pom and 
> 'parent.relativePath' points at no local POM @ line 14, column 13: Network is 
> unreachable -> [Help 2]
>  [ERROR]
>  [ERROR] To see the full stack trace of the errors, re-run Maven with the 
> -e switch.
>  [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>  [ERROR]
>  [ERROR] For more information about the errors and possible solutions, 
> please read the following articles:
>  [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
>  [ERROR] [Help 2] 
> http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
>
> However it’s not a network issue because…
>
>  phil@Phils-MacBook-Pro nifi % curl 
> https://repo1.maven.org/maven2/org/apache/apache/25/apache-25.pom
>  
>
>  

Re: [discuss] nifi 1.16.1

[Mike Thomsen]

I have a few PRs for standardizing and updating our dependencies like
Jackson and Avro. Might be good to get those included in 1.16.1

On Mon, Apr 4, 2022 at 3:02 PM Joe Witt  wrote:
>
> Team,
>
> Sorry for the delays in wrapping up the 1.16 release from earlier last
> week.  We had great vote turnout and it was definitely a large release in
> terms of features/etc..
>
> Watching evolution on the 1.17 line since it is clear we can benefit from a
> 1.16.1 release so I've started preparing that and will closely monitor
> candidates for inclusion.
>
> The branch is pushed and is called 'support/nifi-1.16'.  I've already
> pulled in everything on 1.17 that looks like it can work in a pure 3rd
> digit release.  Some of this makes sense due to bugs found and or natural
> concerns of scanners for things like springshell/etc..  In any case it is
> always good to tighten up the release lines if time permits.
>
> I'll probably start the release process tomorrow or Wednesday.  Will watch
> for anything that comes in.
>
> Thanks

Re: Missing maven dependencies when building nifi

[Mike Thomsen]

Did you run "mvn install" from the root of the project before
attempting to run a full test of the code base?

On Mon, Apr 4, 2022 at 9:36 AM Phil H  wrote:
>
> Hi there,
>
> So I got past this issue by running "mvn test" from the root of the nifi
> tree.  After much compiling and testing, I am now getting the attached
> error.  I think I am following the contributor guide in having my local
> branch track origin/main (currently "4a90334728: NIFI-9846 Implement
> pagination listing for Azure List processors").
>
> Is this some environmental issue on my machine somehow?  Or does the main
> branch contain compilation errors, and I should be tracking something else
> for my own branch?  As I said in my previous email, I am new to github, so
> I'm not sure of the conventions there.
>
> Thanks in advance
> Phil
>
> (PS: this occur whethers I use JDK1.8 or JDK11, the two options I have
> installed)
>
>
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.9.0:compile
> (default-compile) on project nifi-h2-database-migrator: Compilation
> failure: Compilation failure:
> [ERROR]
> D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[19,35]
> error: package org.apache.nifi.org.h2.jdbcx does not exist
> [ERROR]
> D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[37,8]
> error: cannot find symbol
> [ERROR]   symbol:   class JdbcDataSource
> [ERROR]   location: class H2DatabaseMigrator
> [ERROR]
> D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[37,49]
> error: cannot find symbol
> [ERROR]   symbol:   class JdbcDataSource
> [ERROR]   location: class H2DatabaseMigrator
>
> On Sat, Apr 2, 2022 at 10:20 PM Phil H  wrote:
>
> > Hi there,
> >
> > Trying to build the nifi git repo for the first time and I'm getting the
> > message below
> >
> > I'm used to working in offline development environments, so I'm a bit lost
> > with this online dependency stuff. I note that
> > https://repository.apache.org/snapshots/org/apache/nifi/ doesn't contain
> > any similar subdirectories to that in the URL.
> >
> > Help!
> >
> > Regards,
> > Phil
> >
> > [WARNING] The POM for org.apache.nifi:nifi-utils:jar:1.16.0-SNAPSHOT is
> > missing, no dependency information available
> > Downloading from apache.snapshots:
> > https://repository.apache.org/snapshots/org/apache/nifi/nifi-utils/1.16.0-SNAPSHOT/nifi-utils-1.16.0-SNAPSHOT.jar
> > [INFO]
> > 
> > [INFO] Reactor Summary for nifi-standard-bundle 1.16.0-SNAPSHOT:
> > [INFO]
> > [INFO] nifi-standard-bundle ... SUCCESS [01:07
> > min]
> > [INFO] nifi-standard-utils  FAILURE [
> >  3.218 s]
> > [INFO] nifi-standard-processors ... SKIPPED
> > [INFO] nifi-standard-reporting-tasks .. SKIPPED
> > [INFO] nifi-standard-content-viewer ... SKIPPED
> > [INFO] nifi-jolt-transform-json-ui  SKIPPED
> > [INFO] nifi-standard-nar .. SKIPPED
> > [INFO]
> > 
> > [INFO] BUILD FAILURE
> > [INFO]
> > 
> > [INFO] Total time:  01:43 min
> > [INFO] Finished at: 2022-04-02T21:58:47+10:00
> > [INFO]
> > 
> > [ERROR] Failed to execute goal on project nifi-standard-utils: Could not
> > resolve dependencies for project
> > org.apache.nifi:nifi-standard-utils:jar:1.16.0-SNAPSHOT: Could not find
> > artifact org.apache.nifi:nifi-utils:jar:1.16.0-SNAPSHOT in apache.snapshots
> > (https://repository.apache.org/snapshots) -> [Help 1]
> >

Re: [ANNOUNCE] New Apache NiFi Committer Adam Markovics

[Mike Thomsen]

Congrats, Adam!

On Mon, Mar 28, 2022 at 10:15 AM Pierre Villard
 wrote:
>
> Congrats Adam!
>
> Le lun. 28 mars 2022 à 16:11, Kevin Doran  a écrit :
>
> >  Congrats, Adam!
> >
> > On Mar 28, 2022 at 10:08:24, Otto Fowler  wrote:
> >
> > > Congratulations!
> > >
> > > From: Marton Szasz  
> > > Reply: dev@nifi.apache.org  
> > > Date: March 25, 2022 at 08:50:07
> > > To: dev@nifi.apache.org  
> > > Subject:  [ANNOUNCE] New Apache NiFi Committer Adam Markovics
> > >
> > > Apache NiFi community,
> > >
> > > On behalf of the Apache NiFi PMC, I am very pleased to announce that
> > > Adam Markovics has accepted the PMC's invitation to become a committer
> > > on the Apache NiFi project. We greatly appreciate all of Adam's hard
> > > work and generous contributions to the project. We look forward to
> > > continued involvement in the project.
> > >
> > > Adam has been a consistent contributor to MiNiFi C++ for almost 2
> > > years. His contributions vary from user experience improvements, for
> > > example better diagnostics for incorrect flow designs, and developer
> > > quality of life improvements like improved static analysis and better
> > > test coverage.
> > >
> > > Welcome Adam and congratulations!
> > >
> >

Stacktrace from upgrading to Jackson 2.13.X

[Mike Thomsen]

I hit a snag updating the codebase to Jackson 2.13.X. Stacktrace looks
familiar, like one of the Java 9/11 refactorings that moved code out
of the RT into third party modules. Any ideas?

[ERROR] 
org.apache.nifi.registry.security.authorization.database.TestDatabaseUserGroupProvider.testOnConfiguredAppliesIdentityMappingsToInitialUsers
 Time elapsed: 0 s  <<< ERROR!
java.lang.IllegalStateException: Failed to load ApplicationContext
at 
org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:132)
at 
org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:124)
at 
org.springframework.test.context.support.DependencyInjectionTestExecutionListener.injectDependencies(DependencyInjectionTestExecutionListener.java:118)
at 
org.springframework.test.context.support.DependencyInjectionTestExecutionListener.prepareTestInstance(DependencyInjectionTestExecutionListener.java:83)
at 
org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:248)
at 
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:227)
at 
org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:289)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:291)
at 
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:246)
at 
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at 
org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at 
org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at 
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:364)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:237)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:158)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Caused by: org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'mappingJackson2XmlHttpMessageConverter'
defined in class path resource
[org/springframework/boot/autoconfigure/http/JacksonHttpMessageConvertersConfiguration$MappingJackson2XmlHttpMessageConverterConfiguration.class]:
Bean instantiation via factory method failed; nested exception is
org.springframework.beans.BeanInstantiationException: Failed to
instantiate 
[org.springframework.http.converter.xml.MappingJackson2XmlHttpMessageConverter]:
Factory method 'mappingJackson2XmlHttpMessageConverter' threw
exception; nested exception is
javax.xml.stream.FactoryConfigurationError: Provider for
javax.xml.stream.XMLOutputFactory cannot be found
at 
org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658)
at 
org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at 
org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at 

Re: [VOTE] Release Apache NiFi 1.16.0 (rc3)

[Mike Thomsen]

+1 binding

Ran the full build on Java 17 and tested DeduplicateRecord with a Cassandra DMC.

On Wed, Mar 23, 2022 at 7:45 PM Joey Frazee
 wrote:
>
> +1 (binding)
>
> - Verified checksums and signatures
> - Did full build in Java 11 on Fedora
> - Ran build with rpm profile and tested rpm install
> - Ran some basic flows
>
> > On Mar 23, 2022, at 2:40 PM, Andrew Lim  wrote:
> >
> > +1 (binding)
> >
> > -Ran full clean install on OS X (Catalina 10.15.7, OpenJDK version 
> > 1.8.0_252)
> > -Tested secure NiFi with secure NiFi Registry
> > -Ran basic flows successfully; tested basic versioned flow functionality
> > -Reviewed/tested Core UI fixes and documentation/website updates
> >
> > Drew
> >
> >> On Mar 21, 2022, at 5:41 PM, Joe Witt  wrote:
> >>
> >> Hello,
> >>
> >> I am pleased to be calling this vote for the source release of Apache NiFi
> >> 1.16.0.
> >>
> >> The source zip, including signatures, digests, etc. can be found at:
> >> https://repository.apache.org/content/repositories/orgapachenifi-1198
> >>
> >> The source being voted upon and the convenience binaries can be found at:
> >> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.0/
> >>
> >> A helpful reminder on how the release candidate verification process works:
> >> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >>
> >> The Git tag is nifi-1.16.0-RC3
> >> The Git commit ID is b019a9191f1c83bc7f547dc02c1b679b8936acee
> >> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=b019a9191f1c83bc7f547dc02c1b679b8936acee
> >>
> >> Checksums of nifi-1.16.0-source-release.zip:
> >> SHA256: 2f16cb94df404d1bcc9c32835ba98da8940005a5d7ea5504c155ee42021a221e
> >> SHA512:
> >> cbd95f15cec5ffe506fef204526267b18b77d7266f6dc3e1bbc3c7600aac12e119977f7d8cf93dbbbc86fbb0739ba88aaa11a5381d29a463ec9a0c9a18f4e9e6
> >>
> >> Release artifacts are signed with the following key:
> >> https://people.apache.org/keys/committer/joewitt.asc
> >>
> >> KEYS file available here:
> >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> >>
> >> 401 issues were closed/resolved for this release:
> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350741
> >>
> >> Release note highlights can be found here:
> >> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.0
> >>
> >> The vote will be open for 72 hours.
> >> Please download the release candidate and evaluate the necessary items
> >> including checking hashes, signatures, build from source, and test. Then
> >> please vote:
> >>
> >> [ ] +1 Release this package as nifi-1.16.0
> >> [ ] +0 no opinion
> >> [ ] -1 Do not release this package because...
> >