Re: Long Polling Client

[Clay Teahouse]

Thanks Joe for the pointers.

On Wed, Oct 14, 2020 at 3:31 PM Joe Witt  wrote:

> Right makes sense.  They're waiting for the completion of the response as
> the payload to pass on.  If we need incremental handling of the response
> body then we need to factor that into a given processor.  InvokeHttp or
> something like it is a good candidate.  Almost a sort of 'InvokeHttpRecord'
> kind of processor which uses the format/schema awareness of records to
> frame the response objects and then based on some configurable value passes
> those records on as a flow file.
>
> Thanks
>
> On Wed, Oct 14, 2020 at 1:25 PM Clay Teahouse 
> wrote:
>
>> Hello,
>> I tried both getHTTP and invokeHTTP (but didn't try all options). What I
>> need is to deal with the cases, such as SSE (Server Sent Events) which
>> works with long polling. Meaning the client initiates a connection (via a
>> HTTP request) to the SSE server, the server keeps the connection open (for
>> as long as possible) and streams data back to the client on the same
>> connection. I am able to have the data streamed to the client if I set up a
>> stand alone HTTP client and issue HTTP requests but I am not able to get
>> getHTTP or invokeHTTP work (I don't get any messages back).
>>
>> thanks.
>>
>> On Wed, Oct 14, 2020 at 2:50 PM Joe Witt  wrote:
>>
>>> Clay
>>>
>>> Have you evaluated whether InvokeHTTP will give you the desired behavior
>>> for your case - in particular with a long timeout perhaps?  If you have and
>>> it doesn't do the job do you mean something which initiates a request to an
>>> HTTP server then assumes the response will remain open and it should take
>>> portions of the response and treat each as its own flowfile/object to pass
>>> along?
>>>
>>> Thanks
>>>
>>> On Tue, Oct 13, 2020 at 6:31 PM Clay Teahouse 
>>> wrote:
>>>
 Does NiFi have a processor that can act as a client for a long polling
 server, for example an SSE server?
 More specifically, I want a client that can issue a HTTP GET request to
 a long polling server and accept stream of messages from the server (on the
 same connection).
 If there isn't one, which processor is best to extend to achieve this
 goal?

 thanks.

>>>

Re: Long Polling Client

[Joe Witt]

Right makes sense.  They're waiting for the completion of the response as
the payload to pass on.  If we need incremental handling of the response
body then we need to factor that into a given processor.  InvokeHttp or
something like it is a good candidate.  Almost a sort of 'InvokeHttpRecord'
kind of processor which uses the format/schema awareness of records to
frame the response objects and then based on some configurable value passes
those records on as a flow file.

Thanks

On Wed, Oct 14, 2020 at 1:25 PM Clay Teahouse 
wrote:

> Hello,
> I tried both getHTTP and invokeHTTP (but didn't try all options). What I
> need is to deal with the cases, such as SSE (Server Sent Events) which
> works with long polling. Meaning the client initiates a connection (via a
> HTTP request) to the SSE server, the server keeps the connection open (for
> as long as possible) and streams data back to the client on the same
> connection. I am able to have the data streamed to the client if I set up a
> stand alone HTTP client and issue HTTP requests but I am not able to get
> getHTTP or invokeHTTP work (I don't get any messages back).
>
> thanks.
>
> On Wed, Oct 14, 2020 at 2:50 PM Joe Witt  wrote:
>
>> Clay
>>
>> Have you evaluated whether InvokeHTTP will give you the desired behavior
>> for your case - in particular with a long timeout perhaps?  If you have and
>> it doesn't do the job do you mean something which initiates a request to an
>> HTTP server then assumes the response will remain open and it should take
>> portions of the response and treat each as its own flowfile/object to pass
>> along?
>>
>> Thanks
>>
>> On Tue, Oct 13, 2020 at 6:31 PM Clay Teahouse 
>> wrote:
>>
>>> Does NiFi have a processor that can act as a client for a long polling
>>> server, for example an SSE server?
>>> More specifically, I want a client that can issue a HTTP GET request to
>>> a long polling server and accept stream of messages from the server (on the
>>> same connection).
>>> If there isn't one, which processor is best to extend to achieve this
>>> goal?
>>>
>>> thanks.
>>>
>>

Re: Long Polling Client

[Clay Teahouse]

Hello,
I tried both getHTTP and invokeHTTP (but didn't try all options). What I
need is to deal with the cases, such as SSE (Server Sent Events) which
works with long polling. Meaning the client initiates a connection (via a
HTTP request) to the SSE server, the server keeps the connection open (for
as long as possible) and streams data back to the client on the same
connection. I am able to have the data streamed to the client if I set up a
stand alone HTTP client and issue HTTP requests but I am not able to get
getHTTP or invokeHTTP work (I don't get any messages back).

thanks.

On Wed, Oct 14, 2020 at 2:50 PM Joe Witt  wrote:

> Clay
>
> Have you evaluated whether InvokeHTTP will give you the desired behavior
> for your case - in particular with a long timeout perhaps?  If you have and
> it doesn't do the job do you mean something which initiates a request to an
> HTTP server then assumes the response will remain open and it should take
> portions of the response and treat each as its own flowfile/object to pass
> along?
>
> Thanks
>
> On Tue, Oct 13, 2020 at 6:31 PM Clay Teahouse 
> wrote:
>
>> Does NiFi have a processor that can act as a client for a long polling
>> server, for example an SSE server?
>> More specifically, I want a client that can issue a HTTP GET request to a
>> long polling server and accept stream of messages from the server (on the
>> same connection).
>> If there isn't one, which processor is best to extend to achieve this
>> goal?
>>
>> thanks.
>>
>

Re: [EXT] Re: sslcontext certs

[Peter Wicks (pwicks)]

Micron Confidential

I agree Nathan.  I believe the situation I ran into came about due to bad 
planning.  Users started independently hosting services, and it was only later 
that we realized that a centralized service or variables would be a better 
solution.

It would probably be easier to just go the direction you suggested 

From: Nathan Gough 
Reply-To: "users@nifi.apache.org" 
Date: Wednesday, October 14, 2020 at 1:59 PM
To: "users@nifi.apache.org" 
Subject: Re: [EXT] Re: sslcontext certs

Is there a reason each ListenHTTP has a unique SSLContextService if they're all 
using the same certificates?

If it were me, I'd use a single shared SSLContextService, and when I needed to 
update the certificate in the keystore/truststore, I would change it on disk by 
renaming the old file and putting the new file in place with the original name. 
Now NiFi and the context service refers to the updated certificates and no NiFi 
configuration changed. Does this work for you?

Nathan

On Wed, Oct 14, 2020 at 3:29 PM Peter Wicks (pwicks) 
mailto:pwi...@micron.com>> wrote:
Micron Confidential

I've found this annoying in the past as well. I would not be opposed to an 
additional implementation of the SSLContext that uses the NiFi certs by 
default, though... if it uses the client certificate as well you'd have to make 
it restricted, so as to prevent users from impersonating the servers identity 
when communicating with external services. (A restricted Controller Service?)

--Peter

On 10/14/20, 12:44 PM, "Michael Di Domenico" 
mailto:mdidomeni...@gmail.com>> wrote:

ah, okay that sounds like maybe a step in a good direction, but
doesn't necessarily solve my problem.  What I'm trying to alleviate is
the need to go into nifi to change the certs when they expire.

i'll have to look up parameter contexts, that should at least make it
so there's only one place to make the change.

thanks

On Wed, Oct 14, 2020 at 2:40 PM Joe Witt 
mailto:joe.w...@gmail.com>> wrote:
>
> Michael,
>
> There is not any specific way supported or intended to combine the 
context used by NiFi's own HTTP server with those that would be used by 
processors within the flow.
>
> However, using parameter contexts here is a great way to ensure you have 
only a single place to update for flow internals.  If those values are 
parameterized it should work out nicely.
>
> Thanks
>
> On Wed, Oct 14, 2020 at 11:34 AM Michael Di Domenico 
mailto:mdidomeni...@gmail.com>> wrote:
>>
>> i have a nifi server with several listenhttp modules on different
>> ports.  each one has an sslcontext within it that uses the same certs
>> as the main 443 instance.
>>
>> sadly i changed the cert when expired on the 443 port, but failed to
>> change the sslcontext on the ports.  is there a way to tell the
>> sslcontext on the other ports to just use the same cert that's on the
>> 443 port?
>>
>> what i'm trying to avoid having to do is change the filename in all
>> the contexts to point to the new cert, i'd rather change it in one
>> place and have everything else pick it up
>>
>> using a symlink on the filesystem seemed like one way, but i thought
>> there might be a way to do it in nifi


Micron Confidential


Micron Confidential

Re: [EXT] Re: sslcontext certs

[Nathan Gough]

Is there a reason each ListenHTTP has a unique SSLContextService if they're
all using the same certificates?

If it were me, I'd use a single shared SSLContextService, and when I needed
to update the certificate in the keystore/truststore, I would change it on
disk by renaming the old file and putting the new file in place with the
original name. Now NiFi and the context service refers to the updated
certificates and no NiFi configuration changed. Does this work for you?

Nathan

On Wed, Oct 14, 2020 at 3:29 PM Peter Wicks (pwicks) 
wrote:

> Micron Confidential
>
> I've found this annoying in the past as well. I would not be opposed to an
> additional implementation of the SSLContext that uses the NiFi certs by
> default, though... if it uses the client certificate as well you'd have to
> make it restricted, so as to prevent users from impersonating the servers
> identity when communicating with external services. (A restricted
> Controller Service?)
>
> --Peter
>
> On 10/14/20, 12:44 PM, "Michael Di Domenico" 
> wrote:
>
> ah, okay that sounds like maybe a step in a good direction, but
> doesn't necessarily solve my problem.  What I'm trying to alleviate is
> the need to go into nifi to change the certs when they expire.
>
> i'll have to look up parameter contexts, that should at least make it
> so there's only one place to make the change.
>
> thanks
>
> On Wed, Oct 14, 2020 at 2:40 PM Joe Witt  wrote:
> >
> > Michael,
> >
> > There is not any specific way supported or intended to combine the
> context used by NiFi's own HTTP server with those that would be used by
> processors within the flow.
> >
> > However, using parameter contexts here is a great way to ensure you
> have only a single place to update for flow internals.  If those values are
> parameterized it should work out nicely.
> >
> > Thanks
> >
> > On Wed, Oct 14, 2020 at 11:34 AM Michael Di Domenico <
> mdidomeni...@gmail.com> wrote:
> >>
> >> i have a nifi server with several listenhttp modules on different
> >> ports.  each one has an sslcontext within it that uses the same
> certs
> >> as the main 443 instance.
> >>
> >> sadly i changed the cert when expired on the 443 port, but failed to
> >> change the sslcontext on the ports.  is there a way to tell the
> >> sslcontext on the other ports to just use the same cert that's on
> the
> >> 443 port?
> >>
> >> what i'm trying to avoid having to do is change the filename in all
> >> the contexts to point to the new cert, i'd rather change it in one
> >> place and have everything else pick it up
> >>
> >> using a symlink on the filesystem seemed like one way, but i thought
> >> there might be a way to do it in nifi
>
>
> Micron Confidential
>

Re: Long Polling Client

[Joe Witt]

Clay

Have you evaluated whether InvokeHTTP will give you the desired behavior
for your case - in particular with a long timeout perhaps?  If you have and
it doesn't do the job do you mean something which initiates a request to an
HTTP server then assumes the response will remain open and it should take
portions of the response and treat each as its own flowfile/object to pass
along?

Thanks

On Tue, Oct 13, 2020 at 6:31 PM Clay Teahouse 
wrote:

> Does NiFi have a processor that can act as a client for a long polling
> server, for example an SSE server?
> More specifically, I want a client that can issue a HTTP GET request to a
> long polling server and accept stream of messages from the server (on the
> same connection).
> If there isn't one, which processor is best to extend to achieve this goal?
>
> thanks.
>

Re: [EXT] Re: sslcontext certs

[Peter Wicks (pwicks)]

Micron Confidential

I've found this annoying in the past as well. I would not be opposed to an 
additional implementation of the SSLContext that uses the NiFi certs by 
default, though... if it uses the client certificate as well you'd have to make 
it restricted, so as to prevent users from impersonating the servers identity 
when communicating with external services. (A restricted Controller Service?)

--Peter

On 10/14/20, 12:44 PM, "Michael Di Domenico"  wrote:

ah, okay that sounds like maybe a step in a good direction, but
doesn't necessarily solve my problem.  What I'm trying to alleviate is
the need to go into nifi to change the certs when they expire.

i'll have to look up parameter contexts, that should at least make it
so there's only one place to make the change.

thanks

On Wed, Oct 14, 2020 at 2:40 PM Joe Witt  wrote:
>
> Michael,
>
> There is not any specific way supported or intended to combine the 
context used by NiFi's own HTTP server with those that would be used by 
processors within the flow.
>
> However, using parameter contexts here is a great way to ensure you have 
only a single place to update for flow internals.  If those values are 
parameterized it should work out nicely.
>
> Thanks
>
> On Wed, Oct 14, 2020 at 11:34 AM Michael Di Domenico 
 wrote:
>>
>> i have a nifi server with several listenhttp modules on different
>> ports.  each one has an sslcontext within it that uses the same certs
>> as the main 443 instance.
>>
>> sadly i changed the cert when expired on the 443 port, but failed to
>> change the sslcontext on the ports.  is there a way to tell the
>> sslcontext on the other ports to just use the same cert that's on the
>> 443 port?
>>
>> what i'm trying to avoid having to do is change the filename in all
>> the contexts to point to the new cert, i'd rather change it in one
>> place and have everything else pick it up
>>
>> using a symlink on the filesystem seemed like one way, but i thought
>> there might be a way to do it in nifi


Micron Confidential

Re: sslcontext certs

[Michael Di Domenico]

ah, okay that sounds like maybe a step in a good direction, but
doesn't necessarily solve my problem.  What I'm trying to alleviate is
the need to go into nifi to change the certs when they expire.

i'll have to look up parameter contexts, that should at least make it
so there's only one place to make the change.

thanks

On Wed, Oct 14, 2020 at 2:40 PM Joe Witt  wrote:
>
> Michael,
>
> There is not any specific way supported or intended to combine the context 
> used by NiFi's own HTTP server with those that would be used by processors 
> within the flow.
>
> However, using parameter contexts here is a great way to ensure you have only 
> a single place to update for flow internals.  If those values are 
> parameterized it should work out nicely.
>
> Thanks
>
> On Wed, Oct 14, 2020 at 11:34 AM Michael Di Domenico  
> wrote:
>>
>> i have a nifi server with several listenhttp modules on different
>> ports.  each one has an sslcontext within it that uses the same certs
>> as the main 443 instance.
>>
>> sadly i changed the cert when expired on the 443 port, but failed to
>> change the sslcontext on the ports.  is there a way to tell the
>> sslcontext on the other ports to just use the same cert that's on the
>> 443 port?
>>
>> what i'm trying to avoid having to do is change the filename in all
>> the contexts to point to the new cert, i'd rather change it in one
>> place and have everything else pick it up
>>
>> using a symlink on the filesystem seemed like one way, but i thought
>> there might be a way to do it in nifi

Re: sslcontext certs

[Joe Witt]

Michael,

There is not any specific way supported or intended to combine the context
used by NiFi's own HTTP server with those that would be used by processors
within the flow.

However, using parameter contexts here is a great way to ensure you have
only a single place to update for flow internals.  If those values are
parameterized it should work out nicely.

Thanks

On Wed, Oct 14, 2020 at 11:34 AM Michael Di Domenico 
wrote:

> i have a nifi server with several listenhttp modules on different
> ports.  each one has an sslcontext within it that uses the same certs
> as the main 443 instance.
>
> sadly i changed the cert when expired on the 443 port, but failed to
> change the sslcontext on the ports.  is there a way to tell the
> sslcontext on the other ports to just use the same cert that's on the
> 443 port?
>
> what i'm trying to avoid having to do is change the filename in all
> the contexts to point to the new cert, i'd rather change it in one
> place and have everything else pick it up
>
> using a symlink on the filesystem seemed like one way, but i thought
> there might be a way to do it in nifi
>

sslcontext certs

[Michael Di Domenico]

i have a nifi server with several listenhttp modules on different
ports.  each one has an sslcontext within it that uses the same certs
as the main 443 instance.

sadly i changed the cert when expired on the 443 port, but failed to
change the sslcontext on the ports.  is there a way to tell the
sslcontext on the other ports to just use the same cert that's on the
443 port?

what i'm trying to avoid having to do is change the filename in all
the contexts to point to the new cert, i'd rather change it in one
place and have everything else pick it up

using a symlink on the filesystem seemed like one way, but i thought
there might be a way to do it in nifi

Re: Nifi 1.12 processor is not auto upgrading default processor if it has a different version of custom processor

[Bryan Bende]

I think the issue is that the way the logic works, it takes the class
name from flow.xml and then looks to see which bundles contain that
class name.

In your case, there are two bundles containing
org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService
- nifi-aws-nar 1.12.0
- your custom nar 1.0.0

So when it comes to the component that was from nifi-aws-nar 1.8.0, it
sees there are two bundles to choose from and neither are the exact
bundle from the flow.xml(1.8.0), so it doesn't know which is correct,
and therefore it creates a ghosted component.

I think you should be able to use the "Change Version" feature on the
service to change it from ghosted to 1.12.0, but I'm not 100% sure on
that.

Ideally we should improve the bundle selection logic so that instead
of just looking for bundles with that class name, it also looks for
the referenced group and artifact name and prefers those first.

On Wed, Oct 14, 2020 at 11:56 AM sanjeet rath  wrote:
>
> Hi All,
>
> I am facing one issue during the nifi cluster upgrade to 1.12 version from 
> 1.8 version.
>
> I have a custom processor for AWSCredentialsProviderControllerService 
> controller service. this has been build on top of 1.8 version . the structure 
> for custom processor in flow.xml.gz file is:
>
> AWSCredentialsProviderControllerService100  
> org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService
>
>
> com.xxx.xx1234
> nifi-custom-ping-credentials-controller-service
> 1.0.0
>   
>
>
> There are also default AWSCredentialsProviderControllerService controller 
> service of 1.8version is present which is having below configuration in 
> flow.xml
>
> AWSCredentialsProviderControllerService180
> org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService
>   
> org.apache.nifi
> nifi-aws-nar
> 1.8.0
>
>
> So I am upgrading the nifi cluster , which means putting this flow.xml.gz 
> file from 1.8 cluster to 1.12 cluster .
>
> After the cluster up I am seeing the default 
> AWSCredentialsProviderControllerService(1.8 version) controller is not auto 
> upgraded to 1.12 bundle and getting invalid with error:
>
> Error:
>
> missing controller service validated against "any property" is invalid 
> because controler service of this type 
> org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService,
>  but this is not a valid reporting task type.
>
> log I am seeing :
>
> 2020-10-14 17:14:56,042 ERROR [main] o.a.nifi.controller.ExtensionBuilder 
> Could not create Controller Service of type 
> org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService
>  for ID 25defb18-0175-1000-5bb4-febb1b1a21db due to: Unable to find bundle 
> for coordinate org.apache.nifi:nifi-aws-nar:1.8.0; creating "Ghost" 
> implementation 2020-10-14 17:14:56,042 INFO [main] 
> o.a.nifi.groups.StandardProcessGroup 
> StandardControllerServiceNode[service=GhostControllerService[id=25defb18-0175-1000-5bb4-febb1b1a21db,
>  
> type=org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService],
>  versionedComponentId=null, 
> processGroup=StandardProcessGroup[identifier=8cb90667-0174-1000-8741-3bfe7f19db7f],
>  active=false] added to 
> StandardProcessGroup[identifier=8cb90667-0174-1000-8741-3bfe7f19db7f]
>
> There is no issue in the custom processor 
> (nifi-custom-ping-credentials-controller-service 1.0.0) as 1.0.0 version nar 
> file is present in the 1.12 cluster Also no issue with other 1.8 version 
> processor & controller service , all are auto upgraded to 1.12 version.
>
> Could you please let me know what should be done to avoid this type of issue 
> in upgrade?
>
>
> --
> Sanjeet Kumar Rath,
> mob- +91 8777577470
>

Nifi 1.12 processor is not auto upgrading default processor if it has a different version of custom processor

[sanjeet rath]

Hi All,

I am facing one issue during the nifi cluster upgrade to 1.12 version from
1.8 version.

I have a *custom processor for AWSCredentialsProviderControllerService
controller service. this has been build on top of 1.8* version . the
structure for custom processor in flow.xml.gz file is:

AWSCredentialsProviderControllerService100
org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService

   
com.xxx.xx1234
nifi-custom-ping-credentials-controller-service
*1.0.0*
  


There are also *default AWSCredentialsProviderControllerService *controller
service of 1.8version is present which is having below configuration in
flow.xml

AWSCredentialsProviderControllerService180
org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService
  
org.apache.nifi
nifi-aws-nar
*1.8.0*


So I am upgrading the nifi cluster , which means putting this flow.xml.gz
file from 1.8 cluster to 1.12 cluster .

After the cluster up I am seeing the *default
AWSCredentialsProviderControllerService(1.8 version) controller is not auto
upgraded to 1.12* bundle and getting invalid with error:

Error:

missing controller service validated against "any property" is invalid
because controler service of this type
org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService,
but this is not a valid reporting task type.

log I am seeing :

2020-10-14 17:14:56,042 ERROR [main] o.a.nifi.controller.ExtensionBuilder
Could not create Controller Service of type
org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService
for ID 25defb18-0175-1000-5bb4-febb1b1a21db due to: Unable to find bundle
for coordinate org.apache.nifi:nifi-aws-nar:1.8.0; creating "Ghost"
implementation 2020-10-14 17:14:56,042 INFO [main]
o.a.nifi.groups.StandardProcessGroup
StandardControllerServiceNode[service=GhostControllerService[id=25defb18-0175-1000-5bb4-febb1b1a21db,
type=org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService],
versionedComponentId=null,
processGroup=StandardProcessGroup[identifier=8cb90667-0174-1000-8741-3bfe7f19db7f],
active=false] added to
StandardProcessGroup[identifier=8cb90667-0174-1000-8741-3bfe7f19db7f]

There is *no issue in the custom processor
*(nifi-custom-ping-credentials-controller-service
1.0.0) as 1.0.0 version nar file is present in the 1.12 cluster Also *no
issue with other 1.8 version processor *& controller service , all are auto
upgraded to 1.12 version.

Could you please let me know what should be done to avoid this type of
issue in upgrade?

-- 
Sanjeet Kumar Rath,
mob- +91 8777577470

Re: Clustered nifi issues

[Wyll Ingersoll]

I see what's happening.  The container sets up a /root/.nifi-cli.config file 
that has the required security parameters so that the user doesn't have to 
supply them on the command line.

From: Bryan Bende 
Sent: Wednesday, October 14, 2020 10:45 AM
To: users@nifi.apache.org 
Subject: Re: Clustered nifi issues

The CLI does not use nifi.properties, there are several ways of passing in 
config...

https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#property-argument-handling

On Wed, Oct 14, 2020 at 10:01 AM Wyll Ingersoll 
mailto:wyllys.ingers...@keepertech.com>> wrote:
That makes sense.  It must be reading the keystore/truststore specified in the 
nifi.properties file then?

From: Bryan Bende mailto:bbe...@gmail.com>>
Sent: Wednesday, October 14, 2020 9:59 AM
To: users@nifi.apache.org 
mailto:users@nifi.apache.org>>
Subject: Re: Clustered nifi issues

The get-nodes command calls the REST resource /controller/cluster which 
authorizes against READ on /controller [1], so there is no way you can call 
this in a secure environment without authenticating somehow, which from the CLI 
means specifying a keystore/truststore.

[1] 
https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ControllerResource.java#L857

On Wed, Oct 14, 2020 at 9:26 AM Wyll Ingersoll 
mailto:wyllys.ingers...@keepertech.com>> wrote:
Yes, this is for a secured cluster deployed as a Kubernetes stateful set.  The 
certificate parameters are apparently not needed to just get the status of the 
nodes using the command below.




From: Sushil Kumar mailto:skm@gmail.com>>
Sent: Tuesday, October 13, 2020 4:01 PM
To: users@nifi.apache.org 
mailto:users@nifi.apache.org>>
Subject: Re: Clustered nifi issues

Did you say that the same line of code works fine for secured clusters too.
I asked because nifi-toolkit has a separate set of parameters asking for 
certificates and everything else related to secure clusters.


On Tue, Oct 13, 2020 at 12:14 PM Wyll Ingersoll 
mailto:wyllys.ingers...@keepertech.com>> wrote:

I found that instead of dealing with nifi client certificate hell, the 
nifi-toolkit cli.sh will work just fine for testing the readiness of the 
cluster.  Here is my readiness script which seems to work just fine with in 
kubernetes with the apache/nifi docker container version 1.12.1



#!/bin/bash


$NIFI_TOOLKIT_HOME/bin/cli.sh nifi get-nodes -ot json > /tmp/cluster.state

if [ $? -ne 0 ]; then

cat /tmp/cluster.state

exit 1

fi

STATUS=$(jq -r ".cluster.nodes[] | select((.address==\"$(hostname -f)\") or 
.address==\"localhost\") | .status" /tmp/cluster.state)

if [[ ! $STATUS = "CONNECTED" ]]; then

echo "Node not found with CONNECTED state. Full cluster state:"

jq . /tmp/cluster.state

exit 1

fi



From: Chris Sampson 
mailto:chris.samp...@naimuri.com>>
Sent: Thursday, October 1, 2020 9:03 AM
To: users@nifi.apache.org 
mailto:users@nifi.apache.org>>
Subject: Re: Clustered nifi issues

For info, the probes we currently use for our StatefulSet Pods are:

  *   livenessProbe - tcpSocket to ping the NiFi instance port (e.g. 8080)
  *   readinessProbe - exec command to curl the nifi-api/controller/cluster 
endpoint to check the node's cluster connection status, e.g.:

readinessProbe:
exec:
command:
- bash
- -c
- |
if [ "${SECURE}" = "true" ]; then
INITIAL_ADMIN_SLUG=$(echo "${INITIAL_ADMIN}" | tr '[:upper:]' '[:lower:]' | tr 
' ' '-')

curl -v \
--cacert ${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/nifi-cert.pem \
--cert 
${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/${INITIAL_ADMIN_SLUG}-cert.pem
 \
--key 
${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/${INITIAL_ADMIN_SLUG}-key.pem
 \
https://$(hostname -f):8080/nifi-api/controller/cluster > /tmp/cluster.state
else
curl -kv http://$(hostname -f):8080/nifi-api/controller/cluster > 
/tmp/cluster.state
fi

STATUS=$(jq -r ".cluster.nodes[] | select((.address==\"$(hostname -f)\") or 
.address==\"localhost\") | .status" /tmp/cluster.state)

if [[ ! $STATUS = "CONNECTED" ]]; then
echo "Node not found with CONNECTED state. Full cluster state:"
jq . /tmp/cluster.state
exit 1
fi

Note that INITIAL_ADMIN is the CN of a user with appropriate permissions to 
call the endpoint and for whom our pod contains a set of certificate files in 
the indicated locations (generated from NiFi Toolkit in an init-container 
before the Pod starts); jq utility was added into our customised version of the 
apache/nifi Docker Image.


---
Chris Sampson
IT Consultant
chris.samp...@naimuri.com
[X]

Re: Clustered nifi issues

[Bryan Bende]

The CLI does not use nifi.properties, there are several ways of passing in
config...

https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#property-argument-handling

On Wed, Oct 14, 2020 at 10:01 AM Wyll Ingersoll <
wyllys.ingers...@keepertech.com> wrote:

> That makes sense.  It must be reading the keystore/truststore specified in
> the nifi.properties file then?
> --
> *From:* Bryan Bende 
> *Sent:* Wednesday, October 14, 2020 9:59 AM
> *To:* users@nifi.apache.org 
> *Subject:* Re: Clustered nifi issues
>
> The get-nodes command calls the REST resource /controller/cluster which
> authorizes against READ on /controller [1], so there is no way you can call
> this in a secure environment without authenticating somehow, which from the
> CLI means specifying a keystore/truststore.
>
> [1]
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ControllerResource.java#L857
>
> On Wed, Oct 14, 2020 at 9:26 AM Wyll Ingersoll <
> wyllys.ingers...@keepertech.com> wrote:
>
> Yes, this is for a secured cluster deployed as a Kubernetes stateful set.
> The certificate parameters are apparently not needed to just get the status
> of the nodes using the command below.
>
>
>
> --
> *From:* Sushil Kumar 
> *Sent:* Tuesday, October 13, 2020 4:01 PM
> *To:* users@nifi.apache.org 
> *Subject:* Re: Clustered nifi issues
>
> Did you say that the same line of code works fine for secured clusters
> too.
> I asked because nifi-toolkit has a separate set of parameters asking for
> certificates and everything else related to secure clusters.
>
>
> On Tue, Oct 13, 2020 at 12:14 PM Wyll Ingersoll <
> wyllys.ingers...@keepertech.com> wrote:
>
>
> I found that instead of dealing with nifi client certificate hell, the
> nifi-toolkit cli.sh will work just fine for testing the readiness of the
> cluster.  Here is my readiness script which seems to work just fine with in
> kubernetes with the apache/nifi docker container version 1.12.1
>
>
> #!/bin/bash
>
>
> $NIFI_TOOLKIT_HOME/bin/cli.sh nifi get-nodes -ot json > /tmp/cluster.state
>
> if [ $? -ne 0 ]; then
>
> cat /tmp/cluster.state
>
> exit 1
>
> fi
>
> STATUS=$(jq -r ".cluster.nodes[] | select((.address==\"$(hostname -f)\")
> or .address==\"localhost\") | .status" /tmp/cluster.state)
>
> if [[ ! $STATUS = "CONNECTED" ]]; then
>
> echo "Node not found with CONNECTED state. Full cluster state:"
>
> jq . /tmp/cluster.state
>
> exit 1
>
> fi
>
>
> --
> *From:* Chris Sampson 
> *Sent:* Thursday, October 1, 2020 9:03 AM
> *To:* users@nifi.apache.org 
> *Subject:* Re: Clustered nifi issues
>
> For info, the probes we currently use for our StatefulSet Pods are:
>
>- livenessProbe - tcpSocket to ping the NiFi instance port (e.g. 8080)
>- readinessProbe - exec command to curl the
>nifi-api/controller/cluster endpoint to check the node's cluster connection
>status, e.g.:
>
> readinessProbe:
> exec:
> command:
> - bash
> - -c
> - |
> if [ "${SECURE}" = "true" ]; then
> INITIAL_ADMIN_SLUG=$(echo "${INITIAL_ADMIN}" | tr '[:upper:]' '[:lower:]'
> | tr ' ' '-')
>
> curl -v \
> --cacert ${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/nifi-cert.pem \
> --cert
> ${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/${INITIAL_ADMIN_SLUG}-cert.pem
> \
> --key
> ${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/${INITIAL_ADMIN_SLUG}-key.pem
> \
> https://$(hostname -f):8080/nifi-api/controller/cluster >
> /tmp/cluster.state
> else
> curl -kv http://$(hostname -f):8080/nifi-api/controller/cluster >
> /tmp/cluster.state
> fi
>
> STATUS=$(jq -r ".cluster.nodes[] | select((.address==\"$(hostname -f)\")
> or .address==\"localhost\") | .status" /tmp/cluster.state)
>
> if [[ ! $STATUS = "CONNECTED" ]]; then
> echo "Node not found with CONNECTED state. Full cluster state:"
> jq . /tmp/cluster.state
> exit 1
> fi
>
>
> Note that INITIAL_ADMIN is the CN of a user with appropriate permissions
> to call the endpoint and for whom our pod contains a set of certificate
> files in the indicated locations (generated from NiFi Toolkit in an
> init-container before the Pod starts); jq utility was added into our
> customised version of the apache/nifi Docker Image.
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
> 
>
>
>

Re: Clustered nifi issues

[Wyll Ingersoll]

That makes sense.  It must be reading the keystore/truststore specified in the 
nifi.properties file then?

From: Bryan Bende 
Sent: Wednesday, October 14, 2020 9:59 AM
To: users@nifi.apache.org 
Subject: Re: Clustered nifi issues

The get-nodes command calls the REST resource /controller/cluster which 
authorizes against READ on /controller [1], so there is no way you can call 
this in a secure environment without authenticating somehow, which from the CLI 
means specifying a keystore/truststore.

[1] 
https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ControllerResource.java#L857

On Wed, Oct 14, 2020 at 9:26 AM Wyll Ingersoll 
mailto:wyllys.ingers...@keepertech.com>> wrote:
Yes, this is for a secured cluster deployed as a Kubernetes stateful set.  The 
certificate parameters are apparently not needed to just get the status of the 
nodes using the command below.




From: Sushil Kumar mailto:skm@gmail.com>>
Sent: Tuesday, October 13, 2020 4:01 PM
To: users@nifi.apache.org 
mailto:users@nifi.apache.org>>
Subject: Re: Clustered nifi issues

Did you say that the same line of code works fine for secured clusters too.
I asked because nifi-toolkit has a separate set of parameters asking for 
certificates and everything else related to secure clusters.


On Tue, Oct 13, 2020 at 12:14 PM Wyll Ingersoll 
mailto:wyllys.ingers...@keepertech.com>> wrote:

I found that instead of dealing with nifi client certificate hell, the 
nifi-toolkit cli.sh will work just fine for testing the readiness of the 
cluster.  Here is my readiness script which seems to work just fine with in 
kubernetes with the apache/nifi docker container version 1.12.1



#!/bin/bash


$NIFI_TOOLKIT_HOME/bin/cli.sh nifi get-nodes -ot json > /tmp/cluster.state

if [ $? -ne 0 ]; then

cat /tmp/cluster.state

exit 1

fi

STATUS=$(jq -r ".cluster.nodes[] | select((.address==\"$(hostname -f)\") or 
.address==\"localhost\") | .status" /tmp/cluster.state)

if [[ ! $STATUS = "CONNECTED" ]]; then

echo "Node not found with CONNECTED state. Full cluster state:"

jq . /tmp/cluster.state

exit 1

fi



From: Chris Sampson 
mailto:chris.samp...@naimuri.com>>
Sent: Thursday, October 1, 2020 9:03 AM
To: users@nifi.apache.org 
mailto:users@nifi.apache.org>>
Subject: Re: Clustered nifi issues

For info, the probes we currently use for our StatefulSet Pods are:

  *   livenessProbe - tcpSocket to ping the NiFi instance port (e.g. 8080)
  *   readinessProbe - exec command to curl the nifi-api/controller/cluster 
endpoint to check the node's cluster connection status, e.g.:

readinessProbe:
exec:
command:
- bash
- -c
- |
if [ "${SECURE}" = "true" ]; then
INITIAL_ADMIN_SLUG=$(echo "${INITIAL_ADMIN}" | tr '[:upper:]' '[:lower:]' | tr 
' ' '-')

curl -v \
--cacert ${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/nifi-cert.pem \
--cert 
${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/${INITIAL_ADMIN_SLUG}-cert.pem
 \
--key 
${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/${INITIAL_ADMIN_SLUG}-key.pem
 \
https://$(hostname -f):8080/nifi-api/controller/cluster > /tmp/cluster.state
else
curl -kv http://$(hostname -f):8080/nifi-api/controller/cluster > 
/tmp/cluster.state
fi

STATUS=$(jq -r ".cluster.nodes[] | select((.address==\"$(hostname -f)\") or 
.address==\"localhost\") | .status" /tmp/cluster.state)

if [[ ! $STATUS = "CONNECTED" ]]; then
echo "Node not found with CONNECTED state. Full cluster state:"
jq . /tmp/cluster.state
exit 1
fi

Note that INITIAL_ADMIN is the CN of a user with appropriate permissions to 
call the endpoint and for whom our pod contains a set of certificate files in 
the indicated locations (generated from NiFi Toolkit in an init-container 
before the Pod starts); jq utility was added into our customised version of the 
apache/nifi Docker Image.


---
Chris Sampson
IT Consultant
chris.samp...@naimuri.com
[X]

Re: Clustered nifi issues

[Bryan Bende]

The get-nodes command calls the REST resource /controller/cluster which
authorizes against READ on /controller [1], so there is no way you can call
this in a secure environment without authenticating somehow, which from the
CLI means specifying a keystore/truststore.

[1]
https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ControllerResource.java#L857

On Wed, Oct 14, 2020 at 9:26 AM Wyll Ingersoll <
wyllys.ingers...@keepertech.com> wrote:

> Yes, this is for a secured cluster deployed as a Kubernetes stateful set.
> The certificate parameters are apparently not needed to just get the status
> of the nodes using the command below.
>
>
>
> --
> *From:* Sushil Kumar 
> *Sent:* Tuesday, October 13, 2020 4:01 PM
> *To:* users@nifi.apache.org 
> *Subject:* Re: Clustered nifi issues
>
> Did you say that the same line of code works fine for secured clusters
> too.
> I asked because nifi-toolkit has a separate set of parameters asking for
> certificates and everything else related to secure clusters.
>
>
> On Tue, Oct 13, 2020 at 12:14 PM Wyll Ingersoll <
> wyllys.ingers...@keepertech.com> wrote:
>
>
> I found that instead of dealing with nifi client certificate hell, the
> nifi-toolkit cli.sh will work just fine for testing the readiness of the
> cluster.  Here is my readiness script which seems to work just fine with in
> kubernetes with the apache/nifi docker container version 1.12.1
>
>
> #!/bin/bash
>
>
> $NIFI_TOOLKIT_HOME/bin/cli.sh nifi get-nodes -ot json > /tmp/cluster.state
>
> if [ $? -ne 0 ]; then
>
> cat /tmp/cluster.state
>
> exit 1
>
> fi
>
> STATUS=$(jq -r ".cluster.nodes[] | select((.address==\"$(hostname -f)\")
> or .address==\"localhost\") | .status" /tmp/cluster.state)
>
> if [[ ! $STATUS = "CONNECTED" ]]; then
>
> echo "Node not found with CONNECTED state. Full cluster state:"
>
> jq . /tmp/cluster.state
>
> exit 1
>
> fi
>
>
> --
> *From:* Chris Sampson 
> *Sent:* Thursday, October 1, 2020 9:03 AM
> *To:* users@nifi.apache.org 
> *Subject:* Re: Clustered nifi issues
>
> For info, the probes we currently use for our StatefulSet Pods are:
>
>- livenessProbe - tcpSocket to ping the NiFi instance port (e.g. 8080)
>- readinessProbe - exec command to curl the
>nifi-api/controller/cluster endpoint to check the node's cluster connection
>status, e.g.:
>
> readinessProbe:
> exec:
> command:
> - bash
> - -c
> - |
> if [ "${SECURE}" = "true" ]; then
> INITIAL_ADMIN_SLUG=$(echo "${INITIAL_ADMIN}" | tr '[:upper:]' '[:lower:]'
> | tr ' ' '-')
>
> curl -v \
> --cacert ${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/nifi-cert.pem \
> --cert
> ${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/${INITIAL_ADMIN_SLUG}-cert.pem
> \
> --key
> ${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/${INITIAL_ADMIN_SLUG}-key.pem
> \
> https://$(hostname -f):8080/nifi-api/controller/cluster >
> /tmp/cluster.state
> else
> curl -kv http://$(hostname -f):8080/nifi-api/controller/cluster >
> /tmp/cluster.state
> fi
>
> STATUS=$(jq -r ".cluster.nodes[] | select((.address==\"$(hostname -f)\")
> or .address==\"localhost\") | .status" /tmp/cluster.state)
>
> if [[ ! $STATUS = "CONNECTED" ]]; then
> echo "Node not found with CONNECTED state. Full cluster state:"
> jq . /tmp/cluster.state
> exit 1
> fi
>
>
> Note that INITIAL_ADMIN is the CN of a user with appropriate permissions
> to call the endpoint and for whom our pod contains a set of certificate
> files in the indicated locations (generated from NiFi Toolkit in an
> init-container before the Pod starts); jq utility was added into our
> customised version of the apache/nifi Docker Image.
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
> 
>
>
>

Re: Clustered nifi issues

[Wyll Ingersoll]

Yes, this is for a secured cluster deployed as a Kubernetes stateful set.  The 
certificate parameters are apparently not needed to just get the status of the 
nodes using the command below.




From: Sushil Kumar 
Sent: Tuesday, October 13, 2020 4:01 PM
To: users@nifi.apache.org 
Subject: Re: Clustered nifi issues

Did you say that the same line of code works fine for secured clusters too.
I asked because nifi-toolkit has a separate set of parameters asking for 
certificates and everything else related to secure clusters.


On Tue, Oct 13, 2020 at 12:14 PM Wyll Ingersoll 
mailto:wyllys.ingers...@keepertech.com>> wrote:

I found that instead of dealing with nifi client certificate hell, the 
nifi-toolkit cli.sh will work just fine for testing the readiness of the 
cluster.  Here is my readiness script which seems to work just fine with in 
kubernetes with the apache/nifi docker container version 1.12.1



#!/bin/bash


$NIFI_TOOLKIT_HOME/bin/cli.sh nifi get-nodes -ot json > /tmp/cluster.state

if [ $? -ne 0 ]; then

cat /tmp/cluster.state

exit 1

fi

STATUS=$(jq -r ".cluster.nodes[] | select((.address==\"$(hostname -f)\") or 
.address==\"localhost\") | .status" /tmp/cluster.state)

if [[ ! $STATUS = "CONNECTED" ]]; then

echo "Node not found with CONNECTED state. Full cluster state:"

jq . /tmp/cluster.state

exit 1

fi



From: Chris Sampson 
mailto:chris.samp...@naimuri.com>>
Sent: Thursday, October 1, 2020 9:03 AM
To: users@nifi.apache.org 
mailto:users@nifi.apache.org>>
Subject: Re: Clustered nifi issues

For info, the probes we currently use for our StatefulSet Pods are:

  *   livenessProbe - tcpSocket to ping the NiFi instance port (e.g. 8080)
  *   readinessProbe - exec command to curl the nifi-api/controller/cluster 
endpoint to check the node's cluster connection status, e.g.:

readinessProbe:
exec:
command:
- bash
- -c
- |
if [ "${SECURE}" = "true" ]; then
INITIAL_ADMIN_SLUG=$(echo "${INITIAL_ADMIN}" | tr '[:upper:]' '[:lower:]' | tr 
' ' '-')

curl -v \
--cacert ${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/nifi-cert.pem \
--cert 
${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/${INITIAL_ADMIN_SLUG}-cert.pem
 \
--key 
${NIFI_HOME}/data/conf/certs/${INITIAL_ADMIN_SLUG}/${INITIAL_ADMIN_SLUG}-key.pem
 \
https://$(hostname -f):8080/nifi-api/controller/cluster > /tmp/cluster.state
else
curl -kv http://$(hostname -f):8080/nifi-api/controller/cluster > 
/tmp/cluster.state
fi

STATUS=$(jq -r ".cluster.nodes[] | select((.address==\"$(hostname -f)\") or 
.address==\"localhost\") | .status" /tmp/cluster.state)

if [[ ! $STATUS = "CONNECTED" ]]; then
echo "Node not found with CONNECTED state. Full cluster state:"
jq . /tmp/cluster.state
exit 1
fi

Note that INITIAL_ADMIN is the CN of a user with appropriate permissions to 
call the endpoint and for whom our pod contains a set of certificate files in 
the indicated locations (generated from NiFi Toolkit in an init-container 
before the Pod starts); jq utility was added into our customised version of the 
apache/nifi Docker Image.


---
Chris Sampson
IT Consultant
chris.samp...@naimuri.com
[https://docs.google.com/uc?export=download=1oPtzd0P7DqtuzpjiTRAa6h6coFitpqom=0B9aXwC5rMc6lVlZ2OWpUaVlFVmUwTlZBdjQ0KzAxb1dZS2hJPQ]

Re: Compare two NiFi servers

[Mike Thomsen]

Copy flow.xml.gz from both and either write a script or find a tool
that can look for the processor references. That file is essentially
the master state of what you see on the canvas.

On Tue, Oct 13, 2020 at 5:44 PM Alberto Dominguez
 wrote:
>
> Hello,
>
> I have two environments and one NiFi server in each one. How can I compare
> which flows and processors that I have in each one?
>
> I have until 5 levels of grouping.
>
> NiFi Home --> First --> Second...
>
> Thank you!