ListenWebSocket disconnect event

[Luis]

Hi everyone,

I am using ListenWebSocket in a Flow and works perfect, nevertheless I
need to know when client gets disconnected.

The processor doesn't seem to have the "Disconnect output path" though
it is considered in the source code as far as a I could check.

Does anyone know some custom processor that consider the "Disconnect
output". I am a very basic java programmer so I couldn't figure out how
to add it looking at NIFI source code.

Appreciate any help.

Thanks,

LC

Re: Connecting Controller Services Automatically

[Mark Payne]

It should be hierarchical. The Set of Controller Services to match by name is 
obtained by calling `parentGroup.getControllerServices(true)` where parentGroup 
is the Process Group that components are to be added to.

@Eric, the code that does the matching can be found at [1].

Thanks
Mark


[1] 
https://github.com/apache/nifi/blob/45e626f6a872296cdd08fcfc533c7ce7e3624fca/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L3514



On May 23, 2020, at 8:31 PM, Bryan Bende 
mailto:bbe...@gmail.com>> wrote:

Do you have any controller services in process groups above the “controller 
container”?

I can’t remember if it is based on only the immediate parent, or the entire 
hierarchy.

On Sat, May 23, 2020 at 8:00 PM Andrew Grande 
mailto:apere...@gmail.com>> wrote:
Maybe something is going on with specific types or hierarchies. I've noticed 
DefaultSslContext didn't get assigned, even though it was the only one 
available. Does autowiring logic apply to this one?

Andrew

On Sat, May 23, 2020, 3:54 PM Eric Secules 
mailto:esecu...@gmail.com>> wrote:
Hi Bryan,

I have noticed this behaviour sometimes, but not all the time I am running the 
latest registry and NiFi versions. I haven't found a conclusive pattern but I 
have a hunch that it has to do with having versioned process groups within 
versioned process groups. My deployment strategy is this:

  *   Have an outer process group which only contains controller services, 
called the "Controller Container"
 *   For now I just have one controller service per type of controller 
service.
  *   When deploying, download all production flows inside the Controller 
Container.
  *   I noticed that some of the controller services find their match, but 
others don't leaving me with roughly 70 invalid processors out of 800.

If you could point me in the right direction of the code which is supposed to 
do the matching I might be able to debug better.

Thanks,
Eric

On Sat, May 23, 2020 at 3:27 PM Bryan Bende 
mailto:bbe...@gmail.com>> wrote:
If you use registry >= 0.5.0 And nifi >= 1.10.0, then it will auto select 
external controller services with the same name as long as there is only one of 
the same type with same name (name is not unique).

On Sat, May 23, 2020 at 3:34 PM Andy LoPresto 
mailto:alopre...@apache.org>> wrote:
My position is that we don’t claim completely automated deployment as a 
feature, so manually setting the controller service IDs is not exposed. 
Technically, they are defined in the flow.xml.gz and could be modified by an 
administrator to be static after generation. This would require frequent manual 
manipulation of the flow.xml.gz in various environments and frequent restarts 
of the NiFi service. I do not recommend this.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On May 23, 2020, at 11:05 AM, Andrew Grande 
mailto:apere...@gmail.com>> wrote:

Aren't those IDs generated? How can one enforce it?

Andrew

On Sat, May 23, 2020, 10:53 AM Andy LoPresto 
mailto:alopre...@apache.org>> wrote:
If you want the process to be completely automated, you would have to enforce 
the controller service IDs to be identical across environments. Otherwise 
deployment would need a manual intervention to reference the specific 
controller service in the proper component.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On May 22, 2020, at 3:57 PM, Eric Secules 
mailto:esecu...@gmail.com>> wrote:

Hi Andy,

Given that you have a flow which operates on two different S3 accounts for 
example, how would you do deployment automation? Do you mandate that the 
controller service with the same ID must exist in both a development and 
production environment rather than try to connect a processor to a matching 
controller service?

-Eric

On Fri, May 22, 2020 at 3:44 PM Andy LoPresto 
mailto:alopre...@apache.org>> wrote:
Eric,

I can’t answer all these questions but I would definitely have hesitations 
around building an expectation that there is only one instance of any given 
controller service type in an entire canvas. I can think of numerous flows 
(this may not affect your particular flows, but the concepts still apply) which 
require multiple instances of the same controller service type to be available:

* A flow which invokes a mutually-authenticated TLS HTTP API, consumes data, 
transforms it, and posts it to another mTLS API
* A flow which retrieves objects from one S3 bucket and puts them into an S3 
bucket in a different AWS account
* A flow which connects to one database and retrieves data, transforms it, and 
persists it to another database

If there is only _

Re: Connecting Controller Services Automatically

[Bryan Bende]

Do you have any controller services in process groups above the “controller
container”?

I can’t remember if it is based on only the immediate parent, or the entire
hierarchy.

On Sat, May 23, 2020 at 8:00 PM Andrew Grande  wrote:

> Maybe something is going on with specific types or hierarchies. I've
> noticed DefaultSslContext didn't get assigned, even though it was the only
> one available. Does autowiring logic apply to this one?
>
> Andrew
>
> On Sat, May 23, 2020, 3:54 PM Eric Secules  wrote:
>
>> Hi Bryan,
>>
>> I have noticed this behaviour sometimes, but not all the time I am
>> running the latest registry and NiFi versions. I haven't found a conclusive
>> pattern but I have a hunch that it has to do with having versioned process
>> groups within versioned process groups. My deployment strategy is this:
>>
>>- Have an outer process group which only contains controller
>>services, called the "Controller Container"
>>   - For now I just have one controller service per type of
>>   controller service.
>>   - When deploying, download all production flows inside the
>>Controller Container.
>>- I noticed that some of the controller services find their match,
>>but others don't leaving me with roughly 70 invalid processors out of 800.
>>
>> If you could point me in the right direction of the code which is
>> supposed to do the matching I might be able to debug better.
>>
>> Thanks,
>> Eric
>>
>> On Sat, May 23, 2020 at 3:27 PM Bryan Bende  wrote:
>>
>>> If you use registry >= 0.5.0 And nifi >= 1.10.0, then it will auto
>>> select external controller services with the same name as long as there is
>>> only one of the same type with same name (name is not unique).
>>>
>>> On Sat, May 23, 2020 at 3:34 PM Andy LoPresto 
>>> wrote:
>>>
 My position is that we don’t claim completely automated deployment as a
 feature, so manually setting the controller service IDs is not exposed.
 Technically, they are defined in the flow.xml.gz and could be modified by
 an administrator to be static after generation. This would require frequent
 manual manipulation of the flow.xml.gz in various environments and frequent
 restarts of the NiFi service. I do not recommend this.


 Andy LoPresto
 alopre...@apache.org
 *alopresto.apa...@gmail.com *
 He/Him
 PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

 On May 23, 2020, at 11:05 AM, Andrew Grande  wrote:

 Aren't those IDs generated? How can one enforce it?

 Andrew

 On Sat, May 23, 2020, 10:53 AM Andy LoPresto 
 wrote:

> If you want the process to be completely automated, you would have to
> enforce the controller service IDs to be identical across environments.
> Otherwise deployment would need a manual intervention to reference the
> specific controller service in the proper component.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 22, 2020, at 3:57 PM, Eric Secules  wrote:
>
> Hi Andy,
>
> Given that you have a flow which operates on two different S3 accounts
> for example, how would you do deployment automation? Do you mandate that
> the controller service with the same ID must exist in both a development
> and production environment rather than try to connect a processor to a
> matching controller service?
>
> -Eric
>
> On Fri, May 22, 2020 at 3:44 PM Andy LoPresto 
> wrote:
>
>> Eric,
>>
>> I can’t answer all these questions but I would definitely have
>> hesitations around building an expectation that there is only one 
>> instance
>> of any given controller service type in an entire canvas. I can think of
>> numerous flows (this may not affect your particular flows, but the 
>> concepts
>> still apply) which require multiple instances of the same controller
>> service type to be available:
>>
>> * A flow which invokes a mutually-authenticated TLS HTTP API,
>> consumes data, transforms it, and posts it to another mTLS API
>> * A flow which retrieves objects from one S3 bucket and puts them
>> into an S3 bucket in a different AWS account
>> * A flow which connects to one database and retrieves data,
>> transforms it, and persists it to another database
>>
>> If there is only _one_ StandardSSLContextService,
>> AWSCredentialsProviderControllerService, or DBCPConnectionPool available 
>> in
>> the entire controller, these flows cannot exist.
>>
>> I am not saying the retrieval of new flow versions and the matching
>> of referenced controller services cannot be improved, but I would
>> definitely advise caution before going too far down this path without
>> considering all possible side effects and potential 

Re: Connecting Controller Services Automatically

[Andrew Grande]

Maybe something is going on with specific types or hierarchies. I've
noticed DefaultSslContext didn't get assigned, even though it was the only
one available. Does autowiring logic apply to this one?

Andrew

On Sat, May 23, 2020, 3:54 PM Eric Secules  wrote:

> Hi Bryan,
>
> I have noticed this behaviour sometimes, but not all the time I am running
> the latest registry and NiFi versions. I haven't found a conclusive pattern
> but I have a hunch that it has to do with having versioned process groups
> within versioned process groups. My deployment strategy is this:
>
>- Have an outer process group which only contains controller services,
>called the "Controller Container"
>   - For now I just have one controller service per type of controller
>   service.
>   - When deploying, download all production flows inside the
>Controller Container.
>- I noticed that some of the controller services find their match, but
>others don't leaving me with roughly 70 invalid processors out of 800.
>
> If you could point me in the right direction of the code which is supposed
> to do the matching I might be able to debug better.
>
> Thanks,
> Eric
>
> On Sat, May 23, 2020 at 3:27 PM Bryan Bende  wrote:
>
>> If you use registry >= 0.5.0 And nifi >= 1.10.0, then it will auto select
>> external controller services with the same name as long as there is only
>> one of the same type with same name (name is not unique).
>>
>> On Sat, May 23, 2020 at 3:34 PM Andy LoPresto 
>> wrote:
>>
>>> My position is that we don’t claim completely automated deployment as a
>>> feature, so manually setting the controller service IDs is not exposed.
>>> Technically, they are defined in the flow.xml.gz and could be modified by
>>> an administrator to be static after generation. This would require frequent
>>> manual manipulation of the flow.xml.gz in various environments and frequent
>>> restarts of the NiFi service. I do not recommend this.
>>>
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> He/Him
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On May 23, 2020, at 11:05 AM, Andrew Grande  wrote:
>>>
>>> Aren't those IDs generated? How can one enforce it?
>>>
>>> Andrew
>>>
>>> On Sat, May 23, 2020, 10:53 AM Andy LoPresto 
>>> wrote:
>>>
 If you want the process to be completely automated, you would have to
 enforce the controller service IDs to be identical across environments.
 Otherwise deployment would need a manual intervention to reference the
 specific controller service in the proper component.

 Andy LoPresto
 alopre...@apache.org
 *alopresto.apa...@gmail.com *
 He/Him
 PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

 On May 22, 2020, at 3:57 PM, Eric Secules  wrote:

 Hi Andy,

 Given that you have a flow which operates on two different S3 accounts
 for example, how would you do deployment automation? Do you mandate that
 the controller service with the same ID must exist in both a development
 and production environment rather than try to connect a processor to a
 matching controller service?

 -Eric

 On Fri, May 22, 2020 at 3:44 PM Andy LoPresto 
 wrote:

> Eric,
>
> I can’t answer all these questions but I would definitely have
> hesitations around building an expectation that there is only one instance
> of any given controller service type in an entire canvas. I can think of
> numerous flows (this may not affect your particular flows, but the 
> concepts
> still apply) which require multiple instances of the same controller
> service type to be available:
>
> * A flow which invokes a mutually-authenticated TLS HTTP API, consumes
> data, transforms it, and posts it to another mTLS API
> * A flow which retrieves objects from one S3 bucket and puts them into
> an S3 bucket in a different AWS account
> * A flow which connects to one database and retrieves data, transforms
> it, and persists it to another database
>
> If there is only _one_ StandardSSLContextService,
> AWSCredentialsProviderControllerService, or DBCPConnectionPool available 
> in
> the entire controller, these flows cannot exist.
>
> I am not saying the retrieval of new flow versions and the matching of
> referenced controller services cannot be improved, but I would definitely
> advise caution before going too far down this path without considering all
> possible side effects and potential constraints on future flow 
> development.
>
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 22, 2020, at 3:01 PM, Eric Secules  wrote:
>
> Hello everyone,
>
> I am running into an i

Re: Connecting Controller Services Automatically

[Eric Secules]

Hi Bryan,

I have noticed this behaviour sometimes, but not all the time I am running
the latest registry and NiFi versions. I haven't found a conclusive pattern
but I have a hunch that it has to do with having versioned process groups
within versioned process groups. My deployment strategy is this:

   - Have an outer process group which only contains controller services,
   called the "Controller Container"
  - For now I just have one controller service per type of controller
  service.
  - When deploying, download all production flows inside the Controller
   Container.
   - I noticed that some of the controller services find their match, but
   others don't leaving me with roughly 70 invalid processors out of 800.

If you could point me in the right direction of the code which is supposed
to do the matching I might be able to debug better.

Thanks,
Eric

On Sat, May 23, 2020 at 3:27 PM Bryan Bende  wrote:

> If you use registry >= 0.5.0 And nifi >= 1.10.0, then it will auto select
> external controller services with the same name as long as there is only
> one of the same type with same name (name is not unique).
>
> On Sat, May 23, 2020 at 3:34 PM Andy LoPresto 
> wrote:
>
>> My position is that we don’t claim completely automated deployment as a
>> feature, so manually setting the controller service IDs is not exposed.
>> Technically, they are defined in the flow.xml.gz and could be modified by
>> an administrator to be static after generation. This would require frequent
>> manual manipulation of the flow.xml.gz in various environments and frequent
>> restarts of the NiFi service. I do not recommend this.
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On May 23, 2020, at 11:05 AM, Andrew Grande  wrote:
>>
>> Aren't those IDs generated? How can one enforce it?
>>
>> Andrew
>>
>> On Sat, May 23, 2020, 10:53 AM Andy LoPresto 
>> wrote:
>>
>>> If you want the process to be completely automated, you would have to
>>> enforce the controller service IDs to be identical across environments.
>>> Otherwise deployment would need a manual intervention to reference the
>>> specific controller service in the proper component.
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> He/Him
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On May 22, 2020, at 3:57 PM, Eric Secules  wrote:
>>>
>>> Hi Andy,
>>>
>>> Given that you have a flow which operates on two different S3 accounts
>>> for example, how would you do deployment automation? Do you mandate that
>>> the controller service with the same ID must exist in both a development
>>> and production environment rather than try to connect a processor to a
>>> matching controller service?
>>>
>>> -Eric
>>>
>>> On Fri, May 22, 2020 at 3:44 PM Andy LoPresto 
>>> wrote:
>>>
 Eric,

 I can’t answer all these questions but I would definitely have
 hesitations around building an expectation that there is only one instance
 of any given controller service type in an entire canvas. I can think of
 numerous flows (this may not affect your particular flows, but the concepts
 still apply) which require multiple instances of the same controller
 service type to be available:

 * A flow which invokes a mutually-authenticated TLS HTTP API, consumes
 data, transforms it, and posts it to another mTLS API
 * A flow which retrieves objects from one S3 bucket and puts them into
 an S3 bucket in a different AWS account
 * A flow which connects to one database and retrieves data, transforms
 it, and persists it to another database

 If there is only _one_ StandardSSLContextService,
 AWSCredentialsProviderControllerService, or DBCPConnectionPool available in
 the entire controller, these flows cannot exist.

 I am not saying the retrieval of new flow versions and the matching of
 referenced controller services cannot be improved, but I would definitely
 advise caution before going too far down this path without considering all
 possible side effects and potential constraints on future flow development.



 Andy LoPresto
 alopre...@apache.org
 *alopresto.apa...@gmail.com *
 He/Him
 PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

 On May 22, 2020, at 3:01 PM, Eric Secules  wrote:

 Hello everyone,

 I am running into an issue with automated deployment using nipyapi
 . We would like to be able
 to pull down flows from a registry and have them ready to go once all their
 controller services have been turned on. But there are a few issues.
 Sometimes the flows that we download from the registry reference controller
 service IDs that don't exist on this machine because the 

Re: Connecting Controller Services Automatically

[Andy LoPresto]

Yes, I should have clarified this. Thanks Bryan. This is the solution for the 
generic use case. The original question was about reducing the controller to 
only a single instance of a specific controller service implementation, which 
is how the tangent got started. 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On May 23, 2020, at 3:27 PM, Bryan Bende  wrote:
> 
> If you use registry >= 0.5.0 And nifi >= 1.10.0, then it will auto select 
> external controller services with the same name as long as there is only one 
> of the same type with same name (name is not unique).
> 
> On Sat, May 23, 2020 at 3:34 PM Andy LoPresto  > wrote:
> My position is that we don’t claim completely automated deployment as a 
> feature, so manually setting the controller service IDs is not exposed. 
> Technically, they are defined in the flow.xml.gz and could be modified by an 
> administrator to be static after generation. This would require frequent 
> manual manipulation of the flow.xml.gz in various environments and frequent 
> restarts of the NiFi service. I do not recommend this. 
> 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On May 23, 2020, at 11:05 AM, Andrew Grande > > wrote:
>> 
>> Aren't those IDs generated? How can one enforce it?
>> 
>> Andrew
>> 
>> On Sat, May 23, 2020, 10:53 AM Andy LoPresto > > wrote:
>> If you want the process to be completely automated, you would have to 
>> enforce the controller service IDs to be identical across environments. 
>> Otherwise deployment would need a manual intervention to reference the 
>> specific controller service in the proper component. 
>> 
>> Andy LoPresto
>> alopre...@apache.org 
>> alopresto.apa...@gmail.com 
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>>> On May 22, 2020, at 3:57 PM, Eric Secules >> > wrote:
>>> 
>>> Hi Andy, 
>>> 
>>> Given that you have a flow which operates on two different S3 accounts for 
>>> example, how would you do deployment automation? Do you mandate that the 
>>> controller service with the same ID must exist in both a development and 
>>> production environment rather than try to connect a processor to a matching 
>>> controller service?
>>> 
>>> -Eric
>>> 
>>> On Fri, May 22, 2020 at 3:44 PM Andy LoPresto >> > wrote:
>>> Eric,
>>> 
>>> I can’t answer all these questions but I would definitely have hesitations 
>>> around building an expectation that there is only one instance of any given 
>>> controller service type in an entire canvas. I can think of numerous flows 
>>> (this may not affect your particular flows, but the concepts still apply) 
>>> which require multiple instances of the same controller service type to be 
>>> available: 
>>> 
>>> * A flow which invokes a mutually-authenticated TLS HTTP API, consumes 
>>> data, transforms it, and posts it to another mTLS API
>>> * A flow which retrieves objects from one S3 bucket and puts them into an 
>>> S3 bucket in a different AWS account
>>> * A flow which connects to one database and retrieves data, transforms it, 
>>> and persists it to another database
>>> 
>>> If there is only _one_ StandardSSLContextService, 
>>> AWSCredentialsProviderControllerService, or DBCPConnectionPool available in 
>>> the entire controller, these flows cannot exist. 
>>> 
>>> I am not saying the retrieval of new flow versions and the matching of 
>>> referenced controller services cannot be improved, but I would definitely 
>>> advise caution before going too far down this path without considering all 
>>> possible side effects and potential constraints on future flow development. 
>>>  
>>> 
>>> 
>>> Andy LoPresto
>>> alopre...@apache.org 
>>> alopresto.apa...@gmail.com 
>>> He/Him
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>> 
 On May 22, 2020, at 3:01 PM, Eric Secules >>> > wrote:
 
 Hello everyone,
 
 I am running into an issue with automated deployment using nipyapi 
 . We would like to be able to 
 pull down flows from a registry and have them ready to go once all their 
 controller services have been turned on. But there are a few issues. 
 Sometimes the flows that we download from the registry reference 
 controller service IDs that don't exist on this machine because the flow 
 was developed in a different environment. That's easy enough to fix if 
 there is just one app

Re: Connecting Controller Services Automatically

[Bryan Bende]

If you use registry >= 0.5.0 And nifi >= 1.10.0, then it will auto select
external controller services with the same name as long as there is only
one of the same type with same name (name is not unique).

On Sat, May 23, 2020 at 3:34 PM Andy LoPresto  wrote:

> My position is that we don’t claim completely automated deployment as a
> feature, so manually setting the controller service IDs is not exposed.
> Technically, they are defined in the flow.xml.gz and could be modified by
> an administrator to be static after generation. This would require frequent
> manual manipulation of the flow.xml.gz in various environments and frequent
> restarts of the NiFi service. I do not recommend this.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 23, 2020, at 11:05 AM, Andrew Grande  wrote:
>
> Aren't those IDs generated? How can one enforce it?
>
> Andrew
>
> On Sat, May 23, 2020, 10:53 AM Andy LoPresto  wrote:
>
>> If you want the process to be completely automated, you would have to
>> enforce the controller service IDs to be identical across environments.
>> Otherwise deployment would need a manual intervention to reference the
>> specific controller service in the proper component.
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On May 22, 2020, at 3:57 PM, Eric Secules  wrote:
>>
>> Hi Andy,
>>
>> Given that you have a flow which operates on two different S3 accounts
>> for example, how would you do deployment automation? Do you mandate that
>> the controller service with the same ID must exist in both a development
>> and production environment rather than try to connect a processor to a
>> matching controller service?
>>
>> -Eric
>>
>> On Fri, May 22, 2020 at 3:44 PM Andy LoPresto 
>> wrote:
>>
>>> Eric,
>>>
>>> I can’t answer all these questions but I would definitely have
>>> hesitations around building an expectation that there is only one instance
>>> of any given controller service type in an entire canvas. I can think of
>>> numerous flows (this may not affect your particular flows, but the concepts
>>> still apply) which require multiple instances of the same controller
>>> service type to be available:
>>>
>>> * A flow which invokes a mutually-authenticated TLS HTTP API, consumes
>>> data, transforms it, and posts it to another mTLS API
>>> * A flow which retrieves objects from one S3 bucket and puts them into
>>> an S3 bucket in a different AWS account
>>> * A flow which connects to one database and retrieves data, transforms
>>> it, and persists it to another database
>>>
>>> If there is only _one_ StandardSSLContextService,
>>> AWSCredentialsProviderControllerService, or DBCPConnectionPool available in
>>> the entire controller, these flows cannot exist.
>>>
>>> I am not saying the retrieval of new flow versions and the matching of
>>> referenced controller services cannot be improved, but I would definitely
>>> advise caution before going too far down this path without considering all
>>> possible side effects and potential constraints on future flow development.
>>>
>>>
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> He/Him
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On May 22, 2020, at 3:01 PM, Eric Secules  wrote:
>>>
>>> Hello everyone,
>>>
>>> I am running into an issue with automated deployment using nipyapi
>>> . We would like to be able
>>> to pull down flows from a registry and have them ready to go once all their
>>> controller services have been turned on. But there are a few issues.
>>> Sometimes the flows that we download from the registry reference controller
>>> service IDs that don't exist on this machine because the flow was developed
>>> in a different environment. That's easy enough to fix if there is just one
>>> applicable controller service, but not when there are two or more.
>>>
>>> We have taken the decision to put all our controller services at the top
>>> level and have one of each kind we need, rather than have multiple of the
>>> same controller service attached to individual process groups.
>>>
>>> We are running into a problem where some processors can either connect
>>> to a JSONTreeReader or a CSVReader and there's no indication in the
>>> ProcessorDTO object which type it was originally connected to, just a GUID
>>> of a controller service that doesn't exist in this deployment.
>>>
>>> Would it be possible to include the type or name of the controller
>>> service in the component.config.descriptors section? Are we going about it
>>> the wrong way trying to simplify down to the least number of controller
>>> services?
>>>
>>> Thanks,
>>> Eric
>>>
>>>
>>>
>>
> --
Sent from Gmail Mobile

Re: Connecting Controller Services Automatically

[Andy LoPresto]

My position is that we don’t claim completely automated deployment as a 
feature, so manually setting the controller service IDs is not exposed. 
Technically, they are defined in the flow.xml.gz and could be modified by an 
administrator to be static after generation. This would require frequent manual 
manipulation of the flow.xml.gz in various environments and frequent restarts 
of the NiFi service. I do not recommend this. 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On May 23, 2020, at 11:05 AM, Andrew Grande  wrote:
> 
> Aren't those IDs generated? How can one enforce it?
> 
> Andrew
> 
> On Sat, May 23, 2020, 10:53 AM Andy LoPresto  > wrote:
> If you want the process to be completely automated, you would have to enforce 
> the controller service IDs to be identical across environments. Otherwise 
> deployment would need a manual intervention to reference the specific 
> controller service in the proper component. 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On May 22, 2020, at 3:57 PM, Eric Secules > > wrote:
>> 
>> Hi Andy, 
>> 
>> Given that you have a flow which operates on two different S3 accounts for 
>> example, how would you do deployment automation? Do you mandate that the 
>> controller service with the same ID must exist in both a development and 
>> production environment rather than try to connect a processor to a matching 
>> controller service?
>> 
>> -Eric
>> 
>> On Fri, May 22, 2020 at 3:44 PM Andy LoPresto > > wrote:
>> Eric,
>> 
>> I can’t answer all these questions but I would definitely have hesitations 
>> around building an expectation that there is only one instance of any given 
>> controller service type in an entire canvas. I can think of numerous flows 
>> (this may not affect your particular flows, but the concepts still apply) 
>> which require multiple instances of the same controller service type to be 
>> available: 
>> 
>> * A flow which invokes a mutually-authenticated TLS HTTP API, consumes data, 
>> transforms it, and posts it to another mTLS API
>> * A flow which retrieves objects from one S3 bucket and puts them into an S3 
>> bucket in a different AWS account
>> * A flow which connects to one database and retrieves data, transforms it, 
>> and persists it to another database
>> 
>> If there is only _one_ StandardSSLContextService, 
>> AWSCredentialsProviderControllerService, or DBCPConnectionPool available in 
>> the entire controller, these flows cannot exist. 
>> 
>> I am not saying the retrieval of new flow versions and the matching of 
>> referenced controller services cannot be improved, but I would definitely 
>> advise caution before going too far down this path without considering all 
>> possible side effects and potential constraints on future flow development.  
>> 
>> 
>> Andy LoPresto
>> alopre...@apache.org 
>> alopresto.apa...@gmail.com 
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>>> On May 22, 2020, at 3:01 PM, Eric Secules >> > wrote:
>>> 
>>> Hello everyone,
>>> 
>>> I am running into an issue with automated deployment using nipyapi 
>>> . We would like to be able to 
>>> pull down flows from a registry and have them ready to go once all their 
>>> controller services have been turned on. But there are a few issues. 
>>> Sometimes the flows that we download from the registry reference controller 
>>> service IDs that don't exist on this machine because the flow was developed 
>>> in a different environment. That's easy enough to fix if there is just one 
>>> applicable controller service, but not when there are two or more. 
>>> 
>>> We have taken the decision to put all our controller services at the top 
>>> level and have one of each kind we need, rather than have multiple of the 
>>> same controller service attached to individual process groups.
>>> 
>>> We are running into a problem where some processors can either connect to a 
>>> JSONTreeReader or a CSVReader and there's no indication in the ProcessorDTO 
>>> object which type it was originally connected to, just a GUID of a 
>>> controller service that doesn't exist in this deployment. 
>>> 
>>> Would it be possible to include the type or name of the controller service 
>>> in the component.config.descriptors section? Are we going about it the 
>>> wrong way trying to simplify down to the least number of controller 
>>> services? 
>>> 
>>> Thanks,
>>> Eric
>> 
> 

Re: Connecting Controller Services Automatically

[Andrew Grande]

Aren't those IDs generated? How can one enforce it?

Andrew

On Sat, May 23, 2020, 10:53 AM Andy LoPresto  wrote:

> If you want the process to be completely automated, you would have to
> enforce the controller service IDs to be identical across environments.
> Otherwise deployment would need a manual intervention to reference the
> specific controller service in the proper component.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 22, 2020, at 3:57 PM, Eric Secules  wrote:
>
> Hi Andy,
>
> Given that you have a flow which operates on two different S3 accounts for
> example, how would you do deployment automation? Do you mandate that the
> controller service with the same ID must exist in both a development and
> production environment rather than try to connect a processor to a matching
> controller service?
>
> -Eric
>
> On Fri, May 22, 2020 at 3:44 PM Andy LoPresto 
> wrote:
>
>> Eric,
>>
>> I can’t answer all these questions but I would definitely have
>> hesitations around building an expectation that there is only one instance
>> of any given controller service type in an entire canvas. I can think of
>> numerous flows (this may not affect your particular flows, but the concepts
>> still apply) which require multiple instances of the same controller
>> service type to be available:
>>
>> * A flow which invokes a mutually-authenticated TLS HTTP API, consumes
>> data, transforms it, and posts it to another mTLS API
>> * A flow which retrieves objects from one S3 bucket and puts them into an
>> S3 bucket in a different AWS account
>> * A flow which connects to one database and retrieves data, transforms
>> it, and persists it to another database
>>
>> If there is only _one_ StandardSSLContextService,
>> AWSCredentialsProviderControllerService, or DBCPConnectionPool available in
>> the entire controller, these flows cannot exist.
>>
>> I am not saying the retrieval of new flow versions and the matching of
>> referenced controller services cannot be improved, but I would definitely
>> advise caution before going too far down this path without considering all
>> possible side effects and potential constraints on future flow development.
>>
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On May 22, 2020, at 3:01 PM, Eric Secules  wrote:
>>
>> Hello everyone,
>>
>> I am running into an issue with automated deployment using nipyapi
>> . We would like to be able to
>> pull down flows from a registry and have them ready to go once all their
>> controller services have been turned on. But there are a few issues.
>> Sometimes the flows that we download from the registry reference controller
>> service IDs that don't exist on this machine because the flow was developed
>> in a different environment. That's easy enough to fix if there is just one
>> applicable controller service, but not when there are two or more.
>>
>> We have taken the decision to put all our controller services at the top
>> level and have one of each kind we need, rather than have multiple of the
>> same controller service attached to individual process groups.
>>
>> We are running into a problem where some processors can either connect to
>> a JSONTreeReader or a CSVReader and there's no indication in the
>> ProcessorDTO object which type it was originally connected to, just a GUID
>> of a controller service that doesn't exist in this deployment.
>>
>> Would it be possible to include the type or name of the controller
>> service in the component.config.descriptors section? Are we going about it
>> the wrong way trying to simplify down to the least number of controller
>> services?
>>
>> Thanks,
>> Eric
>>
>>
>>
>

Re: Connecting Controller Services Automatically

[Andy LoPresto]

If you want the process to be completely automated, you would have to enforce 
the controller service IDs to be identical across environments. Otherwise 
deployment would need a manual intervention to reference the specific 
controller service in the proper component. 

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On May 22, 2020, at 3:57 PM, Eric Secules  wrote:
> 
> Hi Andy, 
> 
> Given that you have a flow which operates on two different S3 accounts for 
> example, how would you do deployment automation? Do you mandate that the 
> controller service with the same ID must exist in both a development and 
> production environment rather than try to connect a processor to a matching 
> controller service?
> 
> -Eric
> 
> On Fri, May 22, 2020 at 3:44 PM Andy LoPresto  > wrote:
> Eric,
> 
> I can’t answer all these questions but I would definitely have hesitations 
> around building an expectation that there is only one instance of any given 
> controller service type in an entire canvas. I can think of numerous flows 
> (this may not affect your particular flows, but the concepts still apply) 
> which require multiple instances of the same controller service type to be 
> available: 
> 
> * A flow which invokes a mutually-authenticated TLS HTTP API, consumes data, 
> transforms it, and posts it to another mTLS API
> * A flow which retrieves objects from one S3 bucket and puts them into an S3 
> bucket in a different AWS account
> * A flow which connects to one database and retrieves data, transforms it, 
> and persists it to another database
> 
> If there is only _one_ StandardSSLContextService, 
> AWSCredentialsProviderControllerService, or DBCPConnectionPool available in 
> the entire controller, these flows cannot exist. 
> 
> I am not saying the retrieval of new flow versions and the matching of 
> referenced controller services cannot be improved, but I would definitely 
> advise caution before going too far down this path without considering all 
> possible side effects and potential constraints on future flow development.  
> 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On May 22, 2020, at 3:01 PM, Eric Secules > > wrote:
>> 
>> Hello everyone,
>> 
>> I am running into an issue with automated deployment using nipyapi 
>> . We would like to be able to 
>> pull down flows from a registry and have them ready to go once all their 
>> controller services have been turned on. But there are a few issues. 
>> Sometimes the flows that we download from the registry reference controller 
>> service IDs that don't exist on this machine because the flow was developed 
>> in a different environment. That's easy enough to fix if there is just one 
>> applicable controller service, but not when there are two or more. 
>> 
>> We have taken the decision to put all our controller services at the top 
>> level and have one of each kind we need, rather than have multiple of the 
>> same controller service attached to individual process groups.
>> 
>> We are running into a problem where some processors can either connect to a 
>> JSONTreeReader or a CSVReader and there's no indication in the ProcessorDTO 
>> object which type it was originally connected to, just a GUID of a 
>> controller service that doesn't exist in this deployment. 
>> 
>> Would it be possible to include the type or name of the controller service 
>> in the component.config.descriptors section? Are we going about it the wrong 
>> way trying to simplify down to the least number of controller services? 
>> 
>> Thanks,
>> Eric
> 

Re: Use of SNI routing in Nifi ?

[Pat White]

Thank you Andy, certainly appreciate you looking at this. The use of a
frontend proxy is an excellent point, both to handle the routing as well as
adding isolation for Nifi.

Thanks again for the help.

patw

On Fri, May 22, 2020 at 3:53 PM Andy LoPresto  wrote:

> Thanks Pat. The S2S protocol uses TLS as a component, and attempts to use
> the highest protocol version supported by both endpoints. For Java 8, this
> should be TLSv1.2, and for Java 11, TLSv1.3 (introduced in upcoming NiFi
> 1.12.0).
>
> NiFi itself doesn’t support hosting multiple instances on the same port,
> so the only way I see this being applicable is if a load balancer/reverse
> proxy in front of NiFi + other services attempted to identify and route
> incoming traffic based on SNI.
>
> I tried to craft a realistic scenario for this email but I couldn’t get to
> a point where it made sense. If you have a specific desired scenario, I can
> try to analyze it, but the entire concept of having multiple NiFi services
> or NiFi + other services be exposed on the same port and use SNI to
> differentiate seems unnecessary to me.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 22, 2020, at 12:05 PM, Pat White  wrote:
>
> Hi Andy,
> Thanks very much for the feedback, and my apologies for being vague. I
> have not used SNI so i have some learning to do.
>
> Specific use case we were asked about relates with Nifi to Nifi transfers,
> so not the webservice itself but rather S2S.
> I was wondering if S2S protocol supports SNI, and if so some pointers on
> how to configure that.
>
> patw
>
> On Fri, May 22, 2020 at 1:14 PM Andy LoPresto 
> wrote:
>
>> Hi Pat,
>>
>> Are you asking if NiFi’s internal web server supports SNI or if NiFi
>> processors/framework connecting to external services can resolve SNI? Maybe
>> some more context around your question would help us answer.
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On May 22, 2020, at 9:19 AM, Pat White  wrote:
>>
>> Hi Folks,
>>
>> Has anyone tried using SNI routing with Nifi?
>>
>> I believe Jetty supports the TLS extension for SNI but have not tried
>> using it, would appreciate any feedback if someone has tried this. Thank
>> you.
>>
>>
>>
>>
>