Re: NiFi Registry - DatabaseFlowPersistenceProvider and GitFlowPersistenceProvider simultaniously

[Pierre Villard]

typo: so you can't run the DB and Git providers simultaneously.

Le mar. 16 janv. 2024 à 18:08, Bryan Bende  a écrit :

> Hello,
>
> You can have only one persistence provider defined, so you can run the
> DB and Git providers simultaneously.
>
> Thanks,
>
> Bryan
>
> On Tue, Jan 16, 2024 at 6:52 AM Roman Wesołowski
>  wrote:
> >
> > Hello,
> >
> > We would like to have 3 nodes cluster with Nfii Registry one for each
> environment. So for this, we would like to have one external RDS Postgresql
> for all Registries to make deployments automatically.
> >
> > I am just wondering whether it is possible and working correctly if I
> would like to configure the Nifi Registry to use
> GitFlowPersistenceProvider as well to have all changes visible in
> bitbucket/jira - let's say I will call commit with JIRA-tickets numbers and
> then we can see some changes directly in jira/bitbucket. I suppose in that
> case I will duplicate all changes in DB/git. But what can happen during the
> deployment so it will still work correctly?
> > Maybe am I missing something?
> > Thanks for any advice.
> >
> > Roman
> >
>

Re: NiFi Registry - DatabaseFlowPersistenceProvider and GitFlowPersistenceProvider simultaniously

[Bryan Bende]

Hello,

You can have only one persistence provider defined, so you can run the
DB and Git providers simultaneously.

Thanks,

Bryan

On Tue, Jan 16, 2024 at 6:52 AM Roman Wesołowski
 wrote:
>
> Hello,
>
> We would like to have 3 nodes cluster with Nfii Registry one for each 
> environment. So for this, we would like to have one external RDS Postgresql 
> for all Registries to make deployments automatically.
>
> I am just wondering whether it is possible and working correctly if I would 
> like to configure the Nifi Registry to use  GitFlowPersistenceProvider as 
> well to have all changes visible in bitbucket/jira - let's say I will call 
> commit with JIRA-tickets numbers and then we can see some changes directly in 
> jira/bitbucket. I suppose in that case I will duplicate all changes in 
> DB/git. But what can happen during the deployment so it will still work 
> correctly?
> Maybe am I missing something?
> Thanks for any advice.
>
> Roman
>

Re: Nifi - Content-repo on AWS-EBS volumes

[Phillip Lord]

I just switched a cluster using 3 EBS volumes for cont-repo from gp2 to gp3… 
resolved definite I/O throughput issues.  The change to gp3 was significant 
enough that I might actually reduce from 3 to 2 volumes, perhaps even a single 
volume would be sufficient.

Of course every use case is unique.
On Dec 15, 2023 at 5:37 PM -0500, Gregory M. Foreman 
, wrote:
> Mark:
>
> Got it. Thank you for the help.
>
> Greg
>
> > On Dec 15, 2023, at 4:14 PM, Mark Payne  wrote:
> >
> > Greg,
> >
> > Whether or not multiple content repos will have any impact depends very 
> > much on where your system’s bottleneck is. If your bottleneck is disk I/O, 
> > it will absolutely help. If your bottleneck is CPU, it won’t. If, for 
> > example, you’re running on bare metal and have 48 cores on your machine and 
> > you’re running with spinning disks, you’ll definitely want to use multiple 
> > spinning disks. But if you’re running in AWS on a VM that has 4 cores and 
> > you’re using gp3 EBS volumes, it’s unlikely that multiple content repos 
> > will help.
> >
> > Thanks
> > -Mark
> >
> >
> >
> > > On Dec 15, 2023, at 3:25 PM, Gregory M. Foreman 
> > >  wrote:
> > >
> > > Mark:
> > >
> > > I was just discussing multiple content repos on EBS volumes with a 
> > > colleague. I found your post from a long time ago:
> > >
> > > https://lists.apache.org/thread/nq3mpry0wppzrodmldrcfnxwzp3n1cjv
> > >
> > > “Re #2: I don't know that i've used any SAN to back my repositories other 
> > > than the EBS provided by Amazon EC2. In that environment, I found that 
> > > having one or having multiple repos was essentially equivalent.”
> > >
> > > Does that statement still hold true today? Essentially there is no real 
> > > performance benefit to having multiple content repos on multiple EBS 
> > > volumes?
> > >
> > > Thanks,
> > > Greg
> > >
> > >
> > >
> > > > On Dec 11, 2023, at 8:50 PM, Mark Payne  wrote:
> > > >
> > > > Hey Phil,
> > > >
> > > > NiFi will not spread the content of a single file over multiple 
> > > > partitions. It will write the content of FlowFile 1 to content repo 1, 
> > > > then write the next FlowFile to repo 2, etc. so it does round-robin but 
> > > > does not spread a single FlowFile across multiple repos.
> > > >
> > > > Thanks
> > > > -Mark
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Dec 11, 2023, at 8:45 PM, Phillip Lord  
> > > > > wrote:
> > > > >
> > > > >
> > > > > Hello Nifi comrades,
> > > > >
> > > > > Here's my scenario...
> > > > > Let's say I have a Nifi cluster running on EC2 instances with 
> > > > > attached EBS volumes serving as their repos. They've split up their 
> > > > > content-repos into three content-repos per node(cont1, cont2, cont3). 
> > > > > Each being a dedicated EBS volume. My understanding is that the 
> > > > > content-claims for a single file can potentially span across more 
> > > > > than one of these repos.(correct me if I've lost my mind over the 
> > > > > years)
> > > > > For instance if you have a 1 MB file, and lets say your 
> > > > > max.content.claim.size is 100KB, that's 10 - 100KB claims(ish) 
> > > > > potentially split up across the 3 EBS volumes. So if Nifi is trying 
> > > > > to move that file to S3 or something for instance... it needs to be 
> > > > > read from each of the volumes.
> > > > > Whereas if it was a single EBS volume for the cont-repo... it would 
> > > > > read from the single volume, which I would think would be more 
> > > > > performant? Or does spreading out any IO contention across volumes 
> > > > > provide more of a benefit?
> > > > > I know there's different levels of EBS volumes... but not factoring 
> > > > > that in for right now.
> > > > >
> > > > > Appreciate any insight... trying to determine the best configuration.
> > > > >
> > > > > Thanks,
> > > > > Phil
> > > > >
> > > > >
> > >
> >
>

Re: Nifi - Content-repo on AWS-EBS volumes

[Gregory M. Foreman]

Mark:

Got it.  Thank you for the help.

Greg

> On Dec 15, 2023, at 4:14 PM, Mark Payne  wrote:
> 
> Greg,
> 
> Whether or not multiple content repos will have any impact depends very much 
> on where your system’s bottleneck is. If your bottleneck is disk I/O, it will 
> absolutely help. If your bottleneck is CPU, it won’t. If, for example, you’re 
> running on bare metal and have 48 cores on your machine and you’re running 
> with spinning disks, you’ll definitely want to use multiple spinning disks. 
> But if you’re running in AWS on a VM that has 4 cores and you’re using gp3 
> EBS volumes, it’s unlikely that multiple content repos will help.
> 
> Thanks
> -Mark
> 
> 
> 
>> On Dec 15, 2023, at 3:25 PM, Gregory M. Foreman 
>>  wrote:
>> 
>> Mark:
>> 
>> I was just discussing multiple content repos on EBS volumes with a 
>> colleague.  I found your post from a long time ago:
>> 
>> https://lists.apache.org/thread/nq3mpry0wppzrodmldrcfnxwzp3n1cjv
>> 
>> “Re #2: I don't know that i've used any SAN to back my repositories other 
>> than the EBS provided by Amazon EC2. In that environment, I found that 
>> having one or having multiple repos was essentially equivalent.”
>> 
>> Does that statement still hold true today?  Essentially there is no real 
>> performance benefit to having multiple content repos on multiple EBS volumes?
>> 
>> Thanks,
>> Greg
>> 
>> 
>> 
>>> On Dec 11, 2023, at 8:50 PM, Mark Payne  wrote:
>>> 
>>> Hey Phil,
>>> 
>>> NiFi will not spread the content of a single file over multiple partitions. 
>>> It will write the content of FlowFile 1 to content repo 1, then write the 
>>> next FlowFile to repo 2, etc. so it does round-robin but does not spread a 
>>> single FlowFile across multiple repos.
>>> 
>>> Thanks
>>> -Mark
>>> 
>>> Sent from my iPhone
>>> 
 On Dec 11, 2023, at 8:45 PM, Phillip Lord  wrote:
 
 
 Hello Nifi comrades,
 
 Here's my scenario...
 Let's say I have a Nifi cluster running on EC2 instances with attached EBS 
 volumes serving as their repos.  They've split up their content-repos into 
 three content-repos per node(cont1, cont2, cont3).  Each being a dedicated 
 EBS volume.  My understanding is that the content-claims for a single file 
 can potentially span across more than one of these repos.(correct me if 
 I've lost my mind over the years)
 For instance if you have a 1 MB file, and lets say your 
 max.content.claim.size is 100KB, that's 10 - 100KB claims(ish) potentially 
 split up across the 3 EBS volumes.  So if Nifi is trying to move that file 
 to S3 or something for instance... it needs to be read from each of the 
 volumes.  
 Whereas if it was a single EBS volume for the cont-repo... it would read 
 from the single volume, which I would think would be more performant?  Or 
 does spreading out any IO contention across volumes provide more of a 
 benefit?
 I know there's different levels of EBS volumes... but not factoring that 
 in for right now.
 
 Appreciate any insight... trying to determine the best configuration.  
 
 Thanks,
 Phil
 
 
>> 
> 

Re: Nifi - Content-repo on AWS-EBS volumes

[Mark Payne]

Greg,

Whether or not multiple content repos will have any impact depends very much on 
where your system’s bottleneck is. If your bottleneck is disk I/O, it will 
absolutely help. If your bottleneck is CPU, it won’t. If, for example, you’re 
running on bare metal and have 48 cores on your machine and you’re running with 
spinning disks, you’ll definitely want to use multiple spinning disks. But if 
you’re running in AWS on a VM that has 4 cores and you’re using gp3 EBS 
volumes, it’s unlikely that multiple content repos will help.

Thanks
-Mark



> On Dec 15, 2023, at 3:25 PM, Gregory M. Foreman 
>  wrote:
> 
> Mark:
> 
> I was just discussing multiple content repos on EBS volumes with a colleague. 
>  I found your post from a long time ago:
> 
> https://lists.apache.org/thread/nq3mpry0wppzrodmldrcfnxwzp3n1cjv
> 
> “Re #2: I don't know that i've used any SAN to back my repositories other 
> than the EBS provided by Amazon EC2. In that environment, I found that having 
> one or having multiple repos was essentially equivalent.”
> 
> Does that statement still hold true today?  Essentially there is no real 
> performance benefit to having multiple content repos on multiple EBS volumes?
> 
> Thanks,
> Greg
> 
> 
> 
>> On Dec 11, 2023, at 8:50 PM, Mark Payne  wrote:
>> 
>> Hey Phil,
>> 
>> NiFi will not spread the content of a single file over multiple partitions. 
>> It will write the content of FlowFile 1 to content repo 1, then write the 
>> next FlowFile to repo 2, etc. so it does round-robin but does not spread a 
>> single FlowFile across multiple repos.
>> 
>> Thanks
>> -Mark
>> 
>> Sent from my iPhone
>> 
>>> On Dec 11, 2023, at 8:45 PM, Phillip Lord  wrote:
>>> 
>>> 
>>> Hello Nifi comrades,
>>> 
>>> Here's my scenario...
>>> Let's say I have a Nifi cluster running on EC2 instances with attached EBS 
>>> volumes serving as their repos.  They've split up their content-repos into 
>>> three content-repos per node(cont1, cont2, cont3).  Each being a dedicated 
>>> EBS volume.  My understanding is that the content-claims for a single file 
>>> can potentially span across more than one of these repos.(correct me if 
>>> I've lost my mind over the years)
>>> For instance if you have a 1 MB file, and lets say your 
>>> max.content.claim.size is 100KB, that's 10 - 100KB claims(ish) potentially 
>>> split up across the 3 EBS volumes.  So if Nifi is trying to move that file 
>>> to S3 or something for instance... it needs to be read from each of the 
>>> volumes.  
>>> Whereas if it was a single EBS volume for the cont-repo... it would read 
>>> from the single volume, which I would think would be more performant?  Or 
>>> does spreading out any IO contention across volumes provide more of a 
>>> benefit?
>>> I know there's different levels of EBS volumes... but not factoring that in 
>>> for right now.
>>> 
>>> Appreciate any insight... trying to determine the best configuration.  
>>> 
>>> Thanks,
>>> Phil
>>> 
>>> 
> 

Re: Nifi - Content-repo on AWS-EBS volumes

[Gregory M. Foreman]

Mark:

I was just discussing multiple content repos on EBS volumes with a colleague.  
I found your post from a long time ago:

https://lists.apache.org/thread/nq3mpry0wppzrodmldrcfnxwzp3n1cjv

“Re #2: I don't know that i've used any SAN to back my repositories other than 
the EBS provided by Amazon EC2. In that environment, I found that having one or 
having multiple repos was essentially equivalent.”

Does that statement still hold true today?  Essentially there is no real 
performance benefit to having multiple content repos on multiple EBS volumes?

Thanks,
Greg



> On Dec 11, 2023, at 8:50 PM, Mark Payne  wrote:
> 
> Hey Phil,
> 
> NiFi will not spread the content of a single file over multiple partitions. 
> It will write the content of FlowFile 1 to content repo 1, then write the 
> next FlowFile to repo 2, etc. so it does round-robin but does not spread a 
> single FlowFile across multiple repos.
> 
> Thanks
> -Mark
> 
> Sent from my iPhone
> 
>> On Dec 11, 2023, at 8:45 PM, Phillip Lord  wrote:
>> 
>> 
>> Hello Nifi comrades,
>> 
>> Here's my scenario...
>> Let's say I have a Nifi cluster running on EC2 instances with attached EBS 
>> volumes serving as their repos.  They've split up their content-repos into 
>> three content-repos per node(cont1, cont2, cont3).  Each being a dedicated 
>> EBS volume.  My understanding is that the content-claims for a single file 
>> can potentially span across more than one of these repos.(correct me if I've 
>> lost my mind over the years)
>> For instance if you have a 1 MB file, and lets say your 
>> max.content.claim.size is 100KB, that's 10 - 100KB claims(ish) potentially 
>> split up across the 3 EBS volumes.  So if Nifi is trying to move that file 
>> to S3 or something for instance... it needs to be read from each of the 
>> volumes.  
>> Whereas if it was a single EBS volume for the cont-repo... it would read 
>> from the single volume, which I would think would be more performant?  Or 
>> does spreading out any IO contention across volumes provide more of a 
>> benefit?
>> I know there's different levels of EBS volumes... but not factoring that in 
>> for right now.
>> 
>> Appreciate any insight... trying to determine the best configuration.  
>> 
>> Thanks,
>> Phil
>> 
>> 

RE: [NIFI 1.23.2] Insecure Cipher Provider Algorithm

[Isha Lamboo]

Hi Quentin,

I've encountered similar errors in the past when trying to change the 
encryption algorithm.

Here are two things that may help:


  1.  The password/key needs to be at least 12 characters long before you 
migrate to NIFI_PBKDF2_AES_GCM_256. If it is not, you have to first change the 
password to something long enough with the old algorithm in place. If your key 
is blank, you may have to enter the old default value first: nififtw!
  2.  The command to migrate key algorithm does not support an encrypted 
configuration file. If you have the key encrypted you should replace it with 
the unencrypted version, clear the property  ...sensitivekey.protected=... and 
then migrate. After that you can re-encrypt the configuration using the nifi 
toolkit again.

Regards,

Isha

Van: Quentin HORNEMAN GUTTON 
Verzonden: woensdag 13 december 2023 14:59
Aan: users@nifi.apache.org
Onderwerp: [NIFI 1.23.2] Insecure Cipher Provider Algorithm

You don't often get email from 
qhornemangut...@gmail.com. Learn why this is 
important
Hello,

I'm facing an issue after upgrading NiFi 1.13.2 to 1.23.2.

I have a warn log with Insecure Cipher Provider Algorithm 
[PBEWITHMD5AND256BITAES-CBC-OPENSSL]. I tried to update algorithm with the 
set-sensitive-properties-algorithm command to NIFI_PBKDF2_AES_GCM_256 but I 
have an error message with < Descryption failed with algorithm > caused by < 
pad block corrupted >.

Do you have any informations that could help me ?

Best regards,

Quentin HORNEMAN GUTTON

Re: [NIFI 1.23.2] Insecure Cipher Provider Algorithm

[Christian Wahl]

Hi,

I had the same issue changing the password and the cipher.

It worked for me using the NiFi Toolkit and applying the operation onto both 
the flow.json.gz and the flow.xml.gz
The documentation for the encrypt-config command is here:
https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#encrypt_config_tool

At the end I ran it using docker with a command like this:
docker run -v .:/conf apache/nifi-toolkit:1.21.0 encrypt-config -x -f 
/conf/flow.json.gz -n /conf/nifi.properties -o /conf/nifi.properties.new -s 
secret -A NIFI_ARGON2_AES_GCM_256 

Kind regards,
Christian Wahl



> On 13. Dec 2023, at 15:18, Lars Winderling  wrote:
> 
> Hi Quentin,
> 
> I second these findings. I'm getting the same error on 1.23.2 using the same 
> ciphers.
>   deb: 11
>   java: 17.0.7 Temurin
> 
> Best,
> Lars
> 
> On 23-12-13 14:58, Quentin HORNEMAN GUTTON wrote:
>> Hello,
>> 
>> I’m facing an issue after upgrading NiFi 1.13.2 to 1.23.2.
>> 
>> I have a warn log with Insecure Cipher Provider Algorithm 
>> [PBEWITHMD5AND256BITAES-CBC-OPENSSL]. I tried to update algorithm with the 
>> set-sensitive-properties-algorithm command to NIFI_PBKDF2_AES_GCM_256 but I 
>> have an error message with « Descryption failed with algorithm » caused by « 
>> pad block corrupted ».
>> 
>> Do you have any informations that could help me ?
>> 
>> Best regards,
>> 
>> Quentin HORNEMAN GUTTON
> 

Re: [NIFI 1.23.2] Insecure Cipher Provider Algorithm

[Lars Winderling]

Hi Quentin,

I second these findings. I'm getting the same error on 1.23.2 using the 
same ciphers.

  deb: 11
  java: 17.0.7 Temurin

Best,
Lars

On 23-12-13 14:58, Quentin HORNEMAN GUTTON wrote:

Hello,

I’m facing an issue after upgrading NiFi 1.13.2 to 1.23.2.

I have a warn log with Insecure Cipher Provider Algorithm 
[PBEWITHMD5AND256BITAES-CBC-OPENSSL]. I tried to update algorithm with 
the set-sensitive-properties-algorithm command to 
NIFI_PBKDF2_AES_GCM_256 but I have an error message with « Descryption 
failed with algorithm » caused by « pad block corrupted ».


Do you have any informations that could help me ?

Best regards,

Quentin HORNEMAN GUTTON




OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Nifi - Content-repo on AWS-EBS volumes

[Mark Payne]

Hey Phil,

NiFi will not spread the content of a single file over multiple partitions. It 
will write the content of FlowFile 1 to content repo 1, then write the next 
FlowFile to repo 2, etc. so it does round-robin but does not spread a single 
FlowFile across multiple repos.

Thanks
-Mark

Sent from my iPhone

> On Dec 11, 2023, at 8:45 PM, Phillip Lord  wrote:
> 
> 
> Hello Nifi comrades,
> 
> Here's my scenario...
> Let's say I have a Nifi cluster running on EC2 instances with attached EBS 
> volumes serving as their repos.  They've split up their content-repos into 
> three content-repos per node(cont1, cont2, cont3).  Each being a dedicated 
> EBS volume.  My understanding is that the content-claims for a single file 
> can potentially span across more than one of these repos.(correct me if I've 
> lost my mind over the years)
> For instance if you have a 1 MB file, and lets say your 
> max.content.claim.size is 100KB, that's 10 - 100KB claims(ish) potentially 
> split up across the 3 EBS volumes.  So if Nifi is trying to move that file to 
> S3 or something for instance... it needs to be read from each of the volumes. 
>  
> Whereas if it was a single EBS volume for the cont-repo... it would read from 
> the single volume, which I would think would be more performant?  Or does 
> spreading out any IO contention across volumes provide more of a benefit?
> I know there's different levels of EBS volumes... but not factoring that in 
> for right now.
> 
> Appreciate any insight... trying to determine the best configuration.  
> 
> Thanks,
> Phil
> 
> 

Re: NiFi custom extension and deploy phase on Maven

[Etienne Jouvin]

Exactly.

The configuration on my pom.



org.apache.nifi

nifi-nar-bundles

1.23.2




Offering one without all ASF-isms, it can be nice also..

Thanks

Le sam. 9 déc. 2023 à 19:03, Joe Witt  a écrit :

> Etienne,
>
> Are you using our nifi poms as your parent pom of your extensions?  I do
> wonder if perhaps we should offer one that doesn't tie to all our ASF-isms
>
> Thanks
>
> On Sat, Dec 9, 2023 at 10:54 AM Etienne Jouvin 
> wrote:
>
>> Hello all;
>>
>> Since now, I never push my extensions to any Nexus or Artifactory.
>> But now I must do it.
>>
>> The deploy Maven deploy is in failure and I notice it comes from the
>> configuration for the plugin nexus-staging-maven-plugin.
>>
>> The configuration comes from the artifact nifi, where we can find the
>> following configuration.
>>
>> 
>>
>> org.sonatype.plugins
>>
>> nexus-staging-maven-plugin
>>
>> 1.6.13
>>
>> true
>>
>> 
>>
>> 15
>>
>> repository.apache.org
>>
>> https://repository.apache.org/
>>
>> 
>>
>> 
>>
>>
>> Two things :
>> I did not have any issue when I wanted to push my extension in SNAPSHOT
>> version, except I had to configure the enforcer to not fire exception
>>
>> And to be able to push my extensions in my custom Artifactory, I had to
>> disable it by the following configuration, in the build node at my root
>> pom.xml file :
>>
>> 
>>
>> 
>>
>> org.sonatype.plugins
>>
>> nexus-staging-maven-plugin
>>
>> 
>>
>> true
>>
>> 
>>
>> 
>>
>>
>> What do you think if you put your build configuration inside a profile ?
>> Like this, we will not have to add a custom configuration.
>>
>> Regards
>>
>> Etienne Jouvin
>>
>>

Re: NiFi custom extension and deploy phase on Maven

[Joe Witt]

Etienne,

Are you using our nifi poms as your parent pom of your extensions?  I do
wonder if perhaps we should offer one that doesn't tie to all our ASF-isms

Thanks

On Sat, Dec 9, 2023 at 10:54 AM Etienne Jouvin 
wrote:

> Hello all;
>
> Since now, I never push my extensions to any Nexus or Artifactory.
> But now I must do it.
>
> The deploy Maven deploy is in failure and I notice it comes from the
> configuration for the plugin nexus-staging-maven-plugin.
>
> The configuration comes from the artifact nifi, where we can find the
> following configuration.
>
> 
>
> org.sonatype.plugins
>
> nexus-staging-maven-plugin
>
> 1.6.13
>
> true
>
> 
>
> 15
>
> repository.apache.org
>
> https://repository.apache.org/
>
> 
>
> 
>
>
> Two things :
> I did not have any issue when I wanted to push my extension in SNAPSHOT
> version, except I had to configure the enforcer to not fire exception
>
> And to be able to push my extensions in my custom Artifactory, I had to
> disable it by the following configuration, in the build node at my root
> pom.xml file :
>
> 
>
> 
>
> org.sonatype.plugins
>
> nexus-staging-maven-plugin
>
> 
>
> true
>
> 
>
> 
>
>
> What do you think if you put your build configuration inside a profile ?
> Like this, we will not have to add a custom configuration.
>
> Regards
>
> Etienne Jouvin
>
>

Re: NIFI-REGISTRY Migration

[Dennis Suhari]

Hi,doesn’t  fit your answer but maybe it is an aspect:1) connect your old Nifi registry to a git repo (e.g. use GitHub, GitLab).2) Push your processor groups, processors to old Nifi registry and by this it is automatically pushed to Git Repo.3) Setup new Nifi registry and connect it to Git provider and to pushed repo.4) After that you can pull your old code by using processor groups -> import from external repo within your Nifi Br,DennisVon meinem iPhone gesendetAm 29.09.2023 um 13:30 schrieb e-soci...@gmx.fr:Hello Pierre,

 

Thanks for new option.

 

So it means, the dataflow_1 register in the registry_A, will be register in the same way in registry_B after migration ?

Or needs we also use the URL aliasing ?

 

Regards 


 

Envoyé: vendredi 29 septembre 2023 à 12:41
De: "Pierre Villard" 
À: users@nifi.apache.org
Objet: Re: NIFI-REGISTRY Migration


Hi,
the 

You could also consider using the recently added Toolkit CLI optihons to export all / import all from Registry A to B.

 

Pierre

 


Le ven. 29 sept. 2023 à 12:13, Lars Winderling <lars.winderl...@posteo.de> a écrit :


Hi Minh,

you could try to employ URL aliasing as explained in the docs. This way, the registry will store only an alias instead of the actual registry address. When migrating to another host, make sure to assign the same alias to it, and it should be able to retrieve versioning information properly.
Another option would be to use a persistent host name (same A/ entries, or a CNAME).
This way, I have migrated our registry already a few times.

I hope this helps.
Best,
Lars
 
 

On 23-09-29 10:44, e-soci...@gmx.fr wrote:



 

Hello all,

 

Someone has already migrate their nifi-registry from hostA to hostB (nifi-registry) and keep track the versionning for the nifi client points to the hostB ?

 

Have we got some documentations about migration nifi-registry ?

 

regards 

 

Minh 










 

 

Re: NIFI-REGISTRY Migration

[e-sociaux]

Hello Pierre,

 

Thanks for new option.

 

So it means, the dataflow_1 register in the registry_A, will be register in the same way in registry_B after migration ?

Or needs we also use the URL aliasing ?

 

Regards 


 

Envoyé: vendredi 29 septembre 2023 à 12:41
De: "Pierre Villard" 
À: users@nifi.apache.org
Objet: Re: NIFI-REGISTRY Migration


Hi,
the 

You could also consider using the recently added Toolkit CLI optihons to export all / import all from Registry A to B.

 

Pierre

 


Le ven. 29 sept. 2023 à 12:13, Lars Winderling <lars.winderl...@posteo.de> a écrit :


Hi Minh,

you could try to employ URL aliasing as explained in the docs. This way, the registry will store only an alias instead of the actual registry address. When migrating to another host, make sure to assign the same alias to it, and it should be able to retrieve versioning information properly.
Another option would be to use a persistent host name (same A/ entries, or a CNAME).
This way, I have migrated our registry already a few times.

I hope this helps.
Best,
Lars
 
 

On 23-09-29 10:44, e-soci...@gmx.fr wrote:



 

Hello all,

 

Someone has already migrate their nifi-registry from hostA to hostB (nifi-registry) and keep track the versionning for the nifi client points to the hostB ?

 

Have we got some documentations about migration nifi-registry ?

 

regards 

 

Minh 










 

 

Re: NIFI-REGISTRY Migration

[Pierre Villard]

Hi,

You could also consider using the recently added Toolkit CLI options to
export all / import all from Registry A to B.

Pierre

Le ven. 29 sept. 2023 à 12:13, Lars Winderling 
a écrit :

> Hi Minh,
>
> you could try to employ URL aliasing
> 
> as explained in the docs. This way, the registry will store only an alias
> instead of the actual registry address. When migrating to another host,
> make sure to assign the same alias to it, and it should be able to retrieve
> versioning information properly.
> Another option would be to use a persistent host name (same A/
> entries, or a CNAME).
> This way, I have migrated our registry already a few times.
>
> I hope this helps.
> Best,
> Lars
>
> On 23-09-29 10:44, e-soci...@gmx.fr wrote:
>
>
> Hello all,
>
> Someone has already migrate their nifi-registry from hostA to hostB
> (nifi-registry) and keep track the versionning for the nifi client points
> to the hostB ?
>
> Have we got some documentations about migration nifi-registry ?
>
> regards
>
> Minh
>
>
>

Re: NIFI-REGISTRY Migration

[Lars Winderling]

Hi Minh,

you could try to employ URL aliasing 
 
as explained in the docs. This way, the registry will store only an 
alias instead of the actual registry address. When migrating to another 
host, make sure to assign the same alias to it, and it should be able to 
retrieve versioning information properly.
Another option would be to use a persistent host name (same A/ 
entries, or a CNAME).

This way, I have migrated our registry already a few times.

I hope this helps.
Best,
Lars

On 23-09-29 10:44, e-soci...@gmx.fr wrote:

Hello all,
Someone has already migrate their nifi-registry from hostA to hostB 
(nifi-registry) and keep track the versionning for the nifi client 
points to the hostB ?

Have we got some documentations about migration nifi-registry ?
regards
Minh




OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: NiFi hanging during large sql query

[Mike Thomsen]

The resolution was to manually paginate the query statement with the
appropriate postgres syntax and set the fetchSize to 250 instead of 0,
which is unlimited.

On Mon, Sep 11, 2023 at 8:54 AM  wrote:

>
> Hello Mike,
>
> Could you please give me the details about the resolution ?
> Have you change something in the processor or just changing the sql
> command ?
>
> Regards
>
> *Envoyé:* samedi 2 septembre 2023 à 00:00
> *De:* "Mike Thomsen" 
> *À:* users@nifi.apache.org
> *Objet:* NiFi hanging during large sql query
> I have a three node cluster with an executesqlrecord processor with
> primary execution only. The sql it runs is a straight forward select on a
> table with about 44m records. If I leave it running, after about 10 min the
> node becomes unresponsive and leaves the cluster. The query runs just fine
> in jetbrains data grip on that postgresql server, so I don’t think it’s
> anything weird with the db or query. Any ideas about what could be causing
> this? Even with a high limit like 5m records the query doesn’t lock up the
> NiFi node.
>
> Sent from my iPhone
>
>
>

Re: NiFi hanging during large sql query

[e-sociaux]

 



Hello Mike,

 

Could you please give me the details about the resolution ?

Have you change something in the processor or just changing the sql command ?

 

Regards


 

Envoyé: samedi 2 septembre 2023 à 00:00
De: "Mike Thomsen" 
À: users@nifi.apache.org
Objet: NiFi hanging during large sql query

I have a three node cluster with an executesqlrecord processor with primary execution only. The sql it runs is a straight forward select on a table with about 44m records. If I leave it running, after about 10 min the node becomes unresponsive and leaves the cluster. The query runs just fine in jetbrains data grip on that postgresql server, so I don’t think it’s anything weird with the db or query. Any ideas about what could be causing this? Even with a high limit like 5m records the query doesn’t lock up the NiFi node.

Sent from my iPhone




 

 

Re: [NIFI] ExecuteScript

[e-sociaux]

Hello Quentin,

 

I also used the python with ExecuteScript but it is not native and not really supported.

 

The better thing it is use executeGroovy processor.

 

Minh

 
 

Envoyé: vendredi 8 septembre 2023 à 22:55
De: "Rafael Fracasso" 
À: users@nifi.apache.org
Objet: Re: [NIFI] ExecuteScript


As far as I know, you cannot use any external library on executescript, only the native one on jython env.
 

But you can call native python to execute your script using executestreamcommand

 


On Fri, Sep 8, 2023 at 6:15 AM Quentin HORNEMAN GUTTON <qhornemangut...@gmail.com> wrote:


Hello everyone,
 

I would like to know if it is possible to use the Etree or ElementPath library in an ExecuteScript processor with the "Python" script engine (which is Jython), or if it is possible to install modules for the Jython environment ?

 

I'm using NiFi 1.20.0

 

Best regards,








 

 

Re: NiFi OpenTelemetry JavaAgent

[Eric Secules]

Hi Joe,

Thanks for the fast response, I will circle back to this when I can
(hopefully a week or 2) and update with what the root cause was if I can
find it. There was no smoking gun in the app log as far as I can tell. In
the meantime if anyone has success with this on their own I'd appreciate it
if they shared their experience.

Thanks,
Eric

On Fri, Sep 8, 2023 at 4:22 PM Joe Witt  wrote:

> Eric
>
> Any sign anything in the nifi app log?  Did it perhaps run out of mem and
> hang?
>
> In general such agents should be fine.
>
> Thanks
>
> On Fri, Sep 8, 2023 at 6:15 PM Eric Secules  wrote:
>
>> Hello,
>>
>> I have tried adding opentelemetry to NiFi by running the OpenTelemetry
>> javaagent (
>> https://opentelemetry.io/docs/instrumentation/java/getting-started/#instrumentation)
>> with NiFi and it mostly works. I am able to see spans created by jetty,
>> reddison, etc. However NiFi also dies and gets restarted by the bootstrap
>> program:
>>
>> 2023-09-08 15:04:52,560 WARN [main] org.apache.nifi.bootstrap.RunNiFi
>> Apache NiFi appears to have died. Restarting...
>>
>> I can't seem to find any stack traces or anything to indicate why it fell
>> over in any of the logs. In principle is there anything wrong with running
>> the OpenTelemetry java agent with NiFi? Has anyone successfully done what I
>> am trying before?
>>
>> Thanks,
>> Eric
>>
>

Re: NiFi OpenTelemetry JavaAgent

[Joe Witt]

Eric

Any sign anything in the nifi app log?  Did it perhaps run out of mem and
hang?

In general such agents should be fine.

Thanks

On Fri, Sep 8, 2023 at 6:15 PM Eric Secules  wrote:

> Hello,
>
> I have tried adding opentelemetry to NiFi by running the OpenTelemetry
> javaagent (
> https://opentelemetry.io/docs/instrumentation/java/getting-started/#instrumentation)
> with NiFi and it mostly works. I am able to see spans created by jetty,
> reddison, etc. However NiFi also dies and gets restarted by the bootstrap
> program:
>
> 2023-09-08 15:04:52,560 WARN [main] org.apache.nifi.bootstrap.RunNiFi
> Apache NiFi appears to have died. Restarting...
>
> I can't seem to find any stack traces or anything to indicate why it fell
> over in any of the logs. In principle is there anything wrong with running
> the OpenTelemetry java agent with NiFi? Has anyone successfully done what I
> am trying before?
>
> Thanks,
> Eric
>

Re: [NIFI] ExecuteScript

[Joe Witt]

NiFi 2.0 should get you much further with its Python processor support. Are
you able to build main or would you need help with that?

We could put a dev build somewhere if needed.

Thanks

On Fri, Sep 8, 2023 at 4:28 PM Jeremy Dyer  wrote:

> Rafael is right. This isn’t reasonably possible today. You could hack
> around and make it work but not something I would recommend. I had made an
> Anaconda/VirtualEnv controller service for some python AI stuff we are
> doing at Nvidia like 3 years ago but I’m sure that isn’t compatible with
> the projects current state anymore.
>
> Ultimately you aren’t gonna be able to do this correctly with jython in
> the loop. Especially when you get into loading Python C native libraries,
> like CUDA for example. Not a fault of nifi or it’s design just too many
> edge cases that cannot fairly be expected for NiFi to address.
>
> -Jeremy Dyer
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
> --
> *From:* Rafael Fracasso 
> *Sent:* Friday, September 8, 2023 4:55:40 PM
> *To:* users@nifi.apache.org 
> *Subject:* Re: [NIFI] ExecuteScript
>
> As far as I know, you cannot use any external library on executescript,
> only the native one on jython env.
>
> But you can call native python to execute your script using
> executestreamcommand
>
> On Fri, Sep 8, 2023 at 6:15 AM Quentin HORNEMAN GUTTON <
> qhornemangut...@gmail.com> wrote:
>
> Hello everyone,
>
> I would like to know if it is possible to use the Etree or ElementPath
> library in an ExecuteScript processor with the "Python" script engine
> (which is Jython), or if it is possible to install modules for the Jython
> environment ?
>
> I'm using NiFi 1.20.0
>
> Best regards,
>
>

Re: [NIFI] ExecuteScript

[Jeremy Dyer]

Rafael is right. This isn’t reasonably possible today. You could hack around 
and make it work but not something I would recommend. I had made an 
Anaconda/VirtualEnv controller service for some python AI stuff we are doing at 
Nvidia like 3 years ago but I’m sure that isn’t compatible with the projects 
current state anymore.

Ultimately you aren’t gonna be able to do this correctly with jython in the 
loop. Especially when you get into loading Python C native libraries, like CUDA 
for example. Not a fault of nifi or it’s design just too many edge cases that 
cannot fairly be expected for NiFi to address.

-Jeremy Dyer

Get Outlook for iOS<https://aka.ms/o0ukef>

From: Rafael Fracasso 
Sent: Friday, September 8, 2023 4:55:40 PM
To: users@nifi.apache.org 
Subject: Re: [NIFI] ExecuteScript

As far as I know, you cannot use any external library on executescript, only 
the native one on jython env.

But you can call native python to execute your script using executestreamcommand

On Fri, Sep 8, 2023 at 6:15 AM Quentin HORNEMAN GUTTON 
mailto:qhornemangut...@gmail.com>> wrote:
Hello everyone,

I would like to know if it is possible to use the Etree or ElementPath library 
in an ExecuteScript processor with the "Python" script engine (which is 
Jython), or if it is possible to install modules for the Jython environment ?

I'm using NiFi 1.20.0

Best regards,

Re: [NIFI] ExecuteScript

[Rafael Fracasso]

As far as I know, you cannot use any external library on executescript,
only the native one on jython env.

But you can call native python to execute your script using
executestreamcommand

On Fri, Sep 8, 2023 at 6:15 AM Quentin HORNEMAN GUTTON <
qhornemangut...@gmail.com> wrote:

> Hello everyone,
>
> I would like to know if it is possible to use the Etree or ElementPath
> library in an ExecuteScript processor with the "Python" script engine
> (which is Jython), or if it is possible to install modules for the Jython
> environment ?
>
> I'm using NiFi 1.20.0
>
> Best regards,
>

Re: NiFi hanging during large sql query

[Mike Thomsen]

I don't think so.

On Sat, Sep 2, 2023 at 11:36 AM Mark Payne  wrote:

> Thanks for sharing the solution Mike. Is there something we need to update
> in nifi to prevent this from biting others?
>
> Thanks
> Mark
>
> Sent from my iPhone
>
> On Sep 2, 2023, at 9:48 AM, Joe Witt  wrote:
>
> 
> Nice.  Gald you found it.
>
> On Sat, Sep 2, 2023 at 5:07 AM Mike Thomsen 
> wrote:
>
>> It was the PostgreSQL JDBC driver. If you don't paginate the query
>> aggressively, it will try to load a significant chunk of the table into
>> memory rather than just pulling chunks, even with fetchSize set low.
>>
>> On Fri, Sep 1, 2023 at 6:01 PM Mike Thomsen 
>> wrote:
>>
>>> I have a three node cluster with an executesqlrecord processor with
>>> primary execution only. The sql it runs is a straight forward select on a
>>> table with about 44m records. If I leave it running, after about 10 min the
>>> node becomes unresponsive and leaves the cluster. The query runs just fine
>>> in jetbrains data grip on that postgresql server, so I don’t think it’s
>>> anything weird with the db or query. Any ideas about what could be causing
>>> this? Even with a high limit like 5m records the query doesn’t lock up the
>>> NiFi node.
>>>
>>> Sent from my iPhone
>>
>>

Re: NiFi hanging during large sql query

[Matt Burgess]

When you said "fetchSize set low", I assume you mean non-zero, a zero
will fetch all the rows at once. How did you paginate your query with
ExecuteSQLRecord? I was going to suggest GenerateTableFetch in front
to paginate the queries for you, but it definitely seems like we
should be able to do or configure something that will make PG happy.

Regards,
Matt

On Sat, Sep 2, 2023 at 11:36 AM Mark Payne  wrote:
>
> Thanks for sharing the solution Mike. Is there something we need to update in 
> nifi to prevent this from biting others?
>
> Thanks
> Mark
>
> Sent from my iPhone
>
> On Sep 2, 2023, at 9:48 AM, Joe Witt  wrote:
>
> 
> Nice.  Gald you found it.
>
> On Sat, Sep 2, 2023 at 5:07 AM Mike Thomsen  wrote:
>>
>> It was the PostgreSQL JDBC driver. If you don't paginate the query 
>> aggressively, it will try to load a significant chunk of the table into 
>> memory rather than just pulling chunks, even with fetchSize set low.
>>
>> On Fri, Sep 1, 2023 at 6:01 PM Mike Thomsen  wrote:
>>>
>>> I have a three node cluster with an executesqlrecord processor with primary 
>>> execution only. The sql it runs is a straight forward select on a table 
>>> with about 44m records. If I leave it running, after about 10 min the node 
>>> becomes unresponsive and leaves the cluster. The query runs just fine in 
>>> jetbrains data grip on that postgresql server, so I don’t think it’s 
>>> anything weird with the db or query. Any ideas about what could be causing 
>>> this? Even with a high limit like 5m records the query doesn’t lock up the 
>>> NiFi node.
>>>
>>> Sent from my iPhone

Re: NiFi hanging during large sql query

[Mark Payne]

Thanks for sharing the solution Mike. Is there something we need to update in 
nifi to prevent this from biting others?

Thanks
Mark

Sent from my iPhone

On Sep 2, 2023, at 9:48 AM, Joe Witt  wrote:


Nice.  Gald you found it.

On Sat, Sep 2, 2023 at 5:07 AM Mike Thomsen 
mailto:mikerthom...@gmail.com>> wrote:
It was the PostgreSQL JDBC driver. If you don't paginate the query 
aggressively, it will try to load a significant chunk of the table into memory 
rather than just pulling chunks, even with fetchSize set low.

On Fri, Sep 1, 2023 at 6:01 PM Mike Thomsen 
mailto:mikerthom...@gmail.com>> wrote:
I have a three node cluster with an executesqlrecord processor with primary 
execution only. The sql it runs is a straight forward select on a table with 
about 44m records. If I leave it running, after about 10 min the node becomes 
unresponsive and leaves the cluster. The query runs just fine in jetbrains data 
grip on that postgresql server, so I don’t think it’s anything weird with the 
db or query. Any ideas about what could be causing this? Even with a high limit 
like 5m records the query doesn’t lock up the NiFi node.

Sent from my iPhone

Re: NiFi hanging during large sql query

[Joe Witt]

Nice.  Gald you found it.

On Sat, Sep 2, 2023 at 5:07 AM Mike Thomsen  wrote:

> It was the PostgreSQL JDBC driver. If you don't paginate the query
> aggressively, it will try to load a significant chunk of the table into
> memory rather than just pulling chunks, even with fetchSize set low.
>
> On Fri, Sep 1, 2023 at 6:01 PM Mike Thomsen 
> wrote:
>
>> I have a three node cluster with an executesqlrecord processor with
>> primary execution only. The sql it runs is a straight forward select on a
>> table with about 44m records. If I leave it running, after about 10 min the
>> node becomes unresponsive and leaves the cluster. The query runs just fine
>> in jetbrains data grip on that postgresql server, so I don’t think it’s
>> anything weird with the db or query. Any ideas about what could be causing
>> this? Even with a high limit like 5m records the query doesn’t lock up the
>> NiFi node.
>>
>> Sent from my iPhone
>
>

Re: NiFi hanging during large sql query

[Mike Thomsen]

It was the PostgreSQL JDBC driver. If you don't paginate the query
aggressively, it will try to load a significant chunk of the table into
memory rather than just pulling chunks, even with fetchSize set low.

On Fri, Sep 1, 2023 at 6:01 PM Mike Thomsen  wrote:

> I have a three node cluster with an executesqlrecord processor with
> primary execution only. The sql it runs is a straight forward select on a
> table with about 44m records. If I leave it running, after about 10 min the
> node becomes unresponsive and leaves the cluster. The query runs just fine
> in jetbrains data grip on that postgresql server, so I don’t think it’s
> anything weird with the db or query. Any ideas about what could be causing
> this? Even with a high limit like 5m records the query doesn’t lock up the
> NiFi node.
>
> Sent from my iPhone

Re: NiFi-Registry GitFlowMetaData Error

[Bryan Bende]

Hello,

I believe this issue [1] was just resolved yesterday and will be part of
the upcoming 1.23.1 release that is being discussed.

It was an incompatibility with the git library used by registry and the
underlying ssh library used.

Thanks,

Bryan

[1] https://issues.apache.org/jira/browse/NIFI-11927

On Tue, Aug 15, 2023 at 10:59 AM Fries, Matthias 
wrote:

> Hi guys,
>
>
>
> I upgraded from 1.19.1 to 1.22.0 recently and am now facing issues
> regarding the NiFi-Registry.
>
>
>
> Local changes committed from NiFi are available in NiFi-Registry Flow
> Storage Directory afterwards but not in the underlying git repo.
>
> The connection between NiFi-Registry and GitLab is established via ssh
> key. This is the pre and post upgrade configuration of the Git Flow
> Persistence Provider:
>
>
>
>   
>
>
> org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider
>
>   {{
> pathToFlowStorageDirectory }}
>
>   origin
>
>   
>
>
>
> Running a git push command manually pushes the latest changes to the git
> repo without any issues.
>
>
>
> The nifi-registry-app.log is showing the following:
>
>
>
> 2023-08-15 14:37:30,959 ERROR [GitFlowMetaData Push thread]
> o.a.n.r.p.flow.git.GitFlowMetaData Failed to push commits to origin du
>
> e to org.eclipse.jgit.api.errors.TransportException: 
> ssh://git@git-server:port/pathToRepo/repo.git:
> remote hung up unexpectedly
>
>
>
> Any idea on how to solve this?
>
>
>
> Thanks in advance
>
> Matthias
>
>
>
> Das SVA Mail-System ist mit einem Mailverschluesselungs-Gateway
> ausgestattet. Wenn Sie moechten, dass an Sie gerichtete E-Mails
> verschluesselt werden, senden Sie einfach eine S/MIME-signierte E-Mail oder
> Ihren PGP Public Key an matthias.fr...@sva.de.
>
> The SVA mail system is equipped with an email encryption gateway. If you
> want email sent to you to be encrypted please send a S/MIME signed email or
> your PGP public key to matthias.fr...@sva.de.
>

Re: NiFi not rolling logs

[Lars Winderling]

David,
great recommendations, thanks!
Best, Lars

On 23-07-10 15:59, David Handermann wrote:
It is worth noting that the totalSizeCap can be added at any time to 
an existing NiFi deployment, it is not necessary to wait for an 
updated release.


Also related, the nifi-deprecation.log default setting includes a 100 
MB totalSizeCap to avoid excessive repetition of deprecation messages, 
and that has worked well for several releases.


Regards,
David Handermann

On Sun, Jul 9, 2023 at 3:26 PM Mike Thomsen  
wrote:


The totalcapsize feature will help a lot from what I’ve seen.

Sent from my iPhone


On Jul 8, 2023, at 9:26 AM, Lars Winderling
 wrote:


Hi Mike,

thanks for the advice. Our NiFi instances are running for week,
if not months. Often times until the next update, so the startup
option will bring much benefit, I fear, or am I mistaken. But
looking forward to 1.23!


On 8 July 2023 13:40:15 CEST, Mike Thomsen
 wrote:

Lars,

You should also experiment with cleanHistoryOnStart. I did
some experimentation this morning where I set the maxHistory
to 1 (1 day vs the default of 30 which is 30 days), created a
few fake log files from previous days and NiFi immediately
cleared out those "old files" on startup. I have a Jira
ticket up to fix this for 1.x and 2.x and will likely have it
up today. Should definitely be ready for 1.23

On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling
 wrote:

Dear NiFiers, we have been bugged so much by overflowing
logfiles, and nothing has ever helped. I thought it was
just my lack of skills...especially when NiFi has some
issues and keeps on spilling stacktraces with high
frequency to disk, it eats up space quickly. I have
created cronjobs that rotate logs every minute iff
required, and when almost no space is left, it simply
deletes old files. Will try totalCapSize etc. Thank you
for the pointers! Best, Lars


On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed"
 wrote:

Hi

Please have a look at this old jira:
https://issues.apache.org/jira/browse/NIFI-2203
I have had issues where a processor create a log
message ever 10ms resulting in the disk is being
full. For me it seems like the maxHistory settings
only effect how many files defined by the rolling
patten to be kept. If you have defined it like this:

${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{-MM-dd}.%i.log
MaxHistory only effect the days not the increments
file %i per day. So you can stille have thousands of
files in one day.
The totalSizeCap will delete the oldes files if the
total size hits the cap settings.

The totalSizeCap have been added in the logback.xml
file for nifi-registry where it has been added inside
the rollingPolicy section. I cound not get it to work
inside the rollingPolicy section in nifi but just
added in appender section. See my comment in the
jira: https://issues.apache.org/jira/browse/NIFI-2203

Kind regards
Jens M. Kofoed

Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen
:

Yeah, I'm working through some of it where I have
time. I plan to have a Jira up this weekend. I'm
wondering, though, if we shouldn't consider a
spike for switching to log4j2 in 2.X because I
saw a lot of complaints about logback being
inconsistent in honoring its settings.

On Fri, Jul 7, 2023 at 10:19 PM Joe Witt
 wrote:

H. Interesting.  Can you capture these
bits of fun in a jira?

Thanks

On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen
 wrote:

After doing some research, it appears
that  is a wonky setting
WRT how well it's honored by logback. I
let a GenerateFlowFile > LogAttribute
flow run for a long time, and it just
kept filling up. When I added
 that appeared to force
expected behavior on total log size. We
might want to add the following:

true
50GB

On Fri, Jul 7, 

Re: NiFi not rolling logs

[Mike Thomsen]

The totalcapsize feature will help a lot from what I’ve seen.Sent from my iPhoneOn Jul 8, 2023, at 9:26 AM, Lars Winderling  wrote:
Hi Mike,thanks for the advice. Our NiFi instances are running for week, if not months. Often times until the next update, so the startup option will bring much benefit, I fear, or am I mistaken. But looking forward to 1.23!On 8 July 2023 13:40:15 CEST, Mike Thomsen  wrote:
Lars,You should also experiment with cleanHistoryOnStart. I did some experimentation this morning where I set the maxHistory to 1 (1 day vs the default of 30 which is 30 days), created a few fake log files from previous days and NiFi immediately cleared out those "old files" on startup. I have a Jira ticket up to fix this for 1.x and 2.x and will likely have it up today. Should definitely be ready for 1.23On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling  wrote:
Dear NiFiers, we have been bugged so much by overflowing logfiles, and nothing has ever helped. I thought it was just my lack of skills...especially when NiFi has some issues and keeps on spilling stacktraces with high frequency to disk, it eats up space quickly. I have created cronjobs that rotate logs every minute iff required, and when almost no space is left, it simply deletes old files. Will try totalCapSize etc. Thank you for the pointers! Best, LarsOn 8 July 2023 09:33:41 CEST, "Jens M. Kofoed"  wrote:
HiPlease have a look at this old jira: https://issues.apache.org/jira/browse/NIFI-2203I have had issues where a processor create a log message ever 10ms resulting in the disk is being full. For me it seems like the maxHistory settings only effect how many files defined by the rolling patten to be kept. If you have defined it like this:${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{-MM-dd}.%i.logMaxHistory only effect the days not the increments file %i per day. So you can stille have thousands of files in one day.The totalSizeCap will delete the oldes files if the total size hits the cap settings.The totalSizeCap have been added in the logback.xml file for nifi-registry where it has been added inside the rollingPolicy section. I cound not get it to work inside the rollingPolicy section in nifi but just added in appender section. See my comment in the jira: https://issues.apache.org/jira/browse/NIFI-2203Kind regardsJens M. KofoedDen lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen :Yeah, I'm working through some of it where I have time. I plan to have a Jira up this weekend. I'm wondering, though, if we shouldn't consider a spike for switching to log4j2 in 2.X because I saw a lot of complaints about logback being inconsistent in honoring its settings.On Fri, Jul 7, 2023 at 10:19 PM Joe Witt  wrote:H.  Interesting.  Can you capture these bits of fun in a jira?ThanksOn Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen  wrote:After doing some research, it appears that  is a wonky setting WRT how well it's honored by logback. I let a GenerateFlowFile > LogAttribute flow run for a long time, and it just kept filling up. When I added  that appeared to force expected behavior on total log size. We might want to add the following:true50GBOn Fri, Jul 7, 2023 at 11:33 AM Michael Moser  wrote:Hi Mike,You aren't alone in experiencing this.  I think logback uses a pattern matcher on filename to discover files to delete.  If "something" happens which causes a gap in the date pattern, then the matcher will then fail to pick up and delete files on the other side of that gap.Regards,-- Mike MOn Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen  wrote:We are using the stock configuration, and have noticed that we have a lot of nifi-app* logs that are well beyond the historic data cap of 30 days in logback.xml; some of those logs go back to April. We also have a bunch of 0 byte nifi-user logs and some of the other logs are 0 bytes as well. It looks like logback is rotating based on time, but isn't cleaning up. Is this expected behavior or a problem with the configuration?Thanks,Mike

Re: NiFi not rolling logs

[Lars Winderling]

Hi Mike,

thanks for the advice. Our NiFi instances are running for week, if not months. 
Often times until the next update, so the startup option will bring much 
benefit, I fear, or am I mistaken. But looking forward to 1.23!

On 8 July 2023 13:40:15 CEST, Mike Thomsen  wrote:
>Lars,
>
>You should also experiment with cleanHistoryOnStart. I did some
>experimentation this morning where I set the maxHistory to 1 (1 day vs the
>default of 30 which is 30 days), created a few fake log files from previous
>days and NiFi immediately cleared out those "old files" on startup. I have
>a Jira ticket up to fix this for 1.x and 2.x and will likely have it up
>today. Should definitely be ready for 1.23
>
>On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling 
>wrote:
>
>> Dear NiFiers, we have been bugged so much by overflowing logfiles, and
>> nothing has ever helped. I thought it was just my lack of
>> skills...especially when NiFi has some issues and keeps on spilling
>> stacktraces with high frequency to disk, it eats up space quickly. I have
>> created cronjobs that rotate logs every minute iff required, and when
>> almost no space is left, it simply deletes old files. Will try totalCapSize
>> etc. Thank you for the pointers! Best, Lars
>>
>>
>> On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed" 
>> wrote:
>>
>>> Hi
>>>
>>> Please have a look at this old jira:
>>> https://issues.apache.org/jira/browse/NIFI-2203
>>> I have had issues where a processor create a log message ever 10ms
>>> resulting in the disk is being full. For me it seems like the maxHistory
>>> settings only effect how many files defined by the rolling patten to be
>>> kept. If you have defined it like this:
>>>
>>> ${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{-MM-dd}.%i.log
>>> MaxHistory only effect the days not the increments file %i per day. So
>>> you can stille have thousands of files in one day.
>>> The totalSizeCap will delete the oldes files if the total size hits the
>>> cap settings.
>>>
>>> The totalSizeCap have been added in the logback.xml file for
>>> nifi-registry where it has been added inside the rollingPolicy section. I
>>> cound not get it to work inside the rollingPolicy section in nifi but just
>>> added in appender section. See my comment in the jira:
>>> https://issues.apache.org/jira/browse/NIFI-2203
>>>
>>> Kind regards
>>> Jens M. Kofoed
>>>
>>> Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen <
>>> mikerthom...@gmail.com>:
>>>
 Yeah, I'm working through some of it where I have time. I plan to have a
 Jira up this weekend. I'm wondering, though, if we shouldn't consider a
 spike for switching to log4j2 in 2.X because I saw a lot of complaints
 about logback being inconsistent in honoring its settings.

 On Fri, Jul 7, 2023 at 10:19 PM Joe Witt  wrote:

> H.  Interesting.  Can you capture these bits of fun in a jira?
>
> Thanks
>
> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen 
> wrote:
>
>> After doing some research, it appears that  is a wonky
>> setting WRT how well it's honored by logback. I let a GenerateFlowFile >
>> LogAttribute flow run for a long time, and it just kept filling up. When 
>> I
>> added  that appeared to force expected behavior on total 
>> log
>> size. We might want to add the following:
>>
>> true
>> 50GB
>>
>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser 
>> wrote:
>>
>>> Hi Mike,
>>>
>>> You aren't alone in experiencing this.  I think logback uses a
>>> pattern matcher on filename to discover files to delete.  If "something"
>>> happens which causes a gap in the date pattern, then the matcher will 
>>> then
>>> fail to pick up and delete files on the other side of that gap.
>>>
>>> Regards,
>>> -- Mike M
>>>
>>>
>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen 
>>> wrote:
>>>
 We are using the stock configuration, and have noticed that we have
 a lot of nifi-app* logs that are well beyond the historic data cap of 
 30
 days in logback.xml; some of those logs go back to April. We also have 
 a
 bunch of 0 byte nifi-user logs and some of the other logs are 0 bytes 
 as
 well. It looks like logback is rotating based on time, but isn't 
 cleaning
 up. Is this expected behavior or a problem with the configuration?

 Thanks,

 Mike

>>>

Re: NiFi not rolling logs

[Mike Thomsen]

Lars,

You should also experiment with cleanHistoryOnStart. I did some
experimentation this morning where I set the maxHistory to 1 (1 day vs the
default of 30 which is 30 days), created a few fake log files from previous
days and NiFi immediately cleared out those "old files" on startup. I have
a Jira ticket up to fix this for 1.x and 2.x and will likely have it up
today. Should definitely be ready for 1.23

On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling 
wrote:

> Dear NiFiers, we have been bugged so much by overflowing logfiles, and
> nothing has ever helped. I thought it was just my lack of
> skills...especially when NiFi has some issues and keeps on spilling
> stacktraces with high frequency to disk, it eats up space quickly. I have
> created cronjobs that rotate logs every minute iff required, and when
> almost no space is left, it simply deletes old files. Will try totalCapSize
> etc. Thank you for the pointers! Best, Lars
>
>
> On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed" 
> wrote:
>
>> Hi
>>
>> Please have a look at this old jira:
>> https://issues.apache.org/jira/browse/NIFI-2203
>> I have had issues where a processor create a log message ever 10ms
>> resulting in the disk is being full. For me it seems like the maxHistory
>> settings only effect how many files defined by the rolling patten to be
>> kept. If you have defined it like this:
>>
>> ${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{-MM-dd}.%i.log
>> MaxHistory only effect the days not the increments file %i per day. So
>> you can stille have thousands of files in one day.
>> The totalSizeCap will delete the oldes files if the total size hits the
>> cap settings.
>>
>> The totalSizeCap have been added in the logback.xml file for
>> nifi-registry where it has been added inside the rollingPolicy section. I
>> cound not get it to work inside the rollingPolicy section in nifi but just
>> added in appender section. See my comment in the jira:
>> https://issues.apache.org/jira/browse/NIFI-2203
>>
>> Kind regards
>> Jens M. Kofoed
>>
>> Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen <
>> mikerthom...@gmail.com>:
>>
>>> Yeah, I'm working through some of it where I have time. I plan to have a
>>> Jira up this weekend. I'm wondering, though, if we shouldn't consider a
>>> spike for switching to log4j2 in 2.X because I saw a lot of complaints
>>> about logback being inconsistent in honoring its settings.
>>>
>>> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt  wrote:
>>>
 H.  Interesting.  Can you capture these bits of fun in a jira?

 Thanks

 On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen 
 wrote:

> After doing some research, it appears that  is a wonky
> setting WRT how well it's honored by logback. I let a GenerateFlowFile >
> LogAttribute flow run for a long time, and it just kept filling up. When I
> added  that appeared to force expected behavior on total 
> log
> size. We might want to add the following:
>
> true
> 50GB
>
> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser 
> wrote:
>
>> Hi Mike,
>>
>> You aren't alone in experiencing this.  I think logback uses a
>> pattern matcher on filename to discover files to delete.  If "something"
>> happens which causes a gap in the date pattern, then the matcher will 
>> then
>> fail to pick up and delete files on the other side of that gap.
>>
>> Regards,
>> -- Mike M
>>
>>
>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen 
>> wrote:
>>
>>> We are using the stock configuration, and have noticed that we have
>>> a lot of nifi-app* logs that are well beyond the historic data cap of 30
>>> days in logback.xml; some of those logs go back to April. We also have a
>>> bunch of 0 byte nifi-user logs and some of the other logs are 0 bytes as
>>> well. It looks like logback is rotating based on time, but isn't 
>>> cleaning
>>> up. Is this expected behavior or a problem with the configuration?
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>

Re: NiFi not rolling logs

[Lars Winderling]

Dear NiFiers, we have been bugged so much by overflowing logfiles, and nothing 
has ever helped. I thought it was just my lack of skills...especially when NiFi 
has some issues and keeps on spilling stacktraces with high frequency to disk, 
it eats up space quickly. I have created cronjobs that rotate logs every minute 
iff required, and when almost no space is left, it simply deletes old files. 
Will try totalCapSize etc. Thank you for the pointers! Best, Lars

On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed"  wrote:
>Hi
>
>Please have a look at this old jira:
>https://issues.apache.org/jira/browse/NIFI-2203
>I have had issues where a processor create a log message ever 10ms
>resulting in the disk is being full. For me it seems like the maxHistory
>settings only effect how many files defined by the rolling patten to be
>kept. If you have defined it like this:
>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{-MM-dd}.%i.log
>MaxHistory only effect the days not the increments file %i per day. So you
>can stille have thousands of files in one day.
>The totalSizeCap will delete the oldes files if the total size hits the cap
>settings.
>
>The totalSizeCap have been added in the logback.xml file for nifi-registry
>where it has been added inside the rollingPolicy section. I cound not get
>it to work inside the rollingPolicy section in nifi but just added
>in appender section. See my comment in the jira:
>https://issues.apache.org/jira/browse/NIFI-2203
>
>Kind regards
>Jens M. Kofoed
>
>Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen :
>
>> Yeah, I'm working through some of it where I have time. I plan to have a
>> Jira up this weekend. I'm wondering, though, if we shouldn't consider a
>> spike for switching to log4j2 in 2.X because I saw a lot of complaints
>> about logback being inconsistent in honoring its settings.
>>
>> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt  wrote:
>>
>>> H.  Interesting.  Can you capture these bits of fun in a jira?
>>>
>>> Thanks
>>>
>>> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen 
>>> wrote:
>>>
 After doing some research, it appears that  is a wonky
 setting WRT how well it's honored by logback. I let a GenerateFlowFile >
 LogAttribute flow run for a long time, and it just kept filling up. When I
 added  that appeared to force expected behavior on total log
 size. We might want to add the following:

 true
 50GB

 On Fri, Jul 7, 2023 at 11:33 AM Michael Moser 
 wrote:

> Hi Mike,
>
> You aren't alone in experiencing this.  I think logback uses a pattern
> matcher on filename to discover files to delete.  If "something" happens
> which causes a gap in the date pattern, then the matcher will then fail to
> pick up and delete files on the other side of that gap.
>
> Regards,
> -- Mike M
>
>
> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen 
> wrote:
>
>> We are using the stock configuration, and have noticed that we have a
>> lot of nifi-app* logs that are well beyond the historic data cap of 30 
>> days
>> in logback.xml; some of those logs go back to April. We also have a bunch
>> of 0 byte nifi-user logs and some of the other logs are 0 bytes as well. 
>> It
>> looks like logback is rotating based on time, but isn't cleaning up. Is
>> this expected behavior or a problem with the configuration?
>>
>> Thanks,
>>
>> Mike
>>
>

Re: NiFi not rolling logs

[Jens M. Kofoed]

Hi

Please have a look at this old jira:
https://issues.apache.org/jira/browse/NIFI-2203
I have had issues where a processor create a log message ever 10ms
resulting in the disk is being full. For me it seems like the maxHistory
settings only effect how many files defined by the rolling patten to be
kept. If you have defined it like this:
${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{-MM-dd}.%i.log
MaxHistory only effect the days not the increments file %i per day. So you
can stille have thousands of files in one day.
The totalSizeCap will delete the oldes files if the total size hits the cap
settings.

The totalSizeCap have been added in the logback.xml file for nifi-registry
where it has been added inside the rollingPolicy section. I cound not get
it to work inside the rollingPolicy section in nifi but just added
in appender section. See my comment in the jira:
https://issues.apache.org/jira/browse/NIFI-2203

Kind regards
Jens M. Kofoed

Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen :

> Yeah, I'm working through some of it where I have time. I plan to have a
> Jira up this weekend. I'm wondering, though, if we shouldn't consider a
> spike for switching to log4j2 in 2.X because I saw a lot of complaints
> about logback being inconsistent in honoring its settings.
>
> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt  wrote:
>
>> H.  Interesting.  Can you capture these bits of fun in a jira?
>>
>> Thanks
>>
>> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen 
>> wrote:
>>
>>> After doing some research, it appears that  is a wonky
>>> setting WRT how well it's honored by logback. I let a GenerateFlowFile >
>>> LogAttribute flow run for a long time, and it just kept filling up. When I
>>> added  that appeared to force expected behavior on total log
>>> size. We might want to add the following:
>>>
>>> true
>>> 50GB
>>>
>>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser 
>>> wrote:
>>>
 Hi Mike,

 You aren't alone in experiencing this.  I think logback uses a pattern
 matcher on filename to discover files to delete.  If "something" happens
 which causes a gap in the date pattern, then the matcher will then fail to
 pick up and delete files on the other side of that gap.

 Regards,
 -- Mike M


 On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen 
 wrote:

> We are using the stock configuration, and have noticed that we have a
> lot of nifi-app* logs that are well beyond the historic data cap of 30 
> days
> in logback.xml; some of those logs go back to April. We also have a bunch
> of 0 byte nifi-user logs and some of the other logs are 0 bytes as well. 
> It
> looks like logback is rotating based on time, but isn't cleaning up. Is
> this expected behavior or a problem with the configuration?
>
> Thanks,
>
> Mike
>

Re: NiFi not rolling logs

[Mike Thomsen]

Yeah, I'm working through some of it where I have time. I plan to have a
Jira up this weekend. I'm wondering, though, if we shouldn't consider a
spike for switching to log4j2 in 2.X because I saw a lot of complaints
about logback being inconsistent in honoring its settings.

On Fri, Jul 7, 2023 at 10:19 PM Joe Witt  wrote:

> H.  Interesting.  Can you capture these bits of fun in a jira?
>
> Thanks
>
> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen 
> wrote:
>
>> After doing some research, it appears that  is a wonky
>> setting WRT how well it's honored by logback. I let a GenerateFlowFile >
>> LogAttribute flow run for a long time, and it just kept filling up. When I
>> added  that appeared to force expected behavior on total log
>> size. We might want to add the following:
>>
>> true
>> 50GB
>>
>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser  wrote:
>>
>>> Hi Mike,
>>>
>>> You aren't alone in experiencing this.  I think logback uses a pattern
>>> matcher on filename to discover files to delete.  If "something" happens
>>> which causes a gap in the date pattern, then the matcher will then fail to
>>> pick up and delete files on the other side of that gap.
>>>
>>> Regards,
>>> -- Mike M
>>>
>>>
>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen 
>>> wrote:
>>>
 We are using the stock configuration, and have noticed that we have a
 lot of nifi-app* logs that are well beyond the historic data cap of 30 days
 in logback.xml; some of those logs go back to April. We also have a bunch
 of 0 byte nifi-user logs and some of the other logs are 0 bytes as well. It
 looks like logback is rotating based on time, but isn't cleaning up. Is
 this expected behavior or a problem with the configuration?

 Thanks,

 Mike

>>>

Re: NiFi not rolling logs

[Joe Witt]

H.  Interesting.  Can you capture these bits of fun in a jira?

Thanks

On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen  wrote:

> After doing some research, it appears that  is a wonky
> setting WRT how well it's honored by logback. I let a GenerateFlowFile >
> LogAttribute flow run for a long time, and it just kept filling up. When I
> added  that appeared to force expected behavior on total log
> size. We might want to add the following:
>
> true
> 50GB
>
> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser  wrote:
>
>> Hi Mike,
>>
>> You aren't alone in experiencing this.  I think logback uses a pattern
>> matcher on filename to discover files to delete.  If "something" happens
>> which causes a gap in the date pattern, then the matcher will then fail to
>> pick up and delete files on the other side of that gap.
>>
>> Regards,
>> -- Mike M
>>
>>
>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen 
>> wrote:
>>
>>> We are using the stock configuration, and have noticed that we have a
>>> lot of nifi-app* logs that are well beyond the historic data cap of 30 days
>>> in logback.xml; some of those logs go back to April. We also have a bunch
>>> of 0 byte nifi-user logs and some of the other logs are 0 bytes as well. It
>>> looks like logback is rotating based on time, but isn't cleaning up. Is
>>> this expected behavior or a problem with the configuration?
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>

Re: NiFi not rolling logs

[Michael Moser]

Hi Mike,

You aren't alone in experiencing this.  I think logback uses a pattern
matcher on filename to discover files to delete.  If "something" happens
which causes a gap in the date pattern, then the matcher will then fail to
pick up and delete files on the other side of that gap.

Regards,
-- Mike M


On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen  wrote:

> We are using the stock configuration, and have noticed that we have a lot
> of nifi-app* logs that are well beyond the historic data cap of 30 days in
> logback.xml; some of those logs go back to April. We also have a bunch of 0
> byte nifi-user logs and some of the other logs are 0 bytes as well. It
> looks like logback is rotating based on time, but isn't cleaning up. Is
> this expected behavior or a problem with the configuration?
>
> Thanks,
>
> Mike
>

Re: Nifi Registry v1.22.0 - Unable to create buckets over http behind proxy

[Juan Pablo Gardella]

Ignore, I found the option is available after pressing the settings button.

On Mon, Jun 19, 2023 at 11:04 PM Juan Pablo Gardella <
gardellajuanpa...@gmail.com> wrote:

> Hi,
>
> I am testing the nifi registry behind a proxy and it is unable to create
> buckets. Last time I tested using v 1.14.0 works fine. I am checking logs,
> but do you remember any change that prevents that? The logs show
>
> 2023-06-20 01:43:30,663 INFO [NiFi Registry Web Server-14]
> o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException:
> Access tokens are only issued over HTTPS. Returning Conflict response.
> 2023-06-20 01:43:30,898 INFO [NiFi Registry Web Server-19]
> o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException:
> User authentication/authorization is only supported when running over
> HTTPS.. Returning Conflict response.
>
> From browser I see following requests:
> https://serverhost/nifi-registry-api/access/token/kerberos
>
> Do you know how to make it works with http again?
>
> Thanks
>
>

RE: NiFi 1.19.1 - ExecuteSQL - Decimal Precision above 6 returns "OE-7"

[DSI]

Hi Isha,

That’s precisely what we are doing to overcome the issue.
It’s not the database since the JDBC driver outside NIFI works fine.

Would be cool if NIFI didn’t assume any suitable precision or allow a 
configuration for that…
We have 500+ daily queries that go through that processor, we still don’t know 
the real impact because we only detected two days ago, but yeah we started to 
force a type cast to string as a workaround.

Thank you.

Tiago Sebastião

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 19 de abril de 2023 15:10
To: users@nifi.apache.org
Subject: RE: NiFi 1.19.1 - ExecuteSQL - Decimal Precision above 6 returns "OE-7"

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

The content is not messed up but converted to scientific notation. This is 
valid in json and various other data formats.
Apparently either the database or NiFi considers decimal(x,7) the point at 
which scientific notation is more suitable.

Since you’re going to CSV format, you can convert the values to string with the 
desired format right in the SQL query.

Regards,

Isha

Van: Tiago Luís Sebastião (DSI) 
mailto:tiago.luis.sebast...@cgd.pt>>
Verzonden: woensdag 19 april 2023 12:56
Aan: users@nifi.apache.org<mailto:users@nifi.apache.org>
Onderwerp: NiFi 1.19.1 - ExecuteSQL - Decimal Precision above 6 returns "OE-7"

Hi,

I’m using a standalone deployment (Nifi 1.19.1) that runs directly on the 
server.
I’ve noticed something tricky for the first time…
When executing a query with Nifi processor ExecuteSQL the maximum decimal 
precision that it returns is 6, anything above that returns “0E-7”.

Query:
“
select
  0.000::decimal(38,7) as Test1 -- NOT OK -> Returns 0E-7
,0.000::decimal(38,6) as Test2 – OK -> Returns 0,00
,0.000::decimal(12,7) as Test3 -- Needed -> NOT OK -> Returns 0E-7
,0.000::decimal(12,6) as Test4 -- OK -> Returns 0,00
“

ExecuteSQL . List Queue . View content:
"
Obj___avro.schemaö_{"type":"record","name":"NiFi_ExecuteSQL_Record","namespace":"any.data","fields":[{"name":"TEST1","type":["null","string"]},{"name":"TEST2","type":["null","string"]},{"name":"TEST3","type":["null","string"]},{"name":"TEST4","type":["null","string"]}]}_avro.codec_nulléõ
žSø®²›_yá‚_„_@__0E-7__0.00__0E-7__0.00éõ
žSø®²›_yá‚_„
"

After ExecuteSQL I use a ConvertRecord with AvroReader and CSVRecordSetWriter 
but since the content is already messed up it converts to:
“0E-7; 0.00;0E-7;0.00”

Any idea if this is a known issue or how to solve it?

- ExecuteSQL Specs

Database Connection Pooling Service:  DBCPConnectionPool-Netezza -> IBM Cloud 
Pak for Data System (PostgreSQL)
setSQL: select 0.000::decimal(38,7) as Test1, 0.000::decimal(38,6) as 
Test2, 0.000::decimal(12,7) as Test3, 0.000::decimal(12,6) as Test4
Normalize Table/Column Names: false
Use Avro Logical Types: false
Compression Format: NONE
Default Decimal Precision: 10
Default Decimal Scale: 0

-

Thanks in advance.

Tiago Sebastião

RE: NiFi 1.19.1 - ExecuteSQL - Decimal Precision above 6 returns "OE-7"

[Isha Lamboo]

Hi Tiago,

The content is not messed up but converted to scientific notation. This is 
valid in json and various other data formats.
Apparently either the database or NiFi considers decimal(x,7) the point at 
which scientific notation is more suitable.

Since you’re going to CSV format, you can convert the values to string with the 
desired format right in the SQL query.

Regards,

Isha

Van: Tiago Luís Sebastião (DSI) 
Verzonden: woensdag 19 april 2023 12:56
Aan: users@nifi.apache.org
Onderwerp: NiFi 1.19.1 - ExecuteSQL - Decimal Precision above 6 returns "OE-7"

Hi,

I’m using a standalone deployment (Nifi 1.19.1) that runs directly on the 
server.
I’ve noticed something tricky for the first time…
When executing a query with Nifi processor ExecuteSQL the maximum decimal 
precision that it returns is 6, anything above that returns “0E-7”.

Query:
“
select
  0.000::decimal(38,7) as Test1 -- NOT OK -> Returns 0E-7
,0.000::decimal(38,6) as Test2 – OK -> Returns 0,00
,0.000::decimal(12,7) as Test3 -- Needed -> NOT OK -> Returns 0E-7
,0.000::decimal(12,6) as Test4 -- OK -> Returns 0,00
“

ExecuteSQL . List Queue . View content:
"
Obj___avro.schemaö_{"type":"record","name":"NiFi_ExecuteSQL_Record","namespace":"any.data","fields":[{"name":"TEST1","type":["null","string"]},{"name":"TEST2","type":["null","string"]},{"name":"TEST3","type":["null","string"]},{"name":"TEST4","type":["null","string"]}]}_avro.codec_nulléõ
žSø®²›_yá‚_„_@__0E-7__0.00__0E-7__0.00éõ
žSø®²›_yá‚_„
"

After ExecuteSQL I use a ConvertRecord with AvroReader and CSVRecordSetWriter 
but since the content is already messed up it converts to:
“0E-7; 0.00;0E-7;0.00”

Any idea if this is a known issue or how to solve it?

- ExecuteSQL Specs

Database Connection Pooling Service:  DBCPConnectionPool-Netezza -> IBM Cloud 
Pak for Data System (PostgreSQL)
setSQL: select 0.000::decimal(38,7) as Test1, 0.000::decimal(38,6) as 
Test2, 0.000::decimal(12,7) as Test3, 0.000::decimal(12,6) as Test4
Normalize Table/Column Names: false
Use Avro Logical Types: false
Compression Format: NONE
Default Decimal Precision: 10
Default Decimal Scale: 0

-

Thanks in advance.

Tiago Sebastião

Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue

[Simon Bence]

Hi Josef,

There is a PR made by Mark takes care of this behaviour:  
https://github.com/apache/nifi/pull/7095

I hope this will fit to your needs! If there is something is going unexpected, 
please reply!

Regards,
Bence

> On 2023. Mar 28., at 9:14, Simon Bence  wrote:
> 
> Hi Josef,
> 
> I will take a look as soon as I will have the chance and find you back.
> 
> Regards,
> Bence
> 
>> On 2023. Mar 28., at 8:44, josef.zahn...@swisscom.com wrote:
>> 
>> Hi Bence
>>  
>> We still facing problems with the current NiFi v1.20.0 release and nested 
>> process groups. Would you mind having a look at the issue?
>>  
>> https://issues.apache.org/jira/browse/NIFI-11251
>>  
>> It seems that NiFi is not correctly stopping/starting the processors within 
>> the (nested?) PGs. As workaround we always stop the processors, do the NiFi 
>> registry update and start the processors again. Very annoying as we have 
>> HTTP listeners which stop answering in that period.
>>  
>> Cheers Josef
>>  
>>  
>> From: Zahner Josef, GSB-LR-TRW-LI 
>> Date: Tuesday, 13 December 2022 at 13:53
>> To: users@nifi.apache.org 
>> Subject: Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes 
>> issue
>> 
>> Hi Bence
>>  
>> I mentioned yesterday another issue which we thought would be solved after 
>> the upgrade steps. Sadly it happened again and after an investigation on our 
>> side it seems to be fully reproducible, I’ve opened a Jira Bugticket 
>> https://issues.apache.org/jira/browse/NIFI-10973 with more details. It’s as 
>> well related to NiFi Registry nested flows…
>>  
>> Thanks for taking care of it :-).
>>  
>> Cheers Josef
>>  
>>  
>> From: Simon Bence 
>> Reply to: "users@nifi.apache.org" 
>> Date: Monday, 12 December 2022 at 16:40
>> To: "users@nifi.apache.org" 
>> Subject: Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes 
>> issue
>>  
>> Hi Josef, 
>>  
>> Thanks for the quick update! Meanwhile I tried to reproduce this situation 
>> but based on the known facts I was not able (Note: without any version 
>> change). I think it is connected to the version change indeed, but if you 
>> face with this ever again, please let me know!
>>  
>> Regards,
>> Bence
>> 
>> 
>> 
>> On 2022. Dec 12., at 16:33, > <mailto:josef.zahn...@swisscom.com>> > <mailto:josef.zahn...@swisscom.com>> wrote:
>>  
>> Ok quick update on this. We investigated the changes on the *.snapshot files 
>> on NiFi Registry. It seems that it’s mainly caused due to the fact that the 
>> NiFi Version changes from 1.18.0 to 1.19.1 and some other facts caused due 
>> to the upgrade. After commiting all the child PGs once the “Show Local 
>> Changes” seems to behave as expected.
>> We have seen another issue caused only on the first commit of the child PGs 
>> on v1.19.1, however I’ll not mention them as I’m guessing that this is 
>> highly related to the last commits with the older NiFi version.
>>  
>> Cheers Josef
>>  
>>  
>> From: "Zahner Josef, GSB-LR-TRW-LI" > <mailto:josef.zahn...@swisscom.com>>
>> Date: Monday, 12 December 2022 at 14:44
>> To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" 
>> mailto:users@nifi.apache.org>>
>> Subject: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes 
>> issue
>>  
>> Hi guys
>>  
>> Sorry that I’ve to bother you again. We just upgraded to NiFi 1.19.1 due to 
>> the major issues with the NiFi Registry Client on NiFi 1.18.0.
>>  
>> The following issue still persists in the new version. Let’s say everything 
>> is up-to-date (parent and child PGs) from NiFi Registry perspective, now I 
>> do change eg. a label directly in the parent PG. I do see on that parent PG, 
>> that most likely every connection/service/processor has changed, even though 
>> I’ve changed only one single label size, please check the screenshot for 
>> NiFi Registry Client under “Show Local Changes”.  
>>  
>> 
>>  
>> Is this a cosmetical issue or do we have to worry about that (we had massive 
>> NiFi registry issue on NiFi 1.18.0) ? Because in theory it makes no sense to 
>> that NiFi tells that everything has changed if only one label changes.
>>  
>> Thanks in advance
>> Josef
> 

Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue

[Simon Bence]

Hi Josef,

I will take a look as soon as I will have the chance and find you back.

Regards,
Bence

> On 2023. Mar 28., at 8:44, josef.zahn...@swisscom.com wrote:
> 
> Hi Bence
>  
> We still facing problems with the current NiFi v1.20.0 release and nested 
> process groups. Would you mind having a look at the issue?
>  
> https://issues.apache.org/jira/browse/NIFI-11251
>  
> It seems that NiFi is not correctly stopping/starting the processors within 
> the (nested?) PGs. As workaround we always stop the processors, do the NiFi 
> registry update and start the processors again. Very annoying as we have HTTP 
> listeners which stop answering in that period.
>  
> Cheers Josef
>  
>  
> From: Zahner Josef, GSB-LR-TRW-LI 
> Date: Tuesday, 13 December 2022 at 13:53
> To: users@nifi.apache.org 
> Subject: Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes 
> issue
> 
> Hi Bence
>  
> I mentioned yesterday another issue which we thought would be solved after 
> the upgrade steps. Sadly it happened again and after an investigation on our 
> side it seems to be fully reproducible, I’ve opened a Jira Bugticket 
> https://issues.apache.org/jira/browse/NIFI-10973 with more details. It’s as 
> well related to NiFi Registry nested flows…
>  
> Thanks for taking care of it :-).
>  
> Cheers Josef
>  
>  
> From: Simon Bence 
> Reply to: "users@nifi.apache.org" 
> Date: Monday, 12 December 2022 at 16:40
> To: "users@nifi.apache.org" 
> Subject: Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes 
> issue
>  
> Hi Josef, 
>  
> Thanks for the quick update! Meanwhile I tried to reproduce this situation 
> but based on the known facts I was not able (Note: without any version 
> change). I think it is connected to the version change indeed, but if you 
> face with this ever again, please let me know!
>  
> Regards,
> Bence
> 
> 
> 
> On 2022. Dec 12., at 16:33,  <mailto:josef.zahn...@swisscom.com>>  <mailto:josef.zahn...@swisscom.com>> wrote:
>  
> Ok quick update on this. We investigated the changes on the *.snapshot files 
> on NiFi Registry. It seems that it’s mainly caused due to the fact that the 
> NiFi Version changes from 1.18.0 to 1.19.1 and some other facts caused due to 
> the upgrade. After commiting all the child PGs once the “Show Local Changes” 
> seems to behave as expected.
> We have seen another issue caused only on the first commit of the child PGs 
> on v1.19.1, however I’ll not mention them as I’m guessing that this is highly 
> related to the last commits with the older NiFi version.
>  
> Cheers Josef
>  
>  
> From: "Zahner Josef, GSB-LR-TRW-LI"  <mailto:josef.zahn...@swisscom.com>>
> Date: Monday, 12 December 2022 at 14:44
> To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" 
> mailto:users@nifi.apache.org>>
> Subject: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue
>  
> Hi guys
>  
> Sorry that I’ve to bother you again. We just upgraded to NiFi 1.19.1 due to 
> the major issues with the NiFi Registry Client on NiFi 1.18.0.
>  
> The following issue still persists in the new version. Let’s say everything 
> is up-to-date (parent and child PGs) from NiFi Registry perspective, now I do 
> change eg. a label directly in the parent PG. I do see on that parent PG, 
> that most likely every connection/service/processor has changed, even though 
> I’ve changed only one single label size, please check the screenshot for NiFi 
> Registry Client under “Show Local Changes”.  
>  
> 
>  
> Is this a cosmetical issue or do we have to worry about that (we had massive 
> NiFi registry issue on NiFi 1.18.0) ? Because in theory it makes no sense to 
> that NiFi tells that everything has changed if only one label changes.
>  
> Thanks in advance
> Josef

Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue

[Josef.Zahner1]

Hi Bence

We still facing problems with the current NiFi v1.20.0 release and nested 
process groups. Would you mind having a look at the issue?

https://issues.apache.org/jira/browse/NIFI-11251

It seems that NiFi is not correctly stopping/starting the processors within the 
(nested?) PGs. As workaround we always stop the processors, do the NiFi 
registry update and start the processors again. Very annoying as we have HTTP 
listeners which stop answering in that period.

Cheers Josef


From: Zahner Josef, GSB-LR-TRW-LI 
Date: Tuesday, 13 December 2022 at 13:53
To: users@nifi.apache.org 
Subject: Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes 
issue
Hi Bence

I mentioned yesterday another issue which we thought would be solved after the 
upgrade steps. Sadly it happened again and after an investigation on our side 
it seems to be fully reproducible, I’ve opened a Jira Bugticket 
https://issues.apache.org/jira/browse/NIFI-10973 with more details. It’s as 
well related to NiFi Registry nested flows…

Thanks for taking care of it :-).

Cheers Josef


From: Simon Bence 
Reply to: "users@nifi.apache.org" 
Date: Monday, 12 December 2022 at 16:40
To: "users@nifi.apache.org" 
Subject: Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes 
issue

Hi Josef,

Thanks for the quick update! Meanwhile I tried to reproduce this situation but 
based on the known facts I was not able (Note: without any version change). I 
think it is connected to the version change indeed, but if you face with this 
ever again, please let me know!

Regards,
Bence



On 2022. Dec 12., at 16:33, 
mailto:josef.zahn...@swisscom.com>> 
mailto:josef.zahn...@swisscom.com>> wrote:

Ok quick update on this. We investigated the changes on the *.snapshot files on 
NiFi Registry. It seems that it’s mainly caused due to the fact that the NiFi 
Version changes from 1.18.0 to 1.19.1 and some other facts caused due to the 
upgrade. After commiting all the child PGs once the “Show Local Changes” seems 
to behave as expected.
We have seen another issue caused only on the first commit of the child PGs on 
v1.19.1, however I’ll not mention them as I’m guessing that this is highly 
related to the last commits with the older NiFi version.

Cheers Josef


From: "Zahner Josef, GSB-LR-TRW-LI" 
mailto:josef.zahn...@swisscom.com>>
Date: Monday, 12 December 2022 at 14:44
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Subject: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue

Hi guys

Sorry that I’ve to bother you again. We just upgraded to NiFi 1.19.1 due to the 
major issues with the NiFi Registry Client on NiFi 1.18.0.

The following issue still persists in the new version. Let’s say everything is 
up-to-date (parent and child PGs) from NiFi Registry perspective, now I do 
change eg. a label directly in the parent PG. I do see on that parent PG, that 
most likely every connection/service/processor has changed, even though I’ve 
changed only one single label size, please check the screenshot for NiFi 
Registry Client under “Show Local Changes”.



Is this a cosmetical issue or do we have to worry about that (we had massive 
NiFi registry issue on NiFi 1.18.0) ? Because in theory it makes no sense to 
that NiFi tells that everything has changed if only one label changes.

Thanks in advance
Josef



smime.p7s
Description: S/MIME Cryptographic Signature

Re: NiFi 1.20.0 PutSFTP: SSH Client connection failed -> Timeout expired

[David Handermann]

Hi Josef,

Thanks for the reply, and for highlighting the MaxStartups setting in
OpenSSH. As the documentation notes, MaxStartups limits the number of
unauthenticated connections, so for the purpose of PutSFTP, it is probably
more important to have a sufficiently high number allowed.

I can appreciate the challenge of clear flow design with multiple sources
channeling to a single PutSFTP Processor. Given the Synology NAS support
for 4000 concurrent connections, it sounds like it should be more than
capable of handling the number of SFTP connections from NiFi.

If you run into additional issues, please pass along the details.

Regards,
David Handermann

On Fri, Feb 17, 2023 at 3:44 PM  wrote:

> Hi David
>
>
>
> Thanks a lot for answering here in this topic.
>
>
>
> A few comments to your reply.
>
>
>
>- SSHD config and the “MaxSession” setting -> when we started
>troubleshooting we thought as well that this could help to fix our issue.
>However based on the sshd_config manpage (see below) this setting is
>correlating only to session multiplexing, which I don’t expect that NiFi
>does it. That was one of the reasons why I created my mail. When we started
>using the SFTP Server increased as well the “MaxStartups” to 100 as
>potentially a lot of PutSFTP processors can try to open a connection to the
>SFTP server in parallel.
>
>
> *MaxSessions*
>
> *Specifies the maximum number of open shell, login or subsystem (e.g.
> sftp) sessions permitted per network connection.  Multiple sessions may be
> established by clients that support connection multiplexing.  Setting
> MaxSessions to 1 will effectively disable session multiplexing, whereas
> setting it to 0 will prevent all shell, login and subsystem sessions while
> still permitting forwarding.  The default is 10.*
>
>
>
>- Reason for having 50 PutSFTP instance -> We fetch data from a lot of
>different data sources, each of them are processed and the original files
>will be store on the SFTP server. The reason why we created not a single
>SFTP processor was clarity on the GUI. It would need a lot of local
>Input/Output ports and would mess up the canvas. But you are right, less
>SFTP processor would reduce the load of the SFTP server as we would use
>less total processors to transfer data. However we use a quite powerful
>Synology NAS (SA3600) as SFTP server and based on the datasheet it should
>be possible to handle 4000 concurrent sessions (sadly not more).
>
>
>
> Based on your explanation how NiFi handles SFTP we decided to restart and
> upgrade our NAS to the newest firmware. In the last few hours the issue
> happened only two times, but we will monitor it. It’s very likely that the
> issue is or was related to our SFTP server and not to NiFi.
>
>
>
> Cheers Josef
>
>
>
> *From: *David Handermann 
> *Date: *Thursday, 16 February 2023 at 15:51
> *To: *users@nifi.apache.org 
> *Subject: *Re: NiFi 1.20.0 PutSFTP: SSH Client connection failed ->
> Timeout expired
>
> Hi Josef,
>
>
>
> The FetchSFTP Processor implements some connection reuse, but GetSFTP,
> ListSFTP, and PutSFTP create a new connection for every invocation of the
> processor. I have considered a new approach SFTP components using a
> Controller Service with connection pooling, but it still requires some
> design work prior to implementation.
>
>
>
> Based on the current SFTP processor behavior, it is possible to have
> connection timeouts when the SFTP server is not accepting new connections.
> Every SFTP server is different, but OpenSSH uses the MaxSessions setting in
> sshd_config [1] to limit the number of simultaneous open sessions.
>
>
>
> Using the example of 50 PutSFTP processors connecting to the same server,
> it is very possible to encounter a connection timeout if NiFi triggers all
> of them in rapid succession.
>
>
>
> The Connection Timeout property in PutSFTP controls how long the processor
> will wait before throwing the ClientConnectException. The number of
> Concurrent Tasks for each PutSFTP processor also influences the number of
> simultaneous connections. Increasing the Connection Timeout in PutSFTP may
> hide the problem, but if the destination SFTP server can handle the load,
> it may be helpful to increase the number of maximum sessions.
>
>
>
> On the other hand, is there a reason for having 50 separate instances of
> PutSFTP sending to the same server? It should be possible to design the
> flow and parameterize the destination using FlowFile attributes. Depending
> on the number of CPU cores and available threads, having more SFTP
> connections results in poor perfo

Re: NiFi 1.20.0 PutSFTP: SSH Client connection failed -> Timeout expired

[Josef.Zahner1]

Hi David

Thanks a lot for answering here in this topic.

A few comments to your reply.


  *   SSHD config and the “MaxSession” setting -> when we started 
troubleshooting we thought as well that this could help to fix our issue. 
However based on the sshd_config manpage (see below) this setting is 
correlating only to session multiplexing, which I don’t expect that NiFi does 
it. That was one of the reasons why I created my mail. When we started using 
the SFTP Server increased as well the “MaxStartups” to 100 as potentially a lot 
of PutSFTP processors can try to open a connection to the SFTP server in 
parallel.


MaxSessions
Specifies the maximum number of open shell, login or subsystem (e.g. sftp) 
sessions permitted per network connection.  Multiple sessions may be 
established by clients that support connection multiplexing.  Setting 
MaxSessions to 1 will effectively disable session multiplexing, whereas setting 
it to 0 will prevent all shell, login and subsystem sessions while still 
permitting forwarding.  The default is 10.


  *   Reason for having 50 PutSFTP instance -> We fetch data from a lot of 
different data sources, each of them are processed and the original files will 
be store on the SFTP server. The reason why we created not a single SFTP 
processor was clarity on the GUI. It would need a lot of local Input/Output 
ports and would mess up the canvas. But you are right, less SFTP processor 
would reduce the load of the SFTP server as we would use less total processors 
to transfer data. However we use a quite powerful Synology NAS (SA3600) as SFTP 
server and based on the datasheet it should be possible to handle 4000 
concurrent sessions (sadly not more).

Based on your explanation how NiFi handles SFTP we decided to restart and 
upgrade our NAS to the newest firmware. In the last few hours the issue 
happened only two times, but we will monitor it. It’s very likely that the 
issue is or was related to our SFTP server and not to NiFi.

Cheers Josef

From: David Handermann 
Date: Thursday, 16 February 2023 at 15:51
To: users@nifi.apache.org 
Subject: Re: NiFi 1.20.0 PutSFTP: SSH Client connection failed -> Timeout 
expired
Hi Josef,

The FetchSFTP Processor implements some connection reuse, but GetSFTP, 
ListSFTP, and PutSFTP create a new connection for every invocation of the 
processor. I have considered a new approach SFTP components using a Controller 
Service with connection pooling, but it still requires some design work prior 
to implementation.

Based on the current SFTP processor behavior, it is possible to have connection 
timeouts when the SFTP server is not accepting new connections. Every SFTP 
server is different, but OpenSSH uses the MaxSessions setting in sshd_config 
[1] to limit the number of simultaneous open sessions.

Using the example of 50 PutSFTP processors connecting to the same server, it is 
very possible to encounter a connection timeout if NiFi triggers all of them in 
rapid succession.

The Connection Timeout property in PutSFTP controls how long the processor will 
wait before throwing the ClientConnectException. The number of Concurrent Tasks 
for each PutSFTP processor also influences the number of simultaneous 
connections. Increasing the Connection Timeout in PutSFTP may hide the problem, 
but if the destination SFTP server can handle the load, it may be helpful to 
increase the number of maximum sessions.

On the other hand, is there a reason for having 50 separate instances of 
PutSFTP sending to the same server? It should be possible to design the flow 
and parameterize the destination using FlowFile attributes. Depending on the 
number of CPU cores and available threads, having more SFTP connections results 
in poor performance, so smaller numbers can be better.

Regards,
David Handermann

[1] 
https://linux.die.net/man/5/sshd_config<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinux.die.net%2Fman%2F5%2Fsshd_config=05%7C01%7CJosef.Zahner1%40swisscom.com%7Cca0dea0c3781417181e008db102d598a%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C638121559173781274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=CjGVtb6tqsQsNelhECCV3aNvvFwgZwEYEC9gGiEnWwQ%3D=0>

On Thu, Feb 16, 2023 at 1:33 AM 
mailto:josef.zahn...@swisscom.com>> wrote:
Hi guys

It was upgrade time again on our side, we just upgraded from 1.19.1 to 1.20.0. 
Since 1.20.0 we do see significantly more SSH Connection Timeout errors on the 
PutSFTP processor…

PutSFTP Processor ERROR:
2023-02-16 07:44:07,905 ERROR [Timer-Driven Process Thread-50] 
o.a.nifi.processors.standard.PutSFTP 
PutSFTP[id=12563431-c40a-1af7-b09b-16de27d887b7] Unable to transfer 
StandardFlowFileRecord[uuid=a1adadb1-61f9-414f-99e5-aad4331165ef,claim=StandardContentClaim
 [resourceClaim=StandardResourceClaim[id=1676529847583-196507, 
container=default, section=923], offset=0, 
length=6582249],offset=0,name=myfile.avro.gz,si

Re: NiFi 1.20.0 PutSFTP: SSH Client connection failed -> Timeout expired

[David Handermann]

Hi Josef,

The FetchSFTP Processor implements some connection reuse, but GetSFTP,
ListSFTP, and PutSFTP create a new connection for every invocation of the
processor. I have considered a new approach SFTP components using a
Controller Service with connection pooling, but it still requires some
design work prior to implementation.

Based on the current SFTP processor behavior, it is possible to have
connection timeouts when the SFTP server is not accepting new connections.
Every SFTP server is different, but OpenSSH uses the MaxSessions setting in
sshd_config [1] to limit the number of simultaneous open sessions.

Using the example of 50 PutSFTP processors connecting to the same server,
it is very possible to encounter a connection timeout if NiFi triggers all
of them in rapid succession.

The Connection Timeout property in PutSFTP controls how long the processor
will wait before throwing the ClientConnectException. The number of
Concurrent Tasks for each PutSFTP processor also influences the number of
simultaneous connections. Increasing the Connection Timeout in PutSFTP may
hide the problem, but if the destination SFTP server can handle the load,
it may be helpful to increase the number of maximum sessions.

On the other hand, is there a reason for having 50 separate instances of
PutSFTP sending to the same server? It should be possible to design the
flow and parameterize the destination using FlowFile attributes. Depending
on the number of CPU cores and available threads, having more SFTP
connections results in poor performance, so smaller numbers can be better.

Regards,
David Handermann

[1] https://linux.die.net/man/5/sshd_config

On Thu, Feb 16, 2023 at 1:33 AM  wrote:

> Hi guys
>
>
>
> It was upgrade time again on our side, we just upgraded from 1.19.1 to
> 1.20.0. Since 1.20.0 we do see significantly more SSH Connection Timeout
> errors on the PutSFTP processor…
>
>
>
> PutSFTP Processor ERROR:
>
> 2023-02-16 07:44:07,905 ERROR [Timer-Driven Process Thread-50]
> o.a.nifi.processors.standard.PutSFTP PutSFTP[
> id=12563431-c40a-1af7-b09b-16de27d887b7] Unable to transfer
> StandardFlowFileRecord[uuid=a1adadb1-61f9-414f-99e5-aad4331165ef,
> claim=StandardContentClaim [resourceClaim=StandardResourceClaim[
> id=1676529847583-196507, container=default, section=923], offset=0,
> length=6582249],offset=0,name=myfile.avro.gz,size=6582249] to remote host
> nas.local.com due to
> org.apache.nifi.processors.standard.socket.ClientConnectException: SSH
> Client connection failed [nas.local.com:22]:
> net.schmizz.sshj.transport.TransportException: Timeout expired: 3
> MILLISECONDS; routing to failure
> net.schmizz.sshj.transport.TransportException: Timeout expired: 3
> MILLISECONDS
>
>
>
> I know that David Handermann implemented a fix (
> https://issues.apache.org/jira/browse/NIFI-9989) for SSHJ, but I don’t
> know if it really is related. May be it’s just a configuration issue
> (number of allowed concurrent connections?) on the SFTP Server side.
>
> Let’s make an example, let’s say we do have 50 PutSFTP processors to the
> same destination and all of them are getting data in an interval of 15min.
> Does NiFi keep the SSH connection established for this 50 processors or
> will it be closed after each flow has been transferred? If it isn’t closed
> after each flow, how can we influence the timeout? I see only the two
> timeouts below, which let me assume that it’s closed after each flow… But
> may be one of you guys can bring some light into the dark.
>
>
>
> [image: Background pattern Description automatically generated with low
> confidence]
>
>
>
>
>
> Cheers
>
> Josef
>

Re: NiFi docker container fails to start on VMWare host

[David Early via users]

It was because of this:

https://stackoverflow.com/questions/72841549/container-fails-to-start-insufficient-memory-for-the-java-runtime-environment-t#:~:text=Possible%20reasons%3A%20The%20system%20is,physical%20memory%20or%20swap%20space

The customer had installed docker 18.xx, and we have not had to specify the
version before and this new NiFi 1.19.X version was put out with minimal
testing on our servers (i.e. didn't check with older docker versions.

Upgrading to docker 20.10.13 resolved the issue.

Thanks!


On Mon, Jan 30, 2023 at 11:52 PM Chris Sampson 
wrote:

> Have you tried an earlier version of nifi in the same environment?
>
> Does the image for nifi 1.18.0 boot successfully, for example?
>
> The reason I'd check this is mainly that the apache/nifi convenience
> docker image switched from java 8 to java 11 for nifi version 1.19.0+. I
> think the way in which memory is allocated to the jvm within a docker
> container changed between java 8 and 11, so it might be worth checking to
> see whether an earlier version works.
>
> Is the VMWare environment 64 or 32bit as I think that can also affect how
> the jvm allocates memory (based on a brief search for parts of your error
> message - https://www.baeldung.com/jvm-compressed-oops)?
>
>
> On Tue, 31 Jan 2023, 03:36 Cannon Palms,  wrote:
>
>> Check the resources on the host.
>>
>> By default, the docker container will have all of the available memory on
>> the host machine available to it. If you'd like to ensure that this
>> available memory is at least X, you can use a memory reservation in your
>> docker compose file:
>>
>> ```
>> ...
>> mem_reservation: 4Gi
>> ```
>>
>> etc.
>>
>> You are correct, though, that this is not a Nifi-specific question. You
>> may have more success on a docker-specific forum or platform.
>>
>> Best,
>> Cannon
>>
>>
>> On Mon, Jan 30, 2023, 8:27 PM David Early via users <
>> users@nifi.apache.org> wrote:
>>
>>> Hi all,
>>>
>>> We are deploying several NiFi containers as part of a service for a
>>> customer and we have run into an issue we have not seen before.
>>>
>>> I THINK the problem may be more of a problem with Java/Docker/VMWare
>>> than NiFi per se, but I wanted to ask here in case someone has seen this
>>> before.
>>>
>>> We recently upgraded a small system in Azure to 1.19.1 using the
>>> "official" docker container.  This worked just fine.  The host was an Azure
>>> VM running an older CentOS 7 image.  There were no issues.
>>>
>>> For the current install, we used the same Docker image but it is on prem
>>> with a customer who is using VMWare to create the hosts we are running on.
>>> The hosts (3 of them) are 8 core/64G.
>>>
>>> The error we are getting is:
>>>
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /opt/nifi/nifi-current/conf/bootstrap.conf
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /opt/nifi/nifi-current/conf/bootstrap.conf
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /opt/nifi/nifi-current/conf/nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /opt/nifi/nifi-current/conf/nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /opt/nifi/nifi-current/conf/nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /opt/nifi/nifi-current/conf/nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /opt/nifi/nifi-current/conf/nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /opt/nifi/nifi-current/conf/nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /opt/nifi/nifi-current/conf/nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /home/nifi/.nifi-cli.nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /home/nifi/.nifi-cli.nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /home/nifi/.nifi-cli.nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /home/nifi/.nifi-cli.nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /home/nifi/.nifi-cli.nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /opt/nifi/nifi-current/conf/nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /opt/nifi/nifi-current/conf/nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  /opt/nifi/nifi-current/conf/nifi.properties
 Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
 file  

Re: NiFi docker container fails to start on VMWare host

[Chris Sampson]

Have you tried an earlier version of nifi in the same environment?

Does the image for nifi 1.18.0 boot successfully, for example?

The reason I'd check this is mainly that the apache/nifi convenience docker
image switched from java 8 to java 11 for nifi version 1.19.0+. I think the
way in which memory is allocated to the jvm within a docker container
changed between java 8 and 11, so it might be worth checking to see whether
an earlier version works.

Is the VMWare environment 64 or 32bit as I think that can also affect how
the jvm allocates memory (based on a brief search for parts of your error
message - https://www.baeldung.com/jvm-compressed-oops)?


On Tue, 31 Jan 2023, 03:36 Cannon Palms,  wrote:

> Check the resources on the host.
>
> By default, the docker container will have all of the available memory on
> the host machine available to it. If you'd like to ensure that this
> available memory is at least X, you can use a memory reservation in your
> docker compose file:
>
> ```
> ...
> mem_reservation: 4Gi
> ```
>
> etc.
>
> You are correct, though, that this is not a Nifi-specific question. You
> may have more success on a docker-specific forum or platform.
>
> Best,
> Cannon
>
>
> On Mon, Jan 30, 2023, 8:27 PM David Early via users 
> wrote:
>
>> Hi all,
>>
>> We are deploying several NiFi containers as part of a service for a
>> customer and we have run into an issue we have not seen before.
>>
>> I THINK the problem may be more of a problem with Java/Docker/VMWare than
>> NiFi per se, but I wanted to ask here in case someone has seen this before.
>>
>> We recently upgraded a small system in Azure to 1.19.1 using the
>> "official" docker container.  This worked just fine.  The host was an Azure
>> VM running an older CentOS 7 image.  There were no issues.
>>
>> For the current install, we used the same Docker image but it is on prem
>> with a customer who is using VMWare to create the hosts we are running on.
>> The hosts (3 of them) are 8 core/64G.
>>
>> The error we are getting is:
>>
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/bootstrap.conf
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/bootstrap.conf
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /home/nifi/.nifi-cli.nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /home/nifi/.nifi-cli.nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /home/nifi/.nifi-cli.nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /home/nifi/.nifi-cli.nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /home/nifi/.nifi-cli.nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>>> file  /opt/nifi/nifi-current/conf/nifi.properties
>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: 

Re: NiFi docker container fails to start on VMWare host

[Cannon Palms]

Check the resources on the host.

By default, the docker container will have all of the available memory on
the host machine available to it. If you'd like to ensure that this
available memory is at least X, you can use a memory reservation in your
docker compose file:

```
...
mem_reservation: 4Gi
```

etc.

You are correct, though, that this is not a Nifi-specific question. You may
have more success on a docker-specific forum or platform.

Best,
Cannon


On Mon, Jan 30, 2023, 8:27 PM David Early via users 
wrote:

> Hi all,
>
> We are deploying several NiFi containers as part of a service for a
> customer and we have run into an issue we have not seen before.
>
> I THINK the problem may be more of a problem with Java/Docker/VMWare than
> NiFi per se, but I wanted to ask here in case someone has seen this before.
>
> We recently upgraded a small system in Azure to 1.19.1 using the
> "official" docker container.  This worked just fine.  The host was an Azure
> VM running an older CentOS 7 image.  There were no issues.
>
> For the current install, we used the same Docker image but it is on prem
> with a customer who is using VMWare to create the hosts we are running on.
> The hosts (3 of them) are 8 core/64G.
>
> The error we are getting is:
>
> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file
>>  /opt/nifi/nifi-current/conf/bootstrap.conf
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/bootstrap.conf
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /home/nifi/.nifi-cli.nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /home/nifi/.nifi-cli.nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /home/nifi/.nifi-cli.nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /home/nifi/.nifi-cli.nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /home/nifi/.nifi-cli.nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  /opt/nifi/nifi-current/conf/nifi.properties
>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target
>> file  

Re: Nifi flowfile monitoring

[Guillermo Muñoz]

Hi,

We have a seven nodes cluster with 1.16.3. We haven't been able to
reproduce it. We just know that only it happens in the same nodes of the
cluster, but we havent seen any difference between them.

All the processor groups have only one input port and one output port. We
have single batch per node as "Process Group FlowFile Concurrency"
configuration and Batch output as Process Group Outbound Policy.

Thanks. Regards.

El lun, 9 ene 2023 a las 11:46, Pierre Villard ()
escribió:

> Hi,
>
> We should figure out why this happens as it should not. Which version are
> you using? Is it something you've been able to reproduce?
> How are flow files getting into the process group, and how are they going
> out? Multiple input ports? Multiple output ports?
> Also you configured "Batch" for the outbound policy but what did you
> configure for the flow file concurrency at process group level?
>
> Pierre
>
>
> Le lun. 9 janv. 2023 à 11:38, Guillermo Muñoz Salgado 
> a écrit :
>
>> Hi,
>>
>> Sometimes we have flowfiles stuck without reason in queues before out
>> ports in consumer groups with batch output flowfile concurrency configured.
>> We don't know why, and we haven't been able to find out where the problem
>> is. So our operation teams has to check periodically, and change the
>> flowfile concurrency to streaming to unstack these flowfiles.
>>
>> Thanks to all.
>>
>> El lun, 2 ene 2023 a las 10:02, Pierre Villard (<
>> pierre.villard...@gmail.com>) escribió:
>>
>>> I think it'd be good to take a step back. How can a flow file get stuck
>>> in a relationship? 1 - Is it because you have a self loop failure
>>> relationship? 2 - Is it because you have "dead ends" with a failure
>>> relationship going to a funnel? 3 - or is it because a flow file is failing
>>> and is being rolled back in the input relationship?
>>> With 1, you should leverage the retry feature that has been recently
>>> added
>>> With 2, this is a bad practice but you could indeed using reporting
>>> tasks to check flow files in those specific queues
>>> With 3, bulletins would be generated and I recommend having monitoring
>>> in place for the bulletins (with reporting tasks)
>>>
>>> Le sam. 31 déc. 2022 à 16:49, Jorge Machado  a écrit :
>>>
 Activate prometheus exporter and you will have a metric for  number of
 objects in the queue

 On 31. Dec 2022, at 07:08, nayan sharma 
 wrote:

 Hi Users,
 Is there anyway through which I can monitor or raise alert if any flow
 file got stuck in nifi queue.

 For now operation team needs to manually check for these. If you can
 suggest way through which I can achieve this that would be great.

 Thanks,
 Nayan
 --
 Thanks & Regards,
 Nayan Sharma
  *+91-8095382952*

 
 



>>
>> --
>> Guillermo Muñoz Salgado
>>
>

-- 
Guillermo Muñoz Salgado

Re: Nifi flowfile monitoring

[Pierre Villard]

Hi,

We should figure out why this happens as it should not. Which version are
you using? Is it something you've been able to reproduce?
How are flow files getting into the process group, and how are they going
out? Multiple input ports? Multiple output ports?
Also you configured "Batch" for the outbound policy but what did you
configure for the flow file concurrency at process group level?

Pierre


Le lun. 9 janv. 2023 à 11:38, Guillermo Muñoz Salgado  a
écrit :

> Hi,
>
> Sometimes we have flowfiles stuck without reason in queues before out
> ports in consumer groups with batch output flowfile concurrency configured.
> We don't know why, and we haven't been able to find out where the problem
> is. So our operation teams has to check periodically, and change the
> flowfile concurrency to streaming to unstack these flowfiles.
>
> Thanks to all.
>
> El lun, 2 ene 2023 a las 10:02, Pierre Villard (<
> pierre.villard...@gmail.com>) escribió:
>
>> I think it'd be good to take a step back. How can a flow file get stuck
>> in a relationship? 1 - Is it because you have a self loop failure
>> relationship? 2 - Is it because you have "dead ends" with a failure
>> relationship going to a funnel? 3 - or is it because a flow file is failing
>> and is being rolled back in the input relationship?
>> With 1, you should leverage the retry feature that has been recently added
>> With 2, this is a bad practice but you could indeed using reporting tasks
>> to check flow files in those specific queues
>> With 3, bulletins would be generated and I recommend having monitoring in
>> place for the bulletins (with reporting tasks)
>>
>> Le sam. 31 déc. 2022 à 16:49, Jorge Machado  a écrit :
>>
>>> Activate prometheus exporter and you will have a metric for  number of
>>> objects in the queue
>>>
>>> On 31. Dec 2022, at 07:08, nayan sharma  wrote:
>>>
>>> Hi Users,
>>> Is there anyway through which I can monitor or raise alert if any flow
>>> file got stuck in nifi queue.
>>>
>>> For now operation team needs to manually check for these. If you can
>>> suggest way through which I can achieve this that would be great.
>>>
>>> Thanks,
>>> Nayan
>>> --
>>> Thanks & Regards,
>>> Nayan Sharma
>>>  *+91-8095382952*
>>>
>>> 
>>> 
>>>
>>>
>>>
>
> --
> Guillermo Muñoz Salgado
>

Re: Nifi flowfile monitoring

[Guillermo Muñoz Salgado]

Hi,

Sometimes we have flowfiles stuck without reason in queues before out ports
in consumer groups with batch output flowfile concurrency configured. We
don't know why, and we haven't been able to find out where the problem is.
So our operation teams has to check periodically, and change the flowfile
concurrency to streaming to unstack these flowfiles.

Thanks to all.

El lun, 2 ene 2023 a las 10:02, Pierre Villard ()
escribió:

> I think it'd be good to take a step back. How can a flow file get stuck in
> a relationship? 1 - Is it because you have a self loop failure
> relationship? 2 - Is it because you have "dead ends" with a failure
> relationship going to a funnel? 3 - or is it because a flow file is failing
> and is being rolled back in the input relationship?
> With 1, you should leverage the retry feature that has been recently added
> With 2, this is a bad practice but you could indeed using reporting tasks
> to check flow files in those specific queues
> With 3, bulletins would be generated and I recommend having monitoring in
> place for the bulletins (with reporting tasks)
>
> Le sam. 31 déc. 2022 à 16:49, Jorge Machado  a écrit :
>
>> Activate prometheus exporter and you will have a metric for  number of
>> objects in the queue
>>
>> On 31. Dec 2022, at 07:08, nayan sharma  wrote:
>>
>> Hi Users,
>> Is there anyway through which I can monitor or raise alert if any flow
>> file got stuck in nifi queue.
>>
>> For now operation team needs to manually check for these. If you can
>> suggest way through which I can achieve this that would be great.
>>
>> Thanks,
>> Nayan
>> --
>> Thanks & Regards,
>> Nayan Sharma
>>  *+91-8095382952*
>>
>> 
>> 
>>
>>
>>

-- 
Guillermo Muñoz Salgado

Re: NiFi failing to start

[James McMahon]

Russ and Chris, thank you for responding. I got this to work. It turned out
to be a combination of different issues. One was finding the proper
arguments for tls-toolkit so that my CN would be set properly in my certs.
The second was configuration settings in my nifi.properties file.

For that first challenge, these were the tls-toolkit commands I used that
worked:
./bin/tls-toolkit.sh standalone -n 'ec2-52-4-149-72.compute-1.amazonaws.com'
--certificateAuthorityHostname 'ec2-52-4-149-72.compute-1.amazonaws.com'
./bin/tls-toolkit.sh standalone -B  -C "CN=admin, OU=NIFI"
Once I did this, I used openssl to query my pkcs12 file, verifying that my
CN now properly included ec2-52-4-149-72.compute-1.amazonaws.com rather
than *localhost*. This is an excerpt of what I found:

Command: openssl pkcs12 -info -nodes -in CN=admin_OU=NIFI.p12

Enter Import Password:

MAC Iteration 102400

MAC verified OK

PKCS7 Data

...

*subject=/OU=NIFI/CN=admin*

*issuer=/OU=NIFI/CN=ec2-52-4-149-72.compute-1.amazonaws.com
*

-BEGIN CERTIFICATE-

...

-END CERTIFICATE-

Certificate bag

Bag Attributes: 

*subject=/OU=NIFI/CN=ec2-52-4-149-72.compute-1.amazonaws.com
*

*issuer=/OU=NIFI/CN=ec2-52-4-149-72.compute-1.amazonaws.com
*

-BEGIN CERTIFICATE-

…..

-END CERTIFICATE-

For that second impediment, I eventually got it to work using these entries
in my nifi.properties:
nifi.authorizer.configuration.file=/opt/nifi/config_resources/authorizers.xml
#nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.login.identity.provider.configuration.file=

I'm still not entirely sure why I don't need to have a
login-identity-providers.xml file, but suspect it is because I now employ
TLS to present my identity, maybe?
My authorizers.xml looks like this:




file-user-group-provider
org.apache.nifi.authorization.FileUserGroupProvider
./conf/users.xml


CN=admin,
OU=NiFi


file-access-policy-provider

org.apache.nifi.authorization.FileAccessPolicyProvider
file-user-group-provider
./conf/authorizations.xml
CN=admin, OU=NiFi

CN=admin, OU=NiFi



managed-authorizer

org.apache.nifi.authorization.StandardManagedAuthorizer
file-access-policy-provider



Thanks again for your replies.
Jim


On Wed, Dec 28, 2022 at 2:48 PM Russell Bateman 
wrote:

> In case you or someone else wishes only to run, develop, start, stop,
> start over, etc., and doesn't care to authenticate a (non-production)
> installation, I have followed this since NiFi 1.14 and last used it for
> 1.19:
>
> https://www.javahotchocolate.com/notes/nifi.html#20210716
>
> If this doesn't work it's usually because the properties file has become
> too modified. Start over with a fresh download.
>
> Russ
>
>
> On 12/28/22 12:03, Chris Sampson wrote:
>
> I think you will need to remove/comment out the references to
> single-user-provider in authorisers.xml and login-providers.xml as well as
> removing it from nifi.properties (see the comments in these files as
> they're provided in the nifi distributions).
>
> If you are using 2-way TLS authentication then I don't think you need to
> configure anything else, but remember that all of your nifi instances in
> your cluster (if applicable) will need to trust one another's certificates
> along with all user certificates - the easiest way of doing this is
> typically to trust a common CA that issues all the nifi instance and user
> certs. This could be nifi-toolkit, but beware that the CA used by toolkit
> is auto-generated on startup, so you need to retain and configure the same
> CA for toolkit of you plan to use it to issue new certs in future.
>
> On Wed, 28 Dec 2022, 17:32 James McMahon,  wrote:
>
>> I continue to experience errors when I try to start my nifi 1.16.3
>> instance. I have followed this guide in an effort to use the toolkit to
>> generate self-0signed certs for user admin, signed by a nifi truststore:
>>
>> Apache NiFi Walkthroughs
>> 
>>
>> I seem to be having issues with this in my nifi.properties:
>> nifi.security.user.authorizer=single-user-authorizer
>>
>> When I set it to nothing, it tells me this is required. When I set it to
>> single-user-authorizer, this error results in the log:
>>  Error creating bean with name 'authorizer': FactoryBean threw exception
>> on object creation; nested exception is java.lang.Exception: The specified
>> authorizer 'single-user-authorizer' could not be found.
>>
>> I suspect my authorizers.xml and/or my login-identity-providers.xml files
>> are misconfigured. How should those two config files be structured if I
>> wish to run a secure nifi instance where mith my self-signed certs,
>> 

Re: Nifi flowfile monitoring

[Pierre Villard]

I think it'd be good to take a step back. How can a flow file get stuck in
a relationship? 1 - Is it because you have a self loop failure
relationship? 2 - Is it because you have "dead ends" with a failure
relationship going to a funnel? 3 - or is it because a flow file is failing
and is being rolled back in the input relationship?
With 1, you should leverage the retry feature that has been recently added
With 2, this is a bad practice but you could indeed using reporting tasks
to check flow files in those specific queues
With 3, bulletins would be generated and I recommend having monitoring in
place for the bulletins (with reporting tasks)

Le sam. 31 déc. 2022 à 16:49, Jorge Machado  a écrit :

> Activate prometheus exporter and you will have a metric for  number of
> objects in the queue
>
> On 31. Dec 2022, at 07:08, nayan sharma  wrote:
>
> Hi Users,
> Is there anyway through which I can monitor or raise alert if any flow
> file got stuck in nifi queue.
>
> For now operation team needs to manually check for these. If you can
> suggest way through which I can achieve this that would be great.
>
> Thanks,
> Nayan
> --
> Thanks & Regards,
> Nayan Sharma
>  *+91-8095382952*
>
> 
> 
>
>
>

Re: Nifi flowfile monitoring

[Jorge Machado]

Activate prometheus exporter and you will have a metric for  number of objects 
in the queue

> On 31. Dec 2022, at 07:08, nayan sharma  wrote:
> 
> Hi Users,
> Is there anyway through which I can monitor or raise alert if any flow file 
> got stuck in nifi queue. 
> 
> For now operation team needs to manually check for these. If you can suggest 
> way through which I can achieve this that would be great.
> 
> Thanks,
> Nayan
> -- 
> Thanks & Regards,
> Nayan Sharma
>  +91-8095382952
> 
>   
> 

Re: NiFi failing to start

[Russell Bateman]

In case you or someone else wishes only to run, develop, start, stop, 
start over, etc., and doesn't care to authenticate a (non-production) 
installation, I have followed this since NiFi 1.14 and last used it for 
1.19:


https://www.javahotchocolate.com/notes/nifi.html#20210716

If this doesn't work it's usually because the properties file has become 
too modified. Start over with a fresh download.


Russ


On 12/28/22 12:03, Chris Sampson wrote:
I think you will need to remove/comment out the references to 
single-user-provider in authorisers.xml and login-providers.xml as 
well as removing it from nifi.properties (see the comments in these 
files as they're provided in the nifi distributions).


If you are using 2-way TLS authentication then I don't think you need 
to configure anything else, but remember that all of your nifi 
instances in your cluster (if applicable) will need to trust one 
another's certificates along with all user certificates - the easiest 
way of doing this is typically to trust a common CA that issues all 
the nifi instance and user certs. This could be nifi-toolkit, but 
beware that the CA used by toolkit is auto-generated on startup, so 
you need to retain and configure the same CA for toolkit of you plan 
to use it to issue new certs in future.


On Wed, 28 Dec 2022, 17:32 James McMahon,  wrote:

I continue to experience errors when I try to start my nifi 1.16.3
instance. I have followed this guide in an effort to use the
toolkit to generate self-0signed certs for user admin, signed by a
nifi truststore:

Apache NiFi Walkthroughs


I seem to be having issues with this in my nifi.properties:
nifi.security.user.authorizer=single-user-authorizer

When I set it to nothing, it tells me this is required. When I set
it to single-user-authorizer, this error results in the log:
 Error creating bean with name 'authorizer': FactoryBean threw
exception on object creation; nested exception is
java.lang.Exception: The specified authorizer
'single-user-authorizer' could not be found.

I suspect my authorizers.xml and/or my
login-identity-providers.xml files are misconfigured. How should
those two config files be structured if I wish to run a secure
nifi instance where mith my self-signed certs, generated using the
nifi toolkit?

Re: NiFi failing to start

[Chris Sampson]

I think you will need to remove/comment out the references to
single-user-provider in authorisers.xml and login-providers.xml as well as
removing it from nifi.properties (see the comments in these files as
they're provided in the nifi distributions).

If you are using 2-way TLS authentication then I don't think you need to
configure anything else, but remember that all of your nifi instances in
your cluster (if applicable) will need to trust one another's certificates
along with all user certificates - the easiest way of doing this is
typically to trust a common CA that issues all the nifi instance and user
certs. This could be nifi-toolkit, but beware that the CA used by toolkit
is auto-generated on startup, so you need to retain and configure the same
CA for toolkit of you plan to use it to issue new certs in future.

On Wed, 28 Dec 2022, 17:32 James McMahon,  wrote:

> I continue to experience errors when I try to start my nifi 1.16.3
> instance. I have followed this guide in an effort to use the toolkit to
> generate self-0signed certs for user admin, signed by a nifi truststore:
>
> Apache NiFi Walkthroughs
> 
>
> I seem to be having issues with this in my nifi.properties:
> nifi.security.user.authorizer=single-user-authorizer
>
> When I set it to nothing, it tells me this is required. When I set it to
> single-user-authorizer, this error results in the log:
>  Error creating bean with name 'authorizer': FactoryBean threw exception
> on object creation; nested exception is java.lang.Exception: The specified
> authorizer 'single-user-authorizer' could not be found.
>
> I suspect my authorizers.xml and/or my login-identity-providers.xml files
> are misconfigured. How should those two config files be structured if I
> wish to run a secure nifi instance where mith my self-signed certs,
> generated using the nifi toolkit?
>

Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue

[Simon Bence]

Hi Josef,

I assigned it to myself and will take a deeper look shortly. Understanding the 
reasons and fixing it might need more time and I am asking for your patience.

Best regards,
Bence

> On 2022. Dec 13., at 13:53,  
>  wrote:
> 
> Hi Bence
>  
> I mentioned yesterday another issue which we thought would be solved after 
> the upgrade steps. Sadly it happened again and after an investigation on our 
> side it seems to be fully reproducible, I’ve opened a Jira Bugticket 
> https://issues.apache.org/jira/browse/NIFI-10973 
> <https://issues.apache.org/jira/browse/NIFI-10973> with more details. It’s as 
> well related to NiFi Registry nested flows…
>  
> Thanks for taking care of it :-).
>  
> Cheers Josef
>  
>  
> From: Simon Bence mailto:simonbence@gmail.com>>
> Reply to: "users@nifi.apache.org <mailto:users@nifi.apache.org>" 
> mailto:users@nifi.apache.org>>
> Date: Monday, 12 December 2022 at 16:40
> To: "users@nifi.apache.org" 
> Subject: Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes 
> issue
>  
> Hi Josef, 
>  
> Thanks for the quick update! Meanwhile I tried to reproduce this situation 
> but based on the known facts I was not able (Note: without any version 
> change). I think it is connected to the version change indeed, but if you 
> face with this ever again, please let me know!
>  
> Regards,
> Bence
> 
> 
>> On 2022. Dec 12., at 16:33, > <mailto:josef.zahn...@swisscom.com>> > <mailto:josef.zahn...@swisscom.com>> wrote:
>>  
>> Ok quick update on this. We investigated the changes on the *.snapshot files 
>> on NiFi Registry. It seems that it’s mainly caused due to the fact that the 
>> NiFi Version changes from 1.18.0 to 1.19.1 and some other facts caused due 
>> to the upgrade. After commiting all the child PGs once the “Show Local 
>> Changes” seems to behave as expected.
>> We have seen another issue caused only on the first commit of the child PGs 
>> on v1.19.1, however I’ll not mention them as I’m guessing that this is 
>> highly related to the last commits with the older NiFi version.
>>  
>> Cheers Josef
>>  
>>  
>> From: "Zahner Josef, GSB-LR-TRW-LI" > <mailto:josef.zahn...@swisscom.com>>
>> Date: Monday, 12 December 2022 at 14:44
>> To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" 
>> mailto:users@nifi.apache.org>>
>> Subject: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes 
>> issue
>>  
>> Hi guys
>>  
>> Sorry that I’ve to bother you again. We just upgraded to NiFi 1.19.1 due to 
>> the major issues with the NiFi Registry Client on NiFi 1.18.0.
>>  
>> The following issue still persists in the new version. Let’s say everything 
>> is up-to-date (parent and child PGs) from NiFi Registry perspective, now I 
>> do change eg. a label directly in the parent PG. I do see on that parent PG, 
>> that most likely every connection/service/processor has changed, even though 
>> I’ve changed only one single label size, please check the screenshot for 
>> NiFi Registry Client under “Show Local Changes”.  
>>  
>> 
>>  
>> Is this a cosmetical issue or do we have to worry about that (we had massive 
>> NiFi registry issue on NiFi 1.18.0) ? Because in theory it makes no sense to 
>> that NiFi tells that everything has changed if only one label changes.
>>  
>> Thanks in advance
>> Josef

Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue

[Josef.Zahner1]

Hi Bence

I mentioned yesterday another issue which we thought would be solved after the 
upgrade steps. Sadly it happened again and after an investigation on our side 
it seems to be fully reproducible, I’ve opened a Jira Bugticket 
https://issues.apache.org/jira/browse/NIFI-10973 with more details. It’s as 
well related to NiFi Registry nested flows…

Thanks for taking care of it :-).

Cheers Josef


From: Simon Bence 
Reply to: "users@nifi.apache.org" 
Date: Monday, 12 December 2022 at 16:40
To: "users@nifi.apache.org" 
Subject: Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes 
issue

Hi Josef,

Thanks for the quick update! Meanwhile I tried to reproduce this situation but 
based on the known facts I was not able (Note: without any version change). I 
think it is connected to the version change indeed, but if you face with this 
ever again, please let me know!

Regards,
Bence


On 2022. Dec 12., at 16:33, 
mailto:josef.zahn...@swisscom.com>> 
mailto:josef.zahn...@swisscom.com>> wrote:

Ok quick update on this. We investigated the changes on the *.snapshot files on 
NiFi Registry. It seems that it’s mainly caused due to the fact that the NiFi 
Version changes from 1.18.0 to 1.19.1 and some other facts caused due to the 
upgrade. After commiting all the child PGs once the “Show Local Changes” seems 
to behave as expected.
We have seen another issue caused only on the first commit of the child PGs on 
v1.19.1, however I’ll not mention them as I’m guessing that this is highly 
related to the last commits with the older NiFi version.

Cheers Josef


From: "Zahner Josef, GSB-LR-TRW-LI" 
mailto:josef.zahn...@swisscom.com>>
Date: Monday, 12 December 2022 at 14:44
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Subject: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue

Hi guys

Sorry that I’ve to bother you again. We just upgraded to NiFi 1.19.1 due to the 
major issues with the NiFi Registry Client on NiFi 1.18.0.

The following issue still persists in the new version. Let’s say everything is 
up-to-date (parent and child PGs) from NiFi Registry perspective, now I do 
change eg. a label directly in the parent PG. I do see on that parent PG, that 
most likely every connection/service/processor has changed, even though I’ve 
changed only one single label size, please check the screenshot for NiFi 
Registry Client under “Show Local Changes”.



Is this a cosmetical issue or do we have to worry about that (we had massive 
NiFi registry issue on NiFi 1.18.0) ? Because in theory it makes no sense to 
that NiFi tells that everything has changed if only one label changes.

Thanks in advance
Josef



smime.p7s
Description: S/MIME Cryptographic Signature

Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue

[Simon Bence]

Hi Josef,

Thanks for the quick update! Meanwhile I tried to reproduce this situation but 
based on the known facts I was not able (Note: without any version change). I 
think it is connected to the version change indeed, but if you face with this 
ever again, please let me know!

Regards,
Bence

> On 2022. Dec 12., at 16:33,  
>  wrote:
> 
> Ok quick update on this. We investigated the changes on the *.snapshot files 
> on NiFi Registry. It seems that it’s mainly caused due to the fact that the 
> NiFi Version changes from 1.18.0 to 1.19.1 and some other facts caused due to 
> the upgrade. After commiting all the child PGs once the “Show Local Changes” 
> seems to behave as expected.
> We have seen another issue caused only on the first commit of the child PGs 
> on v1.19.1, however I’ll not mention them as I’m guessing that this is highly 
> related to the last commits with the older NiFi version.
>  
> Cheers Josef
>  
>  
> From: "Zahner Josef, GSB-LR-TRW-LI"  >
> Date: Monday, 12 December 2022 at 14:44
> To: "users@nifi.apache.org " 
> mailto:users@nifi.apache.org>>
> Subject: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue
>  
> Hi guys
>  
> Sorry that I’ve to bother you again. We just upgraded to NiFi 1.19.1 due to 
> the major issues with the NiFi Registry Client on NiFi 1.18.0.
>  
> The following issue still persists in the new version. Let’s say everything 
> is up-to-date (parent and child PGs) from NiFi Registry perspective, now I do 
> change eg. a label directly in the parent PG. I do see on that parent PG, 
> that most likely every connection/service/processor has changed, even though 
> I’ve changed only one single label size, please check the screenshot for NiFi 
> Registry Client under “Show Local Changes”.  
>  
> 
>  
> Is this a cosmetical issue or do we have to worry about that (we had massive 
> NiFi registry issue on NiFi 1.18.0) ? Because in theory it makes no sense to 
> that NiFi tells that everything has changed if only one label changes.
>  
> Thanks in advance
> Josef

Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

[Mark Payne]

I’ve not followed everything goin on in this thread. But just to offer a bit of 
clarification: If the flow.json.gz exists then the flow.xml.gz is ignored. But 
if the flow.json.gz file is not found, it will automatically fall back to the 
flow.xml.gz

So if you were to remove/rename the flow.json.gz it would definitely pick up 
the XML file and process that. Otherwise, it will ignore it.

Thanks
-Mark


On Dec 7, 2022, at 4:24 AM, Tiago Luís Sebastião (DSI) 
 wrote:

Hi Isha,

I did not delete the flow.xml.gz because I think that since the version 1.16.0, 
if I’m not mistaken, when the flow.json.xml was released it replaced completely 
the flow.xml.gz, so any changes made on flow.xml.gz wouldn’t make any 
difference, so knowing that, I assumed there shouldn’t be anything 
configured/dependent on the xml file. Also, the error didn’t seem related at 
all to the flow, it seemed related to not being able to decrypt the 
flow.json.gz with the old algorithm and encrypt with the new algorithm.

Thanks for the help.

Regards.

Tiago Sebastião

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 7 de dezembro de 2022 08:42
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

Thanks for updating and sharing your solution.

I noticed you only encrypted the flow.json.gz file. Did you also delete the xml 
version in previous attempts?
It makes me wonder if the migrating fails for the flow.xml.gz specifically, 
then deleting the xml during migration might be an easier fix.

Regards,

Isha


Van: Tiago Luís Sebastião (DSI) 
mailto:tiago.luis.sebast...@cgd.pt>>
Verzonden: dinsdag 6 december 2022 17:24
Aan: users@nifi.apache.org<mailto:users@nifi.apache.org>
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi Isha,

I had already tried that and didn’t work also.

But I tried the last thing I wrote on the previous email and it worked… Not 
happy with this solution but the issue (encryption/warnings) is solved I guess.

Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL

With nifi Stopped:

  1.  Encrypt new files on the side:

/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 
-sPasswordUsedOnNifiProperties -x -v


  1.  Backup the flow.json.gz and rename the new one:

mv /apps/nifi-1.18.0/conf/flow.json.gz 
/apps/nifi-1.18.0/conf/flow_bk202212.json.gz
mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz


  1.  Change nifi.properties file:

nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256


  1.  Start nifi


No errors found and no warnings…

Tiago Sebastião

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

I’ve had a similar experience with migrating the flow encryption algorithm and 
in fact, some of them are still on the old one. The nifi.sh commands to update 
the sensitive properties key and algorithm are very tricky to use, because they 
update the nifi.properties file even if the migration fails for the flow.xml.gz 
and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following 
ones to fail because it tried to decrypt using the new algorithm. I needed to 
reset the nifi.properties file everytime.

Another thing I’ve noticed is that it doesn’t support the property file 
protection scheme that NiFi has. If your sensitive props key is encrypted you 
need to enter the raw value and make sure the 
nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.

These steps have mostly worked for me:


  1.  Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another 
dir
  2.  Unprotect the sensitive properties key:

 *   Replace the encrypted key with the raw one
 *   Empty this property: nifi.sensitive.props.key.protected

  1.  Check that the algorithm is still the old one: 
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
  2.  Check the length of the raw key, it needs to be 12 characters or longer 
to migr

RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

[DSI]

Hi Isha,

I did not delete the flow.xml.gz because I think that since the version 1.16.0, 
if I’m not mistaken, when the flow.json.xml was released it replaced completely 
the flow.xml.gz, so any changes made on flow.xml.gz wouldn’t make any 
difference, so knowing that, I assumed there shouldn’t be anything 
configured/dependent on the xml file. Also, the error didn’t seem related at 
all to the flow, it seemed related to not being able to decrypt the 
flow.json.gz with the old algorithm and encrypt with the new algorithm.

Thanks for the help.

Regards.

Tiago Sebastião

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 7 de dezembro de 2022 08:42
To: users@nifi.apache.org
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

Thanks for updating and sharing your solution.

I noticed you only encrypted the flow.json.gz file. Did you also delete the xml 
version in previous attempts?
It makes me wonder if the migrating fails for the flow.xml.gz specifically, 
then deleting the xml during migration might be an easier fix.

Regards,

Isha


Van: Tiago Luís Sebastião (DSI) 
mailto:tiago.luis.sebast...@cgd.pt>>
Verzonden: dinsdag 6 december 2022 17:24
Aan: users@nifi.apache.org<mailto:users@nifi.apache.org>
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi Isha,

I had already tried that and didn’t work also.

But I tried the last thing I wrote on the previous email and it worked… Not 
happy with this solution but the issue (encryption/warnings) is solved I guess.

Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL

With nifi Stopped:

  1.  Encrypt new files on the side:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s 
PasswordUsedOnNifiProperties -x -v


  1.  Backup the flow.json.gz and rename the new one:

mv /apps/nifi-1.18.0/conf/flow.json.gz 
/apps/nifi-1.18.0/conf/flow_bk202212.json.gz

mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz


  1.  Change nifi.properties file:
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256


  1.  Start nifi

No errors found and no warnings…

Tiago Sebastião

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

I’ve had a similar experience with migrating the flow encryption algorithm and 
in fact, some of them are still on the old one. The nifi.sh commands to update 
the sensitive properties key and algorithm are very tricky to use, because they 
update the nifi.properties file even if the migration fails for the flow.xml.gz 
and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following 
ones to fail because it tried to decrypt using the new algorithm. I needed to 
reset the nifi.properties file everytime.

Another thing I’ve noticed is that it doesn’t support the property file 
protection scheme that NiFi has. If your sensitive props key is encrypted you 
need to enter the raw value and make sure the 
nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.

These steps have mostly worked for me:


  1.  Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another 
dir
  2.  Unprotect the sensitive properties key:

 *   Replace the encrypted key with the raw one
 *   Empty this property: nifi.sensitive.props.key.protected

  1.  Check that the algorithm is still the old one: 
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
  2.  Check the length of the raw key, it needs to be 12 characters or longer 
to migrate to the new algorithm.
  3.  If the key is empty or short:

 *   ./bin/nifi.sh set-sensitive-properties-key 
PasswordUsedOnNifiProperties (run from the main dir)
 *   Check the output for any failures and if they occur, revert the 
nifi.properties file and fix any errors
 *   Try to start nifi with the new sensitive properties key

  1.  If it works, stop nifi and update the algorithm:

 *   ./bin/nifi.sh set-sensitive-properties-algorithm 
NIFI_PBKDF2_AES_GCM_256
 *   Check 

RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

[Isha Lamboo]

Hi Tiago,

Thanks for updating and sharing your solution.

I noticed you only encrypted the flow.json.gz file. Did you also delete the xml 
version in previous attempts?
It makes me wonder if the migrating fails for the flow.xml.gz specifically, 
then deleting the xml during migration might be an easier fix.

Regards,

Isha


Van: Tiago Luís Sebastião (DSI) 
Verzonden: dinsdag 6 december 2022 17:24
Aan: users@nifi.apache.org
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi Isha,

I had already tried that and didn’t work also.

But I tried the last thing I wrote on the previous email and it worked… Not 
happy with this solution but the issue (encryption/warnings) is solved I guess.

Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL

With nifi Stopped:

  1.  Encrypt new files on the side:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s 
PasswordUsedOnNifiProperties -x -v


  1.  Backup the flow.json.gz and rename the new one:

mv /apps/nifi-1.18.0/conf/flow.json.gz 
/apps/nifi-1.18.0/conf/flow_bk202212.json.gz

mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz


  1.  Change nifi.properties file:
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256


  1.  Start nifi

No errors found and no warnings…

Tiago Sebastião
Tlf: 218456542 | Ext. 556542 | 
tiago.luis.sebast...@cgd.pt<mailto:tiago.luis.sebast...@cgd.pt>
[cid:image001.jpg@01D90A1F.72B3A4E0]
DSI - Data Office – USI6.2 Unidade de Serviços de Dados
www.cgd.pt<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cgd.pt%2F=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=37%2FWY7BK17LoCCmr7ii1I0w7FcW65j%2F2A43ckkt1mok%3D=0>
Antes de imprimir esta mensagem, pense no meio ambiente.
Esta mensagem e-mail, assim como os ficheiros eventualmente anexos, é reservada 
aos seus destinatários, e pode conter informação confidencial ou estar sujeita 
a restrições legais. Se não é o seu destinatário ou se recebeu esta mensagem 
por motivo de erro, solicitamos que não faça qualquer uso ou divulgação do seu 
conteúdo e proceda à eliminação permanente desta mensagem e respetivos anexos.
Caixa Geral de Depósitos, S.A. | Sede Social: Av. João XXI, 63, 1000-300 LISBOA 
| Capital Social 3.844.143.735,00 € | CRCL e Contribuinte 500 960 046

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

I’ve had a similar experience with migrating the flow encryption algorithm and 
in fact, some of them are still on the old one. The nifi.sh commands to update 
the sensitive properties key and algorithm are very tricky to use, because they 
update the nifi.properties file even if the migration fails for the flow.xml.gz 
and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following 
ones to fail because it tried to decrypt using the new algorithm. I needed to 
reset the nifi.properties file everytime.

Another thing I’ve noticed is that it doesn’t support the property file 
protection scheme that NiFi has. If your sensitive props key is encrypted you 
need to enter the raw value and make sure the 
nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.

These steps have mostly worked for me:


  1.  Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another 
dir
  2.  Unprotect the sensitive properties key:

 *   Replace the encrypted key with the raw one
 *   Empty this property: nifi.sensitive.props.key.protected

  1.  Check that the algorithm is still the old one: 
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
  2.  Check the length of the raw key, it needs to be 12 characters or longer 
to migrate to the new algorithm.
  3.  If the key is empty or short:

 *   ./bin/nifi.sh set-sensitive-properties-key 
PasswordUsedOnNifiProperties (run from the main dir)
 *   Check the output for any failures and if they occur, revert the 
nifi.properties file and fix any errors
 *   Try to start nifi with the new sensitive properties key

  1.  If it works, st

RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

[DSI]

Hi Isha,

I had already tried that and didn’t work also.

But I tried the last thing I wrote on the previous email and it worked… Not 
happy with this solution but the issue (encryption/warnings) is solved I guess.

Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL

With nifi Stopped:

1.   Encrypt new files on the side:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s 
PasswordUsedOnNifiProperties -x -v


2.   Backup the flow.json.gz and rename the new one:

mv /apps/nifi-1.18.0/conf/flow.json.gz 
/apps/nifi-1.18.0/conf/flow_bk202212.json.gz

mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz


3.   Change nifi.properties file:
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256


4.   Start nifi

No errors found and no warnings…

Tiago Sebastião
Tlf: 218456542 | Ext. 556542 | 
tiago.luis.sebast...@cgd.pt<mailto:tiago.luis.sebast...@cgd.pt>
[cid:image002.jpg@01D52816.782219F0]

DSI - Data Office – USI6.2 Unidade de Serviços de Dados
www.cgd.pt<http://www.cgd.pt/>

Antes de imprimir esta mensagem, pense no meio ambiente.
Esta mensagem e-mail, assim como os ficheiros eventualmente anexos, é reservada 
aos seus destinatários, e pode conter informação confidencial ou estar sujeita 
a restrições legais. Se não é o seu destinatário ou se recebeu esta mensagem 
por motivo de erro, solicitamos que não faça qualquer uso ou divulgação do seu 
conteúdo e proceda à eliminação permanente desta mensagem e respetivos anexos.
Caixa Geral de Depósitos, S.A. | Sede Social: Av. João XXI, 63, 1000-300 LISBOA 
| Capital Social 3.844.143.735,00 € | CRCL e Contribuinte 500 960 046

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

I’ve had a similar experience with migrating the flow encryption algorithm and 
in fact, some of them are still on the old one. The nifi.sh commands to update 
the sensitive properties key and algorithm are very tricky to use, because they 
update the nifi.properties file even if the migration fails for the flow.xml.gz 
and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following 
ones to fail because it tried to decrypt using the new algorithm. I needed to 
reset the nifi.properties file everytime.

Another thing I’ve noticed is that it doesn’t support the property file 
protection scheme that NiFi has. If your sensitive props key is encrypted you 
need to enter the raw value and make sure the 
nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.

These steps have mostly worked for me:


  1.  Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another 
dir
  2.  Unprotect the sensitive properties key:

 *   Replace the encrypted key with the raw one
 *   Empty this property: nifi.sensitive.props.key.protected

  1.  Check that the algorithm is still the old one: 
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
  2.  Check the length of the raw key, it needs to be 12 characters or longer 
to migrate to the new algorithm.
  3.  If the key is empty or short:

 *   ./bin/nifi.sh set-sensitive-properties-key 
PasswordUsedOnNifiProperties (run from the main dir)
 *   Check the output for any failures and if they occur, revert the 
nifi.properties file and fix any errors
 *   Try to start nifi with the new sensitive properties key

  1.  If it works, stop nifi and update the algorithm:

 *   ./bin/nifi.sh set-sensitive-properties-algorithm 
NIFI_PBKDF2_AES_GCM_256
 *   Check the output for any failures and if they occur, revert the 
nifi.properties file and fix any errors
 *   Try to start nifi with the new algorithm

  1.  Stop nifi, encrypt the sensitive properties key (or restore it if you 
didn’t need to change the password)

 *   Use the encrypt-config.sh from the nifi-toolkit, specify output files 
so you can check and compare.
 *   Beware: this tool doesn’t seem to know about flow.json.gz, so only use 
it to change the config files.

I hope this helps you find the solution.

Regards,

Isha

Van: Tiago Luís Sebastião (DSI) 
mailto:tiago.luis.sebast...@cgd.pt>>
Verzonden: donderdag 24 november 2022 16:19
Aan: users@nifi.apache.org<mailto:users@nifi.apache.org>
Onderwerp: RE: NiFi 1.18.0 S

Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows

[Kevin Doran]

 I believe as soon as this is merged Joe Witt is planning to prepare the
1.19.1 RC candidate and open the vote. We should have a release by next
week unless something comes up


On Dec 1, 2022 at 07:44:29, josef.zahn...@swisscom.com wrote:

> Hi Bence
>
>
>
> Ok got it, thanks a lot, So we will try to sync as less as possible. Let’s
> hope that we will see a NiFi 1.19.1 soon…
>
>
>
> Cheers Josef
>
>
>
> *From: *Simon Bence 
> *Reply to: *"users@nifi.apache.org" 
> *Date: *Thursday, 1 December 2022 at 13:42
> *To: *"users@nifi.apache.org" 
> *Subject: *Re: NiFi Registry Bug which brakes the flow sync with NiFi
> 1.18.0 (and same version of the registry) on nested flows
>
>
>
> Hi Josef,
>
>
>
> I would like to inform you that there is an adjustment for the ticket I
> mentioned in my previous mail, which is needed for the proper behaviour of
> the function in every case. The change is in PR phase but I hope it will be
> merged soon: https://github.com/apache/nifi/pull/6741 I would not
> recommend to go forward without this additional change!
>
>
>
> Regards,
>
> Bence
>
>
>
> On 2022. Nov 29., at 16:21, Simon Bence  wrote:
>
>
>
> Hi Josef,
>
>
>
> Your welcome! In your case it looks like the information is lost during
> transformations of the internal representation.
>
>
>
> For technical details, here you can take a look on the code changes:
> https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2
>
>
>
> Regards,
>
> Bence
>
>
>
> On 2022. Nov 29., at 16:08,  <
> josef.zahn...@swisscom.com> wrote:
>
>
>
> Hi Bence, Joe
>
>
>
> Thank you guys for your fast response.
>
>
>
> @Bence you are right, the whole issue seems to be related to nested flows
> which we try to sync. We are relying heavily on this as we are doing the
> integration on one NiFi system and then sync it to production via the NiFi
> Registry. Our parent flow contains multiple nested flows.
>
>
>
> Ok, as NIFI-10874 will be included in the next release we will skip the
> 1.19.0 and wait for 1.19.1 / 1.20.0. Sadly it will probably take more than
> a few days until we see the next release... Do you have an idea why the
> flow with the nested flow could lose the storageLocation? Because, we just
> change a small thing in one of the nested flows and suddenly one of the
> other nested flows have been losing it’s storageLocation in the parent flow
> and from this point the whole flow was broken. Now we are scared to change
> anything as it could brake anytime again.
>
>
>
> Cheers Josef
>
>
>
>
>
>
>
> *From: *Simon Bence 
> *Reply to: *"users@nifi.apache.org" 
> *Date: *Tuesday, 29 November 2022 at 13:31
> *To: *"users@nifi.apache.org" 
> *Subject: *Re: NiFi Registry Bug which brakes the flow sync with NiFi
> 1.18.0 (and same version of the registry) on nested flows
>
>
>
> Hi Josef,
>
>
>
> Thanks for your patience!
>
>
>
> I took a deeper look on what you were writing. In general, this is a sign
> for the case where the registry client cannot find the nested flow. As far
> as I understood in your case this happens when you are having a sync. Based
> on this you may be hitting
> https://issues.apache.org/jira/browse/NIFI-10874 which is not part of
> NiFi 1.19.0. The next release should contain the fix that addresses your
> issue.
>
>
>
> Regards,
>
> Bence
>
>
>
>
> On 2022. Nov 28., at 17:48, Simon Bence  wrote:
>
> Hi Josef,
>
> Thank you for raising the attention to this.
>
> The storageLocation is a new concept to generalise the information used to
> locate nested flows in a versioned flow. In general it can be there and
> does not cause issues, as of now NiFi falls back the previous way to locate
> nested flows. As Joe mentioned, there is a fix in 1.19, but I need to
> double check if it relates to this situation. Please give me some time, I
> will find you back.
>
> Regards,
> Bence
>
>
>
> On 2022. Nov 28., at 17:17, Joe Witt  wrote:
>
> Josef
>
> Sorry for the challenges you've hit there.  I do think in 1.18 we had a
> bug/regression as we refactored our registry client.  That issue should be
> resolved in 1.19 which just went live thanks to
> https://issues.apache.org/jira/browse/NIFI-10787.  However, I am not
> positive if this will solve the scenario you've hit now but please if able
> try it out.
>
> Thanks
>
> On Mon, Nov 28, 2022 at 8:50 AM  wrote:
> Hi guys
>
>
>
> We had the following bug twice already and it broke the whole sync with
&g

Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows

[Josef.Zahner1]

Hi Bence

Ok got it, thanks a lot, So we will try to sync as less as possible. Let’s hope 
that we will see a NiFi 1.19.1 soon…

Cheers Josef

From: Simon Bence 
Reply to: "users@nifi.apache.org" 
Date: Thursday, 1 December 2022 at 13:42
To: "users@nifi.apache.org" 
Subject: Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and 
same version of the registry) on nested flows

Hi Josef,

I would like to inform you that there is an adjustment for the ticket I 
mentioned in my previous mail, which is needed for the proper behaviour of the 
function in every case. The change is in PR phase but I hope it will be merged 
soon: https://github.com/apache/nifi/pull/6741 I would not recommend to go 
forward without this additional change!

Regards,
Bence


On 2022. Nov 29., at 16:21, Simon Bence 
mailto:simonbence@gmail.com>> wrote:

Hi Josef,

Your welcome! In your case it looks like the information is lost during 
transformations of the internal representation.

For technical details, here you can take a look on the code changes: 
https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2

Regards,
Bence


On 2022. Nov 29., at 16:08, 
mailto:josef.zahn...@swisscom.com>> 
mailto:josef.zahn...@swisscom.com>> wrote:

Hi Bence, Joe

Thank you guys for your fast response.

@Bence you are right, the whole issue seems to be related to nested flows which 
we try to sync. We are relying heavily on this as we are doing the integration 
on one NiFi system and then sync it to production via the NiFi Registry. Our 
parent flow contains multiple nested flows.

Ok, as NIFI-10874 will be included in the next release we will skip the 1.19.0 
and wait for 1.19.1 / 1.20.0. Sadly it will probably take more than a few days 
until we see the next release... Do you have an idea why the flow with the 
nested flow could lose the storageLocation? Because, we just change a small 
thing in one of the nested flows and suddenly one of the other nested flows 
have been losing it’s storageLocation in the parent flow and from this point 
the whole flow was broken. Now we are scared to change anything as it could 
brake anytime again.

Cheers Josef



From: Simon Bence mailto:simonbence@gmail.com>>
Reply to: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Date: Tuesday, 29 November 2022 at 13:31
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Subject: Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and 
same version of the registry) on nested flows

Hi Josef,

Thanks for your patience!

I took a deeper look on what you were writing. In general, this is a sign for 
the case where the registry client cannot find the nested flow. As far as I 
understood in your case this happens when you are having a sync. Based on this 
you may be hitting https://issues.apache.org/jira/browse/NIFI-10874 which is 
not part of NiFi 1.19.0. The next release should contain the fix that addresses 
your issue.

Regards,
Bence



On 2022. Nov 28., at 17:48, Simon Bence 
mailto:simonbence@gmail.com>> wrote:

Hi Josef,

Thank you for raising the attention to this.

The storageLocation is a new concept to generalise the information used to 
locate nested flows in a versioned flow. In general it can be there and does 
not cause issues, as of now NiFi falls back the previous way to locate nested 
flows. As Joe mentioned, there is a fix in 1.19, but I need to double check if 
it relates to this situation. Please give me some time, I will find you back.

Regards,
Bence



On 2022. Nov 28., at 17:17, Joe Witt 
mailto:joe.w...@gmail.com>> wrote:

Josef

Sorry for the challenges you've hit there.  I do think in 1.18 we had a 
bug/regression as we refactored our registry client.  That issue should be 
resolved in 1.19 which just went live thanks to 
https://issues.apache.org/jira/browse/NIFI-10787.  However, I am not positive 
if this will solve the scenario you've hit now but please if able try it out.

Thanks

On Mon, Nov 28, 2022 at 8:50 AM 
mailto:josef.zahn...@swisscom.com>> wrote:
Hi guys



We had the following bug twice already and it broke the whole sync with the 
NiFi registry for the Flow/PG. First time was directly after we have upgraded 
from NiFi 1.15.3 to 1.18.0, but we ignored it as we thought it could be because 
of the upgrade, however it occurred again after a few NiFi Registry commits on 
NiFi 1.18.0… The error was the following when we tried to change the flow 
version or when we tried to start from scratch with that version from the NiFi 
Registry, so the version was broken in the NiFi Registry:









We investigated the last NiFi Registry Commit and we saw in our GIT repo (we 
sync the “flow_storage” to GIT) on line 4847 that NiFi removed a 
“storageLocation” from another PG which makes no sense at all. W

Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows

[Simon Bence]

Hi Josef,

I would like to inform you that there is an adjustment for the ticket I 
mentioned in my previous mail, which is needed for the proper behaviour of the 
function in every case. The change is in PR phase but I hope it will be merged 
soon: https://github.com/apache/nifi/pull/6741 
<https://github.com/apache/nifi/pull/6741> I would not recommend to go forward 
without this additional change!

Regards,
Bence

> On 2022. Nov 29., at 16:21, Simon Bence  wrote:
> 
> Hi Josef,
> 
> Your welcome! In your case it looks like the information is lost during 
> transformations of the internal representation.
> 
> For technical details, here you can take a look on the code changes: 
> https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2
>  
> <https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2>
> 
> Regards,
> Bence
> 
>> On 2022. Nov 29., at 16:08, > <mailto:josef.zahn...@swisscom.com>> > <mailto:josef.zahn...@swisscom.com>> wrote:
>> 
>> Hi Bence, Joe
>>  
>> Thank you guys for your fast response.
>>  
>> @Bence you are right, the whole issue seems to be related to nested flows 
>> which we try to sync. We are relying heavily on this as we are doing the 
>> integration on one NiFi system and then sync it to production via the NiFi 
>> Registry. Our parent flow contains multiple nested flows.
>>  
>> Ok, as NIFI-10874 will be included in the next release we will skip the 
>> 1.19.0 and wait for 1.19.1 / 1.20.0. Sadly it will probably take more than a 
>> few days until we see the next release... Do you have an idea why the flow 
>> with the nested flow could lose the storageLocation? Because, we just change 
>> a small thing in one of the nested flows and suddenly one of the other 
>> nested flows have been losing it’s storageLocation in the parent flow and 
>> from this point the whole flow was broken. Now we are scared to change 
>> anything as it could brake anytime again.
>>  
>> Cheers Josef
>>  
>>  
>>  
>> From: Simon Bence > <mailto:simonbence@gmail.com>>
>> Reply to: "users@nifi.apache.org <mailto:users@nifi.apache.org>" 
>> mailto:users@nifi.apache.org>>
>> Date: Tuesday, 29 November 2022 at 13:31
>> To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" 
>> mailto:users@nifi.apache.org>>
>> Subject: Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 
>> (and same version of the registry) on nested flows
>>  
>> Hi Josef, 
>>  
>> Thanks for your patience!
>>  
>> I took a deeper look on what you were writing. In general, this is a sign 
>> for the case where the registry client cannot find the nested flow. As far 
>> as I understood in your case this happens when you are having a sync. Based 
>> on this you may be hitting https://issues.apache.org/jira/browse/NIFI-10874 
>> <https://issues.apache.org/jira/browse/NIFI-10874> which is not part of NiFi 
>> 1.19.0. The next release should contain the fix that addresses your issue.
>>  
>> Regards,
>> Bence
>> 
>> 
>>> On 2022. Nov 28., at 17:48, Simon Bence >> <mailto:simonbence@gmail.com>> wrote:
>>> 
>>> Hi Josef,
>>> 
>>> Thank you for raising the attention to this.
>>> 
>>> The storageLocation is a new concept to generalise the information used to 
>>> locate nested flows in a versioned flow. In general it can be there and 
>>> does not cause issues, as of now NiFi falls back the previous way to locate 
>>> nested flows. As Joe mentioned, there is a fix in 1.19, but I need to 
>>> double check if it relates to this situation. Please give me some time, I 
>>> will find you back.
>>> 
>>> Regards,
>>> Bence 
>>> 
>>> 
>>>> On 2022. Nov 28., at 17:17, Joe Witt >>> <mailto:joe.w...@gmail.com>> wrote:
>>>> 
>>>> Josef
>>>> 
>>>> Sorry for the challenges you've hit there.  I do think in 1.18 we had a 
>>>> bug/regression as we refactored our registry client.  That issue should be 
>>>> resolved in 1.19 which just went live thanks to 
>>>> https://issues.apache.org/jira/browse/NIFI-10787 
>>>> <https://issues.apache.org/jira/browse/NIFI-10787>.  However, I am not 
>>>> positive if this will solve the scenario you've hit now but please if able 
>>>> try it out.
>>>> 
>>>> Thanks
>>>> 
>>>

Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows

[Simon Bence]

Hi Josef,

Your welcome! In your case it looks like the information is lost during 
transformations of the internal representation.

For technical details, here you can take a look on the code changes: 
https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2 
<https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2>

Regards,
Bence

> On 2022. Nov 29., at 16:08,  
>  wrote:
> 
> Hi Bence, Joe
>  
> Thank you guys for your fast response.
>  
> @Bence you are right, the whole issue seems to be related to nested flows 
> which we try to sync. We are relying heavily on this as we are doing the 
> integration on one NiFi system and then sync it to production via the NiFi 
> Registry. Our parent flow contains multiple nested flows.
>  
> Ok, as NIFI-10874 will be included in the next release we will skip the 
> 1.19.0 and wait for 1.19.1 / 1.20.0. Sadly it will probably take more than a 
> few days until we see the next release... Do you have an idea why the flow 
> with the nested flow could lose the storageLocation? Because, we just change 
> a small thing in one of the nested flows and suddenly one of the other nested 
> flows have been losing it’s storageLocation in the parent flow and from this 
> point the whole flow was broken. Now we are scared to change anything as it 
> could brake anytime again.
>  
> Cheers Josef
>  
>  
>  
> From: Simon Bence mailto:simonbence@gmail.com>>
> Reply to: "users@nifi.apache.org <mailto:users@nifi.apache.org>" 
> mailto:users@nifi.apache.org>>
> Date: Tuesday, 29 November 2022 at 13:31
> To: "users@nifi.apache.org" 
> Subject: Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 
> (and same version of the registry) on nested flows
>  
> Hi Josef, 
>  
> Thanks for your patience!
>  
> I took a deeper look on what you were writing. In general, this is a sign for 
> the case where the registry client cannot find the nested flow. As far as I 
> understood in your case this happens when you are having a sync. Based on 
> this you may be hitting https://issues.apache.org/jira/browse/NIFI-10874 
> <https://issues.apache.org/jira/browse/NIFI-10874> which is not part of NiFi 
> 1.19.0. The next release should contain the fix that addresses your issue.
>  
> Regards,
> Bence
> 
> 
>> On 2022. Nov 28., at 17:48, Simon Bence > <mailto:simonbence@gmail.com>> wrote:
>> 
>> Hi Josef,
>> 
>> Thank you for raising the attention to this.
>> 
>> The storageLocation is a new concept to generalise the information used to 
>> locate nested flows in a versioned flow. In general it can be there and does 
>> not cause issues, as of now NiFi falls back the previous way to locate 
>> nested flows. As Joe mentioned, there is a fix in 1.19, but I need to double 
>> check if it relates to this situation. Please give me some time, I will find 
>> you back.
>> 
>> Regards,
>> Bence 
>> 
>> 
>>> On 2022. Nov 28., at 17:17, Joe Witt >> <mailto:joe.w...@gmail.com>> wrote:
>>> 
>>> Josef
>>> 
>>> Sorry for the challenges you've hit there.  I do think in 1.18 we had a 
>>> bug/regression as we refactored our registry client.  That issue should be 
>>> resolved in 1.19 which just went live thanks to 
>>> https://issues.apache.org/jira/browse/NIFI-10787.  However, I am not 
>>> positive if this will solve the scenario you've hit now but please if able 
>>> try it out.
>>> 
>>> Thanks
>>> 
>>> On Mon, Nov 28, 2022 at 8:50 AM  wrote:
>>> Hi guys
>>> 
>>>  
>>> 
>>> We had the following bug twice already and it broke the whole sync with the 
>>> NiFi registry for the Flow/PG. First time was directly after we have 
>>> upgraded from NiFi 1.15.3 to 1.18.0, but we ignored it as we thought it 
>>> could be because of the upgrade, however it occurred again after a few NiFi 
>>> Registry commits on NiFi 1.18.0… The error was the following when we tried 
>>> to change the flow version or when we tried to start from scratch with that 
>>> version from the NiFi Registry, so the version was broken in the NiFi 
>>> Registry:
>>> 
>>>  
>>> 
>>> 
>>> 
>>>  
>>> 
>>>  
>>> 
>>> We investigated the last NiFi Registry Commit and we saw in our GIT repo 
>>> (we sync the “flow_storage” to GIT) on line 4847 that NiFi removed a 
>>> “storageLocation” from another PG which makes no sense at all. We c

Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows

[Josef.Zahner1]

Hi Bence, Joe

Thank you guys for your fast response.

@Bence you are right, the whole issue seems to be related to nested flows which 
we try to sync. We are relying heavily on this as we are doing the integration 
on one NiFi system and then sync it to production via the NiFi Registry. Our 
parent flow contains multiple nested flows.

Ok, as NIFI-10874 will be included in the next release we will skip the 1.19.0 
and wait for 1.19.1 / 1.20.0. Sadly it will probably take more than a few days 
until we see the next release... Do you have an idea why the flow with the 
nested flow could lose the storageLocation? Because, we just change a small 
thing in one of the nested flows and suddenly one of the other nested flows 
have been losing it’s storageLocation in the parent flow and from this point 
the whole flow was broken. Now we are scared to change anything as it could 
brake anytime again.

Cheers Josef



From: Simon Bence 
Reply to: "users@nifi.apache.org" 
Date: Tuesday, 29 November 2022 at 13:31
To: "users@nifi.apache.org" 
Subject: Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and 
same version of the registry) on nested flows

Hi Josef,

Thanks for your patience!

I took a deeper look on what you were writing. In general, this is a sign for 
the case where the registry client cannot find the nested flow. As far as I 
understood in your case this happens when you are having a sync. Based on this 
you may be hitting https://issues.apache.org/jira/browse/NIFI-10874 which is 
not part of NiFi 1.19.0. The next release should contain the fix that addresses 
your issue.

Regards,
Bence


On 2022. Nov 28., at 17:48, Simon Bence 
mailto:simonbence@gmail.com>> wrote:

Hi Josef,

Thank you for raising the attention to this.

The storageLocation is a new concept to generalise the information used to 
locate nested flows in a versioned flow. In general it can be there and does 
not cause issues, as of now NiFi falls back the previous way to locate nested 
flows. As Joe mentioned, there is a fix in 1.19, but I need to double check if 
it relates to this situation. Please give me some time, I will find you back.

Regards,
Bence


On 2022. Nov 28., at 17:17, Joe Witt 
mailto:joe.w...@gmail.com>> wrote:

Josef

Sorry for the challenges you've hit there.  I do think in 1.18 we had a 
bug/regression as we refactored our registry client.  That issue should be 
resolved in 1.19 which just went live thanks to 
https://issues.apache.org/jira/browse/NIFI-10787.  However, I am not positive 
if this will solve the scenario you've hit now but please if able try it out.

Thanks

On Mon, Nov 28, 2022 at 8:50 AM  wrote:
Hi guys



We had the following bug twice already and it broke the whole sync with the 
NiFi registry for the Flow/PG. First time was directly after we have upgraded 
from NiFi 1.15.3 to 1.18.0, but we ignored it as we thought it could be because 
of the upgrade, however it occurred again after a few NiFi Registry commits on 
NiFi 1.18.0… The error was the following when we tried to change the flow 
version or when we tried to start from scratch with that version from the NiFi 
Registry, so the version was broken in the NiFi Registry:









We investigated the last NiFi Registry Commit and we saw in our GIT repo (we 
sync the “flow_storage” to GIT) on line 4847 that NiFi removed a 
“storageLocation” from another PG which makes no sense at all. We changed 
nothing there and especially why should NiFi remove only the storageLocation 
line… We have one specialty, as we have nested NiFi Registry flows, so one of 
the flows where the storageLocation has been removed was such a nested flow.











Luckily we were able to resolve the error. We tried to add the line and commit 
it to GIT plus we dropped the the database to repopulate the DB, however it was 
broken again after a commit from NiFi. So we tried to manually create a new 
fake version on bucket.yml in the corresponding bucket folder and added as well 
the line in the snapshot again. We dropped then the DB and restarted NiFi 
Registry and voilà it was working again.



However it was a nightmare to get it working again as the flow was completely 
broken, we couldn’t checkout the affected version at all.



Any thoughts on this? Shall I fill a Jira Ticket? The problem is, we can’t 
really reproduce it, it looks like it happens randomly. As you could imagine, 
we can’t share our template as it contains a lot of confidential material.



Cheers Josef






smime.p7s
Description: S/MIME Cryptographic Signature

Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows

[Simon Bence]

Hi Josef,

Thanks for your patience!

I took a deeper look on what you were writing. In general, this is a sign for 
the case where the registry client cannot find the nested flow. As far as I 
understood in your case this happens when you are having a sync. Based on this 
you may be hitting https://issues.apache.org/jira/browse/NIFI-10874 
 which is not part of NiFi 
1.19.0. The next release should contain the fix that addresses your issue.

Regards,
Bence

> On 2022. Nov 28., at 17:48, Simon Bence  wrote:
> 
> Hi Josef,
> 
> Thank you for raising the attention to this.
> 
> The storageLocation is a new concept to generalise the information used to 
> locate nested flows in a versioned flow. In general it can be there and does 
> not cause issues, as of now NiFi falls back the previous way to locate nested 
> flows. As Joe mentioned, there is a fix in 1.19, but I need to double check 
> if it relates to this situation. Please give me some time, I will find you 
> back.
> 
> Regards,
> Bence 
> 
>> On 2022. Nov 28., at 17:17, Joe Witt  wrote:
>> 
>> Josef
>> 
>> Sorry for the challenges you've hit there.  I do think in 1.18 we had a 
>> bug/regression as we refactored our registry client.  That issue should be 
>> resolved in 1.19 which just went live thanks to 
>> https://issues.apache.org/jira/browse/NIFI-10787.  However, I am not 
>> positive if this will solve the scenario you've hit now but please if able 
>> try it out.
>> 
>> Thanks
>> 
>> On Mon, Nov 28, 2022 at 8:50 AM  wrote:
>> Hi guys
>> 
>>  
>> 
>> We had the following bug twice already and it broke the whole sync with the 
>> NiFi registry for the Flow/PG. First time was directly after we have 
>> upgraded from NiFi 1.15.3 to 1.18.0, but we ignored it as we thought it 
>> could be because of the upgrade, however it occurred again after a few NiFi 
>> Registry commits on NiFi 1.18.0… The error was the following when we tried 
>> to change the flow version or when we tried to start from scratch with that 
>> version from the NiFi Registry, so the version was broken in the NiFi 
>> Registry:
>> 
>>  
>> 
>> 
>> 
>>  
>> 
>>  
>> 
>> We investigated the last NiFi Registry Commit and we saw in our GIT repo (we 
>> sync the “flow_storage” to GIT) on line 4847 that NiFi removed a 
>> “storageLocation” from another PG which makes no sense at all. We changed 
>> nothing there and especially why should NiFi remove only the storageLocation 
>> line… We have one specialty, as we have nested NiFi Registry flows, so one 
>> of the flows where the storageLocation has been removed was such a nested 
>> flow.
>> 
>>  
>> 
>>  
>> 
>> 
>> 
>>  
>> 
>>  
>> 
>> Luckily we were able to resolve the error. We tried to add the line and 
>> commit it to GIT plus we dropped the the database to repopulate the DB, 
>> however it was broken again after a commit from NiFi. So we tried to 
>> manually create a new fake version on bucket.yml in the corresponding bucket 
>> folder and added as well the line in the snapshot again. We dropped then the 
>> DB and restarted NiFi Registry and voilà it was working again.
>> 
>>  
>> 
>> However it was a nightmare to get it working again as the flow was 
>> completely broken, we couldn’t checkout the affected version at all.
>> 
>>  
>> 
>> Any thoughts on this? Shall I fill a Jira Ticket? The problem is, we can’t 
>> really reproduce it, it looks like it happens randomly. As you could 
>> imagine, we can’t share our template as it contains a lot of confidential 
>> material.
>> 
>>  
>> 
>> Cheers Josef
>> 
>>  
>> 
> 

Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows

[Simon Bence]

Hi Josef,

Thank you for raising the attention to this.

The storageLocation is a new concept to generalise the information used to 
locate nested flows in a versioned flow. In general it can be there and does 
not cause issues, as of now NiFi falls back the previous way to locate nested 
flows. As Joe mentioned, there is a fix in 1.19, but I need to double check if 
it relates to this situation. Please give me some time, I will find you back.

Regards,
Bence 

> On 2022. Nov 28., at 17:17, Joe Witt  wrote:
> 
> Josef
> 
> Sorry for the challenges you've hit there.  I do think in 1.18 we had a 
> bug/regression as we refactored our registry client.  That issue should be 
> resolved in 1.19 which just went live thanks to 
> https://issues.apache.org/jira/browse/NIFI-10787 
> .  However, I am not 
> positive if this will solve the scenario you've hit now but please if able 
> try it out.
> 
> Thanks
> 
> On Mon, Nov 28, 2022 at 8:50 AM  > wrote:
> Hi guys
> 
>  
> 
> We had the following bug twice already and it broke the whole sync with the 
> NiFi registry for the Flow/PG. First time was directly after we have upgraded 
> from NiFi 1.15.3 to 1.18.0, but we ignored it as we thought it  could be 
> because of the upgrade, however it occurred again after a few NiFi Registry 
> commits on NiFi 1.18.0… The error was the following when we tried to change 
> the flow version or when we tried to start from scratch with that version 
> from the NiFi Registry, so the version was broken in the NiFi Registry:
> 
>  
> 
> 
> 
>  
> 
>  
> 
> We investigated the last NiFi Registry Commit and we saw in our GIT repo (we 
> sync the “flow_storage” to GIT) on line 4847 that NiFi removed a 
> “storageLocation” from another PG which makes no sense at all. We changed 
> nothing there and especially why should NiFi remove only the storageLocation 
> line… We have one specialty, as we have nested NiFi Registry flows, so one of 
> the flows where the storageLocation has been removed was such a nested flow.
> 
>  
> 
>  
> 
> 
> 
>  
> 
>  
> 
> Luckily we were able to resolve the error. We tried to add the line and 
> commit it to GIT plus we dropped the the database to repopulate the DB, 
> however it was broken again after a commit from NiFi. So we tried to manually 
> create a new fake version on bucket.yml in the corresponding bucket folder 
> and added as well the line in the snapshot again. We dropped then the DB and 
> restarted NiFi Registry and voilà it was working again.
> 
>  
> 
> However it was a nightmare to get it working again as the flow was completely 
> broken, we couldn’t checkout the affected version at all.
> 
>  
> 
> Any thoughts on this? Shall I fill a Jira Ticket? The problem is, we can’t 
> really reproduce it, it looks like it happens randomly. As you could imagine, 
> we can’t share our template as it contains a lot of confidential material.
> 
>  
> 
> Cheers Josef
> 
>  
> 

RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

[Isha Lamboo]

Hi Tiago,

I’ve had a similar experience with migrating the flow encryption algorithm and 
in fact, some of them are still on the old one. The nifi.sh commands to update 
the sensitive properties key and algorithm are very tricky to use, because they 
update the nifi.properties file even if the migration fails for the flow.xml.gz 
and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following 
ones to fail because it tried to decrypt using the new algorithm. I needed to 
reset the nifi.properties file everytime.

Another thing I’ve noticed is that it doesn’t support the property file 
protection scheme that NiFi has. If your sensitive props key is encrypted you 
need to enter the raw value and make sure the 
nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.

These steps have mostly worked for me:


  1.  Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another 
dir
  2.  Unprotect the sensitive properties key:
 *   Replace the encrypted key with the raw one
 *   Empty this property: nifi.sensitive.props.key.protected
  3.  Check that the algorithm is still the old one: 
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
  4.  Check the length of the raw key, it needs to be 12 characters or longer 
to migrate to the new algorithm.
  5.  If the key is empty or short:
 *   ./bin/nifi.sh set-sensitive-properties-key 
PasswordUsedOnNifiProperties (run from the main dir)
 *   Check the output for any failures and if they occur, revert the 
nifi.properties file and fix any errors
 *   Try to start nifi with the new sensitive properties key
  6.  If it works, stop nifi and update the algorithm:
 *   ./bin/nifi.sh set-sensitive-properties-algorithm 
NIFI_PBKDF2_AES_GCM_256
 *   Check the output for any failures and if they occur, revert the 
nifi.properties file and fix any errors
 *   Try to start nifi with the new algorithm
  7.  Stop nifi, encrypt the sensitive properties key (or restore it if you 
didn’t need to change the password)
 *   Use the encrypt-config.sh from the nifi-toolkit, specify output files 
so you can check and compare.
 *   Beware: this tool doesn’t seem to know about flow.json.gz, so only use 
it to change the config files.

I hope this helps you find the solution.

Regards,

Isha

Van: Tiago Luís Sebastião (DSI) 
Verzonden: donderdag 24 november 2022 16:19
Aan: users@nifi.apache.org
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi again,

Sorry for not following up but other priorities came ahead…

Basically it’s still not working, I’ve tried several combinations and I still 
keep getting:
“Failed to process Flow Configuration [./conf/flow.xml.gz]
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm 
[AES/GCM/NoPadding]”

After reading some documentation, for this purpose, assuming that the password 
configured in the nifi.properties file is “PasswordUsedOnNifiProperties”…

I’ve tried and failed:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties

I’ve tried and failed by setting algorithm to empty string:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties

I’ve tried and failed using the new toolkit (I was using toolkit version 
1.13.3):
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.xml.gz -n /apps/nifi-1.18.0/conf/nifi.properties -s 
PasswordUsedOnNifiProperties -A NIFI_ARGON2_AES_GCM_256 -x -v

I’ve tried and failed doing the same but generating new files to debug:
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s 
PasswordUsedOnNifiProperties -x -v

In this last one I noticed that the file flow2.json.gz got its passwords 
encrypted differently and a longer encrypt also.

I’m kind of wondering if I can use this last command to generate these files on 
the side and then manipulate the nifi.properties file by changing the algorithm 
to NIFI_PBKDF2_AES_GCM_256 since it seems it is already encrypted despite the 
known errors/warnings, then I would rename these new files to the older ones 
and start nifi with:

  1.  New flow.json.gz file (apparently encrypted with NIFI_PBKDF2_AES_GCM_256 
algorithm)
  2.  New nifi.properties file (with nifi.sensitive.props.algorithm property 
manipulated to NIFI_PBKDF2_AES_GCM_256)

Since, 
unfortunately<https://eu

RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

[DSI]

Hi again,

Sorry for not following up but other priorities came ahead…

Basically it’s still not working, I’ve tried several combinations and I still 
keep getting:
“Failed to process Flow Configuration [./conf/flow.xml.gz]
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm 
[AES/GCM/NoPadding]”

After reading some documentation, for this purpose, assuming that the password 
configured in the nifi.properties file is “PasswordUsedOnNifiProperties”…

I’ve tried and failed:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties

I’ve tried and failed by setting algorithm to empty string:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties

I’ve tried and failed using the new toolkit (I was using toolkit version 
1.13.3):
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.xml.gz -n /apps/nifi-1.18.0/conf/nifi.properties -s 
PasswordUsedOnNifiProperties -A NIFI_ARGON2_AES_GCM_256 -x -v

I’ve tried and failed doing the same but generating new files to debug:
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s 
PasswordUsedOnNifiProperties -x -v

In this last one I noticed that the file flow2.json.gz got its passwords 
encrypted differently and a longer encrypt also.

I’m kind of wondering if I can use this last command to generate these files on 
the side and then manipulate the nifi.properties file by changing the algorithm 
to NIFI_PBKDF2_AES_GCM_256 since it seems it is already encrypted despite the 
known errors/warnings, then I would rename these new files to the older ones 
and start nifi with:

1.   New flow.json.gz file (apparently encrypted with 
NIFI_PBKDF2_AES_GCM_256 algorithm)

2.   New nifi.properties file (with nifi.sensitive.props.algorithm property 
manipulated to NIFI_PBKDF2_AES_GCM_256)

Since, 
unfortunately<https://www.google.com/search?sxsrf=ALiCzsYtztPHersBtg21lqlpGlc7DZ9CUw:1669302812266=unfortunately=1=X=2ahUKEwiq4pbJjcf7AhXKzqQKHfK8DNwQkeECKAB6BAgGEAE>,
 Im getting nowhere with this and I need to migrate to version 1.18.0 in order 
to apply the bugfix that changes the 
serverConnectorFactory.setNeedClientAuth(wantClientAuth) to 
serverConnectorFactory.setWantClientAuth(wantClientAuth) I am needing help in a 
consequent situation.
In order to reduce the size of the log generated from the deprecation warnings 
(WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider 
Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] 
generate salt requested) I’ve tried to LEVEL OFF that Warning from 
stateless-logback.xml file without success.

On  the tag  and   I changed the tag 
 so that I could see the full class name (although without success 
also…)
%date %level [%thread] %logger{40} %msg%n
to
%date %level [%thread] %logger{140} %msg%n

On the stateless-logback.xml I inserted the following:



It’s not working and I don’t understand why, the class name seems to be correct 
but I keep getting the same WARN.

Sorry for the long email…

Regards.
Tiago Sebastião

From: Tiago Luís Sebastião (DSI)
Sent: 28 de outubro de 2022 09:48
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi David,

It’s a standalone deployment and runs directly on the server.

Yes the command updated the flow.xml.gz/flow.json.gz and nifi.properties 
settings.

Maybe I messed up the nifi.sensitive.props.key, I’ll run some more tests.

Thanks for your help.
 Tiago
From: David Handermann [mailto:exceptionfact...@apache.org]
Sent: 27 de outubro de 2022 16:50
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi Tiago,

The initial warning for the Insecure Cipher Provider Algorithm indicates the 
use of the deprecated setting as mentioned previously.

The set-sensitive-properties-algorithm command looks correct, and should have 
updated the flow.xml.gz, flow.json.gz, and nifi.properties settings.

The Decryption Failed message indicates that the nifi.sensitive.props.key value 
does not match the value used to encrypt the flow configuration, or that the 
algorithm does not match.

Can you provide some additional details about the NiFi installation? Is this a 
standalone or clustered deployment, and is it running in a containerized 
environment, or di

Re: Nifi unable to list/empty queue.

[Mark Payne]

Hello,

For access to data, because it’s considered more sensitive than the flow 
definition, both the user and the nifi node accessing the data must be granted 
permissions to view and modify data. Did you give the nodes permissions to view 
and modify data?

Also moving this from security@ to users@ mailing list, as this is more of a 
use-based question. The security@ mailing list should be used for sensitive 
topics such as potential vulnerabilities, etc.

Thanks
Mark


Sent from my iPhone

On Nov 22, 2022, at 4:58 AM, lemontree <715733...@qq.com> wrote:



Hello

we have secured Nifi cluster in 1.1.7 with 3 nodes. When we click to list or 
empty queue on connection, there is error message



Insufficient Permissions

Node 192.168.106.5:9443 is unable to fulfill this request due to: Unable to 
modify the data for Processor with ID 682706ca-08e4-3d90-9b6a-5f845573299f. 
Contact the system administrator. Contact the system administrator.


the error request:

  1.
请求 URL:
https://192.168.106.5:9443/nifi-api/flowfile-queues/f7b68394-8e95-3e87-902f-90a74d3d8a42/listing-requests
  2.
请求方法:
POST
  3.
状态代码:
403 Forbidden



We grant user policy to view and modify data, but no success. Admin user got 
the same error.message


we use the managed-authorizer configed as this : 
nifi.security.user.authorizer=managed-authorizer;

very strangely, other action policy work normally, such as create 
processor...





Regards

Re: NiFi component status history not showing more than a days worth of data

[Simon Bence]

Hi Michael,

I did not have time to take a deeper look on this, but I had a fix for cleanup 
related issues lately: https://github.com/apache/nifi/pull/6374 


As for the size of the stored data, it might be perfectly normal. The amount of 
data primarily depends on the time window you want to monitor* and the amount 
of components you have in the flow. One of the reasons to add this 
implementations was actually to do not take up a lot of heap space in case of 
bigger flows. As for clusters, as far as I remember all the QuestDB instances 
on all the nodes are working separately, thus you should have the same 
footprint for a single node setup with the same flow. If you experience a 
different behaviour, I would happily take a look.

As for the 24hours, I will take a look.

* Please note that even if you set the time window for 1 day, you might have up 
to two days of data in the QuestDb as the partitioning is day based and it can 
drop only full days, after they run out of the monitored time window.

Regards,
Bence


> On 2022. Nov 16., at 9:20, Garland, Michael R  
> wrote:
> 
> Further investigation has shown that for the Node Status History, in order to 
> get it working, I have had to disconnect 4 out of 6 nodes from the cluster in 
> order to get any node level status history to be displayed without occurring 
> a parse error.  This seems to suggest the QuestDB implementation in NiFi 
> cannot scale well for multi-clustered deployments with many processors.  The 
> total size of the QuestDB files on each node is 377MB, which is after approx. 
> 2 days of NiFi running.
> 
> The issue around component level status history only showing a maximum of 
> 24hrs worth of history still remains.
> 
> Can a bug be raised on this? 
> 
> On 2022/11/14 16:24:04 "Garland, Michael R" wrote: 
> > Hi, 
> > 
> > We're trying to make use of the 'EmbeddedQuestDbStatusHistoryRepository' 
> > implementation of the Status History Repository within NiFi 1.17.0, but 
> > cannot seem to get this feature working as what is described within the 
> > NiFi admin guide.
> 
> > 
> > What we are seeing is that the component Status History viewer within NiFi 
> > only ever displays 1 days' worth of data, regardless of the configuration 
> > of the  'nifi.status.repository.questdb.persist.component.days' value 
> > (leaving the default value of 3)
> 
> > 
> > A second point, clicking on the 'Node Status History' button from the 
> > hamburger symbol doesn't load any information on a 6 node cluster that has 
> > been running for a few days and has ~600MB worth of QuestDB data on each 
> > node (with the 'nifi.status.repository.questdb.persist.node.days' value 
> > left at its default of 14 days).  After approx. 75 seconds, NiFi displays a 
> > parse error, with no information in the logs that would explain the cause.  
> > Has anyone seen this behaviour?
> 
> > 
> > Many thanks, 
> > 
> > Michael 
> > 
> 

RE: NiFi component status history not showing more than a days worth of data

[Garland, Michael R]

Further investigation has shown that for the Node Status History, in order to 
get it working, I have had to disconnect 4 out of 6 nodes from the cluster in 
order to get any node level status history to be displayed without occurring a 
parse error.  This seems to suggest the QuestDB implementation in NiFi cannot 
scale well for multi-clustered deployments with many processors.  The total 
size of the QuestDB files on each node is 377MB, which is after approx. 2 days 
of NiFi running.
The issue around component level status history only showing a maximum of 24hrs 
worth of history still remains.
Can a bug be raised on this?

On 2022/11/14 16:24:04 "Garland, Michael R" wrote:
> Hi,
>
> We're trying to make use of the 'EmbeddedQuestDbStatusHistoryRepository' 
> implementation of the Status History Repository within NiFi 1.17.0, but 
> cannot seem to get this feature working as what is described within the NiFi 
> admin guide.

>
> What we are seeing is that the component Status History viewer within NiFi 
> only ever displays 1 days' worth of data, regardless of the configuration of 
> the  'nifi.status.repository.questdb.persist.component.days' value (leaving 
> the default value of 3)

>
> A second point, clicking on the 'Node Status History' button from the 
> hamburger symbol doesn't load any information on a 6 node cluster that has 
> been running for a few days and has ~600MB worth of QuestDB data on each node 
> (with the 'nifi.status.repository.questdb.persist.node.days' value left at 
> its default of 14 days).  After approx. 75 seconds, NiFi displays a parse 
> error, with no information in the logs that would explain the cause.  Has 
> anyone seen this behaviour?

>
> Many thanks,
>
> Michael
>

Re: Nifi with Nginx(ssl) reverse proxy

[Joe Witt]

Thanks!

On Tue, Nov 15, 2022 at 10:06 AM Ben .T.George 
wrote:

> Hello,
>
> I have achieved this with nginx - reverse proxy method.
>
> Regards,
> Ben
>
> On Sun, Nov 13, 2022 at 7:22 PM Ben .T.George 
> wrote:
>
>> Hello,
>>
>> i have a nifi instance running fine on port 8443, i would like to
>> configure SSL and configureit to listen on 443,
>>
>> i was trying to do that with nginx and somehow it is not working for me.
>>
>> what i would like to achieve
>> User->https(port 443)->nifi(port 8443)
>>
>> below are my conf details
>> # web properties #
>> #
>> nifi.web.http.host=
>> nifi.web.http.port=
>> nifi.web.http.network.interface.default=
>> #
>> nifi.web.https.host=
>> nifi.web.https.port=8443
>> nifi.web.https.network.interface.default=
>> nifi.web.jetty.working.directory=./work/jetty
>> nifi.web.jetty.threads=200
>> nifi.web.max.header.size=16 KB
>> nifi.web.proxy.context.path=
>> nifi.web.proxy.host=
>> nifi.web.max.content.size=
>> nifi.web.max.requests.per.second=3
>> nifi.web.max.access.token.requests.per.second=25
>> nifi.web.request.timeout=60 secs
>> nifi.web.request.ip.whitelist=
>> nifi.web.should.send.server.version=true
>> nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i"
>> "%{User-Agent}i"
>>
>> Nginx:
>>
>> server {
>> listen   443 ssl http2 default_server;
>> server_name sftplx.example.com;
>>
>> ssl_certificate "/etc/pki/nginx/example.com.cer";
>> ssl_certificate_key "/etc/pki/nginx/privatekey.pem";
>> error_page 404 /404.html;
>> location = /40x.html {
>> }
>>
>> error_page 500 502 503 504 /50x.html;
>> location = /50x.html {
>> }
>>
>> location /nifi {
>> proxy_pass https://localhost:8443/nifi;
>> proxy_ssl_server_name on;
>> proxy_set_header Host  sftplx.example.com ;
>> proxy_set_header X-Real-IP $remote_addr;
>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>> proxy_set_header X-Forwarded-Proto $scheme;
>>
>> }
>>
>>
>> Thanks & Regards,
>> Ben
>>
>
>
> --
> Yours Sincerely
> Ben.T.George
> Phone : +965 - 50629829 / 94100799
>
> *" Live like you will die tomorrow, learn like you will live forever "*
>

Re: Nifi with Nginx(ssl) reverse proxy

[Ben .T.George]

Hello,

I have achieved this with nginx - reverse proxy method.

Regards,
Ben

On Sun, Nov 13, 2022 at 7:22 PM Ben .T.George  wrote:

> Hello,
>
> i have a nifi instance running fine on port 8443, i would like to
> configure SSL and configureit to listen on 443,
>
> i was trying to do that with nginx and somehow it is not working for me.
>
> what i would like to achieve
> User->https(port 443)->nifi(port 8443)
>
> below are my conf details
> # web properties #
> #
> nifi.web.http.host=
> nifi.web.http.port=
> nifi.web.http.network.interface.default=
> #
> nifi.web.https.host=
> nifi.web.https.port=8443
> nifi.web.https.network.interface.default=
> nifi.web.jetty.working.directory=./work/jetty
> nifi.web.jetty.threads=200
> nifi.web.max.header.size=16 KB
> nifi.web.proxy.context.path=
> nifi.web.proxy.host=
> nifi.web.max.content.size=
> nifi.web.max.requests.per.second=3
> nifi.web.max.access.token.requests.per.second=25
> nifi.web.request.timeout=60 secs
> nifi.web.request.ip.whitelist=
> nifi.web.should.send.server.version=true
> nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i"
> "%{User-Agent}i"
>
> Nginx:
>
> server {
> listen   443 ssl http2 default_server;
> server_name sftplx.example.com;
>
> ssl_certificate "/etc/pki/nginx/example.com.cer";
> ssl_certificate_key "/etc/pki/nginx/privatekey.pem";
> error_page 404 /404.html;
> location = /40x.html {
> }
>
> error_page 500 502 503 504 /50x.html;
> location = /50x.html {
> }
>
> location /nifi {
> proxy_pass https://localhost:8443/nifi;
> proxy_ssl_server_name on;
> proxy_set_header Host  sftplx.example.com ;
> proxy_set_header X-Real-IP $remote_addr;
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_set_header X-Forwarded-Proto $scheme;
>
> }
>
>
> Thanks & Regards,
> Ben
>


-- 
Yours Sincerely
Ben.T.George
Phone : +965 - 50629829 / 94100799

*" Live like you will die tomorrow, learn like you will live forever "*

Re: NiFi on AWS EC2

[James McMahon]

I used the command
docker exec -it nifi /bin/bash
to review what I understand to be the nifi directories and config files in
the container.

I notice nifi in the container is in many ways not configured to Apache
NiFi recommendations for optimal performance. For example, the
nifi.properties repo params all refer to repos placed on one common disk
device (the one where the container lives, presumably).

I've configured external ebs volumes that I've mounted on my instance. One
for content_repository, one for flowfile_repository, and likewise for
database and provenance repositories. I'd like to have the containerized
nifi write to and read from those so that I don't bottleneck performance
reading and writing to the same device for repos.

I need to persist my changes to nifi config files. How does one avoid
making changes in nifi.properties and the like that are lost when the
docker container is stopped, deleted, and a new one instantiated?

I need to leverage external resources when nifi runs within my container.
How do we direct nifi in the container to use those external resources
outside of the container to host content_repository, etc etc?

Thank you in advance for any help.
Jim

On Tue, Nov 8, 2022 at 10:28 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Jim,
>
> You're welcome! Thanks for following up and confirming the solution, great
> collaborative effort!
>
> Regard,
> David Handermann
>
>
>
>
> On Tue, Nov 8, 2022, 7:25 PM James McMahon  wrote:
>
>> That was it. Adding the port to the docker run command proxy got me to
>> the promised land. I was then able to use the userid and password from the
>> docker log to access nifi on my ec2 instance.
>>
>> David, Dmitry - thank you so much. This was a huge help to me, and I hope
>> it will help others trying the same approach in the future.
>> Jim
>>
>> On Tue, Nov 8, 2022 at 8:13 PM David Handermann <
>> exceptionfact...@apache.org> wrote:
>>
>>> It may also be necessary to include the port in the host variable:
>>>
>>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=
>>> ec2-3-238-27-220.compute-1.amazonaws.com:8443 -d apache/nifi:latest
>>>
>>> It is possible to access the configuration and logs files using an
>>> interactive shell with the following Docker command:
>>>
>>> docker exec -it nifi /bin/bash
>>>
>>> Regards,
>>> David Handermann
>>>
>>> On Tue, Nov 8, 2022 at 7:09 PM Dmitry Stepanov 
>>> wrote:
>>>
 Make sure you use your full domain name
 ec2-3-238-27-220.compute-1.amazonaws.com
 David shorten it in his code

 On November 8, 2022 5:57:26 p.m. James McMahon 
 wrote:

> Thank you, David. I’ve made that change, adding the proxy host
> specification on the docker command line. I continue to get the same error
> message. Is it possible I need to indicate my key on the docker command
> line too?
>
> Related, how can one access nifi.properties and the usual nifi config
> files, as well as the family of nifi-app.log files and bootstrap.conf, 
> when
> nifi is running inside a docker container?
>
> Thanks again for sticking with this. I feel like we’re getting closer.
> Jim
>
> On Tue, Nov 8, 2022 at 7:31 PM David Handermann <
> exceptionfact...@apache.org> wrote:
>
>> Hi Jim,
>>
>> Good adjustment on the security group inbound rules.
>>
>> The error page is the result of NiFi receiving an unexpected HTTP
>> Host header, not matching one of the expected values.
>>
>> For this to work, it is possible to pass the external DNS name as the
>> value of the NIFI_WEB_PROXY_HOST environment variable. This can be
>> specified in the docker run command as follows:
>>
>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
>> amazonaws.com -d apache/nifi:latest
>>
>> That will allow NiFi to accept the Host header from the browser, and
>> then present the login screen.
>>
>> Regards,
>> David Handermann
>>
>> On Tue, Nov 8, 2022 at 6:06 PM James McMahon 
>> wrote:
>>
>>> Hi David. This is very helpful, thank you. I feel like I am close,
>>> but I get an error. My Inbound Rules for my security group now include:
>>> 8443 TCP (MyIP)/32
>>> 443 TCP (MyIP)/32
>>> 22 TCP (MyIP)/32
>>>
>>> In my browser - I tried both Edge and Chrome - I use this
>>> URL:
>>> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
>>> I have also tried with /nifi at the tail end.
>>>
>>> I get this error:
>>>
>>> *System Error*
>>>
>>> *The request contained an invalid host header
>>> [ec2-3-238-27-220.compute-1.amazonaws.com:8443
>>> ] in the request
>>> [/]. Check for request manipulation or third-party intercept.*
>>>
>>> *Valid host headers are [empty] or:*
>>>
>>>- *127.0.0.1*
>>>- *127.0.0.1:8443 

Re: NiFi on AWS EC2

[David Handermann]

Jim,

You're welcome! Thanks for following up and confirming the solution, great
collaborative effort!

Regard,
David Handermann




On Tue, Nov 8, 2022, 7:25 PM James McMahon  wrote:

> That was it. Adding the port to the docker run command proxy got me to the
> promised land. I was then able to use the userid and password from the
> docker log to access nifi on my ec2 instance.
>
> David, Dmitry - thank you so much. This was a huge help to me, and I hope
> it will help others trying the same approach in the future.
> Jim
>
> On Tue, Nov 8, 2022 at 8:13 PM David Handermann <
> exceptionfact...@apache.org> wrote:
>
>> It may also be necessary to include the port in the host variable:
>>
>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=
>> ec2-3-238-27-220.compute-1.amazonaws.com:8443 -d apache/nifi:latest
>>
>> It is possible to access the configuration and logs files using an
>> interactive shell with the following Docker command:
>>
>> docker exec -it nifi /bin/bash
>>
>> Regards,
>> David Handermann
>>
>> On Tue, Nov 8, 2022 at 7:09 PM Dmitry Stepanov 
>> wrote:
>>
>>> Make sure you use your full domain name
>>> ec2-3-238-27-220.compute-1.amazonaws.com
>>> David shorten it in his code
>>>
>>> On November 8, 2022 5:57:26 p.m. James McMahon 
>>> wrote:
>>>
 Thank you, David. I’ve made that change, adding the proxy host
 specification on the docker command line. I continue to get the same error
 message. Is it possible I need to indicate my key on the docker command
 line too?

 Related, how can one access nifi.properties and the usual nifi config
 files, as well as the family of nifi-app.log files and bootstrap.conf, when
 nifi is running inside a docker container?

 Thanks again for sticking with this. I feel like we’re getting closer.
 Jim

 On Tue, Nov 8, 2022 at 7:31 PM David Handermann <
 exceptionfact...@apache.org> wrote:

> Hi Jim,
>
> Good adjustment on the security group inbound rules.
>
> The error page is the result of NiFi receiving an unexpected HTTP Host
> header, not matching one of the expected values.
>
> For this to work, it is possible to pass the external DNS name as the
> value of the NIFI_WEB_PROXY_HOST environment variable. This can be
> specified in the docker run command as follows:
>
> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
> amazonaws.com -d apache/nifi:latest
>
> That will allow NiFi to accept the Host header from the browser, and
> then present the login screen.
>
> Regards,
> David Handermann
>
> On Tue, Nov 8, 2022 at 6:06 PM James McMahon 
> wrote:
>
>> Hi David. This is very helpful, thank you. I feel like I am close,
>> but I get an error. My Inbound Rules for my security group now include:
>> 8443 TCP (MyIP)/32
>> 443 TCP (MyIP)/32
>> 22 TCP (MyIP)/32
>>
>> In my browser - I tried both Edge and Chrome - I use this
>> URL:
>> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
>> I have also tried with /nifi at the tail end.
>>
>> I get this error:
>>
>> *System Error*
>>
>> *The request contained an invalid host header
>> [ec2-3-238-27-220.compute-1.amazonaws.com:8443
>> ] in the request
>> [/]. Check for request manipulation or third-party intercept.*
>>
>> *Valid host headers are [empty] or:*
>>
>>- *127.0.0.1*
>>- *127.0.0.1:8443 *
>>- *localhost*
>>- *localhost:8443*
>>- *[::1]*
>>- *[::1]:8443*
>>- *7f661ae687d7*
>>- *7f661ae687d7:8443*
>>- *172.17.0.2*
>>- *172.17.0.2:8443 *
>>
>>
>> Does this mean I have formed the URL incorrectly?
>>
>> I also see that I had to add an exception to permit https. When I
>> created the instance, I created my own pem key pair. It is not signed by
>> any CA. For a self-signed key pair like this, do I need to install a key 
>> in
>> my browser security store to avoid adding that exception?
>>
>> Thank you for helping me get that much closer.
>> Jim
>>
>> On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
>> exceptionfact...@apache.org> wrote:
>>
>>> Hi Jim,
>>>
>>> Thanks for the reply and additional background.
>>>
>>> The instructions are dated March 2021, which is prior to the release
>>> of NiFi 1.14.0. In particular, the run command is no longer accurate 
>>> with
>>> the default NiFi container image.
>>>
>>> The current Docker Hub instructions [1] show the basic command needed
>>>
>>> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>>>
>>> In addition, any references to port 8080 in the AWS Security Group
>>> rules should be changed to 

Re: NiFi on AWS EC2

[James McMahon]

That was it. Adding the port to the docker run command proxy got me to the
promised land. I was then able to use the userid and password from the
docker log to access nifi on my ec2 instance.

David, Dmitry - thank you so much. This was a huge help to me, and I hope
it will help others trying the same approach in the future.
Jim

On Tue, Nov 8, 2022 at 8:13 PM David Handermann 
wrote:

> It may also be necessary to include the port in the host variable:
>
> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=
> ec2-3-238-27-220.compute-1.amazonaws.com:8443 -d apache/nifi:latest
>
> It is possible to access the configuration and logs files using an
> interactive shell with the following Docker command:
>
> docker exec -it nifi /bin/bash
>
> Regards,
> David Handermann
>
> On Tue, Nov 8, 2022 at 7:09 PM Dmitry Stepanov 
> wrote:
>
>> Make sure you use your full domain name
>> ec2-3-238-27-220.compute-1.amazonaws.com
>> David shorten it in his code
>>
>> On November 8, 2022 5:57:26 p.m. James McMahon 
>> wrote:
>>
>>> Thank you, David. I’ve made that change, adding the proxy host
>>> specification on the docker command line. I continue to get the same error
>>> message. Is it possible I need to indicate my key on the docker command
>>> line too?
>>>
>>> Related, how can one access nifi.properties and the usual nifi config
>>> files, as well as the family of nifi-app.log files and bootstrap.conf, when
>>> nifi is running inside a docker container?
>>>
>>> Thanks again for sticking with this. I feel like we’re getting closer.
>>> Jim
>>>
>>> On Tue, Nov 8, 2022 at 7:31 PM David Handermann <
>>> exceptionfact...@apache.org> wrote:
>>>
 Hi Jim,

 Good adjustment on the security group inbound rules.

 The error page is the result of NiFi receiving an unexpected HTTP Host
 header, not matching one of the expected values.

 For this to work, it is possible to pass the external DNS name as the
 value of the NIFI_WEB_PROXY_HOST environment variable. This can be
 specified in the docker run command as follows:

 docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
 amazonaws.com -d apache/nifi:latest

 That will allow NiFi to accept the Host header from the browser, and
 then present the login screen.

 Regards,
 David Handermann

 On Tue, Nov 8, 2022 at 6:06 PM James McMahon 
 wrote:

> Hi David. This is very helpful, thank you. I feel like I am close, but
> I get an error. My Inbound Rules for my security group now include:
> 8443 TCP (MyIP)/32
> 443 TCP (MyIP)/32
> 22 TCP (MyIP)/32
>
> In my browser - I tried both Edge and Chrome - I use this
> URL:
> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
> I have also tried with /nifi at the tail end.
>
> I get this error:
>
> *System Error*
>
> *The request contained an invalid host header
> [ec2-3-238-27-220.compute-1.amazonaws.com:8443
> ] in the request
> [/]. Check for request manipulation or third-party intercept.*
>
> *Valid host headers are [empty] or:*
>
>- *127.0.0.1*
>- *127.0.0.1:8443 *
>- *localhost*
>- *localhost:8443*
>- *[::1]*
>- *[::1]:8443*
>- *7f661ae687d7*
>- *7f661ae687d7:8443*
>- *172.17.0.2*
>- *172.17.0.2:8443 *
>
>
> Does this mean I have formed the URL incorrectly?
>
> I also see that I had to add an exception to permit https. When I
> created the instance, I created my own pem key pair. It is not signed by
> any CA. For a self-signed key pair like this, do I need to install a key 
> in
> my browser security store to avoid adding that exception?
>
> Thank you for helping me get that much closer.
> Jim
>
> On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
> exceptionfact...@apache.org> wrote:
>
>> Hi Jim,
>>
>> Thanks for the reply and additional background.
>>
>> The instructions are dated March 2021, which is prior to the release
>> of NiFi 1.14.0. In particular, the run command is no longer accurate with
>> the default NiFi container image.
>>
>> The current Docker Hub instructions [1] show the basic command needed
>>
>> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>>
>> In addition, any references to port 8080 in the AWS Security Group
>> rules should be changed to 8443. The security group rules for port 80 and
>> 18080 should be removed.
>>
>> The instructions that allow plain HTTP access to NiFi on port 8080
>> should NEVER be followed, as this exposes unfiltered and unauthenticated
>> access.
>>
>> Following those changes, it should be possible to access the NiFi UI
>> using the AWS URL:
>>

Re: NiFi on AWS EC2

[James McMahon]

Yes sir, I did. I used the full public domain name.

On Tue, Nov 8, 2022 at 8:08 PM Dmitry Stepanov  wrote:

> Make sure you use your full domain name
> ec2-3-238-27-220.compute-1.amazonaws.com
> David shorten it in his code
>
> On November 8, 2022 5:57:26 p.m. James McMahon 
> wrote:
>
>> Thank you, David. I’ve made that change, adding the proxy host
>> specification on the docker command line. I continue to get the same error
>> message. Is it possible I need to indicate my key on the docker command
>> line too?
>>
>> Related, how can one access nifi.properties and the usual nifi config
>> files, as well as the family of nifi-app.log files and bootstrap.conf, when
>> nifi is running inside a docker container?
>>
>> Thanks again for sticking with this. I feel like we’re getting closer.
>> Jim
>>
>> On Tue, Nov 8, 2022 at 7:31 PM David Handermann <
>> exceptionfact...@apache.org> wrote:
>>
>>> Hi Jim,
>>>
>>> Good adjustment on the security group inbound rules.
>>>
>>> The error page is the result of NiFi receiving an unexpected HTTP Host
>>> header, not matching one of the expected values.
>>>
>>> For this to work, it is possible to pass the external DNS name as the
>>> value of the NIFI_WEB_PROXY_HOST environment variable. This can be
>>> specified in the docker run command as follows:
>>>
>>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
>>> amazonaws.com -d apache/nifi:latest
>>>
>>> That will allow NiFi to accept the Host header from the browser, and
>>> then present the login screen.
>>>
>>> Regards,
>>> David Handermann
>>>
>>> On Tue, Nov 8, 2022 at 6:06 PM James McMahon 
>>> wrote:
>>>
 Hi David. This is very helpful, thank you. I feel like I am close, but
 I get an error. My Inbound Rules for my security group now include:
 8443 TCP (MyIP)/32
 443 TCP (MyIP)/32
 22 TCP (MyIP)/32

 In my browser - I tried both Edge and Chrome - I use this
 URL:
 https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
 I have also tried with /nifi at the tail end.

 I get this error:

 *System Error*

 *The request contained an invalid host header
 [ec2-3-238-27-220.compute-1.amazonaws.com:8443
 ] in the request
 [/]. Check for request manipulation or third-party intercept.*

 *Valid host headers are [empty] or:*

- *127.0.0.1*
- *127.0.0.1:8443 *
- *localhost*
- *localhost:8443*
- *[::1]*
- *[::1]:8443*
- *7f661ae687d7*
- *7f661ae687d7:8443*
- *172.17.0.2*
- *172.17.0.2:8443 *


 Does this mean I have formed the URL incorrectly?

 I also see that I had to add an exception to permit https. When I
 created the instance, I created my own pem key pair. It is not signed by
 any CA. For a self-signed key pair like this, do I need to install a key in
 my browser security store to avoid adding that exception?

 Thank you for helping me get that much closer.
 Jim

 On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
 exceptionfact...@apache.org> wrote:

> Hi Jim,
>
> Thanks for the reply and additional background.
>
> The instructions are dated March 2021, which is prior to the release
> of NiFi 1.14.0. In particular, the run command is no longer accurate with
> the default NiFi container image.
>
> The current Docker Hub instructions [1] show the basic command needed
>
> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>
> In addition, any references to port 8080 in the AWS Security Group
> rules should be changed to 8443. The security group rules for port 80 and
> 18080 should be removed.
>
> The instructions that allow plain HTTP access to NiFi on port 8080
> should NEVER be followed, as this exposes unfiltered and unauthenticated
> access.
>
> Following those changes, it should be possible to access the NiFi UI
> using the AWS URL:
>
> https://ec2...amazonaws.com:8443
>
> The default installation will generate a username and password, which
> can be found in the container logs:
>
> docker logs nifi | grep Generated
>
> Regards,
> David Handermann
>
> [1] https://hub.docker.com/r/apache/nifi
>
> On Tue, Nov 8, 2022 at 4:00 PM James McMahon 
> wrote:
>
>> Hi and thank you, David and Dmitry. In my case I was following this
>> example,
>>
>> https://joeygoksu.com/software/apache-nifi-on-aws/
>>
>> which results in NiFi installed within a container. So to answer one
>> of your questions, I don’t yet know how or where to find nifi.properties 
>> in
>> the container framework. I don’t seem to have the usual /opt/nifi/…..
>> directories on my ec2 instance. Any idea where I need to look 

Re: NiFi on AWS EC2

[David Handermann]

It may also be necessary to include the port in the host variable:

docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=
ec2-3-238-27-220.compute-1.amazonaws.com:8443 -d apache/nifi:latest

It is possible to access the configuration and logs files using an
interactive shell with the following Docker command:

docker exec -it nifi /bin/bash

Regards,
David Handermann

On Tue, Nov 8, 2022 at 7:09 PM Dmitry Stepanov  wrote:

> Make sure you use your full domain name
> ec2-3-238-27-220.compute-1.amazonaws.com
> David shorten it in his code
>
> On November 8, 2022 5:57:26 p.m. James McMahon 
> wrote:
>
>> Thank you, David. I’ve made that change, adding the proxy host
>> specification on the docker command line. I continue to get the same error
>> message. Is it possible I need to indicate my key on the docker command
>> line too?
>>
>> Related, how can one access nifi.properties and the usual nifi config
>> files, as well as the family of nifi-app.log files and bootstrap.conf, when
>> nifi is running inside a docker container?
>>
>> Thanks again for sticking with this. I feel like we’re getting closer.
>> Jim
>>
>> On Tue, Nov 8, 2022 at 7:31 PM David Handermann <
>> exceptionfact...@apache.org> wrote:
>>
>>> Hi Jim,
>>>
>>> Good adjustment on the security group inbound rules.
>>>
>>> The error page is the result of NiFi receiving an unexpected HTTP Host
>>> header, not matching one of the expected values.
>>>
>>> For this to work, it is possible to pass the external DNS name as the
>>> value of the NIFI_WEB_PROXY_HOST environment variable. This can be
>>> specified in the docker run command as follows:
>>>
>>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
>>> amazonaws.com -d apache/nifi:latest
>>>
>>> That will allow NiFi to accept the Host header from the browser, and
>>> then present the login screen.
>>>
>>> Regards,
>>> David Handermann
>>>
>>> On Tue, Nov 8, 2022 at 6:06 PM James McMahon 
>>> wrote:
>>>
 Hi David. This is very helpful, thank you. I feel like I am close, but
 I get an error. My Inbound Rules for my security group now include:
 8443 TCP (MyIP)/32
 443 TCP (MyIP)/32
 22 TCP (MyIP)/32

 In my browser - I tried both Edge and Chrome - I use this
 URL:
 https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
 I have also tried with /nifi at the tail end.

 I get this error:

 *System Error*

 *The request contained an invalid host header
 [ec2-3-238-27-220.compute-1.amazonaws.com:8443
 ] in the request
 [/]. Check for request manipulation or third-party intercept.*

 *Valid host headers are [empty] or:*

- *127.0.0.1*
- *127.0.0.1:8443 *
- *localhost*
- *localhost:8443*
- *[::1]*
- *[::1]:8443*
- *7f661ae687d7*
- *7f661ae687d7:8443*
- *172.17.0.2*
- *172.17.0.2:8443 *


 Does this mean I have formed the URL incorrectly?

 I also see that I had to add an exception to permit https. When I
 created the instance, I created my own pem key pair. It is not signed by
 any CA. For a self-signed key pair like this, do I need to install a key in
 my browser security store to avoid adding that exception?

 Thank you for helping me get that much closer.
 Jim

 On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
 exceptionfact...@apache.org> wrote:

> Hi Jim,
>
> Thanks for the reply and additional background.
>
> The instructions are dated March 2021, which is prior to the release
> of NiFi 1.14.0. In particular, the run command is no longer accurate with
> the default NiFi container image.
>
> The current Docker Hub instructions [1] show the basic command needed
>
> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>
> In addition, any references to port 8080 in the AWS Security Group
> rules should be changed to 8443. The security group rules for port 80 and
> 18080 should be removed.
>
> The instructions that allow plain HTTP access to NiFi on port 8080
> should NEVER be followed, as this exposes unfiltered and unauthenticated
> access.
>
> Following those changes, it should be possible to access the NiFi UI
> using the AWS URL:
>
> https://ec2...amazonaws.com:8443
>
> The default installation will generate a username and password, which
> can be found in the container logs:
>
> docker logs nifi | grep Generated
>
> Regards,
> David Handermann
>
> [1] https://hub.docker.com/r/apache/nifi
>
> On Tue, Nov 8, 2022 at 4:00 PM James McMahon 
> wrote:
>
>> Hi and thank you, David and Dmitry. In my case I was following this
>> example,
>>
>> 

Re: NiFi on AWS EC2

[Dmitry Stepanov]

Make sure you use your full domain name
ec2-3-238-27-220.compute-1.amazonaws.com
David shorten it in his code

On November 8, 2022 5:57:26 p.m. James McMahon  wrote:
Thank you, David. I’ve made that change, adding the proxy host 
specification on the docker command line. I continue to get the same error 
message. Is it possible I need to indicate my key on the docker command 
line too?


Related, how can one access nifi.properties and the usual nifi config 
files, as well as the family of nifi-app.log files and bootstrap.conf, when 
nifi is running inside a docker container?


Thanks again for sticking with this. I feel like we’re getting closer.
Jim

On Tue, Nov 8, 2022 at 7:31 PM David Handermann 
 wrote:

Hi Jim,

Good adjustment on the security group inbound rules.

The error page is the result of NiFi receiving an unexpected HTTP Host 
header, not matching one of the expected values.


For this to work, it is possible to pass the external DNS name as the value 
of the NIFI_WEB_PROXY_HOST environment variable. This can be specified in 
the docker run command as follows:


docker run --name nifi -p 8443:8443 -e 
NIFI_WEB_PROXY_HOST=ec2...amazonaws.com -d apache/nifi:latest


That will allow NiFi to accept the Host header from the browser, and then 
present the login screen.


Regards,
David Handermann

On Tue, Nov 8, 2022 at 6:06 PM James McMahon  wrote:
Hi David. This is very helpful, thank you. I feel like I am close, but I 
get an error. My Inbound Rules for my security group now include:

8443 TCP (MyIP)/32
443 TCP (MyIP)/32
22 TCP (MyIP)/32

In my browser - I tried both Edge and Chrome - I use this
URL:
https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
I have also tried with /nifi at the tail end.

I get this error:
System Error
The request contained an invalid host header 
[ec2-3-238-27-220.compute-1.amazonaws.com:8443] in the request [/]. Check 
for request manipulation or third-party intercept.

Valid host headers are [empty] or:
127.0.0.1
127.0.0.1:8443
localhost
localhost:8443
[::1]
[::1]:8443
7f661ae687d7
7f661ae687d7:8443
172.17.0.2
172.17.0.2:8443


Does this mean I have formed the URL incorrectly?


I also see that I had to add an exception to permit https. When I created 
the instance, I created my own pem key pair. It is not signed by any CA. 
For a self-signed key pair like this, do I need to install a key in my 
browser security store to avoid adding that exception?



Thank you for helping me get that much closer.
Jim

On Tue, Nov 8, 2022 at 5:13 PM David Handermann 
 wrote:

Hi Jim,

Thanks for the reply and additional background.

The instructions are dated March 2021, which is prior to the release of 
NiFi 1.14.0. In particular, the run command is no longer accurate with the 
default NiFi container image.


The current Docker Hub instructions [1] show the basic command needed

docker run --name nifi -p 8443:8443 -d apache/nifi:latest

In addition, any references to port 8080 in the AWS Security Group rules 
should be changed to 8443. The security group rules for port 80 and 18080 
should be removed.


The instructions that allow plain HTTP access to NiFi on port 8080 should 
NEVER be followed, as this exposes unfiltered and unauthenticated access.


Following those changes, it should be possible to access the NiFi UI using 
the AWS URL:


https://ec2...amazonaws.com:8443

The default installation will generate a username and password, which can 
be found in the container logs:


docker logs nifi | grep Generated

Regards,
David Handermann

[1] https://hub.docker.com/r/apache/nifi

On Tue, Nov 8, 2022 at 4:00 PM James McMahon  wrote:
Hi and thank you, David and Dmitry. In my case I was following this example,

https://joeygoksu.com/software/apache-nifi-on-aws/

which results in NiFi installed within a container. So to answer one of 
your questions, I don’t yet know how or where to find nifi.properties in 
the container framework. I don’t seem to have the usual /opt/nifi/….. 
directories on my ec2 instance. Any idea where I need to look for that?


These ports are open by my security group Inbound Rules: 22 to MyIP, 80, 
8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.


I am able to Putty into my instance as ec2-user with my ppk file, which I 
created using putty tools from the original pem key pair. When I do putty 
in, under /opt I find three subdirectories: aws, containerd, and rh. 
Nothing nifi under any of the three that I can see so far.


I start my docker instance with this command:
docker run —name nifi -p 18080:8080 -d apache/nifi:latest

I can do a ps -ef and see running nifi processes. But I don’t yet know how 
to get to the nifi logs or properties file.


You mentioned using using localhost to get to the canvas UI. This confuses 
me. Nifi is running on my EC2 instance - a linux host without a browser. 
I’m in a browser on my laptop. How would localhost in my browser get me to 
my EC2 instance running nifi?


This is the URL I’m using in my browser:

Re: NiFi on AWS EC2

[James McMahon]

Thank you, David. I’ve made that change, adding the proxy host
specification on the docker command line. I continue to get the same error
message. Is it possible I need to indicate my key on the docker command
line too?

Related, how can one access nifi.properties and the usual nifi config
files, as well as the family of nifi-app.log files and bootstrap.conf, when
nifi is running inside a docker container?

Thanks again for sticking with this. I feel like we’re getting closer.
Jim

On Tue, Nov 8, 2022 at 7:31 PM David Handermann 
wrote:

> Hi Jim,
>
> Good adjustment on the security group inbound rules.
>
> The error page is the result of NiFi receiving an unexpected HTTP Host
> header, not matching one of the expected values.
>
> For this to work, it is possible to pass the external DNS name as the
> value of the NIFI_WEB_PROXY_HOST environment variable. This can be
> specified in the docker run command as follows:
>
> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
> amazonaws.com -d apache/nifi:latest
>
> That will allow NiFi to accept the Host header from the browser, and then
> present the login screen.
>
> Regards,
> David Handermann
>
> On Tue, Nov 8, 2022 at 6:06 PM James McMahon  wrote:
>
>> Hi David. This is very helpful, thank you. I feel like I am close, but I
>> get an error. My Inbound Rules for my security group now include:
>> 8443 TCP (MyIP)/32
>> 443 TCP (MyIP)/32
>> 22 TCP (MyIP)/32
>>
>> In my browser - I tried both Edge and Chrome - I use this
>> URL:
>> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
>> I have also tried with /nifi at the tail end.
>>
>> I get this error:
>>
>> *System Error*
>>
>> *The request contained an invalid host header
>> [ec2-3-238-27-220.compute-1.amazonaws.com:8443
>> ] in the request
>> [/]. Check for request manipulation or third-party intercept.*
>>
>> *Valid host headers are [empty] or:*
>>
>>- *127.0.0.1*
>>- *127.0.0.1:8443 *
>>- *localhost*
>>- *localhost:8443*
>>- *[::1]*
>>- *[::1]:8443*
>>- *7f661ae687d7*
>>- *7f661ae687d7:8443*
>>- *172.17.0.2*
>>- *172.17.0.2:8443 *
>>
>>
>> Does this mean I have formed the URL incorrectly?
>>
>> I also see that I had to add an exception to permit https. When I created
>> the instance, I created my own pem key pair. It is not signed by any CA.
>> For a self-signed key pair like this, do I need to install a key in my
>> browser security store to avoid adding that exception?
>>
>> Thank you for helping me get that much closer.
>> Jim
>>
>> On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
>> exceptionfact...@apache.org> wrote:
>>
>>> Hi Jim,
>>>
>>> Thanks for the reply and additional background.
>>>
>>> The instructions are dated March 2021, which is prior to the release of
>>> NiFi 1.14.0. In particular, the run command is no longer accurate with the
>>> default NiFi container image.
>>>
>>> The current Docker Hub instructions [1] show the basic command needed
>>>
>>> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>>>
>>> In addition, any references to port 8080 in the AWS Security Group rules
>>> should be changed to 8443. The security group rules for port 80 and 18080
>>> should be removed.
>>>
>>> The instructions that allow plain HTTP access to NiFi on port 8080
>>> should NEVER be followed, as this exposes unfiltered and unauthenticated
>>> access.
>>>
>>> Following those changes, it should be possible to access the NiFi UI
>>> using the AWS URL:
>>>
>>> https://ec2...amazonaws.com:8443
>>>
>>> The default installation will generate a username and password, which
>>> can be found in the container logs:
>>>
>>> docker logs nifi | grep Generated
>>>
>>> Regards,
>>> David Handermann
>>>
>>> [1] https://hub.docker.com/r/apache/nifi
>>>
>>> On Tue, Nov 8, 2022 at 4:00 PM James McMahon 
>>> wrote:
>>>
 Hi and thank you, David and Dmitry. In my case I was following this
 example,

 https://joeygoksu.com/software/apache-nifi-on-aws/

 which results in NiFi installed within a container. So to answer one of
 your questions, I don’t yet know how or where to find nifi.properties in
 the container framework. I don’t seem to have the usual /opt/nifi/…..
 directories on my ec2 instance. Any idea where I need to look for that?

 These ports are open by my security group Inbound Rules: 22 to MyIP,
 80, 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.

 I am able to Putty into my instance as ec2-user with my ppk file, which
 I created using putty tools from the original pem key pair. When I do putty
 in, under /opt I find three subdirectories: aws, containerd, and rh.
 Nothing nifi under any of the three that I can see so far.

 I start my docker instance with this command:
 docker run —name nifi -p 18080:8080 -d apache/nifi:latest

 I can do a ps -ef 

Re: NiFi on AWS EC2

[David Handermann]

Hi Jim,

Good adjustment on the security group inbound rules.

The error page is the result of NiFi receiving an unexpected HTTP Host
header, not matching one of the expected values.

For this to work, it is possible to pass the external DNS name as the value
of the NIFI_WEB_PROXY_HOST environment variable. This can be specified in
the docker run command as follows:

docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
amazonaws.com -d apache/nifi:latest

That will allow NiFi to accept the Host header from the browser, and then
present the login screen.

Regards,
David Handermann

On Tue, Nov 8, 2022 at 6:06 PM James McMahon  wrote:

> Hi David. This is very helpful, thank you. I feel like I am close, but I
> get an error. My Inbound Rules for my security group now include:
> 8443 TCP (MyIP)/32
> 443 TCP (MyIP)/32
> 22 TCP (MyIP)/32
>
> In my browser - I tried both Edge and Chrome - I use this
> URL:
> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
> I have also tried with /nifi at the tail end.
>
> I get this error:
>
> *System Error*
>
> *The request contained an invalid host header
> [ec2-3-238-27-220.compute-1.amazonaws.com:8443
> ] in the request
> [/]. Check for request manipulation or third-party intercept.*
>
> *Valid host headers are [empty] or:*
>
>- *127.0.0.1*
>- *127.0.0.1:8443 *
>- *localhost*
>- *localhost:8443*
>- *[::1]*
>- *[::1]:8443*
>- *7f661ae687d7*
>- *7f661ae687d7:8443*
>- *172.17.0.2*
>- *172.17.0.2:8443 *
>
>
> Does this mean I have formed the URL incorrectly?
>
> I also see that I had to add an exception to permit https. When I created
> the instance, I created my own pem key pair. It is not signed by any CA.
> For a self-signed key pair like this, do I need to install a key in my
> browser security store to avoid adding that exception?
>
> Thank you for helping me get that much closer.
> Jim
>
> On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
> exceptionfact...@apache.org> wrote:
>
>> Hi Jim,
>>
>> Thanks for the reply and additional background.
>>
>> The instructions are dated March 2021, which is prior to the release of
>> NiFi 1.14.0. In particular, the run command is no longer accurate with the
>> default NiFi container image.
>>
>> The current Docker Hub instructions [1] show the basic command needed
>>
>> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>>
>> In addition, any references to port 8080 in the AWS Security Group rules
>> should be changed to 8443. The security group rules for port 80 and 18080
>> should be removed.
>>
>> The instructions that allow plain HTTP access to NiFi on port 8080 should
>> NEVER be followed, as this exposes unfiltered and unauthenticated access.
>>
>> Following those changes, it should be possible to access the NiFi UI
>> using the AWS URL:
>>
>> https://ec2...amazonaws.com:8443
>>
>> The default installation will generate a username and password, which can
>> be found in the container logs:
>>
>> docker logs nifi | grep Generated
>>
>> Regards,
>> David Handermann
>>
>> [1] https://hub.docker.com/r/apache/nifi
>>
>> On Tue, Nov 8, 2022 at 4:00 PM James McMahon 
>> wrote:
>>
>>> Hi and thank you, David and Dmitry. In my case I was following this
>>> example,
>>>
>>> https://joeygoksu.com/software/apache-nifi-on-aws/
>>>
>>> which results in NiFi installed within a container. So to answer one of
>>> your questions, I don’t yet know how or where to find nifi.properties in
>>> the container framework. I don’t seem to have the usual /opt/nifi/…..
>>> directories on my ec2 instance. Any idea where I need to look for that?
>>>
>>> These ports are open by my security group Inbound Rules: 22 to MyIP, 80,
>>> 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
>>>
>>> I am able to Putty into my instance as ec2-user with my ppk file, which
>>> I created using putty tools from the original pem key pair. When I do putty
>>> in, under /opt I find three subdirectories: aws, containerd, and rh.
>>> Nothing nifi under any of the three that I can see so far.
>>>
>>> I start my docker instance with this command:
>>> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>>>
>>> I can do a ps -ef and see running nifi processes. But I don’t yet know
>>> how to get to the nifi logs or properties file.
>>>
>>> You mentioned using using localhost to get to the canvas UI. This
>>> confuses me. Nifi is running on my EC2 instance - a linux host without a
>>> browser. I’m in a browser on my laptop. How would localhost in my browser
>>> get me to my EC2 instance running nifi?
>>>
>>> This is the URL I’m using in my browser:
>>> http://ec2-3-238-27-220.compute-1.amazonaws.com
>>> (that url changes with each Stop/Start of my instance. I’ve yet to
>>> investigate how to get AWS to stop changing that IP, but I know it can be
>>> done).
>>>
>>> The browser replies with: ec2…….amazonaws 

Re: NiFi on AWS EC2

[James McMahon]

Hi David. This is very helpful, thank you. I feel like I am close, but I
get an error. My Inbound Rules for my security group now include:
8443 TCP (MyIP)/32
443 TCP (MyIP)/32
22 TCP (MyIP)/32

In my browser - I tried both Edge and Chrome - I use this
URL:
https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
I have also tried with /nifi at the tail end.

I get this error:

*System Error*

*The request contained an invalid host header
[ec2-3-238-27-220.compute-1.amazonaws.com:8443
] in the request
[/]. Check for request manipulation or third-party intercept.*

*Valid host headers are [empty] or:*

   - *127.0.0.1*
   - *127.0.0.1:8443 *
   - *localhost*
   - *localhost:8443*
   - *[::1]*
   - *[::1]:8443*
   - *7f661ae687d7*
   - *7f661ae687d7:8443*
   - *172.17.0.2*
   - *172.17.0.2:8443 *


Does this mean I have formed the URL incorrectly?

I also see that I had to add an exception to permit https. When I created
the instance, I created my own pem key pair. It is not signed by any CA.
For a self-signed key pair like this, do I need to install a key in my
browser security store to avoid adding that exception?

Thank you for helping me get that much closer.
Jim

On Tue, Nov 8, 2022 at 5:13 PM David Handermann 
wrote:

> Hi Jim,
>
> Thanks for the reply and additional background.
>
> The instructions are dated March 2021, which is prior to the release of
> NiFi 1.14.0. In particular, the run command is no longer accurate with the
> default NiFi container image.
>
> The current Docker Hub instructions [1] show the basic command needed
>
> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>
> In addition, any references to port 8080 in the AWS Security Group rules
> should be changed to 8443. The security group rules for port 80 and 18080
> should be removed.
>
> The instructions that allow plain HTTP access to NiFi on port 8080 should
> NEVER be followed, as this exposes unfiltered and unauthenticated access.
>
> Following those changes, it should be possible to access the NiFi UI using
> the AWS URL:
>
> https://ec2...amazonaws.com:8443
>
> The default installation will generate a username and password, which can
> be found in the container logs:
>
> docker logs nifi | grep Generated
>
> Regards,
> David Handermann
>
> [1] https://hub.docker.com/r/apache/nifi
>
> On Tue, Nov 8, 2022 at 4:00 PM James McMahon  wrote:
>
>> Hi and thank you, David and Dmitry. In my case I was following this
>> example,
>>
>> https://joeygoksu.com/software/apache-nifi-on-aws/
>>
>> which results in NiFi installed within a container. So to answer one of
>> your questions, I don’t yet know how or where to find nifi.properties in
>> the container framework. I don’t seem to have the usual /opt/nifi/…..
>> directories on my ec2 instance. Any idea where I need to look for that?
>>
>> These ports are open by my security group Inbound Rules: 22 to MyIP, 80,
>> 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
>>
>> I am able to Putty into my instance as ec2-user with my ppk file, which I
>> created using putty tools from the original pem key pair. When I do putty
>> in, under /opt I find three subdirectories: aws, containerd, and rh.
>> Nothing nifi under any of the three that I can see so far.
>>
>> I start my docker instance with this command:
>> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>>
>> I can do a ps -ef and see running nifi processes. But I don’t yet know
>> how to get to the nifi logs or properties file.
>>
>> You mentioned using using localhost to get to the canvas UI. This
>> confuses me. Nifi is running on my EC2 instance - a linux host without a
>> browser. I’m in a browser on my laptop. How would localhost in my browser
>> get me to my EC2 instance running nifi?
>>
>> This is the URL I’m using in my browser:
>> http://ec2-3-238-27-220.compute-1.amazonaws.com
>> (that url changes with each Stop/Start of my instance. I’ve yet to
>> investigate how to get AWS to stop changing that IP, but I know it can be
>> done).
>>
>> The browser replies with: ec2…….amazonaws refused to connect.
>>
>> I can ping my laptop IP address from the putty terminal where I am logged
>> in to my instance. I cannot ping the Public DNS of my instance from
>> Powershell on my laptop. Again, that Public DNS is
>> ec2-3-238-27-220.compute-1.amazonaws.com
>>
>> Any help is much appreciated.
>> Jim
>>
>>
>>
>> On Tue, Nov 8, 2022 at 3:03 PM David Handermann <
>> exceptionfact...@apache.org> wrote:
>>
>>> Hi Jim,
>>>
>>> NiFi 1.14.0 and following default to HTTPS on port 8443, listening on
>>> the localhost address. The nifi.web.https.host can be changed to blank in
>>> order to listen on all interfaces, but the default HTTPS setting with
>>> authenticated required should be retained.
>>>
>>> Can you provide the version of NiFi and some additional details on the
>>> nifi.web values from nifi.properties?
>>>
>>> Regards,

Re: NiFi on AWS EC2

[Mike Thomsen]

> won't render if you're using Edge or IE to access.

The old, discontinued Edge had that problem, but Edge has worked just
fine for the last 2 years or so since it was redone on top of
Chromium.

IMO if you're going to use Chromium with NiFi, the best experience
seems to be Brave.

On Tue, Nov 8, 2022 at 4:21 PM Patrick Timmins  wrote:
>
> In addition to the other suggestions, the last time I checked, the HTML5
> of the NiFi interface won't render if you're using Edge or IE to
> access.  Brave, Chrome, Firefox etc will work, however.
>
> On 11/8/2022 1:53 PM, James McMahon wrote:
> > Has anyone successfully configured NiFi on AWS, and accessed it from a
> > browser on a Windows desktop? I’ve tried following a few links to do
> > this. I’ve verified that my instance security group allows access to
> > 8080 via its inbound rules. I’ve putty’ed into the instance via ssh
> > port 22 to verify that there are no firewall restrictions. But still I
> > get a message to the effect that the server rejected the connection
> > request. Can anyone recommend a link that describes a success path for
> > this?
> > Thanks in advance for your help.
> > Jim

Re: NiFi on AWS EC2

[David Handermann]

Hi Jim,

Thanks for the reply and additional background.

The instructions are dated March 2021, which is prior to the release of
NiFi 1.14.0. In particular, the run command is no longer accurate with the
default NiFi container image.

The current Docker Hub instructions [1] show the basic command needed

docker run --name nifi -p 8443:8443 -d apache/nifi:latest

In addition, any references to port 8080 in the AWS Security Group rules
should be changed to 8443. The security group rules for port 80 and 18080
should be removed.

The instructions that allow plain HTTP access to NiFi on port 8080 should
NEVER be followed, as this exposes unfiltered and unauthenticated access.

Following those changes, it should be possible to access the NiFi UI using
the AWS URL:

https://ec2...amazonaws.com:8443

The default installation will generate a username and password, which can
be found in the container logs:

docker logs nifi | grep Generated

Regards,
David Handermann

[1] https://hub.docker.com/r/apache/nifi

On Tue, Nov 8, 2022 at 4:00 PM James McMahon  wrote:

> Hi and thank you, David and Dmitry. In my case I was following this
> example,
>
> https://joeygoksu.com/software/apache-nifi-on-aws/
>
> which results in NiFi installed within a container. So to answer one of
> your questions, I don’t yet know how or where to find nifi.properties in
> the container framework. I don’t seem to have the usual /opt/nifi/…..
> directories on my ec2 instance. Any idea where I need to look for that?
>
> These ports are open by my security group Inbound Rules: 22 to MyIP, 80,
> 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
>
> I am able to Putty into my instance as ec2-user with my ppk file, which I
> created using putty tools from the original pem key pair. When I do putty
> in, under /opt I find three subdirectories: aws, containerd, and rh.
> Nothing nifi under any of the three that I can see so far.
>
> I start my docker instance with this command:
> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>
> I can do a ps -ef and see running nifi processes. But I don’t yet know how
> to get to the nifi logs or properties file.
>
> You mentioned using using localhost to get to the canvas UI. This confuses
> me. Nifi is running on my EC2 instance - a linux host without a browser.
> I’m in a browser on my laptop. How would localhost in my browser get me to
> my EC2 instance running nifi?
>
> This is the URL I’m using in my browser:
> http://ec2-3-238-27-220.compute-1.amazonaws.com
> (that url changes with each Stop/Start of my instance. I’ve yet to
> investigate how to get AWS to stop changing that IP, but I know it can be
> done).
>
> The browser replies with: ec2…….amazonaws refused to connect.
>
> I can ping my laptop IP address from the putty terminal where I am logged
> in to my instance. I cannot ping the Public DNS of my instance from
> Powershell on my laptop. Again, that Public DNS is
> ec2-3-238-27-220.compute-1.amazonaws.com
>
> Any help is much appreciated.
> Jim
>
>
>
> On Tue, Nov 8, 2022 at 3:03 PM David Handermann <
> exceptionfact...@apache.org> wrote:
>
>> Hi Jim,
>>
>> NiFi 1.14.0 and following default to HTTPS on port 8443, listening on the
>> localhost address. The nifi.web.https.host can be changed to blank in order
>> to listen on all interfaces, but the default HTTPS setting with
>> authenticated required should be retained.
>>
>> Can you provide the version of NiFi and some additional details on the
>> nifi.web values from nifi.properties?
>>
>> Regards,
>> David Handermann
>>
>> On Tue, Nov 8, 2022 at 1:54 PM James McMahon 
>> wrote:
>>
>>> Has anyone successfully configured NiFi on AWS, and accessed it from a
>>> browser on a Windows desktop? I’ve tried following a few links to do this.
>>> I’ve verified that my instance security group allows access to 8080 via its
>>> inbound rules. I’ve putty’ed into the instance via ssh port 22 to verify
>>> that there are no firewall restrictions. But still I get a message to the
>>> effect that the server rejected the connection request. Can anyone
>>> recommend a link that describes a success path for this?
>>> Thanks in advance for your help.
>>> Jim
>>>
>>

Re: NiFi on AWS EC2

[James McMahon]

Hi and thank you, David and Dmitry. In my case I was following this
example,

https://joeygoksu.com/software/apache-nifi-on-aws/

which results in NiFi installed within a container. So to answer one of
your questions, I don’t yet know how or where to find nifi.properties in
the container framework. I don’t seem to have the usual /opt/nifi/…..
directories on my ec2 instance. Any idea where I need to look for that?

These ports are open by my security group Inbound Rules: 22 to MyIP, 80,
8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.

I am able to Putty into my instance as ec2-user with my ppk file, which I
created using putty tools from the original pem key pair. When I do putty
in, under /opt I find three subdirectories: aws, containerd, and rh.
Nothing nifi under any of the three that I can see so far.

I start my docker instance with this command:
docker run —name nifi -p 18080:8080 -d apache/nifi:latest

I can do a ps -ef and see running nifi processes. But I don’t yet know how
to get to the nifi logs or properties file.

You mentioned using using localhost to get to the canvas UI. This confuses
me. Nifi is running on my EC2 instance - a linux host without a browser.
I’m in a browser on my laptop. How would localhost in my browser get me to
my EC2 instance running nifi?

This is the URL I’m using in my browser:
http://ec2-3-238-27-220.compute-1.amazonaws.com
(that url changes with each Stop/Start of my instance. I’ve yet to
investigate how to get AWS to stop changing that IP, but I know it can be
done).

The browser replies with: ec2…….amazonaws refused to connect.

I can ping my laptop IP address from the putty terminal where I am logged
in to my instance. I cannot ping the Public DNS of my instance from
Powershell on my laptop. Again, that Public DNS is
ec2-3-238-27-220.compute-1.amazonaws.com

Any help is much appreciated.
Jim



On Tue, Nov 8, 2022 at 3:03 PM David Handermann 
wrote:

> Hi Jim,
>
> NiFi 1.14.0 and following default to HTTPS on port 8443, listening on the
> localhost address. The nifi.web.https.host can be changed to blank in order
> to listen on all interfaces, but the default HTTPS setting with
> authenticated required should be retained.
>
> Can you provide the version of NiFi and some additional details on the
> nifi.web values from nifi.properties?
>
> Regards,
> David Handermann
>
> On Tue, Nov 8, 2022 at 1:54 PM James McMahon  wrote:
>
>> Has anyone successfully configured NiFi on AWS, and accessed it from a
>> browser on a Windows desktop? I’ve tried following a few links to do this.
>> I’ve verified that my instance security group allows access to 8080 via its
>> inbound rules. I’ve putty’ed into the instance via ssh port 22 to verify
>> that there are no firewall restrictions. But still I get a message to the
>> effect that the server rejected the connection request. Can anyone
>> recommend a link that describes a success path for this?
>> Thanks in advance for your help.
>> Jim
>>
>

Re: NiFi on AWS EC2

[Patrick Timmins]

In addition to the other suggestions, the last time I checked, the HTML5 
of the NiFi interface won't render if you're using Edge or IE to 
access.  Brave, Chrome, Firefox etc will work, however.


On 11/8/2022 1:53 PM, James McMahon wrote:
Has anyone successfully configured NiFi on AWS, and accessed it from a 
browser on a Windows desktop? I’ve tried following a few links to do 
this. I’ve verified that my instance security group allows access to 
8080 via its inbound rules. I’ve putty’ed into the instance via ssh 
port 22 to verify that there are no firewall restrictions. But still I 
get a message to the effect that the server rejected the connection 
request. Can anyone recommend a link that describes a success path for 
this?

Thanks in advance for your help.
Jim

Re: NiFi on AWS EC2

[Dmitry Stepanov]

We did.

Since NiFi is HTTPS by default (recent versions) you need to open ports 
8443 or 443(default HTTPS)

Try opening those and see what happens

Cheers,

Dima Stepanov

Re: NiFi on AWS EC2

[David Handermann]

Hi Jim,

NiFi 1.14.0 and following default to HTTPS on port 8443, listening on the
localhost address. The nifi.web.https.host can be changed to blank in order
to listen on all interfaces, but the default HTTPS setting with
authenticated required should be retained.

Can you provide the version of NiFi and some additional details on the
nifi.web values from nifi.properties?

Regards,
David Handermann

On Tue, Nov 8, 2022 at 1:54 PM James McMahon  wrote:

> Has anyone successfully configured NiFi on AWS, and accessed it from a
> browser on a Windows desktop? I’ve tried following a few links to do this.
> I’ve verified that my instance security group allows access to 8080 via its
> inbound rules. I’ve putty’ed into the instance via ssh port 22 to verify
> that there are no firewall restrictions. But still I get a message to the
> effect that the server rejected the connection request. Can anyone
> recommend a link that describes a success path for this?
> Thanks in advance for your help.
> Jim
>

Re: NiFi Registry Event Logging - Missing User Principal on Some Events

[Shawn Weeks]

I’ll start a ticket and try and get a PR submitted this week.

Thanks
Shawn

> On Nov 1, 2022, at 8:05 AM, Bryan Bende  wrote:
> 
> Seems like it was just missed in here:
> 
> https://github.com/apache/nifi/blob/main/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/event/EventFactory.java
> 
> No JIRA that I know of.
> 
> On Mon, Oct 31, 2022 at 10:58 AM Shawn Weeks  
> wrote:
>> 
>> I’m running NiFi Registry 1.16.3 and I’ve noticed that the event logging 
>> does not log the user principal who created or deleted a user. Other actions 
>> like CREATE_BUCKET, CREATE_FLOW_VERSION, DELETE_BUCKET, etc do log the user 
>> principal. I was wondering if any existing Jira’s covered this? If not I’ll 
>> get one started and see if I can contribute the fix for this. Security 
>> requirements on one of my projects requires always logging the user 
>> performing an action.
>> 
>> Thanks
>> Shawn

Re: nifi-api with a server secured with Microsoft AD

[Jens M. Kofoed]

Hi David

It's also possible to configure authorizers.xml to both handle LDAP and
local users (file-access) so you can have both. It's using
composite-configurable-user-group-provider. Just remember that nifi is case
sentitive, so the what you specify as the user, should match exactly what
nifi sees in the certificate. Some certificates are created with a space
after commas like ", ou=", and others don'ts ",ou=".
I'm using this where all the regular users are from AD, but you can still
create local users which is using a certificate.
from authorizers.xml:


file-user-group-provider
org.apache.nifi.authorization.FileUserGroupProvider
./conf/localAuth/users.xml

cn=name, ou=users, dc=foo,
dc=bar


ldap-user-group-provider
org.apache.nifi.ldap.tenants.LdapUserGroupProvider
...


composite-configurable-user-group-provider

org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider
file-user-group-provider
ldap-user-group-provider


file-access-policy-provider

org.apache.nifi.authorization.FileAccessPolicyProvider
composite-configurable-user-group-provider
./conf/localAuth/authorizations.xml
cn=name, ou=users, dc=foo,
dc=bar


NIFI-CLUSTER01-NODES


managed-authorizer

org.apache.nifi.authorization.StandardManagedAuthorizer
file-access-policy-provider



>From nifi.properties:
nifi.security.user.authorizer=managed-authorizer


In this way, you can create certificates your self, from the CA you used to
create the certificates for nifi. and create these as internal local users
and still use AD.

Kind regards
Jens M. Kofoed


Den tir. 1. nov. 2022 kl. 16.47 skrev David Early via users <
users@nifi.apache.org>:

> Mike and Shawn,  thanks for the feedback, have not had a chance to try
> either, but appreciate your help.  Will be trying the cert this week, will
> reach out to the AD managers about a more direct AD solution.
>
> Dave
>
> On Sat, Oct 29, 2022 at 7:10 PM Mike Thomsen 
> wrote:
>
>> David,
>>
>> Another option you might want to explore is having AD generate client
>> certificates for your users.
>>
>> On Sat, Oct 29, 2022 at 12:01 PM Shawn Weeks 
>> wrote:
>> >
>> > NiFi should always accept a cert at the rest api if you provide one. If
>> your using curl just add the “--key” and “--cert” and call whatever api url
>> your trying directly. You’ll need to make sure that the cert your using is
>> signed by the same local CA that NiFi is set to trust and that you’ve added
>> a user in NiFi that matches the common name on the cert or whatever regex
>> you set for “nifi.security.identity.mapping.value.pattern”
>> >
>> > Thanks
>> > Shawn
>> >
>> > > On Oct 28, 2022, at 3:55 PM, David Early via users <
>> users@nifi.apache.org> wrote:
>> > >
>> > > Hi all,
>> > >
>> > > We have a 3 node cluster secured with Microsort AD for the first time.
>> > >
>> > > I need access to the REST api.  The nifi-api/access/token does not
>> work in this case.
>> > >
>> > > We did use a local CA for certificate generation on the servers.
>> > >
>> > > I am reading that it is possible to do certificate based auth to the
>> apiwe need this in a script (python) to run on a remote server which is
>> checking for old flowfiles that can get stuck in a few places.
>> > >
>> > > Can I use cert based API connection when using AD as the main
>> authentication/authorization for the ui?
>> > >
>> > > Anything special that needs to be done?  I've just not used certs
>> with the api before, but we have used cert based site to site on other
>> systems and it works fine.  Just not sure how to do it with nipyapi or just
>> from curl on the cli.
>> > >
>> > > David
>> >
>>
>
>
> --
> David Early, Ph.D.
> david.ea...@grokstream.com
> 720-470-7460 Cell
>
>

Re: nifi-api with a server secured with Microsoft AD

[David Early via users]

Mike and Shawn,  thanks for the feedback, have not had a chance to try
either, but appreciate your help.  Will be trying the cert this week, will
reach out to the AD managers about a more direct AD solution.

Dave

On Sat, Oct 29, 2022 at 7:10 PM Mike Thomsen  wrote:

> David,
>
> Another option you might want to explore is having AD generate client
> certificates for your users.
>
> On Sat, Oct 29, 2022 at 12:01 PM Shawn Weeks 
> wrote:
> >
> > NiFi should always accept a cert at the rest api if you provide one. If
> your using curl just add the “--key” and “--cert” and call whatever api url
> your trying directly. You’ll need to make sure that the cert your using is
> signed by the same local CA that NiFi is set to trust and that you’ve added
> a user in NiFi that matches the common name on the cert or whatever regex
> you set for “nifi.security.identity.mapping.value.pattern”
> >
> > Thanks
> > Shawn
> >
> > > On Oct 28, 2022, at 3:55 PM, David Early via users <
> users@nifi.apache.org> wrote:
> > >
> > > Hi all,
> > >
> > > We have a 3 node cluster secured with Microsort AD for the first time.
> > >
> > > I need access to the REST api.  The nifi-api/access/token does not
> work in this case.
> > >
> > > We did use a local CA for certificate generation on the servers.
> > >
> > > I am reading that it is possible to do certificate based auth to the
> apiwe need this in a script (python) to run on a remote server which is
> checking for old flowfiles that can get stuck in a few places.
> > >
> > > Can I use cert based API connection when using AD as the main
> authentication/authorization for the ui?
> > >
> > > Anything special that needs to be done?  I've just not used certs with
> the api before, but we have used cert based site to site on other systems
> and it works fine.  Just not sure how to do it with nipyapi or just from
> curl on the cli.
> > >
> > > David
> >
>


-- 
David Early, Ph.D.
david.ea...@grokstream.com
720-470-7460 Cell

Re: NiFi Registry Event Logging - Missing User Principal on Some Events

[Bryan Bende]

Seems like it was just missed in here:

https://github.com/apache/nifi/blob/main/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/event/EventFactory.java

No JIRA that I know of.

On Mon, Oct 31, 2022 at 10:58 AM Shawn Weeks  wrote:
>
> I’m running NiFi Registry 1.16.3 and I’ve noticed that the event logging does 
> not log the user principal who created or deleted a user. Other actions like 
> CREATE_BUCKET, CREATE_FLOW_VERSION, DELETE_BUCKET, etc do log the user 
> principal. I was wondering if any existing Jira’s covered this? If not I’ll 
> get one started and see if I can contribute the fix for this. Security 
> requirements on one of my projects requires always logging the user 
> performing an action.
>
> Thanks
> Shawn

Re: nifi-api with a server secured with Microsoft AD

[Mike Thomsen]

David,

Another option you might want to explore is having AD generate client
certificates for your users.

On Sat, Oct 29, 2022 at 12:01 PM Shawn Weeks  wrote:
>
> NiFi should always accept a cert at the rest api if you provide one. If your 
> using curl just add the “--key” and “--cert” and call whatever api url your 
> trying directly. You’ll need to make sure that the cert your using is signed 
> by the same local CA that NiFi is set to trust and that you’ve added a user 
> in NiFi that matches the common name on the cert or whatever regex you set 
> for “nifi.security.identity.mapping.value.pattern”
>
> Thanks
> Shawn
>
> > On Oct 28, 2022, at 3:55 PM, David Early via users  
> > wrote:
> >
> > Hi all,
> >
> > We have a 3 node cluster secured with Microsort AD for the first time.
> >
> > I need access to the REST api.  The nifi-api/access/token does not work in 
> > this case.
> >
> > We did use a local CA for certificate generation on the servers.
> >
> > I am reading that it is possible to do certificate based auth to the 
> > apiwe need this in a script (python) to run on a remote server which is 
> > checking for old flowfiles that can get stuck in a few places.
> >
> > Can I use cert based API connection when using AD as the main 
> > authentication/authorization for the ui?
> >
> > Anything special that needs to be done?  I've just not used certs with the 
> > api before, but we have used cert based site to site on other systems and 
> > it works fine.  Just not sure how to do it with nipyapi or just from curl 
> > on the cli.
> >
> > David
>