Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

[Mark Payne]

I’ve not followed everything goin on in this thread. But just to offer a bit of 
clarification: If the flow.json.gz exists then the flow.xml.gz is ignored. But 
if the flow.json.gz file is not found, it will automatically fall back to the 
flow.xml.gz

So if you were to remove/rename the flow.json.gz it would definitely pick up 
the XML file and process that. Otherwise, it will ignore it.

Thanks
-Mark


On Dec 7, 2022, at 4:24 AM, Tiago Luís Sebastião (DSI) 
 wrote:

Hi Isha,

I did not delete the flow.xml.gz because I think that since the version 1.16.0, 
if I’m not mistaken, when the flow.json.xml was released it replaced completely 
the flow.xml.gz, so any changes made on flow.xml.gz wouldn’t make any 
difference, so knowing that, I assumed there shouldn’t be anything 
configured/dependent on the xml file. Also, the error didn’t seem related at 
all to the flow, it seemed related to not being able to decrypt the 
flow.json.gz with the old algorithm and encrypt with the new algorithm.

Thanks for the help.

Regards.

Tiago Sebastião

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 7 de dezembro de 2022 08:42
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

Thanks for updating and sharing your solution.

I noticed you only encrypted the flow.json.gz file. Did you also delete the xml 
version in previous attempts?
It makes me wonder if the migrating fails for the flow.xml.gz specifically, 
then deleting the xml during migration might be an easier fix.

Regards,

Isha


Van: Tiago Luís Sebastião (DSI) 
mailto:tiago.luis.sebast...@cgd.pt>>
Verzonden: dinsdag 6 december 2022 17:24
Aan: users@nifi.apache.org<mailto:users@nifi.apache.org>
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi Isha,

I had already tried that and didn’t work also.

But I tried the last thing I wrote on the previous email and it worked… Not 
happy with this solution but the issue (encryption/warnings) is solved I guess.

Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL

With nifi Stopped:

  1.  Encrypt new files on the side:

/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 
-sPasswordUsedOnNifiProperties -x -v


  1.  Backup the flow.json.gz and rename the new one:

mv /apps/nifi-1.18.0/conf/flow.json.gz 
/apps/nifi-1.18.0/conf/flow_bk202212.json.gz
mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz


  1.  Change nifi.properties file:

nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256


  1.  Start nifi


No errors found and no warnings…

Tiago Sebastião

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

I’ve had a similar experience with migrating the flow encryption algorithm and 
in fact, some of them are still on the old one. The nifi.sh commands to update 
the sensitive properties key and algorithm are very tricky to use, because they 
update the nifi.properties file even if the migration fails for the flow.xml.gz 
and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following 
ones to fail because it tried to decrypt using the new algorithm. I needed to 
reset the nifi.properties file everytime.

Another thing I’ve noticed is that it doesn’t support the property file 
protection scheme that NiFi has. If your sensitive props key is encrypted you 
need to enter the raw value and make sure the 
nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.

These steps have mostly worked for me:


  1.  Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another 
dir
  2.  Unprotect the sensitive properties key:

 *   Replace the encrypted key with the raw one
 *   Empty this property: nifi.sensitive.props.key.protected

  1.  Check that the algorithm is still the old one: 
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
  2.  Check the length of the raw key, it needs to be 12 characters or longer 
to migr

RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

[DSI]

Hi Isha,

I did not delete the flow.xml.gz because I think that since the version 1.16.0, 
if I’m not mistaken, when the flow.json.xml was released it replaced completely 
the flow.xml.gz, so any changes made on flow.xml.gz wouldn’t make any 
difference, so knowing that, I assumed there shouldn’t be anything 
configured/dependent on the xml file. Also, the error didn’t seem related at 
all to the flow, it seemed related to not being able to decrypt the 
flow.json.gz with the old algorithm and encrypt with the new algorithm.

Thanks for the help.

Regards.

Tiago Sebastião

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 7 de dezembro de 2022 08:42
To: users@nifi.apache.org
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

Thanks for updating and sharing your solution.

I noticed you only encrypted the flow.json.gz file. Did you also delete the xml 
version in previous attempts?
It makes me wonder if the migrating fails for the flow.xml.gz specifically, 
then deleting the xml during migration might be an easier fix.

Regards,

Isha


Van: Tiago Luís Sebastião (DSI) 
mailto:tiago.luis.sebast...@cgd.pt>>
Verzonden: dinsdag 6 december 2022 17:24
Aan: users@nifi.apache.org<mailto:users@nifi.apache.org>
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi Isha,

I had already tried that and didn’t work also.

But I tried the last thing I wrote on the previous email and it worked… Not 
happy with this solution but the issue (encryption/warnings) is solved I guess.

Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL

With nifi Stopped:

  1.  Encrypt new files on the side:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s 
PasswordUsedOnNifiProperties -x -v


  1.  Backup the flow.json.gz and rename the new one:

mv /apps/nifi-1.18.0/conf/flow.json.gz 
/apps/nifi-1.18.0/conf/flow_bk202212.json.gz

mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz


  1.  Change nifi.properties file:
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256


  1.  Start nifi

No errors found and no warnings…

Tiago Sebastião

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

I’ve had a similar experience with migrating the flow encryption algorithm and 
in fact, some of them are still on the old one. The nifi.sh commands to update 
the sensitive properties key and algorithm are very tricky to use, because they 
update the nifi.properties file even if the migration fails for the flow.xml.gz 
and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following 
ones to fail because it tried to decrypt using the new algorithm. I needed to 
reset the nifi.properties file everytime.

Another thing I’ve noticed is that it doesn’t support the property file 
protection scheme that NiFi has. If your sensitive props key is encrypted you 
need to enter the raw value and make sure the 
nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.

These steps have mostly worked for me:


  1.  Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another 
dir
  2.  Unprotect the sensitive properties key:

 *   Replace the encrypted key with the raw one
 *   Empty this property: nifi.sensitive.props.key.protected

  1.  Check that the algorithm is still the old one: 
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
  2.  Check the length of the raw key, it needs to be 12 characters or longer 
to migrate to the new algorithm.
  3.  If the key is empty or short:

 *   ./bin/nifi.sh set-sensitive-properties-key 
PasswordUsedOnNifiProperties (run from the main dir)
 *   Check the output for any failures and if they occur, revert the 
nifi.properties file and fix any errors
 *   Try to start nifi with the new sensitive properties key

  1.  If it works, stop nifi and update the algorithm:

 *   ./bin/nifi.sh set-sensitive-properties-algorithm 
NIFI_PBKDF2_AES_GCM_256
 *   Check 

RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

[Isha Lamboo]

Hi Tiago,

Thanks for updating and sharing your solution.

I noticed you only encrypted the flow.json.gz file. Did you also delete the xml 
version in previous attempts?
It makes me wonder if the migrating fails for the flow.xml.gz specifically, 
then deleting the xml during migration might be an easier fix.

Regards,

Isha


Van: Tiago Luís Sebastião (DSI) 
Verzonden: dinsdag 6 december 2022 17:24
Aan: users@nifi.apache.org
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi Isha,

I had already tried that and didn’t work also.

But I tried the last thing I wrote on the previous email and it worked… Not 
happy with this solution but the issue (encryption/warnings) is solved I guess.

Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL

With nifi Stopped:

  1.  Encrypt new files on the side:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s 
PasswordUsedOnNifiProperties -x -v


  1.  Backup the flow.json.gz and rename the new one:

mv /apps/nifi-1.18.0/conf/flow.json.gz 
/apps/nifi-1.18.0/conf/flow_bk202212.json.gz

mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz


  1.  Change nifi.properties file:
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256


  1.  Start nifi

No errors found and no warnings…

Tiago Sebastião
Tlf: 218456542 | Ext. 556542 | 
tiago.luis.sebast...@cgd.pt<mailto:tiago.luis.sebast...@cgd.pt>
[cid:image001.jpg@01D90A1F.72B3A4E0]
DSI - Data Office – USI6.2 Unidade de Serviços de Dados
www.cgd.pt<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cgd.pt%2F=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=37%2FWY7BK17LoCCmr7ii1I0w7FcW65j%2F2A43ckkt1mok%3D=0>
Antes de imprimir esta mensagem, pense no meio ambiente.
Esta mensagem e-mail, assim como os ficheiros eventualmente anexos, é reservada 
aos seus destinatários, e pode conter informação confidencial ou estar sujeita 
a restrições legais. Se não é o seu destinatário ou se recebeu esta mensagem 
por motivo de erro, solicitamos que não faça qualquer uso ou divulgação do seu 
conteúdo e proceda à eliminação permanente desta mensagem e respetivos anexos.
Caixa Geral de Depósitos, S.A. | Sede Social: Av. João XXI, 63, 1000-300 LISBOA 
| Capital Social 3.844.143.735,00 € | CRCL e Contribuinte 500 960 046

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

I’ve had a similar experience with migrating the flow encryption algorithm and 
in fact, some of them are still on the old one. The nifi.sh commands to update 
the sensitive properties key and algorithm are very tricky to use, because they 
update the nifi.properties file even if the migration fails for the flow.xml.gz 
and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following 
ones to fail because it tried to decrypt using the new algorithm. I needed to 
reset the nifi.properties file everytime.

Another thing I’ve noticed is that it doesn’t support the property file 
protection scheme that NiFi has. If your sensitive props key is encrypted you 
need to enter the raw value and make sure the 
nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.

These steps have mostly worked for me:


  1.  Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another 
dir
  2.  Unprotect the sensitive properties key:

 *   Replace the encrypted key with the raw one
 *   Empty this property: nifi.sensitive.props.key.protected

  1.  Check that the algorithm is still the old one: 
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
  2.  Check the length of the raw key, it needs to be 12 characters or longer 
to migrate to the new algorithm.
  3.  If the key is empty or short:

 *   ./bin/nifi.sh set-sensitive-properties-key 
PasswordUsedOnNifiProperties (run from the main dir)
 *   Check the output for any failures and if they occur, revert the 
nifi.properties file and fix any errors
 *   Try to start nifi with the new sensitive properties key

  1.  If it works, st

RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

[DSI]

Hi Isha,

I had already tried that and didn’t work also.

But I tried the last thing I wrote on the previous email and it worked… Not 
happy with this solution but the issue (encryption/warnings) is solved I guess.

Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL

With nifi Stopped:

1.   Encrypt new files on the side:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s 
PasswordUsedOnNifiProperties -x -v


2.   Backup the flow.json.gz and rename the new one:

mv /apps/nifi-1.18.0/conf/flow.json.gz 
/apps/nifi-1.18.0/conf/flow_bk202212.json.gz

mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz


3.   Change nifi.properties file:
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256


4.   Start nifi

No errors found and no warnings…

Tiago Sebastião
Tlf: 218456542 | Ext. 556542 | 
tiago.luis.sebast...@cgd.pt<mailto:tiago.luis.sebast...@cgd.pt>
[cid:image002.jpg@01D52816.782219F0]

DSI - Data Office – USI6.2 Unidade de Serviços de Dados
www.cgd.pt<http://www.cgd.pt/>

Antes de imprimir esta mensagem, pense no meio ambiente.
Esta mensagem e-mail, assim como os ficheiros eventualmente anexos, é reservada 
aos seus destinatários, e pode conter informação confidencial ou estar sujeita 
a restrições legais. Se não é o seu destinatário ou se recebeu esta mensagem 
por motivo de erro, solicitamos que não faça qualquer uso ou divulgação do seu 
conteúdo e proceda à eliminação permanente desta mensagem e respetivos anexos.
Caixa Geral de Depósitos, S.A. | Sede Social: Av. João XXI, 63, 1000-300 LISBOA 
| Capital Social 3.844.143.735,00 € | CRCL e Contribuinte 500 960 046

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

I’ve had a similar experience with migrating the flow encryption algorithm and 
in fact, some of them are still on the old one. The nifi.sh commands to update 
the sensitive properties key and algorithm are very tricky to use, because they 
update the nifi.properties file even if the migration fails for the flow.xml.gz 
and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following 
ones to fail because it tried to decrypt using the new algorithm. I needed to 
reset the nifi.properties file everytime.

Another thing I’ve noticed is that it doesn’t support the property file 
protection scheme that NiFi has. If your sensitive props key is encrypted you 
need to enter the raw value and make sure the 
nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.

These steps have mostly worked for me:


  1.  Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another 
dir
  2.  Unprotect the sensitive properties key:

 *   Replace the encrypted key with the raw one
 *   Empty this property: nifi.sensitive.props.key.protected

  1.  Check that the algorithm is still the old one: 
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
  2.  Check the length of the raw key, it needs to be 12 characters or longer 
to migrate to the new algorithm.
  3.  If the key is empty or short:

 *   ./bin/nifi.sh set-sensitive-properties-key 
PasswordUsedOnNifiProperties (run from the main dir)
 *   Check the output for any failures and if they occur, revert the 
nifi.properties file and fix any errors
 *   Try to start nifi with the new sensitive properties key

  1.  If it works, stop nifi and update the algorithm:

 *   ./bin/nifi.sh set-sensitive-properties-algorithm 
NIFI_PBKDF2_AES_GCM_256
 *   Check the output for any failures and if they occur, revert the 
nifi.properties file and fix any errors
 *   Try to start nifi with the new algorithm

  1.  Stop nifi, encrypt the sensitive properties key (or restore it if you 
didn’t need to change the password)

 *   Use the encrypt-config.sh from the nifi-toolkit, specify output files 
so you can check and compare.
 *   Beware: this tool doesn’t seem to know about flow.json.gz, so only use 
it to change the config files.

I hope this helps you find the solution.

Regards,

Isha

Van: Tiago Luís Sebastião (DSI) 
mailto:tiago.luis.sebast...@cgd.pt>>
Verzonden: donderdag 24 november 2022 16:19
Aan: users@nifi.apache.org<mailto:users@nifi.apache.org>
Onderwerp: RE: NiFi 1.18.0 S

RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

[Isha Lamboo]

Hi Tiago,

I’ve had a similar experience with migrating the flow encryption algorithm and 
in fact, some of them are still on the old one. The nifi.sh commands to update 
the sensitive properties key and algorithm are very tricky to use, because they 
update the nifi.properties file even if the migration fails for the flow.xml.gz 
and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following 
ones to fail because it tried to decrypt using the new algorithm. I needed to 
reset the nifi.properties file everytime.

Another thing I’ve noticed is that it doesn’t support the property file 
protection scheme that NiFi has. If your sensitive props key is encrypted you 
need to enter the raw value and make sure the 
nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.

These steps have mostly worked for me:


  1.  Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another 
dir
  2.  Unprotect the sensitive properties key:
 *   Replace the encrypted key with the raw one
 *   Empty this property: nifi.sensitive.props.key.protected
  3.  Check that the algorithm is still the old one: 
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
  4.  Check the length of the raw key, it needs to be 12 characters or longer 
to migrate to the new algorithm.
  5.  If the key is empty or short:
 *   ./bin/nifi.sh set-sensitive-properties-key 
PasswordUsedOnNifiProperties (run from the main dir)
 *   Check the output for any failures and if they occur, revert the 
nifi.properties file and fix any errors
 *   Try to start nifi with the new sensitive properties key
  6.  If it works, stop nifi and update the algorithm:
 *   ./bin/nifi.sh set-sensitive-properties-algorithm 
NIFI_PBKDF2_AES_GCM_256
 *   Check the output for any failures and if they occur, revert the 
nifi.properties file and fix any errors
 *   Try to start nifi with the new algorithm
  7.  Stop nifi, encrypt the sensitive properties key (or restore it if you 
didn’t need to change the password)
 *   Use the encrypt-config.sh from the nifi-toolkit, specify output files 
so you can check and compare.
 *   Beware: this tool doesn’t seem to know about flow.json.gz, so only use 
it to change the config files.

I hope this helps you find the solution.

Regards,

Isha

Van: Tiago Luís Sebastião (DSI) 
Verzonden: donderdag 24 november 2022 16:19
Aan: users@nifi.apache.org
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi again,

Sorry for not following up but other priorities came ahead…

Basically it’s still not working, I’ve tried several combinations and I still 
keep getting:
“Failed to process Flow Configuration [./conf/flow.xml.gz]
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm 
[AES/GCM/NoPadding]”

After reading some documentation, for this purpose, assuming that the password 
configured in the nifi.properties file is “PasswordUsedOnNifiProperties”…

I’ve tried and failed:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties

I’ve tried and failed by setting algorithm to empty string:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties

I’ve tried and failed using the new toolkit (I was using toolkit version 
1.13.3):
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.xml.gz -n /apps/nifi-1.18.0/conf/nifi.properties -s 
PasswordUsedOnNifiProperties -A NIFI_ARGON2_AES_GCM_256 -x -v

I’ve tried and failed doing the same but generating new files to debug:
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s 
PasswordUsedOnNifiProperties -x -v

In this last one I noticed that the file flow2.json.gz got its passwords 
encrypted differently and a longer encrypt also.

I’m kind of wondering if I can use this last command to generate these files on 
the side and then manipulate the nifi.properties file by changing the algorithm 
to NIFI_PBKDF2_AES_GCM_256 since it seems it is already encrypted despite the 
known errors/warnings, then I would rename these new files to the older ones 
and start nifi with:

  1.  New flow.json.gz file (apparently encrypted with NIFI_PBKDF2_AES_GCM_256 
algorithm)
  2.  New nifi.properties file (with nifi.sensitive.props.algorithm property 
manipulated to NIFI_PBKDF2_AES_GCM_256)

Since, 
unfortunately<https://eu

RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

[DSI]

Hi again,

Sorry for not following up but other priorities came ahead…

Basically it’s still not working, I’ve tried several combinations and I still 
keep getting:
“Failed to process Flow Configuration [./conf/flow.xml.gz]
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm 
[AES/GCM/NoPadding]”

After reading some documentation, for this purpose, assuming that the password 
configured in the nifi.properties file is “PasswordUsedOnNifiProperties”…

I’ve tried and failed:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties

I’ve tried and failed by setting algorithm to empty string:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties

I’ve tried and failed using the new toolkit (I was using toolkit version 
1.13.3):
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.xml.gz -n /apps/nifi-1.18.0/conf/nifi.properties -s 
PasswordUsedOnNifiProperties -A NIFI_ARGON2_AES_GCM_256 -x -v

I’ve tried and failed doing the same but generating new files to debug:
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s 
PasswordUsedOnNifiProperties -x -v

In this last one I noticed that the file flow2.json.gz got its passwords 
encrypted differently and a longer encrypt also.

I’m kind of wondering if I can use this last command to generate these files on 
the side and then manipulate the nifi.properties file by changing the algorithm 
to NIFI_PBKDF2_AES_GCM_256 since it seems it is already encrypted despite the 
known errors/warnings, then I would rename these new files to the older ones 
and start nifi with:

1.   New flow.json.gz file (apparently encrypted with 
NIFI_PBKDF2_AES_GCM_256 algorithm)

2.   New nifi.properties file (with nifi.sensitive.props.algorithm property 
manipulated to NIFI_PBKDF2_AES_GCM_256)

Since, 
unfortunately<https://www.google.com/search?sxsrf=ALiCzsYtztPHersBtg21lqlpGlc7DZ9CUw:1669302812266=unfortunately=1=X=2ahUKEwiq4pbJjcf7AhXKzqQKHfK8DNwQkeECKAB6BAgGEAE>,
 Im getting nowhere with this and I need to migrate to version 1.18.0 in order 
to apply the bugfix that changes the 
serverConnectorFactory.setNeedClientAuth(wantClientAuth) to 
serverConnectorFactory.setWantClientAuth(wantClientAuth) I am needing help in a 
consequent situation.
In order to reduce the size of the log generated from the deprecation warnings 
(WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider 
Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] 
generate salt requested) I’ve tried to LEVEL OFF that Warning from 
stateless-logback.xml file without success.

On  the tag  and   I changed the tag 
 so that I could see the full class name (although without success 
also…)
%date %level [%thread] %logger{40} %msg%n
to
%date %level [%thread] %logger{140} %msg%n

On the stateless-logback.xml I inserted the following:



It’s not working and I don’t understand why, the class name seems to be correct 
but I keep getting the same WARN.

Sorry for the long email…

Regards.
Tiago Sebastião

From: Tiago Luís Sebastião (DSI)
Sent: 28 de outubro de 2022 09:48
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi David,

It’s a standalone deployment and runs directly on the server.

Yes the command updated the flow.xml.gz/flow.json.gz and nifi.properties 
settings.

Maybe I messed up the nifi.sensitive.props.key, I’ll run some more tests.

Thanks for your help.
 Tiago
From: David Handermann [mailto:exceptionfact...@apache.org]
Sent: 27 de outubro de 2022 16:50
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi Tiago,

The initial warning for the Insecure Cipher Provider Algorithm indicates the 
use of the deprecated setting as mentioned previously.

The set-sensitive-properties-algorithm command looks correct, and should have 
updated the flow.xml.gz, flow.json.gz, and nifi.properties settings.

The Decryption Failed message indicates that the nifi.sensitive.props.key value 
does not match the value used to encrypt the flow configuration, or that the 
algorithm does not match.

Can you provide some additional details about the NiFi installation? Is this a 
standalone or clustered deployment, and is it running in a containerized 
environment, or di

RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

[DSI]

Hi David,

It’s a standalone deployment and runs directly on the server.

Yes the command updated the flow.xml.gz/flow.json.gz and nifi.properties 
settings.

Maybe I messed up the nifi.sensitive.props.key, I’ll run some more tests.

Thanks for your help.
 Tiago
From: David Handermann [mailto:exceptionfact...@apache.org]
Sent: 27 de outubro de 2022 16:50
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi Tiago,

The initial warning for the Insecure Cipher Provider Algorithm indicates the 
use of the deprecated setting as mentioned previously.

The set-sensitive-properties-algorithm command looks correct, and should have 
updated the flow.xml.gz, flow.json.gz, and nifi.properties settings.

The Decryption Failed message indicates that the nifi.sensitive.props.key value 
does not match the value used to encrypt the flow configuration, or that the 
algorithm does not match.

Can you provide some additional details about the NiFi installation? Is this a 
standalone or clustered deployment, and is it running in a containerized 
environment, or directly on a server?

Regards,
David Handermann

On Thu, Oct 27, 2022 at 10:35 AM Tiago Luís Sebastião (DSI) 
mailto:tiago.luis.sebast...@cgd.pt>> wrote:
Hi all,

I'm having the same “problem”.
I upgraded nifi version from 1.17.0 to 1.18.0 and that same warning started to 
appear 500k times a day.
"
WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider 
Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] 
generate salt requested
"

A already had nifi.sensitive.props.key value defined from when we migrated to 
1.15.3.

With Nifi STOPPED and without changing any configuration on nifi.properties I 
executed the following:

./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256

No errors found there, then I started Nifi and received the following errors:
"
WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web 
server... shutting down.
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm 
[AES/GCM/NoPadding]
"

Since Nifi could not start anymore I reversed it...
Now Im kind of stuck with this warning...
Anyone knows what Im doing wrong?

Tiago

From: David Handermann 
[mailto:exceptionfact...@apache.org<mailto:exceptionfact...@apache.org>]
Sent: 19 de outubro de 2022 13:41
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi Mike,

The deprecation warning is not related to NIFI-10567 or Sensitive Dynamic 
Properties.

Deprecation logging is a new feature added in NiFi 1.18.0 to highlight 
components and features that are targeted for removal in future major releases. 
The current administrator's guide has more details on deprecation logging. [1] 
Deprecation warnings do not impact operational behavior, but they do identify 
configuration settings that should be changed.

In this particular case, the deprecation is related to the use of the insecure 
algorithm.  NiFi 1.14.0 and following introduced new Sensitive Properties Key 
Algorithm settings, which should be used instead of the historical default 
value indicated in the warning. The new default value is 
NIFI_PBKDF2_AES_GCM_256, additional supported options are listed in the 
administrator's guide, [2] along with the command that can be run to update the 
Sensitive Properties Key Algorithm. [3]

Feel free to follow up if you have additional questions.

Regards,
David Handermann

[1] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#deprecation-logging
[2] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#property-encryption-algorithms
[3] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-algorithm

On Wed, Oct 19, 2022 at 7:28 AM Mike S 
<88msha...@gmail.com<mailto:88msha...@gmail.com>> wrote:
I upgraded from 1.16.2 to 1.18.0 and now see this warning in the log file.

WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider 
Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] 
generate salt requested
org.apache.nifi.deprecation.log.DeprecationException: Reference Class 
[org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider] ClassLoader 
[org.apache.nifi.nar.NarClassLoader[./work/nar/framework/nifi-framework-nar-1.18.0.nar-unpacked]]
I read this here.


NIFI-10567<https://issues.apache.org/jira/browse/NIFI-10567> Corrects the 
parsing of Sensitive Dynamic Properties read from the XML version of the flow 
configuration, in absence of the JSON version.

The issue surfaces when upgrading to NiFi 1.17.0 or 1.18.0 from a version older 
than 1.16.0. The issue also requires the presence of a Parameter Context with a 
Sensitive value assigned to a component with a Sensitive Property. Upgr

Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

[David Handermann]

Hi Tiago,

The initial warning for the Insecure Cipher Provider Algorithm indicates
the use of the deprecated setting as mentioned previously.

The set-sensitive-properties-algorithm command looks correct, and should
have updated the flow.xml.gz, flow.json.gz, and nifi.properties settings.

The Decryption Failed message indicates that the nifi.sensitive.props.key
value does not match the value used to encrypt the flow configuration, or
that the algorithm does not match.

Can you provide some additional details about the NiFi installation? Is
this a standalone or clustered deployment, and is it running in a
containerized environment, or directly on a server?

Regards,
David Handermann

On Thu, Oct 27, 2022 at 10:35 AM Tiago Luís Sebastião (DSI) <
tiago.luis.sebast...@cgd.pt> wrote:

> Hi all,
>
>
>
> I'm having the same “problem”.
>
> I upgraded nifi version from 1.17.0 to 1.18.0 and that same warning
> started to appear 500k times a day.
>
> "
>
> WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider
> Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
> generate salt requested
>
> "
>
>
>
> A already had nifi.sensitive.props.key value defined from when we migrated
> to 1.15.3.
>
>
>
> With Nifi STOPPED and without changing any configuration on
> nifi.properties I executed the following:
>
>
>
> ./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
>
>
>
> No errors found there, then I started Nifi and received the following
> errors:
>
> "
>
> WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web
> server... shutting down.
>
> org.apache.nifi.encrypt.EncryptionException: Decryption Failed with
> Algorithm [AES/GCM/NoPadding]
>
> "
>
>
>
> Since Nifi could not start anymore I reversed it...
>
> Now Im kind of stuck with this warning...
>
> Anyone knows what Im doing wrong?
>
>
>
> Tiago
>
>
>
> *From:* David Handermann [mailto:exceptionfact...@apache.org]
> *Sent:* 19 de outubro de 2022 13:41
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
>
>
>
> Hi Mike,
>
>
>
> The deprecation warning is not related to NIFI-10567 or Sensitive Dynamic
> Properties.
>
>
>
> Deprecation logging is a new feature added in NiFi 1.18.0 to highlight
> components and features that are targeted for removal in future major
> releases. The current administrator's guide has more details on deprecation
> logging. [1] Deprecation warnings do not impact operational behavior, but
> they do identify configuration settings that should be changed.
>
>
>
> In this particular case, the deprecation is related to the use of the
> insecure algorithm.  NiFi 1.14.0 and following introduced new Sensitive
> Properties Key Algorithm settings, which should be used instead of the
> historical default value indicated in the warning. The new default value is
> NIFI_PBKDF2_AES_GCM_256, additional supported options are listed in the
> administrator's guide, [2] along with the command that can be run to update
> the Sensitive Properties Key Algorithm. [3]
>
>
>
> Feel free to follow up if you have additional questions.
>
>
>
> Regards,
>
> David Handermann
>
>
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#deprecation-logging
>
> [2]
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#property-encryption-algorithms
>
> [3]
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-algorithm
>
>
>
> On Wed, Oct 19, 2022 at 7:28 AM Mike S <88msha...@gmail.com> wrote:
>
> I upgraded from 1.16.2 to 1.18.0 and now see this warning in the log file.
>
>
>
> WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider
> Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
> generate salt requested
> org.apache.nifi.deprecation.log.DeprecationException: Reference Class
> [org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider] ClassLoader
> [org.apache.nifi.nar.NarClassLoader[./work/nar/framework/nifi-framework-nar-1.18.0.nar-unpacked]]
>
> I read this here.
>
>
>
> NIFI-10567 <https://issues.apache.org/jira/browse/NIFI-10567> Corrects
> the parsing of Sensitive Dynamic Properties read from the XML version of
> the flow configuration, in absence of the JSON version.
>
> The issue surfaces when upgrading to NiFi 1.17.0 or 1.18.0 from a version
> older than 1.16.0. The issue also requires the presence of a Parameter
> Context with a Sensitive value assigned to a component with a Sensitive
> Property. Upgrading from 1.16.0 and following is not a problem.
>
> It appears that all my ListS3 processors using sensitive properties are
> working.
>
> Is this related since 1.16.2 has the latest flow.json.gz file?
>
>
>
> Mike
>
>

RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

[DSI]

Hi all,

I'm having the same “problem”.
I upgraded nifi version from 1.17.0 to 1.18.0 and that same warning started to 
appear 500k times a day.
"
WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider 
Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] 
generate salt requested
"

A already had nifi.sensitive.props.key value defined from when we migrated to 
1.15.3.

With Nifi STOPPED and without changing any configuration on nifi.properties I 
executed the following:

./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256

No errors found there, then I started Nifi and received the following errors:
"
WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web 
server... shutting down.
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm 
[AES/GCM/NoPadding]
"

Since Nifi could not start anymore I reversed it...
Now Im kind of stuck with this warning...
Anyone knows what Im doing wrong?

Tiago

From: David Handermann [mailto:exceptionfact...@apache.org]
Sent: 19 de outubro de 2022 13:41
To: users@nifi.apache.org
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi Mike,

The deprecation warning is not related to NIFI-10567 or Sensitive Dynamic 
Properties.

Deprecation logging is a new feature added in NiFi 1.18.0 to highlight 
components and features that are targeted for removal in future major releases. 
The current administrator's guide has more details on deprecation logging. [1] 
Deprecation warnings do not impact operational behavior, but they do identify 
configuration settings that should be changed.

In this particular case, the deprecation is related to the use of the insecure 
algorithm.  NiFi 1.14.0 and following introduced new Sensitive Properties Key 
Algorithm settings, which should be used instead of the historical default 
value indicated in the warning. The new default value is 
NIFI_PBKDF2_AES_GCM_256, additional supported options are listed in the 
administrator's guide, [2] along with the command that can be run to update the 
Sensitive Properties Key Algorithm. [3]

Feel free to follow up if you have additional questions.

Regards,
David Handermann

[1] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#deprecation-logging
[2] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#property-encryption-algorithms
[3] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-algorithm

On Wed, Oct 19, 2022 at 7:28 AM Mike S 
<88msha...@gmail.com<mailto:88msha...@gmail.com>> wrote:
I upgraded from 1.16.2 to 1.18.0 and now see this warning in the log file.

WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider 
Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] 
generate salt requested
org.apache.nifi.deprecation.log.DeprecationException: Reference Class 
[org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider] ClassLoader 
[org.apache.nifi.nar.NarClassLoader[./work/nar/framework/nifi-framework-nar-1.18.0.nar-unpacked]]
I read this here.


NIFI-10567<https://issues.apache.org/jira/browse/NIFI-10567> Corrects the 
parsing of Sensitive Dynamic Properties read from the XML version of the flow 
configuration, in absence of the JSON version.

The issue surfaces when upgrading to NiFi 1.17.0 or 1.18.0 from a version older 
than 1.16.0. The issue also requires the presence of a Parameter Context with a 
Sensitive value assigned to a component with a Sensitive Property. Upgrading 
from 1.16.0 and following is not a problem.

It appears that all my ListS3 processors using sensitive properties are working.

Is this related since 1.16.2 has the latest flow.json.gz file?



Mike

Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

[David Handermann]

Hi Mike,

The deprecation warning is not related to NIFI-10567 or Sensitive Dynamic
Properties.

Deprecation logging is a new feature added in NiFi 1.18.0 to highlight
components and features that are targeted for removal in future major
releases. The current administrator's guide has more details on deprecation
logging. [1] Deprecation warnings do not impact operational behavior, but
they do identify configuration settings that should be changed.

In this particular case, the deprecation is related to the use of the
insecure algorithm.  NiFi 1.14.0 and following introduced new Sensitive
Properties Key Algorithm settings, which should be used instead of the
historical default value indicated in the warning. The new default value is
NIFI_PBKDF2_AES_GCM_256, additional supported options are listed in the
administrator's guide, [2] along with the command that can be run to update
the Sensitive Properties Key Algorithm. [3]

Feel free to follow up if you have additional questions.

Regards,
David Handermann

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#deprecation-logging
[2]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#property-encryption-algorithms
[3]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-algorithm

On Wed, Oct 19, 2022 at 7:28 AM Mike S <88msha...@gmail.com> wrote:

> I upgraded from 1.16.2 to 1.18.0 and now see this warning in the log file.
>
> WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider
> Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
> generate salt requested
> org.apache.nifi.deprecation.log.DeprecationException: Reference Class
> [org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider] ClassLoader
> [org.apache.nifi.nar.NarClassLoader[./work/nar/framework/nifi-framework-nar-1.18.0.nar-unpacked]]
>
> I read this here.
>
> NIFI-10567  Corrects
> the parsing of Sensitive Dynamic Properties read from the XML version of
> the flow configuration, in absence of the JSON version.
>
> The issue surfaces when upgrading to NiFi 1.17.0 or 1.18.0 from a version
> older than 1.16.0. The issue also requires the presence of a Parameter
> Context with a Sensitive value assigned to a component with a Sensitive
> Property. Upgrading from 1.16.0 and following is not a problem.
>
> It appears that all my ListS3 processors using sensitive properties are
> working.
>
> Is this related since 1.16.2 has the latest flow.json.gz file?
>
>
> Mike
>

Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

[David Handermann]

Josef,

You're welcome, and thanks for the understanding reply.

I updated the Migration Guidance based on your suggestion, highlighting the
issue when upgrading from 1.15.3 and earlier to 1.17.0 or 1.18.0.

I am also tracking the SFTP issue described in NIFI-9989. This issue is
resolved in the upstream SSHJ main branch, but the resolution for NiFi is
waiting on a new release of the library. This will definitely be
incorporated in upcoming releases, as timing allows.

Thanks again for your reply and community participation!

Regards,
David Handermann

[1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance

On Fri, Oct 14, 2022 at 1:38 AM  wrote:

> Thank you guys for fixing it and for providing the workaround. We searched
> in the release notes for any hints regarding the migration from 1.15.3 to
> 1.18.0 and found nothing, may be a short notice would be great that a
> upgrade from <.1.16.0 to 1.17.0/1.18.0 could brake some of the sensitive
> properties. Not everybody can install every release, so it would help
> others as well …
>
>
>
> We don’t know yet how we will proceed. We are dealing as well with the
> SFTP SSH_MSG_UNIMPLEMENTED message [1], but we wanted to test the newest
> nifi version as there are a lot of bugfixes and new features. Due to the
> complexity of our configuration it’s very likely that we run into an
> upgrade issue which then needs to be fixed first. This is not a complaint,
> it’s just a fact due to the complexity and because of the numbers of
> changes on each and every nifi release in our case more or less on every
> new release where we jump on, we saw some sort of a new introduced issue.
>
>
>
> We just love nifi and the community, it’s really an amazing tool!
>
>
>
> Cheers Josef
>
>
>
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-9989
>
>
>
> *From: *David Handermann 
> *Reply to: *"users@nifi.apache.org" 
> *Date: *Thursday, 13 October 2022 at 23:47
> *To: *"users@nifi.apache.org" 
> *Subject: *Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
>
>
>
> Thanks for reporting this issue Josef, and thanks Mark for outlining the
> background and workaround steps.
>
>
>
> I submitted the the following pull request to address the problem:
>
> https://github.com/apache/nifi/pull/6524
>
> Regards,
>
> David Handermann
>
>
>
> On Thu, Oct 13, 2022 at 8:04 AM Mark Payne  wrote:
>
> Hey Josef,
>
>
>
> I’m sorry about the trouble. It looks like this issue was reported here
> [1]. We are looking into a fix for it.
>
>
>
> Fortunately, if you don’t want to wait for the fix there is a workaround
> available.
>
>
>
> The work around is to follow these steps:
>
> 1. Instead of jumping straight to 1.18, update first to 1.16.4
>
> 2. Start NiFi and wait for it to start up. Ensure that all looks healthy.
>
> 3. Shutdown NiFi
>
> 4. Upgrade to 1.18, ensuring that you copy over the conf/flow.json.gz file
> from 1.16.4
>
>
>
> So essentially, you’d need to upgrade from 1.15 to 1.16, and then to 1.18.
>
>
>
> The reason this works is that prior to 1.16, we stored the flow in
> conf/flow.xml.gz. But in 1.16 we updated that to flow.json.gz - and also
> kept around flow.xml.gz in order to make this change seemless.
>
> But it looks like when Sensitive Dynamic Properties was added, there was a
> bug that caused us to not properly load things from flow.xml.gz, only from
> flow.json.gz.
>
> So, if you upgrade first to 1.16.4, you’ll end up with a flow.json.gz that
> you can then copy over to your 1.18 instance.
>
>
>
> I know this is not ideal, and I apologize for that. But if you’re looking
> to upgrade right away this will be quicker than waiting for a resolution of
> NIFI-10567.
>
>
>
> Thanks!
>
> -Mark
>
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-10567
>
>
>
>
>
> On Oct 13, 2022, at 8:28 AM, josef.zahn...@swisscom.com wrote:
>
>
>
> I just found this blog
> https://exceptionfactory.com/posts/2022/08/02/implementing-apache-nifi-support-for-sensitive-dynamic-properties/
>  about
> the jira ticket
> https://issues.apache.org/jira/browse/NIFI-9957?jql=text%20~%20%22sensitive%20dynamical%22
>  .
> What we found out it is that the controller DBCPConnectionPool is fine with
> the password as well as the invokeHTTP. So for the ones where sensitive
> dynamic properties has been enabled the migration of the password was
> successful, but not for the others…
>
>
>
> Cheers Josef
>
>
>
>
>
> *From: *"Zahner Josef, GSB-LR-TRW-LI" 
> *Date: *Thursday, 13 October 2022 at 13:59
> *To: *"users

Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

[Josef.Zahner1]

Thank you guys for fixing it and for providing the workaround. We searched in 
the release notes for any hints regarding the migration from 1.15.3 to 1.18.0 
and found nothing, may be a short notice would be great that a upgrade from 
<.1.16.0 to 1.17.0/1.18.0 could brake some of the sensitive properties. Not 
everybody can install every release, so it would help others as well …

We don’t know yet how we will proceed. We are dealing as well with the SFTP 
SSH_MSG_UNIMPLEMENTED message [1], but we wanted to test the newest nifi 
version as there are a lot of bugfixes and new features. Due to the complexity 
of our configuration it’s very likely that we run into an upgrade issue which 
then needs to be fixed first. This is not a complaint, it’s just a fact due to 
the complexity and because of the numbers of changes on each and every nifi 
release in our case more or less on every new release where we jump on, we saw 
some sort of a new introduced issue.

We just love nifi and the community, it’s really an amazing tool!

Cheers Josef


[1] https://issues.apache.org/jira/browse/NIFI-9989

From: David Handermann 
Reply to: "users@nifi.apache.org" 
Date: Thursday, 13 October 2022 at 23:47
To: "users@nifi.apache.org" 
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

Thanks for reporting this issue Josef, and thanks Mark for outlining the 
background and workaround steps.

I submitted the the following pull request to address the problem:

https://github.com/apache/nifi/pull/6524
Regards,
David Handermann

On Thu, Oct 13, 2022 at 8:04 AM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Hey Josef,

I’m sorry about the trouble. It looks like this issue was reported here [1]. We 
are looking into a fix for it.

Fortunately, if you don’t want to wait for the fix there is a workaround 
available.

The work around is to follow these steps:
1. Instead of jumping straight to 1.18, update first to 1.16.4
2. Start NiFi and wait for it to start up. Ensure that all looks healthy.
3. Shutdown NiFi
4. Upgrade to 1.18, ensuring that you copy over the conf/flow.json.gz file from 
1.16.4

So essentially, you’d need to upgrade from 1.15 to 1.16, and then to 1.18.

The reason this works is that prior to 1.16, we stored the flow in 
conf/flow.xml.gz. But in 1.16 we updated that to flow.json.gz - and also kept 
around flow.xml.gz in order to make this change seemless.
But it looks like when Sensitive Dynamic Properties was added, there was a bug 
that caused us to not properly load things from flow.xml.gz, only from 
flow.json.gz.
So, if you upgrade first to 1.16.4, you’ll end up with a flow.json.gz that you 
can then copy over to your 1.18 instance.

I know this is not ideal, and I apologize for that. But if you’re looking to 
upgrade right away this will be quicker than waiting for a resolution of 
NIFI-10567.

Thanks!
-Mark

[1] https://issues.apache.org/jira/browse/NIFI-10567



On Oct 13, 2022, at 8:28 AM, 
josef.zahn...@swisscom.com<mailto:josef.zahn...@swisscom.com> wrote:

I just found this blog 
https://exceptionfactory.com/posts/2022/08/02/implementing-apache-nifi-support-for-sensitive-dynamic-properties/
 about the jira ticket 
https://issues.apache.org/jira/browse/NIFI-9957?jql=text%20~%20%22sensitive%20dynamical%22
 . What we found out it is that the controller DBCPConnectionPool is fine with 
the password as well as the invokeHTTP. So for the ones where sensitive dynamic 
properties has been enabled the migration of the password was successful, but 
not for the others…

Cheers Josef


From: "Zahner Josef, GSB-LR-TRW-LI" 
mailto:josef.zahn...@swisscom.com>>
Date: Thursday, 13 October 2022 at 13:59
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Subject: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi guys

We just upgraded from NiFi 1.15.3 to 1.18.0. We have somehow a migration issue, 
it seems that all our sensitive properties are broken with 1.18.0. Check my 
screenshot below, It’s related to controller services as well as to processors. 
All sensitive properties shows us an error. If we replace the password it’s 
fine, so it seems that the password got corrupt due to the upgrade. Any hints? 
It leads to a ton of work, we have hundreds of processors with a hardcoded 
password… I’ve seen that we can use external password providers, do we have to 
migrate somehow?










Cheers Josef



smime.p7s
Description: S/MIME Cryptographic Signature

Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

[David Handermann]

Thanks for reporting this issue Josef, and thanks Mark for outlining the
background and workaround steps.

I submitted the the following pull request to address the problem:

https://github.com/apache/nifi/pull/6524

Regards,
David Handermann

On Thu, Oct 13, 2022 at 8:04 AM Mark Payne  wrote:

> Hey Josef,
>
> I’m sorry about the trouble. It looks like this issue was reported here
> [1]. We are looking into a fix for it.
>
> Fortunately, if you don’t want to wait for the fix there is a workaround
> available.
>
> The work around is to follow these steps:
> 1. Instead of jumping straight to 1.18, update first to 1.16.4
> 2. Start NiFi and wait for it to start up. Ensure that all looks healthy.
> 3. Shutdown NiFi
> 4. Upgrade to 1.18, ensuring that you copy over the conf/flow.json.gz file
> from 1.16.4
>
> So essentially, you’d need to upgrade from 1.15 to 1.16, and then to 1.18.
>
> The reason this works is that prior to 1.16, we stored the flow in
> conf/flow.xml.gz. But in 1.16 we updated that to flow.json.gz - and also
> kept around flow.xml.gz in order to make this change seemless.
> But it looks like when Sensitive Dynamic Properties was added, there was a
> bug that caused us to not properly load things from flow.xml.gz, only from
> flow.json.gz.
> So, if you upgrade first to 1.16.4, you’ll end up with a flow.json.gz that
> you can then copy over to your 1.18 instance.
>
> I know this is not ideal, and I apologize for that. But if you’re looking
> to upgrade right away this will be quicker than waiting for a resolution of
> NIFI-10567.
>
> Thanks!
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-10567
>
>
> On Oct 13, 2022, at 8:28 AM, josef.zahn...@swisscom.com wrote:
>
> I just found this blog
> https://exceptionfactory.com/posts/2022/08/02/implementing-apache-nifi-support-for-sensitive-dynamic-properties/
>  about the jira ticket
> https://issues.apache.org/jira/browse/NIFI-9957?jql=text%20~%20%22sensitive%20dynamical%22
>  . What we found out it is that the controller DBCPConnectionPool is fine
> with the password as well as the invokeHTTP. So for the ones where
> sensitive dynamic properties has been enabled the migration of the password
> was successful, but not for the others…
>
> Cheers Josef
>
>
> *From: *"Zahner Josef, GSB-LR-TRW-LI" 
> *Date: *Thursday, 13 October 2022 at 13:59
> *To: *"users@nifi.apache.org" 
> *Subject: *NiFi 1.18.0 Sensitive Property broken after Upgrade
>
> Hi guys
>
> We just upgraded from NiFi 1.15.3 to 1.18.0. We have somehow a migration
> issue, it seems that all our sensitive properties are broken with 1.18.0.
> Check my screenshot below, It’s related to controller services as well as
> to processors. All sensitive properties shows us an error. If we replace
> the password it’s fine, so it seems that the password got corrupt due to
> the upgrade. Any hints? It leads to a ton of work, we have hundreds of
> processors with a hardcoded password… I’ve seen that we can use external
> password providers, do we have to migrate somehow?
>
>
>
> 
>
>
>
> 
>
>
> Cheers Josef
>
>
>

Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

[Mark Payne]

Hey Josef,

I’m sorry about the trouble. It looks like this issue was reported here [1]. We 
are looking into a fix for it.

Fortunately, if you don’t want to wait for the fix there is a workaround 
available.

The work around is to follow these steps:
1. Instead of jumping straight to 1.18, update first to 1.16.4
2. Start NiFi and wait for it to start up. Ensure that all looks healthy.
3. Shutdown NiFi
4. Upgrade to 1.18, ensuring that you copy over the conf/flow.json.gz file from 
1.16.4

So essentially, you’d need to upgrade from 1.15 to 1.16, and then to 1.18.

The reason this works is that prior to 1.16, we stored the flow in 
conf/flow.xml.gz. But in 1.16 we updated that to flow.json.gz - and also kept 
around flow.xml.gz in order to make this change seemless.
But it looks like when Sensitive Dynamic Properties was added, there was a bug 
that caused us to not properly load things from flow.xml.gz, only from 
flow.json.gz.
So, if you upgrade first to 1.16.4, you’ll end up with a flow.json.gz that you 
can then copy over to your 1.18 instance.

I know this is not ideal, and I apologize for that. But if you’re looking to 
upgrade right away this will be quicker than waiting for a resolution of 
NIFI-10567.

Thanks!
-Mark

[1] https://issues.apache.org/jira/browse/NIFI-10567


On Oct 13, 2022, at 8:28 AM, 
josef.zahn...@swisscom.com wrote:

I just found this blog 
https://exceptionfactory.com/posts/2022/08/02/implementing-apache-nifi-support-for-sensitive-dynamic-properties/
 about the jira ticket 
https://issues.apache.org/jira/browse/NIFI-9957?jql=text%20~%20%22sensitive%20dynamical%22
 . What we found out it is that the controller DBCPConnectionPool is fine with 
the password as well as the invokeHTTP. So for the ones where sensitive dynamic 
properties has been enabled the migration of the password was successful, but 
not for the others…

Cheers Josef


From: "Zahner Josef, GSB-LR-TRW-LI" 
mailto:josef.zahn...@swisscom.com>>
Date: Thursday, 13 October 2022 at 13:59
To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Subject: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi guys

We just upgraded from NiFi 1.15.3 to 1.18.0. We have somehow a migration issue, 
it seems that all our sensitive properties are broken with 1.18.0. Check my 
screenshot below, It’s related to controller services as well as to processors. 
All sensitive properties shows us an error. If we replace the password it’s 
fine, so it seems that the password got corrupt due to the upgrade. Any hints? 
It leads to a ton of work, we have hundreds of processors with a hardcoded 
password… I’ve seen that we can use external password providers, do we have to 
migrate somehow?










Cheers Josef