Hi Tiago,
I’ve had a similar experience with migrating the flow encryption algorithm and
in fact, some of them are still on the old one. The nifi.sh commands to update
the sensitive properties key and algorithm are very tricky to use, because they
update the nifi.properties file even if the migration fails for the flow.xml.gz
and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following
ones to fail because it tried to decrypt using the new algorithm. I needed to
reset the nifi.properties file everytime.
Another thing I’ve noticed is that it doesn’t support the property file
protection scheme that NiFi has. If your sensitive props key is encrypted you
need to enter the raw value and make sure the
nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.
These steps have mostly worked for me:
1. Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another
dir
2. Unprotect the sensitive properties key:
* Replace the encrypted key with the raw one
* Empty this property: nifi.sensitive.props.key.protected
3. Check that the algorithm is still the old one:
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
4. Check the length of the raw key, it needs to be 12 characters or longer
to migrate to the new algorithm.
5. If the key is empty or short:
* ./bin/nifi.sh set-sensitive-properties-key
PasswordUsedOnNifiProperties (run from the main dir)
* Check the output for any failures and if they occur, revert the
nifi.properties file and fix any errors
* Try to start nifi with the new sensitive properties key
6. If it works, stop nifi and update the algorithm:
* ./bin/nifi.sh set-sensitive-properties-algorithm
NIFI_PBKDF2_AES_GCM_256
* Check the output for any failures and if they occur, revert the
nifi.properties file and fix any errors
* Try to start nifi with the new algorithm
7. Stop nifi, encrypt the sensitive properties key (or restore it if you
didn’t need to change the password)
* Use the encrypt-config.sh from the nifi-toolkit, specify output files
so you can check and compare.
* Beware: this tool doesn’t seem to know about flow.json.gz, so only use
it to change the config files.
I hope this helps you find the solution.
Regards,
Isha
Van: Tiago Luís Sebastião (DSI) <[email protected]>
Verzonden: donderdag 24 november 2022 16:19
Aan: [email protected]
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi again,
Sorry for not following up but other priorities came ahead…
Basically it’s still not working, I’ve tried several combinations and I still
keep getting:
“Failed to process Flow Configuration [./conf/flow.xml.gz]
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm
[AES/GCM/NoPadding]”
After reading some documentation, for this purpose, assuming that the password
configured in the nifi.properties file is “PasswordUsedOnNifiProperties”…
I’ve tried and failed:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties
I’ve tried and failed by setting algorithm to empty string:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties
I’ve tried and failed using the new toolkit (I was using toolkit version
1.13.3):
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f
/apps/nifi-1.18.0/conf/flow.xml.gz -n /apps/nifi-1.18.0/conf/nifi.properties -s
PasswordUsedOnNifiProperties -A NIFI_ARGON2_AES_GCM_256 -x -v
I’ve tried and failed doing the same but generating new files to debug:
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n
/apps/nifi-1.18.0/conf/nifi.properties -o
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s
PasswordUsedOnNifiProperties -x -v
In this last one I noticed that the file flow2.json.gz got its passwords
encrypted differently and a longer encrypt also.
I’m kind of wondering if I can use this last command to generate these files on
the side and then manipulate the nifi.properties file by changing the algorithm
to NIFI_PBKDF2_AES_GCM_256 since it seems it is already encrypted despite the
known errors/warnings, then I would rename these new files to the older ones
and start nifi with:
1. New flow.json.gz file (apparently encrypted with NIFI_PBKDF2_AES_GCM_256
algorithm)
2. New nifi.properties file (with nifi.sensitive.props.algorithm property
manipulated to NIFI_PBKDF2_AES_GCM_256)
Since,
unfortunately<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fsearch%3Fsxsrf%3DALiCzsYtztPHersBtg21lqlpGlc7DZ9CUw%3A1669302812266%26q%3Dunfortunately%26spell%3D1%26sa%3DX%26ved%3D2ahUKEwiq4pbJjcf7AhXKzqQKHfK8DNwQkeECKAB6BAgGEAE&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C14fdf6863d254d1163dd08dace2f467c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638049003651712131%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QkSjVNecwZDyUuzgbK4F9n4DneE8e5vD81BjfHtf224%3D&reserved=0>,
Im getting nowhere with this and I need to migrate to version 1.18.0 in order
to apply the bugfix that changes the
serverConnectorFactory.setNeedClientAuth(wantClientAuth) to
serverConnectorFactory.setWantClientAuth(wantClientAuth) I am needing help in a
consequent situation.
In order to reduce the size of the log generated from the deprecation warnings
(WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider
Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
generate salt requested) I’ve tried to LEVEL OFF that Warning from
stateless-logback.xml file without success.
On the tag <appender name="APP_FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender"> and <appender
name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> I changed the tag
<pattern> so that I could see the full class name (although without success
also…)
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
to
<pattern>%date %level [%thread] %logger{140} %msg%n</pattern>
On the stateless-logback.xml I inserted the following:
<logger name="org.apache.nifi" level="INFO"/>
<logger
name="deprecation.org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider"
level="OFF" />
It’s not working and I don’t understand why, the class name seems to be correct
but I keep getting the same WARN.
Sorry for the long email…
Regards.
Tiago Sebastião
From: Tiago Luís Sebastião (DSI)
Sent: 28 de outubro de 2022 09:48
To: [email protected]<mailto:[email protected]>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi David,
It’s a standalone deployment and runs directly on the server.
Yes the command updated the flow.xml.gz/flow.json.gz and nifi.properties
settings.
Maybe I messed up the nifi.sensitive.props.key, I’ll run some more tests.
Thanks for your help.
Tiago
From: David Handermann [mailto:[email protected]]
Sent: 27 de outubro de 2022 16:50
To: [email protected]<mailto:[email protected]>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Tiago,
The initial warning for the Insecure Cipher Provider Algorithm indicates the
use of the deprecated setting as mentioned previously.
The set-sensitive-properties-algorithm command looks correct, and should have
updated the flow.xml.gz, flow.json.gz, and nifi.properties settings.
The Decryption Failed message indicates that the nifi.sensitive.props.key value
does not match the value used to encrypt the flow configuration, or that the
algorithm does not match.
Can you provide some additional details about the NiFi installation? Is this a
standalone or clustered deployment, and is it running in a containerized
environment, or directly on a server?
Regards,
David Handermann
On Thu, Oct 27, 2022 at 10:35 AM Tiago Luís Sebastião (DSI)
<[email protected]<mailto:[email protected]>> wrote:
Hi all,
I'm having the same “problem”.
I upgraded nifi version from 1.17.0 to 1.18.0 and that same warning started to
appear 500k times a day.
"
WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider
Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
generate salt requested
"
A already had nifi.sensitive.props.key value defined from when we migrated to
1.15.3.
With Nifi STOPPED and without changing any configuration on nifi.properties I
executed the following:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
No errors found there, then I started Nifi and received the following errors:
"
WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web
server... shutting down.
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm
[AES/GCM/NoPadding]
"
Since Nifi could not start anymore I reversed it...
Now Im kind of stuck with this warning...
Anyone knows what Im doing wrong?
Tiago
From: David Handermann
[mailto:[email protected]<mailto:[email protected]>]
Sent: 19 de outubro de 2022 13:41
To: [email protected]<mailto:[email protected]>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Mike,
The deprecation warning is not related to NIFI-10567 or Sensitive Dynamic
Properties.
Deprecation logging is a new feature added in NiFi 1.18.0 to highlight
components and features that are targeted for removal in future major releases.
The current administrator's guide has more details on deprecation logging. [1]
Deprecation warnings do not impact operational behavior, but they do identify
configuration settings that should be changed.
In this particular case, the deprecation is related to the use of the insecure
algorithm. NiFi 1.14.0 and following introduced new Sensitive Properties Key
Algorithm settings, which should be used instead of the historical default
value indicated in the warning. The new default value is
NIFI_PBKDF2_AES_GCM_256, additional supported options are listed in the
administrator's guide, [2] along with the command that can be run to update the
Sensitive Properties Key Algorithm. [3]
Feel free to follow up if you have additional questions.
Regards,
David Handermann
[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#deprecation-logging<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23deprecation-logging&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C14fdf6863d254d1163dd08dace2f467c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638049003651712131%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=e3TvRGGVBqr3SufZI4QqwBMIZcjrZYfatDeSG1BgBlw%3D&reserved=0>
[2]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#property-encryption-algorithms<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23property-encryption-algorithms&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C14fdf6863d254d1163dd08dace2f467c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638049003651712131%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=l9DP6913pDQq1wOp0x00iO982WpN21hdwB%2FjnlzKhhk%3D&reserved=0>
[3]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-algorithm<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23updating-the-sensitive-properties-algorithm&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C14fdf6863d254d1163dd08dace2f467c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638049003651712131%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YfPuNMaiCCRNLz4eHiUO%2BvN7l2rh%2B7Ce9yCdfOFn%2B3w%3D&reserved=0>
On Wed, Oct 19, 2022 at 7:28 AM Mike S
<[email protected]<mailto:[email protected]>> wrote:
I upgraded from 1.16.2 to 1.18.0 and now see this warning in the log file.
WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider
Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
generate salt requested
org.apache.nifi.deprecation.log.DeprecationException: Reference Class
[org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider] ClassLoader
[org.apache.nifi.nar.NarClassLoader[./work/nar/framework/nifi-framework-nar-1.18.0.nar-unpacked]]
I read this here.
NIFI-10567<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-10567&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C14fdf6863d254d1163dd08dace2f467c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638049003651712131%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vyvgkBNyto%2Bt7YdHFPugG3kRP9QLGA%2BlCCsQVWE1Gm0%3D&reserved=0>
Corrects the parsing of Sensitive Dynamic Properties read from the XML version
of the flow configuration, in absence of the JSON version.
The issue surfaces when upgrading to NiFi 1.17.0 or 1.18.0 from a version older
than 1.16.0. The issue also requires the presence of a Parameter Context with a
Sensitive value assigned to a component with a Sensitive Property. Upgrading
from 1.16.0 and following is not a problem.
It appears that all my ListS3 processors using sensitive properties are working.
Is this related since 1.16.2 has the latest flow.json.gz file?
Mike
