Thanks Juan. A couple notes: Using the same plaintext value for multiple keys will not cause a technical problem, but it is bad security practice and is strongly discouraged. It would not be the source of the issue here (however, you need to use a fully-formed AES key for the provenance encryption key, and it’s unlikely that would be the same value or format as a password for the sensitive properties. That can cause other problems later on).
As you are using the plain WriteAheadProvenanceRepository and not the EncryptedWriteAheadProvenanceRepository, you do not need to provide (and in fact, they are currently ignored) any properties for nifi.provenance.encryption.*. So you can remove those lines entirely (and probably should just for clarity and not to confuse anyone else who looks at these properties). If you want to use the encrypted repository, you’ll need to change the repository implementation (see step-by-step details in the link I provided earlier). The nested exception was that one of the encrypted properties did not contain the “||” delimiter. From visual inspection, it appears that all properties you have listed here do contain the delimiter. That exception is only thrown in one condition, and that is a simple string contains check for the delimiter. Are you sure these are the only encrypted values in your nifi.properties file, and that you are referencing the correct file? Can you look for any other entries of the form “nifi.xyz.protected=“? You mentioned that it generates two unique entries for “nifi.provenance.repository.encryption.key” and you remove the plaintext one. Are you sure that is being removed? If the system believes that property is encrypted (as indicated by the nifi.provenance.repository.encryption.key.protected=aes/gcm/256” line following it) and tries to decrypt the plaintext value, that would cause the exception to be thrown. Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Dec 9, 2019, at 2:22 PM, Juan Pablo Gardella <[email protected]> > wrote: > > Thanks for answering my questions Andy, > > Below are the sensitive properties: > > # Provenance Repository Properties > nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository > nifi.provenance.repository.debug.frequency=1_000_000 > nifi.provenance.repository.encryption.key=fbRg/ZgK7U8qJcrU||4nI1n1aRD0Tooq7TLSTyVDhkmX8 > nifi.provenance.repository.encryption.key.protected=aes/gcm/256 > nifi.provenance.repository.encryption.key.provider.location= > nifi.provenance.repository.encryption.key.id > <http://nifi.provenance.repository.encryption.key.id/>= > # security properties # > nifi.sensitive.props.key=jtZiGY+mZyHPQIc1||/IJnMQBBXKN7VNkwMf6Oo7vZmAs > nifi.sensitive.props.key.protected=aes/gcm/256 > nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL > nifi.sensitive.props.provider=BC > nifi.sensitive.props.additional.keys= > > nifi.security.keystore=/opt/certs/keystore.jks > nifi.security.keystoreType=JKS > nifi.security.keystorePasswd=GuuOm4fyK6yvo76H||av/NQmH7Hw8qK9k0NOMRSjp08tw+walt4D5JLpYPiCHG/Z7DDq5QZ+ui/dKOXxtapH76Gjpt3hMwmP0 > nifi.security.keystorePasswd.protected=aes/gcm/256 > nifi.security.keyPasswd=y4spsJvsy5Fzc3Uq||Q1vMntNgfLLMMSJuyPNn8+9aHlH+banQy82Ly0qrLWf6hNUTNgA+akyh86rlf2J5XZCONL3JCLX6mY0 > nifi.security.keyPasswd.protected=aes/gcm/256 > nifi.security.truststore=/opt/certs/truststore.jks > nifi.security.truststoreType=JKS > nifi.security.truststorePasswd=9r+fyOSjRUXQLcZG||YwAtPYorADqHSKFUmU4H3SbyqvYqqYNZiGidgCOUCibPdP2jiEAMGtLt5xyFsMcNPm5Pye2qXEioLR8 > nifi.security.truststorePasswd.protected=aes/gcm/256 > > These properties are generated by the toolkit. I using the same value for > nifi.sensitive.props.key value and the > nifi.provenance.repository.encryption.key, I was not aware they should be > different. Could be that the problem? > > Juan > > On Mon, 9 Dec 2019 at 08:20, Andy LoPresto <[email protected] > <mailto:[email protected]>> wrote: > Hi Juan, > > The error you are getting is saying that one of the protected properties is > not of the expected format. While the Sensitive Property Provider mechanism > is extensible (see NIFI-5481 [1] for additional options being added), the > only natively supported one in 1.10.0 is AES/GCM encryption. This requires > the sensitive properties to be in the format > > Wl9bXjSWX5DXs4Gm||EDnf18wwAAMJFckgNNfkRWiA4daSDWJCuRvSsbe99AaefQrkpmSqehJtyJGgEbhn402zSyztXi1EGPU > > Where the segment preceding the “||” delimiter is the Base64-encoded 16 byte > initialization vector (IV), which is random and unique for each property, and > the segment following the delimiter is the Base64-encoded cipher text. > > The error states that when NiFi tries to decrypt one of the five encrypted > properties (it does not specify which in this case), it is not encoded in the > proper form. Assuming you are using a strong key for > nifi.bootstrap.sensitive.key in conf/bootstrap.conf, you can share the > nifi.properties file with the encoded and encrypted values with this list to > be verified for format, as no one will be able to decrypt them. However, if > you do not wish to share them, please validate that they are all of the > format specified above and encrypted with the same key that is present in > bootstrap.conf. > > Another thing I noted is that you are replacing the nifi.sensitive.props.key > value and the nifi.provenance.repository.encryption.key value with the same > environment variable. These keys should not have the same value. The > provenance repository key is designed to protect the provenance repository on > disk and be rotated/migrated automatically. The formatting and provision of > these keys is documented in the User Guide [2]. The key can be present in > plaintext (raw hexadecimal encoding) or encrypted as any other sensitive > configuration value in the nifi.properties file. > > The nifi.sensitive.props.key value is a password or other key derivation > material used by NiFi to derive a strong key to encrypt the sensitive > _property_ values - this means things like database passwords, FTP server > passwords, keystore passwords, etc. that the NiFi flow uses and persists in > an encrypted format in the flow.xml.gz file. > > If you believe the sensitive properties key you are injecting into the file > is in the correct format (encoded as described above), check the value of > your master key to ensure it is the same key that encrypted that value. If > you are injecting a plaintext value like “my_bad_sensitive_props_password”, > you must remove the master key from the bootstrap.conf file and ensure there > is no sibling property present called NiFi.sensitive.props.key.protected > which indicates that the value must be decrypted. > > I.e. the existing section like: > > nifi.sensitive.props.key=xPqEWK8a34r19J4z||UOFzOfZE/NQK4Xua8WWblf1/Ld+Pf7eQ1zg0U/qYW2sPwxyhhOXWwQmrUft6qA > nifi.sensitive.props.key.protected=aes/gcm/128 > > Should change to look like: > > nifi.sensitive.props.key=my_bad_sensitive_props_password > NiFi.sensitive.props.key.protected= # or remove this line entirely > > > [1] https://github.com/apache/nifi/pull/3672 > <https://github.com/apache/nifi/pull/3672> > [2] > https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#encrypted-provenance > > <https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#encrypted-provenance> > > > Andy LoPresto > [email protected] <mailto:[email protected]> > [email protected] <mailto:[email protected]> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Dec 8, 2019, at 8:01 PM, Juan Pablo Gardella <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hello all, >> >> I am trying to protect plain text passwords. I am using the latest docker >> image (1.10.0), and edited manually nifi.sensitive.props.key as below >> >> sed -i -e >> "s|^nifi.sensitive.props.key=.*$|nifi.sensitive.props.key=${NIFI_SENSITIVE_PROPS_KEY}|" >> /opt/nifi/nifi-current/conf/nifi.properties >> sed -i -e >> "s|^nifi.provenance.repository.encryption.key=.*$|nifi.provenance.repository.encryption.key=${NIFI_SENSITIVE_PROPS_KEY}|" >> /opt/nifi/nifi-current/conf/nifi.properties >> >> (this command for some reason does not update the file inside the >> Dockerfile, I have to do inside the container). >> >> After updated that property, I run following command inside the container: >> >> bash /opt/nifi/nifi-toolkit-current/bin/encrypt-config.sh -n >> /opt/nifi/nifi-current/conf/nifi.properties -b >> /opt/nifi/nifi-current/conf/bootstrap.conf -a >> /opt/nifi/nifi-current/conf/authorizers.xml -l >> /opt/nifi/nifi-current/conf/login-identity-providers.xml >> >> It prompts to put a master password and after that, I restart[1] the >> container but it failed to start with below error: >> >> nifi | 2019-12-08 18:57:31,777 INFO [main] >> o.a.nifi.properties.NiFiPropertiesLoader Loaded 162 properties from >> /opt/nifi/nifi-current/./conf/nifi.properties >> nifi | 2019-12-08 18:57:31,933 INFO [main] >> o.a.n.properties.ProtectedNiFiProperties There are 5 protected properties of >> 5 sensitive properties (100%) >> nifi | 2019-12-08 18:57:31,935 ERROR [main] org.apache.nifi.NiFi >> Failure to launch NiFi due to java.lang.IllegalArgumentException: There was >> an issue decrypting protected properties >> nifi | java.lang.IllegalArgumentException: There was an issue >> decrypting protected properties >> nifi | at org.apache.nifi.NiFi.initializeProperties(NiFi.java:341) >> nifi | at >> org.apache.nifi.NiFi.convertArgumentsToValidatedNiFiProperties(NiFi.java:309) >> nifi | at org.apache.nifi.NiFi.main(NiFi.java:300) >> nifi | Caused by: java.lang.IllegalArgumentException: The cipher >> text does not contain the delimiter || -- it should be of the form >> Base64(IV) || Base64(cipherText) >> nifi | at >> org.apache.nifi.properties.AESSensitivePropertyProvider.unprotect(AESSensitivePropertyProvider.java:217) >> nifi | at >> org.apache.nifi.properties.ProtectedNiFiProperties.unprotectValue(ProtectedNiFiProperties.java:524) >> nifi | at >> org.apache.nifi.properties.ProtectedNiFiProperties.getUnprotectedProperties(ProtectedNiFiProperties.java:343) >> nifi | at >> org.apache.nifi.properties.NiFiPropertiesLoader.load(NiFiPropertiesLoader.java:209) >> nifi | at >> org.apache.nifi.properties.NiFiPropertiesLoader.load(NiFiPropertiesLoader.java:223) >> nifi | at >> org.apache.nifi.properties.NiFiPropertiesLoader.loadDefault(NiFiPropertiesLoader.java:130) >> nifi | at >> org.apache.nifi.properties.NiFiPropertiesLoader.get(NiFiPropertiesLoader.java:241) >> nifi | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> nifi | at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> nifi | at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> nifi | at java.lang.reflect.Method.invoke(Method.java:498) >> nifi | at org.apache.nifi.NiFi.initializeProperties(NiFi.java:336) >> nifi | ... 2 common frames omitted >> >> Any idea why it is failing? >> >> Thanks, >> Juan >> >> [1] Actually, after that command two entries are generated to >> nifi.provenance.repository.encryption.key= in the file, one with the plain >> text and the other encrypted. I have to remove manually the plain text one. >
