A lot of this specific information is buried in the Admin Guide, but I agree with your thoughts in general.
Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 17, 2017, at 2:36 PM, Russell Bateman <[email protected]> wrote: > > I thank all of you for this exhaustive treatment. Sometimes I regret that > what flies around in forum threads never makes it into the docs. > > On 02/17/2017 02:32 PM, Andy LoPresto wrote: >> Russ, >> >> Both Bryan and Joe are exactly correct in their responses. To add some >> context, each sensitive property value is encrypted using the specified >> algorithm in nifi.properties. By default, and therefore on any unmodified >> system, this is PBEWITHMD5AND256BITAES-CBC-OPENSSL, which means that the >> actual cipher applied is AES with CBC mode of operation [1] using a 256-bit >> key which is derived from the provided “nifi.sensitive.props.key” value >> (which is really a password, not a true key) using the OpenSSL >> EVP_BytesToKey [2] key derivation function (KDF) — a modified version of >> PKCS #5 v1.5’s MD5-based PBKDF1 [3] which is identical if the combined >> length of the key and IV is less than or equal to an MD5 digest and >> non-standard if greater. >> >> A random 16 byte salt is generated during the encrypt process (the salt >> length is actually set to be the block length of the cipher in question — >> AES is always 16 bytes, but if you choose to use DES for example, this value >> would be 8 bytes), and this salt is provided, along with an iteration count >> (1000 by default) to the KDF. Internally to the application, this process >> occurs within the Jasypt library (slated for removal in NIFI-3116 [4]) but >> you can see a compatible Java native implementation of this process in the >> ConfigEncryptionTool I wrote [5]. >> >> If the “nifi.sensitive.props.key” value is empty, a hard-coded password is >> used as Bryan mentioned. This is why ALL DOCUMENTATION should recommend >> setting that value to a unique and strong passphrase prior to deployment [6]. >> >> Once the key is derived, the sensitive value is encrypted and the salt and >> cipher text are concatenated and then encoded in hexadecimal. This output is >> wrapped in “enc{“ and “}” tokens to denote the value is encrypted, and then >> stored in the flow.xml.gz. As Joe and Bryan pointed out, the sensitive value >> (in plaintext or encrypted form) is never sent over the REST API to any >> client, including the UI. When editing a processor property that is >> sensitive, the UI displays a static placeholder. >> >> If two values are the same (i.e. the same password in an EncryptContent >> processor encrypting and an EncryptContent processor decrypting later in the >> same flow), the two encrypted values will be completely different in the >> flow.xml.gz even though they were encrypted with the same key. This is >> because of the random salt value mentioned above. >> >> The encrypt-config.sh script [7] in nifi-toolkit exposes the functionality >> to “migrate” the flow.xml.gz encrypted values to be encrypted with a new >> key. For example, if you read the above and feel uncomfortable with your >> values encrypted with the default hard-coded password rather than a unique >> passphrase, you can run this tool and provide a new passphrase, and it will >> update nifi.properties with the new passphrase (and encrypt it if you have >> encrypted configuration [8] enabled) and decrypt all values in the >> flow.xml.gz and re-encrypt them with the new key and re-populate them. >> >> A small note on the migration tool — you may notice that after running it, >> identical values will have identical encrypted values. This design decision >> was made because the performance tradeoff of not re-performing the KDF with >> a unique salt and cipher initialization on each value is immense, and the >> threat model is weak because the key is already present on the same system. >> This decision was further impacted by the process by which Jasypt derives >> the keys. After NIFI-3116 is completed, I feel confident we can improve the >> security of the cipher texts by using unique IVs without incurring the >> substantial penalty currently levied by the library structure. >> >> Throughout this explanation, you may have had concerns about some of the >> decisions made (algorithms selected, default values, etc.). There are >> ongoing efforts to improve these points, as security is an evolving process >> and attacks and computational processing availability to attackers is always >> increasing. More of this is available in relevant Jiras [9][10][11] and the >> Security Features Roadmap on the wiki [12], but in general I am looking to >> harden the system by restricting the algorithms used by default and >> available in general and increasing the cost of all key derivations (both in >> time and memory hardness via PBKDF2, bcrypt, and scrypt [13]). Obviously the >> obstacles in this effort are backwards compatibility and legacy support. It >> is a balance, but with the support of the community, I see us continuing to >> move forward with regards to security while making the user experience even >> easier. >> >> I hope that provided some helpful information in response to your question. >> >> >> [1] >> https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_Block_Chaining_.28CBC.29 >> [2] https://www.openssl.org/docs/man1.1.0/crypto/EVP_BytesToKey.html >> [3] https://tools.ietf.org/html/rfc2898#section-5.1 >> [4] https://issues.apache.org/jira/browse/NIFI-3116 >> [5] >> https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy#L555 >> [6] >> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi >> [7] >> https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-assembly/src/main/resources/bin/encrypt-config.sh >> [8] >> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#encrypted-passwords-in-configuration-files >> [9] https://issues.apache.org/jira/browse/NIFI-2656 >> [10] https://issues.apache.org/jira/browse/NIFI-2653 >> [11] https://issues.apache.org/jira/browse/NIFI-1465 >> <https://issues.apache.org/jira/browse/NIFI-1465> >> <https://issues.apache.org/jira/browse/NIFI-1480 >> <https://issues.apache.org/jira/browse/NIFI-1480>> >> [12] >> https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap >> <https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap> >> [13] https://github.com/apache/nifi/pull/201 >> <https://github.com/apache/nifi/pull/201> >> >> >> Andy LoPresto >> [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> /[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>/ >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >>> On Feb 17, 2017, at 11:40 AM, Bryan Bende <[email protected] >>> <mailto:[email protected]> <mailto:[email protected] >>> <mailto:[email protected]>>> wrote: >>> >>> Russell, >>> >>> Sensitive property values are stored in the flow.xml.gz in encrypted >>> form. The encryption is based on the value of >>> nifi.sensitive.props.key= and if you haven't set one there is a >>> default in the code. If you have the same sensitive properties key in >>> both instances than you can copy the flow, or if you didn't set one in >>> both instances which means you have the same. If you had different >>> keys then one instance wouldn't be able to decrypt the values from the >>> other instance. >>> >>> Thanks, >>> >>> Bryan >>> >>> On Fri, Feb 17, 2017 at 2:35 PM, Russell Bateman <[email protected] >>> <mailto:[email protected]> <mailto:[email protected] >>> <mailto:[email protected]>>> wrote: >>>> >>>> 1. Where are the values of sensitive-value properties stored? >>>> 2. How do they work? (if not obvious from #1) >>>> 3. Can /flow.xml.g//z/ be copied wholesale or in part to another >>>> instance of NiFi? >>>> >>>> Thanks, >>>> >>>> Russ
signature.asc
Description: Message signed with OpenPGP using GPGMail
