Russ, Both Bryan and Joe are exactly correct in their responses. To add some context, each sensitive property value is encrypted using the specified algorithm in nifi.properties. By default, and therefore on any unmodified system, this is PBEWITHMD5AND256BITAES-CBC-OPENSSL, which means that the actual cipher applied is AES with CBC mode of operation [1] using a 256-bit key which is derived from the provided “nifi.sensitive.props.key” value (which is really a password, not a true key) using the OpenSSL EVP_BytesToKey [2] key derivation function (KDF) — a modified version of PKCS #5 v1.5’s MD5-based PBKDF1 [3] which is identical if the combined length of the key and IV is less than or equal to an MD5 digest and non-standard if greater.
A random 16 byte salt is generated during the encrypt process (the salt length
is actually set to be the block length of the cipher in question — AES is
always 16 bytes, but if you choose to use DES for example, this value would be
8 bytes), and this salt is provided, along with an iteration count (1000 by
default) to the KDF. Internally to the application, this process occurs within
the Jasypt library (slated for removal in NIFI-3116 [4]) but you can see a
compatible Java native implementation of this process in the
ConfigEncryptionTool I wrote [5].
If the “nifi.sensitive.props.key” value is empty, a hard-coded password is used
as Bryan mentioned. This is why ALL DOCUMENTATION should recommend setting that
value to a unique and strong passphrase prior to deployment [6].
Once the key is derived, the sensitive value is encrypted and the salt and
cipher text are concatenated and then encoded in hexadecimal. This output is
wrapped in “enc{“ and “}” tokens to denote the value is encrypted, and then
stored in the flow.xml.gz. As Joe and Bryan pointed out, the sensitive value
(in plaintext or encrypted form) is never sent over the REST API to any client,
including the UI. When editing a processor property that is sensitive, the UI
displays a static placeholder.
If two values are the same (i.e. the same password in an EncryptContent
processor encrypting and an EncryptContent processor decrypting later in the
same flow), the two encrypted values will be completely different in the
flow.xml.gz even though they were encrypted with the same key. This is because
of the random salt value mentioned above.
The encrypt-config.sh script [7] in nifi-toolkit exposes the functionality to
“migrate” the flow.xml.gz encrypted values to be encrypted with a new key. For
example, if you read the above and feel uncomfortable with your values
encrypted with the default hard-coded password rather than a unique passphrase,
you can run this tool and provide a new passphrase, and it will update
nifi.properties with the new passphrase (and encrypt it if you have encrypted
configuration [8] enabled) and decrypt all values in the flow.xml.gz and
re-encrypt them with the new key and re-populate them.
A small note on the migration tool — you may notice that after running it,
identical values will have identical encrypted values. This design decision was
made because the performance tradeoff of not re-performing the KDF with a
unique salt and cipher initialization on each value is immense, and the threat
model is weak because the key is already present on the same system. This
decision was further impacted by the process by which Jasypt derives the keys.
After NIFI-3116 is completed, I feel confident we can improve the security of
the cipher texts by using unique IVs without incurring the substantial penalty
currently levied by the library structure.
Throughout this explanation, you may have had concerns about some of the
decisions made (algorithms selected, default values, etc.). There are ongoing
efforts to improve these points, as security is an evolving process and attacks
and computational processing availability to attackers is always increasing.
More of this is available in relevant Jiras [9][10][11] and the Security
Features Roadmap on the wiki [12], but in general I am looking to harden the
system by restricting the algorithms used by default and available in general
and increasing the cost of all key derivations (both in time and memory
hardness via PBKDF2, bcrypt, and scrypt [13]). Obviously the obstacles in this
effort are backwards compatibility and legacy support. It is a balance, but
with the support of the community, I see us continuing to move forward with
regards to security while making the user experience even easier.
I hope that provided some helpful information in response to your question.
[1]
https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_Block_Chaining_.28CBC.29
<https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_Block_Chaining_.28CBC.29>
[2] https://www.openssl.org/docs/man1.1.0/crypto/EVP_BytesToKey.html
<https://www.openssl.org/docs/man1.1.0/crypto/EVP_BytesToKey.html>
[3] https://tools.ietf.org/html/rfc2898#section-5.1
<https://tools.ietf.org/html/rfc2898#section-5.1>
[4] https://issues.apache.org/jira/browse/NIFI-3116
<https://issues.apache.org/jira/browse/NIFI-3116>
[5]
https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy#L555
<https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy#L555>
[6]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi
<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi>
[7]
https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-assembly/src/main/resources/bin/encrypt-config.sh
<https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-assembly/src/main/resources/bin/encrypt-config.sh>
[8]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#encrypted-passwords-in-configuration-files
<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#encrypted-passwords-in-configuration-files>
[9] https://issues.apache.org/jira/browse/NIFI-2656
<https://issues.apache.org/jira/browse/NIFI-2656>
[10] https://issues.apache.org/jira/browse/NIFI-2653
<https://issues.apache.org/jira/browse/NIFI-2653>
[11] https://issues.apache.org/jira/browse/NIFI-1465
<https://issues.apache.org/jira/browse/NIFI-1480>
[12] https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap
<https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap>
[13] https://github.com/apache/nifi/pull/201
<https://github.com/apache/nifi/pull/201>
Andy LoPresto
[email protected]
[email protected]
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Feb 17, 2017, at 11:40 AM, Bryan Bende <[email protected]> wrote:
>
> Russell,
>
> Sensitive property values are stored in the flow.xml.gz in encrypted
> form. The encryption is based on the value of
> nifi.sensitive.props.key= and if you haven't set one there is a
> default in the code. If you have the same sensitive properties key in
> both instances than you can copy the flow, or if you didn't set one in
> both instances which means you have the same. If you had different
> keys then one instance wouldn't be able to decrypt the values from the
> other instance.
>
> Thanks,
>
> Bryan
>
> On Fri, Feb 17, 2017 at 2:35 PM, Russell Bateman <[email protected]>
> wrote:
>>
>> 1. Where are the values of sensitive-value properties stored?
>> 2. How do they work? (if not obvious from #1)
>> 3. Can /flow.xml.g//z/ be copied wholesale or in part to another
>> instance of NiFi?
>>
>> Thanks,
>>
>> Russ
>>
>>
>>
signature.asc
Description: Message signed with OpenPGP using GPGMail
