I thank all of you for this exhaustive treatment. Sometimes I regret
that what flies around in forum threads never makes it into the docs.
On 02/17/2017 02:32 PM, Andy LoPresto wrote:
Russ,
Both Bryan and Joe are exactly correct in their responses. To add some
context, each sensitive property value is encrypted using the
specified algorithm in nifi.properties. By default, and therefore on
any unmodified system, this is PBEWITHMD5AND256BITAES-CBC-OPENSSL,
which means that the actual cipher applied is AES with CBC mode of
operation [1] using a 256-bit key which is derived from the provided
“nifi.sensitive.props.key” value (which is really a password, not a
true key) using the OpenSSL EVP_BytesToKey [2] key derivation function
(KDF) — a modified version of PKCS #5 v1.5’s MD5-based PBKDF1 [3]
which is identical if the combined length of the key and IV is less
than or equal to an MD5 digest and non-standard if greater.
A random 16 byte salt is generated during the encrypt process (the
salt length is actually set to be the block length of the cipher in
question — AES is always 16 bytes, but if you choose to use DES for
example, this value would be 8 bytes), and this salt is provided,
along with an iteration count (1000 by default) to the KDF. Internally
to the application, this process occurs within the Jasypt library
(slated for removal in NIFI-3116 [4]) but you can see a compatible
Java native implementation of this process in the ConfigEncryptionTool
I wrote [5].
If the “nifi.sensitive.props.key” value is empty, a hard-coded
password is used as Bryan mentioned. This is why ALL DOCUMENTATION
should recommend setting that value to a unique and strong passphrase
prior to deployment [6].
Once the key is derived, the sensitive value is encrypted and the salt
and cipher text are concatenated and then encoded in hexadecimal. This
output is wrapped in “enc{“ and “}” tokens to denote the value is
encrypted, and then stored in the flow.xml.gz. As Joe and Bryan
pointed out, the sensitive value (in plaintext or encrypted form) is
never sent over the REST API to any client, including the UI. When
editing a processor property that is sensitive, the UI displays a
static placeholder.
If two values are the same (i.e. the same password in an
EncryptContent processor encrypting and an EncryptContent processor
decrypting later in the same flow), the two encrypted values will be
completely different in the flow.xml.gz even though they were
encrypted with the same key. This is because of the random salt value
mentioned above.
The encrypt-config.sh script [7] in nifi-toolkit exposes the
functionality to “migrate” the flow.xml.gz encrypted values to be
encrypted with a new key. For example, if you read the above and feel
uncomfortable with your values encrypted with the default hard-coded
password rather than a unique passphrase, you can run this tool and
provide a new passphrase, and it will update nifi.properties with the
new passphrase (and encrypt it if you have encrypted configuration [8]
enabled) and decrypt all values in the flow.xml.gz and re-encrypt them
with the new key and re-populate them.
A small note on the migration tool — you may notice that after running
it, identical values will have identical encrypted values. This design
decision was made because the performance tradeoff of not
re-performing the KDF with a unique salt and cipher initialization on
each value is immense, and the threat model is weak because the key is
already present on the same system. This decision was further impacted
by the process by which Jasypt derives the keys. After NIFI-3116 is
completed, I feel confident we can improve the security of the cipher
texts by using unique IVs without incurring the substantial penalty
currently levied by the library structure.
Throughout this explanation, you may have had concerns about some of
the decisions made (algorithms selected, default values, etc.). There
are ongoing efforts to improve these points, as security is an
evolving process and attacks and computational processing availability
to attackers is always increasing. More of this is available in
relevant Jiras [9][10][11] and the Security Features Roadmap on the
wiki [12], but in general I am looking to harden the system by
restricting the algorithms used by default and available in general
and increasing the cost of all key derivations (both in time and
memory hardness via PBKDF2, bcrypt, and scrypt [13]). Obviously the
obstacles in this effort are backwards compatibility and legacy
support. It is a balance, but with the support of the community, I see
us continuing to move forward with regards to security while making
the user experience even easier.
I hope that provided some helpful information in response to your
question.
[1]
https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_Block_Chaining_.28CBC.29
[2] https://www.openssl.org/docs/man1.1.0/crypto/EVP_BytesToKey.html
[3] https://tools.ietf.org/html/rfc2898#section-5.1
[4] https://issues.apache.org/jira/browse/NIFI-3116
[5]
https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy#L555
[6]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi
[7]
https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-assembly/src/main/resources/bin/encrypt-config.sh
[8]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#encrypted-passwords-in-configuration-files
[9] https://issues.apache.org/jira/browse/NIFI-2656
[10] https://issues.apache.org/jira/browse/NIFI-2653
[11] https://issues.apache.org/jira/browse/NIFI-1465
<https://issues.apache.org/jira/browse/NIFI-1480>
[12]
https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap
[13] https://github.com/apache/nifi/pull/201
Andy LoPresto
[email protected] <mailto:[email protected]>
/[email protected] <mailto:[email protected]>/
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Feb 17, 2017, at 11:40 AM, Bryan Bende <[email protected]
<mailto:[email protected]>> wrote:
Russell,
Sensitive property values are stored in the flow.xml.gz in encrypted
form. The encryption is based on the value of
nifi.sensitive.props.key= and if you haven't set one there is a
default in the code. If you have the same sensitive properties key in
both instances than you can copy the flow, or if you didn't set one in
both instances which means you have the same. If you had different
keys then one instance wouldn't be able to decrypt the values from the
other instance.
Thanks,
Bryan
On Fri, Feb 17, 2017 at 2:35 PM, Russell Bateman
<[email protected] <mailto:[email protected]>> wrote:
1. Where are the values of sensitive-value properties stored?
2. How do they work? (if not obvious from #1)
3. Can /flow.xml.g//z/ be copied wholesale or in part to another
instance of NiFi?
Thanks,
Russ
