[
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15416516#comment-15416516
]
ASF GitHub Bot commented on NIFI-1831:
--------------------------------------
GitHub user alopresto opened a pull request:
https://github.com/apache/nifi/pull/834
NIFI-1831 Implemented encrypted configuration capabilities
This is a "beta" PR -- there are still some edge cases I want to clean up
but I wanted to get eyes on it because it will be a large change. Specific
areas of focus for review:
* LICENSE
* `AESSensitivePropertyProvider` -- class which actually performs
encryption/decryption
* `ConfigEncryptionTool` -- command-line class to make the process of
encrypting the configuration values easy
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/alopresto/nifi NIFI-1831
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/834.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #834
----
commit b75cdb0742fd814e0c9c7831e8e0bcbdc48c1223
Author: Andy LoPresto <[email protected]>
Date: 2016-06-11T04:06:55Z
NIFI-1831 Cleaned up trailing whitespace. (+1 squashed commit)
Squashed commits:
[7fc7e0a] NIFI-1831 Adapted Apache Commons Hex and Base64 encoding/decoding
classes and implemented in nifi-properties (avoiding extraneous Maven
dependencies).
Added sanity checks in unit tests.
Added utility join method in StringUtils.
commit f9f173099b026eb78b2fe16c6ad15ab1bdeed8d3
Author: Andy LoPresto <[email protected]>
Date: 2016-06-11T04:18:37Z
NIFI-1831 Added interface and first implementation of
SensitivePropertyProvider.
commit 354596a2e3ea9cde41bd12c0ff4d145e1ab910c4
Author: Andy LoPresto <[email protected]>
Date: 2016-06-11T04:19:13Z
NIFI-1831 Added skeleton for regression tests and new unit tests for
sensitive property loading.
commit 836ee1e84c2b352ee96c8831e02178d95396400c
Author: Andy LoPresto <[email protected]>
Date: 2016-07-14T20:31:17Z
NIFI-1831 Added initial logic in NiFiProperties to detect sensitive
property keys and return list of sensitive properties.
Added unit tests (and resource).
commit dafa426d36e2bb86361ee8c14681addfa31677f9
Author: Andy LoPresto <[email protected]>
Date: 2016-07-14T20:39:17Z
NIFI-1831 Moved Base64 and Hex sanity tests into correct package.
commit eee44e107ddbecbacd8cb44b9c94fd0bbef5ec60
Author: Andy LoPresto <[email protected]>
Date: 2016-07-14T22:58:01Z
NIFI-1831 Added BouncyCastle dependency to nifi-properties module.
commit a0286b55eb1c1f469ae9612a51a3aadb4198f175
Author: Andy LoPresto <[email protected]>
Date: 2016-07-14T22:59:06Z
NIFI-1831 Added SensitivePropertyProtectionException.
Updated unprotect method contract in interface and implementation with new
exception.
Added unprotect logic.
commit 064b2d14c088f7e984af625658011fe9310a759b
Author: Andy LoPresto <[email protected]>
Date: 2016-07-14T22:59:34Z
NIFI-1831 Added unit tests for sensitive property encryption/decryption.
commit 46a7b1b902672843f10758edbe763dcd35f1ffc7
Author: Andy LoPresto <[email protected]>
Date: 2016-07-15T19:11:50Z
NIFI-1831 Finished logic for sensitive property encryption/decryption.
Added negative/edge case unit tests for sensitive property
encryption/decryption.
commit 2d8ec30adfb6f46e1a4302c0fe074d7b132a3cda
Author: Andy LoPresto <[email protected]>
Date: 2016-07-15T19:17:28Z
NIFI-1831 Excluded additional keys property from self-inclusion to avoid
recursive/protection issue.
commit 723523059a1c8fffe90ddbf29a9eb8ac3e2ec32a
Author: Andy LoPresto <[email protected]>
Date: 2016-07-15T19:33:20Z
NIFI-1831 Removed unnecessary duplication of Base64 and Hex encoders as
BouncyCastle includes implementations.
commit b81f89fa8bb20d96d4d2ff1a45d15c2271e88ad6
Author: Andy LoPresto <[email protected]>
Date: 2016-07-15T19:35:53Z
NIFI-1831 Removed unnecessary reference to commons-codec from NOTICE files
as the duplicate code is removed.
commit 323756119e6841e4b6bab234f793386ea1ce13a5
Author: Andy LoPresto <[email protected]>
Date: 2016-07-15T19:36:57Z
NIFI-1831 Removed unnecessary nifi-properties NOTICE file as the duplicate
code is removed.
commit 0798cd12d2712132285284209be51ac40bfd970b
Author: Andy LoPresto <[email protected]>
Date: 2016-07-15T21:25:32Z
NIFI-1831 Fixed tests which were not deterministic (shuffling IV and
ciphertext could occasionally result in same value, rendering successful
decryption and lacking expected exception).
Added license to AESSensitivePropertyProviderTest.
commit 7d9c3a8159a4d06d0a96364e87141de10a298e33
Author: Andy LoPresto <[email protected]>
Date: 2016-07-15T23:20:59Z
NIFI-1831 Added logic to retrieve protected keys and their protection
mechanism from properties.
Added logic to determine percentage of sensitive keys currently protected.
Added unit tests.
Added failing unit test for successful transparent retrieval of protected
value.
commit 6e817f98b6d0e3408afc98102e1657c239dcc1c0
Author: Andy LoPresto <[email protected]>
Date: 2016-07-18T23:42:03Z
NIFI-1831 Improved NiFiProperties Javadoc.
commit 86d298c4781294e82b0ea8f04c9595249811b6eb
Author: Andy LoPresto <[email protected]>
Date: 2016-07-18T23:42:57Z
NIFI-1831 Added accessor method for implementation-specific key to
interface and implementation.
Added unit test.
commit ad0c178796b6674425959032065bdb1e62bea51f
Author: Andy LoPresto <[email protected]>
Date: 2016-07-19T06:35:19Z
NIFI-1831 Added empty AES SPP constructor and key setter.
Added unit test to verify key cannot be changed once set.
Added SPP factory.
Added unit tests for factory.
commit 9160e2cf4ae9d2c21be7381196af99696b7f66c2
Author: Andy LoPresto <[email protected]>
Date: 2016-07-19T21:47:55Z
NIFI-1831 Performed minor refactoring and added unit test.
commit 2433f225bb362c6f22476689f348a3858a361d26
Author: Andy LoPresto <[email protected]>
Date: 2016-07-20T01:36:47Z
NIFI-1831 Implemented logic to unprotect retrieved properties values
transparently.
Refactored AESSensitivePropertyProvider setKey logic out of constructor.
Changed SensitivePropertyProtectionException to RuntimeException.
Added internal localProviderCache to NiFiProperties to handle
initialization and provider registration.
Added logic to short-circuit infinite loop on protected property lookup.
Added unit tests.
Added test resources.
commit d60b6f8528572ad7c830f0285f4ddcd2c5c2a9f8
Author: Andy LoPresto <[email protected]>
Date: 2016-07-20T03:37:23Z
NIFI-1831 Added logic to handle malformed AES-protected property values.
Added unit test.
Added test resource.
commit 7088aac387fd8ae4a66aca274285d736bcf94b3b
Author: Andy LoPresto <[email protected]>
Date: 2016-07-20T23:37:11Z
NIFI-1831 Added unit tests for getProperty(key, defaultValue).
commit 97c1a2037636aa3219a230030fa35ec9ae653db9
Author: Andy LoPresto <[email protected]>
Date: 2016-07-21T06:52:50Z
NIFI-1831 Added necessity checks before instantiating & registering SPP in
NiFiProperties.
Resolved Maven vs. IDE test issues (different JREs with different JCE
policies).
Resolved regression test issues (unnecessary loading of SPP without BC
provided).
commit bacb9bda03c8964225ab3a460823cd5ef74c64bd
Author: Andy LoPresto <[email protected]>
Date: 2016-07-21T17:47:42Z
NIFI-1831 Added capability for external class to set NiFiProperties
protection key (once).
Added unit tests.
commit 606456feaedb2b8f4df69d0e955762d1b7e5a9de
Author: Andy LoPresto <[email protected]>
Date: 2016-07-22T01:15:51Z
NIFI-1831 Added SLF4J test dependency.
commit 7b94b50a079918974855d72547b6c495f4fded85
Author: Andy LoPresto <[email protected]>
Date: 2016-07-22T01:19:00Z
NIFI-1831 Added logic to lazily instantiate the SensitivePropertyProvider
in NiFiProperties#getInstance().
Added logic to get raw property directly for protection keys.
Added logic to read bootstrap-injected key.
Added unit tests.
commit c1aee4bfb566c25a0b9b16cd6acf9d6c3fe8478f
Author: Andy LoPresto <[email protected]>
Date: 2016-07-22T01:20:27Z
NIFI-1831 Added logic to inject bootstrap-provided protection key into
NiFiProperties on NiFi startup.
Added regression test for no key provided in args for properties with no
protected properties (legacy default).
Added unit tests.
Added test resources.
commit 0163870b69a03c18c0b42faf3bfa406dc27b62ba
Author: Andy LoPresto <[email protected]>
Date: 2016-07-22T02:38:24Z
NIFI-1831 Added custom TestAppender to intercept log messages.
Configured logback-test.xml for NiFiTest.
Added logic to handle bootstrap-provided key.
Added logic to format and validate hex keys.
Added unit tests.
commit 4bf2f76ce40ea9dcf13230a7115299a0b20945ce
Author: Andy LoPresto <[email protected]>
Date: 2016-07-22T04:29:28Z
NIFI-1831 Moved NiFiGroovyTest and associated resources from nifi-jetty
module to nifi-runtime.
commit ab0a904e5db2386e45928d47a40683ddb3b24a90
Author: Andy LoPresto <[email protected]>
Date: 2016-07-22T04:30:26Z
NIFI-1831 Reverted nifi-jetty pom.xml and added Logback dependency with
test scope to nifi-runtime pom.xml.
----
> Allow encrypted passwords in configuration files
> ------------------------------------------------
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Configuration, Core Framework
> Affects Versions: 0.6.1
> Reporter: Andy LoPresto
> Assignee: Andy LoPresto
> Priority: Critical
> Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best
> practice. While file access can be restricted through OS permissions, these
> configuration files can be accidentally checked into source control, shared
> or deployed to multiple instances, etc.
> NiFi should allow a deployer to provide an encrypted password in the
> configuration file to minimize exposure of the passwords. On application
> start-up, NiFi should decrypt the passwords in memory. NiFi should also
> include a utility to encrypt the raw passwords (and optionally populate the
> configuration files and provide additional metadata in the configuration
> files).
> I am aware this simply shifts the responsibility/delegation of trust from the
> passwords in the properties file to a new location on the same system, but
> mitigating the visibility of the raw passwords in the properties file can be
> one step in a defense in depth approach and is often mandated by security
> policies within organizations using NiFi.
> The key used for encryption should not be hard-coded into the application
> source code, nor should it be universally consistent. The key could be
> determined by reading static information from the deployed system and feeding
> it to a key derivation function based on a cryptographically-secure hash
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce
> upgrade, system migration, and portability issues. These challenges will have
> to be kept in consideration when determining the key derivation process.
> Manual key entry is a possibility, and then the master key would only be
> present in memory, but this prevents automatic reboot on loss of power or
> other recovery scenario.
> This must be backward-compatible to allow systems with plaintext passwords to
> continue operating. Options for achieving this are to only attempt to decrypt
> passwords when a sibling property is present, or to match a specific format.
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv: 0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and
> IV are common test values, and an AEAD cipher is preferable to provide cipher
> text integrity assurances, however OpenSSL does not support the use of AEAD
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and
> with which implementation; the absence of the property would default to a raw
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master)
> alopresto
> 🔓 0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master)
> alopresto
> 🔓 0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master)
> alopresto
> 🔓 0s @ 16:27:09 $ xxd password.enc
> 0000000: 5643 5856 6146 6250 4158 364f 5743 7646 VCXVaFbPAX6OWCvF
> 0000010: 6963 6b76 4a63 7744 3854 6b67 3731 4c76 ickvJcwD8Tkg71Lv
> 0000020: 4d38 6d32 7952 4776 5739 413d 0a M8m2yRGvW9A=.
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master)
> alopresto
> 🔓 0s @ 16:27:16 $ more password.enc
> VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master)
> alopresto
> 🔓 0s @ 16:27:55 $
> {code}
> In {{nifi.properties}}:
> {code}
> nifi.security.keystorePasswd=VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=
> nifi.security.keystorePasswd.encrypted=AES-CBC-256
> {code}
> Example 2: *here the encrypted password has a header tag indicating both that
> it is encrypted and the algorithm used*
> {code:java}
> @Test
> public void testShouldDecryptPassword() throws Exception {
> // Arrange
> KeyedCipherProvider cipherProvider = new AESKeyedCipherProvider()
> final String PLAINTEXT = "thisIsABadPassword"
> logger.info("Expected: ${Hex.encodeHexString(PLAINTEXT.bytes)}")
> final byte[] IV = Hex.decodeHex("0123456789ABCDEFFEDCBA9876543210" as
> char[])
> final byte[] LOCAL_KEY =
> Hex.decodeHex("0123456789ABCDEFFEDCBA9876543210" * 2 as char[])
> // Generated via openssl enc -a
> final String CIPHER_TEXT =
> "VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A="
> byte[] cipherBytes = Base64.decoder.decode(CIPHER_TEXT)
> SecretKey localKey = new SecretKeySpec(LOCAL_KEY, "AES")
> EncryptionMethod encryptionMethod = EncryptionMethod.AES_CBC
> logger.info("Using algorithm: ${encryptionMethod.getAlgorithm()}")
> logger.info("Cipher text: \$nifipw\$${CIPHER_TEXT}
> ${cipherBytes.length + 8}")
> // Act
> Cipher cipher = cipherProvider.getCipher(encryptionMethod, localKey,
> IV, false)
> byte[] recoveredBytes = cipher.doFinal(cipherBytes)
>
> // OpenSSL adds a newline character during encryption
> String recovered = new String(recoveredBytes, "UTF-8").trim()
> logger.info("Recovered: ${recovered}
> ${Hex.encodeHexString(recoveredBytes)}")
> // Assert
> assert PLAINTEXT.equals(recovered)
> }
> {code}
> In {{nifi.properties}}:
> {code}
> nifi.security.keystorePasswd=$nifipw$VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=
> {code}
> Ideally, NiFi would use a pluggable implementation architecture to allow
> users to integrate with a variety of secret management services. There are
> both commercial and open source solutions, including CyberArk Enterprise
> Password Vault [1], Hashicorp Vault [2], and Square Keywhiz [3]. In the
> future, this could also be extended to Hardware Security Modules (HSM) like
> SafeNet Luna [4] and Amazon CloudHSM [5].
> [1]
> http://www.cyberark.com/products/privileged-account-security-solution/enterprise-password-vault/
> [2] https://www.vaultproject.io/
> [3] https://square.github.io/keywhiz/
> [4]
> http://www.safenet-inc.com/data-encryption/hardware-security-modules-hsms/luna-hsms-key-management/luna-sa-network-hsm/
> [5] https://aws.amazon.com/cloudhsm/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
