[
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428998#comment-15428998
]
ASF GitHub Bot commented on NIFI-1831:
--------------------------------------
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/834
@joewitt @mattyb149 @bbende @brosander
As of right now, this PR (commit `ffab01d`) is rebased against the latest
master. The tool is found in
`nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.0.0-SNAPSHOT-bin/nifi-toolkit-1.0.0-SNAPSHOT/`.
Running the command below (for example) will read an existing
_nifi.properties_ file, encrypt all sensitive non-empty values using the
provided _key_, populate those values (and the associated protection schemes --
`x.y.z.protected=aes/gcm/256`) into the new _nifi-encrypted.properties_ file,
and persist the key in _bootstrap.conf_.
* `$ ./bin/encrypt-config.sh -h` -- prints a usage message
* `$ ./bin/encrypt-config.sh -b path/to/bootstrap.conf -n
path/to/nifi.properties -o path/to/nifi-encrypted.properties -p
thisIsABadPropertiesPassword` -- normal use as described above
* `$ ./bin/encrypt-config.sh -b path/to/bootstrap.conf -n
path/to/nifi.properties -o path/to/nifi-encrypted.properties -k
0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210` -- normal use
as described above with raw hex key instead of password
* `$ ./bin/encrypt-config.sh -b path/to/bootstrap.conf -n
path/to/nifi.properties -o path/to/nifi-encrypted.properties` -- normal use as
described above but will prompt for key in secure console read
By default, it considers *sensitive* properties as anything that would be a
password or key:
* `nifi.sensitive.props.key`
* `nifi.security.keystorePasswd`
* `nifi.security.keyPasswd`
* `nifi.security.truststorePasswd`
You can mark additional keys as *sensitive* by including them in a comma or
semi-colon delimited string as follows (do this by hand in the input
_nifi.properties_ before running the tool):
`nifi.sensitive.props.additional.keys=nifi.ui.banner.text`
Example:
*before* -- `~/Workspace/scratch/encrypted-configs/nifi.properties`
```
nifi.ui.banner.text=This is the banner text
...
# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=nifi.ui.banner.text
nifi.security.keystore=keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=thisIsABadKeystorePassword
nifi.security.keyPasswd=thisIsABadKeyPassword
nifi.security.truststore=truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=thisIsABadTruststorePassword
nifi.security.needClientAuth=
nifi.security.user.authorizer=file-provider
nifi.security.user.login.identity.provider=
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=
...
```
*run tool* --
```
hw12203:...assembly/target/nifi-toolkit-1.0.0-SNAPSHOT-bin/nifi-toolkit-1.0.0-SNAPSHOT
(NIFI-1831) alopresto
🔓 167s @ 15:49:34 $ ./bin/encrypt-config.sh -b
~/Workspace/scratch/encrypted-configs/bootstrap.conf -n
~/Workspace/scratch/encrypted-configs/nifi.properties -o
~/Workspace/scratch/encrypted-configs/nifi-encrypted.properties -p
thisIsABadPropertiesPassword
2016-08-19 15:57:48,097 INFO [main]
o.a.nifi.properties.ConfigEncryptionTool Invoked ConfigEncryptionTool with args
[-b,/Users/alopresto/Workspace/scratch/encrypted-configs/bootstrap.conf,-n,/Users/alopresto/Workspace/scratch/encrypted-configs/nifi.properties,-o,/Users/alopresto/Workspace/scratch/encrypted-configs/nifi-encrypted.properties,-p,thisIsABadPropertiesPassword]
2016-08-19 15:57:48,794 INFO [main]
o.a.nifi.properties.NiFiPropertiesLoader Loaded 112 properties from
/Users/alopresto/Workspace/scratch/encrypted-configs/nifi.properties
2016-08-19 15:57:48,796 INFO [main]
o.a.n.properties.ProtectedNiFiProperties Loaded 112 properties (including 0
protection schemes) into ProtectedNiFiProperties
2016-08-19 15:57:48,800 INFO [main]
o.a.nifi.properties.ConfigEncryptionTool Loaded NiFiProperties instance with
112 properties
2016-08-19 15:57:48,805 INFO [main]
o.a.n.properties.ProtectedNiFiProperties Loaded 112 properties (including 0
protection schemes) into ProtectedNiFiProperties
2016-08-19 15:57:49,149 INFO [main] o.a.n.p.AESSensitivePropertyProvider
AES Sensitive Property Provider encrypted a sensitive value successfully
2016-08-19 15:57:49,151 INFO [main]
o.a.nifi.properties.ConfigEncryptionTool Protected nifi.ui.banner.text with
aes/gcm/256 ->
2ZJZaFqqXl62HB5w||I57IDLE7hYJf2vJmrkC29ZjDztRJT00CVV1QkDiGte4VIfUB+n2X
2016-08-19 15:57:49,152 INFO [main]
o.a.nifi.properties.ConfigEncryptionTool Updated protection key
nifi.ui.banner.text.protected
2016-08-19 15:57:49,152 INFO [main] o.a.n.p.AESSensitivePropertyProvider
AES Sensitive Property Provider encrypted a sensitive value successfully
2016-08-19 15:57:49,152 INFO [main]
o.a.nifi.properties.ConfigEncryptionTool Protected nifi.security.keyPasswd with
aes/gcm/256 ->
icyABIxwN8FBBd6Q||GL+I6P5tv4ThiVXcAYkFaPsQLK2jZHyJui02W2YxydXfA0sEfg
2016-08-19 15:57:49,152 INFO [main]
o.a.nifi.properties.ConfigEncryptionTool Updated protection key
nifi.security.keyPasswd.protected
2016-08-19 15:57:49,152 INFO [main] o.a.n.p.AESSensitivePropertyProvider
AES Sensitive Property Provider encrypted a sensitive value successfully
2016-08-19 15:57:49,153 INFO [main]
o.a.nifi.properties.ConfigEncryptionTool Protected nifi.security.keystorePasswd
with aes/gcm/256 ->
eXpm2ylPEzfxpnEz||5Iqj7wReXHudmu/q7/q4vj2ooxFGN0R+o4VjV0BjboZ7zD6tqIEX3/Eq
2016-08-19 15:57:49,153 INFO [main]
o.a.nifi.properties.ConfigEncryptionTool Updated protection key
nifi.security.keystorePasswd.protected
2016-08-19 15:57:49,153 INFO [main] o.a.n.p.AESSensitivePropertyProvider
AES Sensitive Property Provider encrypted a sensitive value successfully
2016-08-19 15:57:49,153 INFO [main]
o.a.nifi.properties.ConfigEncryptionTool Protected
nifi.security.truststorePasswd with aes/gcm/256 ->
LGB8TUHtBTHemkzF||J5IqJpEB2i6QlFRqCEvgdq0koNPJv/oV4i3znyoAfzUV4laNv3TmWkeHFfs
2016-08-19 15:57:49,153 INFO [main]
o.a.nifi.properties.ConfigEncryptionTool Updated protection key
nifi.security.truststorePasswd.protected
2016-08-19 15:57:49,167 INFO [main]
o.a.n.properties.ProtectedNiFiProperties Loaded 115 properties (including 4
protection schemes) into ProtectedNiFiProperties
2016-08-19 15:57:49,167 INFO [main]
o.a.nifi.properties.ConfigEncryptionTool Final result: 115 keys including 4
protected keys
2016-08-19 15:57:49,195 INFO [main]
o.a.n.properties.ProtectedNiFiProperties Loaded 115 properties (including 4
protection schemes) into ProtectedNiFiProperties
hw12203:...assembly/target/nifi-toolkit-1.0.0-SNAPSHOT-bin/nifi-toolkit-1.0.0-SNAPSHOT
(NIFI-1831) alopresto
🔓 495s @ 15:57:50 $
```
*after* -- `nifi-encrypted.properties`
```
nifi.ui.banner.text=2ZJZaFqqXl62HB5w||I57IDLE7hYJf2vJmrkC29ZjDztRJT00CVV1QkDiGte4VIfUB+n2X
nifi.ui.banner.text.protected=aes/gcm/256
...
# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=nifi.ui.banner.text
nifi.security.keystore=keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=eXpm2ylPEzfxpnEz||5Iqj7wReXHudmu/q7/q4vj2ooxFGN0R+o4VjV0BjboZ7zD6tqIEX3/Eq
nifi.security.keystorePasswd.protected=aes/gcm/256
nifi.security.keyPasswd=icyABIxwN8FBBd6Q||GL+I6P5tv4ThiVXcAYkFaPsQLK2jZHyJui02W2YxydXfA0sEfg
nifi.security.keyPasswd.protected=aes/gcm/256
nifi.security.truststore=truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=LGB8TUHtBTHemkzF||J5IqJpEB2i6QlFRqCEvgdq0koNPJv/oV4i3znyoAfzUV4laNv3TmWkeHFfs
nifi.security.truststorePasswd.protected=aes/gcm/256
nifi.security.needClientAuth=
nifi.security.user.authorizer=file-provider
nifi.security.user.login.identity.provider=
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=
```
I am still working on a couple small features like handling secure password
entry to the console instead of just the key (and *then will remove the
debugging of the invocation parameters which includes the key/password*).
Please exercise what is here and report any issues. Thanks.
> Allow encrypted passwords in configuration files
> ------------------------------------------------
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Configuration, Core Framework
> Affects Versions: 0.6.1
> Reporter: Andy LoPresto
> Assignee: Andy LoPresto
> Priority: Critical
> Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best
> practice. While file access can be restricted through OS permissions, these
> configuration files can be accidentally checked into source control, shared
> or deployed to multiple instances, etc.
> NiFi should allow a deployer to provide an encrypted password in the
> configuration file to minimize exposure of the passwords. On application
> start-up, NiFi should decrypt the passwords in memory. NiFi should also
> include a utility to encrypt the raw passwords (and optionally populate the
> configuration files and provide additional metadata in the configuration
> files).
> I am aware this simply shifts the responsibility/delegation of trust from the
> passwords in the properties file to a new location on the same system, but
> mitigating the visibility of the raw passwords in the properties file can be
> one step in a defense in depth approach and is often mandated by security
> policies within organizations using NiFi.
> The key used for encryption should not be hard-coded into the application
> source code, nor should it be universally consistent. The key could be
> determined by reading static information from the deployed system and feeding
> it to a key derivation function based on a cryptographically-secure hash
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce
> upgrade, system migration, and portability issues. These challenges will have
> to be kept in consideration when determining the key derivation process.
> Manual key entry is a possibility, and then the master key would only be
> present in memory, but this prevents automatic reboot on loss of power or
> other recovery scenario.
> This must be backward-compatible to allow systems with plaintext passwords to
> continue operating. Options for achieving this are to only attempt to decrypt
> passwords when a sibling property is present, or to match a specific format.
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv: 0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and
> IV are common test values, and an AEAD cipher is preferable to provide cipher
> text integrity assurances, however OpenSSL does not support the use of AEAD
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and
> with which implementation; the absence of the property would default to a raw
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master)
> alopresto
> 🔓 0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master)
> alopresto
> 🔓 0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master)
> alopresto
> 🔓 0s @ 16:27:09 $ xxd password.enc
> 0000000: 5643 5856 6146 6250 4158 364f 5743 7646 VCXVaFbPAX6OWCvF
> 0000010: 6963 6b76 4a63 7744 3854 6b67 3731 4c76 ickvJcwD8Tkg71Lv
> 0000020: 4d38 6d32 7952 4776 5739 413d 0a M8m2yRGvW9A=.
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master)
> alopresto
> 🔓 0s @ 16:27:16 $ more password.enc
> VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master)
> alopresto
> 🔓 0s @ 16:27:55 $
> {code}
> In {{nifi.properties}}:
> {code}
> nifi.security.keystorePasswd=VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=
> nifi.security.keystorePasswd.encrypted=AES-CBC-256
> {code}
> Example 2: *here the encrypted password has a header tag indicating both that
> it is encrypted and the algorithm used*
> {code:java}
> @Test
> public void testShouldDecryptPassword() throws Exception {
> // Arrange
> KeyedCipherProvider cipherProvider = new AESKeyedCipherProvider()
> final String PLAINTEXT = "thisIsABadPassword"
> logger.info("Expected: ${Hex.encodeHexString(PLAINTEXT.bytes)}")
> final byte[] IV = Hex.decodeHex("0123456789ABCDEFFEDCBA9876543210" as
> char[])
> final byte[] LOCAL_KEY =
> Hex.decodeHex("0123456789ABCDEFFEDCBA9876543210" * 2 as char[])
> // Generated via openssl enc -a
> final String CIPHER_TEXT =
> "VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A="
> byte[] cipherBytes = Base64.decoder.decode(CIPHER_TEXT)
> SecretKey localKey = new SecretKeySpec(LOCAL_KEY, "AES")
> EncryptionMethod encryptionMethod = EncryptionMethod.AES_CBC
> logger.info("Using algorithm: ${encryptionMethod.getAlgorithm()}")
> logger.info("Cipher text: \$nifipw\$${CIPHER_TEXT}
> ${cipherBytes.length + 8}")
> // Act
> Cipher cipher = cipherProvider.getCipher(encryptionMethod, localKey,
> IV, false)
> byte[] recoveredBytes = cipher.doFinal(cipherBytes)
>
> // OpenSSL adds a newline character during encryption
> String recovered = new String(recoveredBytes, "UTF-8").trim()
> logger.info("Recovered: ${recovered}
> ${Hex.encodeHexString(recoveredBytes)}")
> // Assert
> assert PLAINTEXT.equals(recovered)
> }
> {code}
> In {{nifi.properties}}:
> {code}
> nifi.security.keystorePasswd=$nifipw$VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=
> {code}
> Ideally, NiFi would use a pluggable implementation architecture to allow
> users to integrate with a variety of secret management services. There are
> both commercial and open source solutions, including CyberArk Enterprise
> Password Vault [1], Hashicorp Vault [2], and Square Keywhiz [3]. In the
> future, this could also be extended to Hardware Security Modules (HSM) like
> SafeNet Luna [4] and Amazon CloudHSM [5].
> [1]
> http://www.cyberark.com/products/privileged-account-security-solution/enterprise-password-vault/
> [2] https://www.vaultproject.io/
> [3] https://square.github.io/keywhiz/
> [4]
> http://www.safenet-inc.com/data-encryption/hardware-security-modules-hsms/luna-hsms-key-management/luna-sa-network-hsm/
> [5] https://aws.amazon.com/cloudhsm/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
