[
https://issues.apache.org/jira/browse/NIFI-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127422#comment-15127422
]
ASF GitHub Bot commented on NIFI-1257:
--------------------------------------
GitHub user alopresto opened a pull request:
https://github.com/apache/nifi/pull/201
NIFI-1257 and 1259
NIFI-1257
NIFI-1259
Added the mechanics for additional strong `Key Derivation Functions` (KDF)
including `PBKDF2`, `Bcrypt`, and `Scrypt` as well as `keyed ciphers` (raw key;
no KDF). Integrated with `EncryptContent` processor and added new property and
property descriptors. Dependent properties not available until NIFI-1121 is
implemented.
Explanation of KDFs available here: [Apache NiFi
Wiki](https://cwiki.apache.org/confluence/display/NIFI/Key+Derivation+Function+Explanations)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/alopresto/nifi NIFI-1257-cherrypick
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/201.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #201
----
commit ab566da624914d553188495a906a523ce7cbe209
Author: Andy LoPresto <[email protected]>
Date: 2015-12-22T18:12:02Z
NIFI-1257:
Added first skeleton of KeyDeriver functionality but re-evaluating to
handle OpenSSL and NiFi legacy PBE consistently with modern KDFs.
commit aaef4ec6e5551392fdf441fe2bca5f6a924206ea
Author: Andy LoPresto <[email protected]>
Date: 2015-12-22T19:02:27Z
NIFI-1257 Skeleton of key deriver mechanics and moving existing classes
into crypto package.
Added variable substitution with defaults to logback-test.xml. (+11
squashed commits)
Squashed commits:
[5aa2ae2] NIFI-1257:
Minor formatting changes in logback-test.xml.
[af12eb5] NIFI-1257:
Added CipherFactory.
[3e04f5b] NIFI-1257:
Improved Javadoc in NiFi legacy cipher provider and OpenSSL cipher provider.
[bfdc1d7] NIFI-1257:
Added PBKDF2 cipher provider implementation and Java & Groovy unit test
skeletons.
[20adeef] NIFI-1257:
Added KeyedCipherProvider interface.
[302a864] NIFI-1257:
Moved OpenPGP encryptor unit tests to proper package.
[ce5c231] NIFI-1257:
Added test logback file with variable substitution expressions.
[f2475eb] NIFI-1257:
Added Groovy unit test support to allow faster development cycle.
[4004cfd] NIFI-1257:
Added OpenSSL PKCS#5 v1.5 EVP_BytesToKey cipher provider and unit test.
[332b9dc] NIFI-1257:
Moved crypto classes into new package.
Began refactor from KeyDeriver to CipherProvider.
[5277efc] NIFI-1257:
Added variables with default value to logback-test.xml.
commit 07c1e1219153ee3ab5a030f68f0647ad41bdd3b9
Author: Andy LoPresto <[email protected]>
Date: 2016-01-13T20:25:34Z
NIFI-1257 Committed state before refactor to inherit from shared
CipherProvider interface. (+5 squashed commits)
Squashed commits:
[634802c] NIFI-1257 Added new keyed encryption methods and added boolean
field for compatibility with new KDFs.
[b74d23c] NIFI-1257 Added parseKeyLengthFromAlgorithm utility method.
Added unit tests.
[f3cd3b6] NIFI-1257 Moved cipher algorithm parsing utility method to common
util class.
[b0e785e] NIFI-1257 Applied contrib-check fixes for line length.
[94a6ecf] NIFI-1257 Added Apache License to various tests.
commit 106de28eeda3838413226185dfe52e0c83bbf04a
Author: Andy LoPresto <[email protected]>
Date: 2016-01-14T02:00:34Z
NIFI-1257 Implemented PBKDF2 cipher provider.
Added default constructor with strong choices for PBKDF2 cipher provider.
(+12 squashed commits)
Squashed commits:
[19f94a0] NIFI-1257 Implemented NiFiLegacyCipherProvider and added unit
tests.
[efcb5f6] NIFI-1257 Added convenience methods in OpenSSLPKCS5 provider that
ignore desired key lengths (all OpenSSL algorithms specify key length in
algorithm).
[0381a06] NIFI-1257 Moved BC provider loading in test to single static
invocation (decreases test time by ~40%).
[b5a8682] NIFI-1257 Added ASF license to openssl_pbkdf2.rb.
[87cfca0] NIFI-1257 Test cleanup.
[cb3598b] NIFI-1257 Added key length parameter to PBKDF2 cipher provider.
[aa50006] NIFI-1257 Added PRF resolution to PBKDF2 cipher provider.
[9dfb6b0] NIFI-1257 Changed PBKDF2 and OpenSSLPKCS5 implementations to
accept EncryptionMethod instead of individual algorithm and provider
combinations.
[d6156bd] NIFI-1257 Added test resource to generate PBKDF2-derived key and
encrypt content using OpenSSL via Ruby.
[6cee174] NIFI-1257 Removed duplicated test.
[5ff87a4] NIFI-1257 Temporarily commented out other implementations while
interface changes.
[2f1ac0c] NIFI-1257 Added RandomIVPBECipherProvider to allow for
non-deterministic IVs.
commit 3a835c4c9a8390a41090b7a66e905a3985b39977
Author: Andy LoPresto <[email protected]>
Date: 2016-01-17T06:43:25Z
NIFI-1257 Added Bcrypt implementation.
Added unit tests.
Added jBcrypt dependency to pom because BC Bcrypt is OpenBSD-based, not
compatible with standard Bcrypt as defined by the Niels Provos paper.
Added Ruby script in resources for external compatibility testing.
commit a7c9a4a63facb82d5037c47332230c8b9200ca40
Author: Andy LoPresto <[email protected]>
Date: 2016-01-19T03:52:18Z
NIFI-1257 Added Javadoc, unit tests, and utility methods.
Updated PBKDF2 IV check to invalid length, rather than just non-zero check.
(+15 squashed commits)
Squashed commits:
[0c66002] NIFI-1257 Added isKeyedCipher boolean method for EncryptionMethod.
[5d64dfa] NIFI-1257 Added TODO for invalid IV length check.
[5c73364] NIFI-1257 Added test for empty PRF in PBKDF2 constructor.
Added key length check to PBKDF2 cipher provider.
[57ade68] NIFI-1257 Ignored tests for weak iteration count/work factor but
added warning message in constructors.
Changed default PRF to SHA-512.
Added salt and key length check to PBKDF2 cipher provider.
[362828f] NIFI-1257 Added pass-through of IllegalArgumentExceptions in
cipher initialization.
[9f019ed] NIFI-1257 OpenSSL cipher provider ignores requested key length.
[6db634b] NIFI-1257 Added check for salt length and encryption method
presence.
[e6426a9] NIFI-1257 Added utility method for algorithm-specific key length
check.
Added unit tests.
[3503b35] NIFI-1257 Added unit test validating salt format for Bcrypt.
[9ffd358] NIFI-1257 Implemented key length check in Bcrypt cipher provider.
Added unit tests.
[5dc8801] NIFI-1257 Improved Javadoc on CipherUtility.
[a3eb290] NIFI-1257 Added utility method to check key length validity for
cipher families.
Added unit tests.
[51ce74f] NIFI-1257 Added unit tests to enforce strong default values for
Bcrypt work factor and PBKDF2 iteration counts.
[b8dfd4e] NIFI-1257 Corrected license language for jBcrypt in
nifi-standard-nar and nifi-assembly LICENSE files.
[8a85b33] NIFI-1257 Added license info for jBcrypt into nifi-standard-nar
and nifi-assembly LICENSE files.
commit 28243e9663e53fc8232bff21a65f1bd7654e32e0
Author: Andy LoPresto <[email protected]>
Date: 2016-01-21T03:48:45Z
NIFI-1257 Reduced code duplication by invoking KeyedCipherProvider from
BcryptCipherProvider after key derivation. (+4 squashed commits)
Squashed commits:
[1c2f06f] NIFI-1257 Fixed bug in Base64 section of Bcrypt salt regular
expression.
[632104a] NIFI-1257 Added unit test for generateSalt.
[8cc990f] NIFI-1257 Reduced code duplication by invoking
KeyedCipherProvider from PBKDF2CipherProvider after key derivation.
Added unit test for invalid IV.
[75c4690] NIFI-1257 Defined interface for KeyedCipherProvider.
Implemented AES implementation for KeyedCipherProvider.
Added unit tests.
Added Ruby script to test/resources for external compatibility check.
commit 3f0b3ca30423000c50075414d4ca02bb21bc69cd
Author: Andy LoPresto <[email protected]>
Date: 2016-01-22T03:35:43Z
NIFI-1257 Implemented Scrypt cipher provider.
Refactored Scrypt internals and added unit tests.
Added unit test for Scrypt check with correct and incorrect passwords.
Changed check to remove hard-coded derived key length. (+3 squashed commits)
Squashed commits:
[c1d7637] NIFI-1257 Added unit test for Scrypt check with invalid passwords
and hashes.
[f89b2fb] NIFI-1257 Implemented minimum safe parameters calculator and unit
test.
Updated salt format regex.
[f553bce] NIFI-1257 Moved memory calculation helper to Scrypt. (+2 squashed
commits)
Squashed commits:
[13c1b52] NIFI-1257 Cleaned up unit tests and javadoc.
Resolved failing PBKDF2 tests and copied multiple run average logic to
Bcrypt tests. (+6 squashed commits)
Squashed commits:
[b16a5b3] NIFI-1257 Updated default iteration count for PBKDF2 cipher
provider.
[b083b66] NIFI-1257 Refactored parameter encoding to separate method.
[5369fbb] NIFI-1257 Updated Javadoc.
[90feb2a] NIFI-1257 Added check for all-zero IV (common security
vulnerability) in AES cipher provider.
Added unit test.
[7cf6c78] NIFI-1257 Updated Javadoc to be consistent across implementations.
[c0424d6] NIFI-1257 Deprecated no-salt cipher generator for Bcrypt to be
consistent with Scrypt implementation.
Updated unit tests.
[06bed9a] NIFI-1257 Implemented Scrypt cipher provider.
Added unit tests for empty salt. (+7 squashed commits)
Squashed commits:
[130b477] NIFI-1257 Added salt translator from mcrypt format to Java format.
Added unit tests for salt formatting and validation.
[c43ba44] NIFI-1257 Implemented ScryptCipherProvider and added unit test
skeleton with internal consistency and external compatibility tests.
[9db678f] NIFI-1257 Updated Bcrypt unit test to accept new interface method
contract.
[897ad03] NIFI-1257 Added sanity test for Scrypt primitive key derivation
compatibility with external Ruby library.
[4adc3fe] NIFI-1257 Exposed Scrypt parseSalt method as public to allow
ScryptCipherProvider to accept formatted salt externally.
[391cd8b] NIFI-1257 Updated scrypt Ruby script to be consistent with bcrypt
and pbkdf2 scripts in key and IV data handling.
[d167917] NIFI-1257 Exposed desired key length as parameter on Scrypt
underlying implementation. (+2 squashed commits)
Squashed commits:
[7f1feb5] NIFI-1257 Standardized default salt generation to byte[] and
added method contracts in top-level PBECipherProvider interface.
[673f35f] NIFI-1257 Scrypt implementation.
Added default salt length constant and accessor in Scrypt.
Changed reference to UTF-8 to be enum instead of freeform requiring runtime
resolution and redundant exception handling. (+11 squashed commits)
Squashed commits:
[b48ac88] NIFI-1257 Removed salt validation check in Scrypt (spec allows
for empty salt) and replaced with logger warnings. Moved unit test to
ScryptCipherProvider unit test as enforcement will be moved higher up.
[611abf6] NIFI-1257 Made convention & style changes in scrypt code.
[9005573] NIFI-1257 Added salt validation and unit test for Scrypt.
[ff12568] NIFI-1257 Added validation check for password and relevant unit
test.
[6b42e8f] NIFI-1257 Added validation check for N, r, p in Scrypt and
relevant unit tests.
[8ab7e45] NIFI-1257 Added ruby script to test external compatibility for
Scrypt implementation.
[b720b30] NIFI-1257 Combined ScryptUtil (application-facing interface) into
Scrypt class to reduce unnecessary complexity.
Updated unit test references.
[7739349] NIFI-1257 Moved high-memory test vector into separate unit test
with Assume statement limiting execution depending on heap size to allow faster
execution.
[badd2ab] NIFI-1257 Enabled high-memory test vector for Scrypt test.
[765cd38] NIFI-1257 Added surefire block to groovy unit test profile to
enforce 3072 MB heap for Scrypt test.
[637f80b] NIFI-1257 Added local Java implementation of Scrypt KDF (and
underlying PBKDF2 KDF) from Will Glozer.
Added unit tests (including test vectors from paper).
commit f875ecac6c8bf2704a2d662587b7b635e2ee7e39
Author: Andy LoPresto <[email protected]>
Date: 2016-01-29T22:52:12Z
NIFI-1257 Cleaned up interfaces and refactored shared code/unit and
regression tests.
Refactored shared stream handling code. (+1 squashed commit)
Squashed commits:
[4c6de17] NIFI-1257 Added logic to Encryptor to read and write IV to/from
cipher stream.
Added unit tests to cover all PBE KDFs. (+1 squashed commit)
Squashed commits:
[f333fa0] NIFI-1257 Added methods to read and write IV to/from cipher
stream. (+12 squashed commits)
Squashed commits:
[4ece756] NIFI-1257 Resolved all tests in nifi-standard-processors module.
[8163f07] NIFI-1257 Refactored implementation-specific logic out of
PasswordBasedEncryptor to support all KDFs.
Regression tests for legacy still pass.
[a52da26] NIFI-1257 Added salt read/write logic to NifiLegacy and OpenSSL
cipher providers.
Added common salt read/write logic to RandomIVPBECipherProvider.
Changed RandomIVPBECipherProvider from interface to abstract class.
Updated strong KDF implementations.
[1407afe] NIFI-1257 Added common cipher processing code to CipherUtility.
[442495b] NIFI-1257 Added regression tests for PasswordBasedEncryptor with
legacy OpenSSL compatibility before integrating cipher provider logic.
[d0410c0] NIFI-1257 Added regression test for PasswordBasedEncryptor before
integrating cipher provider logic.
[e586933] NIFI-1257 Removed legacy unit test (superseded by Groovy unit
test).
[170314a] NIFI-1257 Removed unnecessary EncryptionMethod enumerations from
unit test.
[196d560] NIFI-1257 Added ASF 2.0 license to CipherProfiderFactory unit
test.
[36e95ba] NIFI-1257 Removed unnecessary KeyDeriver interface.
[9c4eb07] NIFI-1257 Added default (NONE) KDF enum value.
Renamed CipherFactory to CipherProviderFactory.
Added unit test for registered KDF resolution from factory.
[f006985] NIFI-1257 Added CipherProvider marker interface which PBE and
keyed cipher providers share.
commit d45dc69b3981ed6194a90aedbf603122d5c47c18
Author: Andy LoPresto <[email protected]>
Date: 2016-01-30T06:03:25Z
NIFI-1257 Added IV read/write to KeyedCipherProvider and changed from
interface to abstract class. (+3 squashed commits)
Squashed commits:
[0ee5175] NIFI-1257 Moved null checks for encryption method and password to
constructor to de-duplicate and catch earlier.
[6ed4142] NIFI-1257 Moved stream handling code to CipherUtility to allow
reuse in KeyedCipherProvider.
[a78b5db] NIFI-1257 Added null checks for encryption method and password.
commit 48b3463319a8fc620a9229a28253e4c99f4afc8e
Author: Andy LoPresto <[email protected]>
Date: 2016-01-30T07:27:46Z
NIFI-1257 Added processor logic to invoke keyed cipher.
Added unit tests. (+5 squashed commits)
Squashed commits:
[28b2224] NIFI-1257 Fixed contrib-check issues.
[143b88f] NIFI-1257 Added helper method on KDF enum to indicate strength of
KDF.
Added EncryptContent processor property for raw hex key (always visible
until NIFI-1121).
Added validations for KDF (keyed and PBE) and hex key.
Added unit tests for EncryptContent processor validations.
[3a8367b] NIFI-1257 Added utility method to return list of valid key
lengths for algorithm.
Added unit tests.
[04d6d16] NIFI-1257 Added description to allowable values for KDF and
encryption method in EncryptContent processor.
[98595d8] NIFI-1257 Added KeyedEncryptor and unit tests.
commit 3c410d8f33b3ceaf5dfd79db355cf087edec80c8
Author: Andy LoPresto <[email protected]>
Date: 2016-02-02T01:25:28Z
NIFI-1257 Fixed casing of Scrypt.java.
----
> Provide additional KDFs for EncryptContent
> ------------------------------------------
>
> Key: NIFI-1257
> URL: https://issues.apache.org/jira/browse/NIFI-1257
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 0.4.0
> Reporter: Andy LoPresto
> Assignee: Andy LoPresto
> Priority: Critical
> Labels: encryption, security
> Fix For: 0.5.0
>
>
> Currently, the two key derivation functions (KDF) supported are NiFi Legacy
> (1000 iterations of MD5 digest over a password and optional salt) and OpenSSL
> PKCS#5 v1.5 (a single iteration of MD5 digest over a password and optional
> salt).
> Both of these are very weak -- they use a deprecated cryptographic hash
> function (CHF) with known weakness and susceptibility to collisions (with
> demonstrated attacks) and a non-configurable and tightly coupled iteration
> count to derive the key and IV.
> Current best practice KDFs (with work factor recommendations) are as follows:
> * PBKDF2 with variable hash function (SHA1, SHA256, SHA384, SHA512, or
> ideally HMAC variants of these functions) and variable iteration count (in
> the 10k - 1M range).
> * bcrypt with work factor of 12 - 16
> * scrypt with work factor of (2^14 - 2^20, 8, 1)
> The salt and iteration count should be stored alongside the hashed record
> (bcrypt handles this natively).
> Notes:
> * http://wildlyinaccurate.com/bcrypt-choosing-a-work-factor/
> * http://blog.ircmaxell.com/2012/12/seven-ways-to-screw-up-bcrypt.html
> *
> http://security.stackexchange.com/questions/17207/recommended-of-rounds-for-bcrypt
> *
> http://security.stackexchange.com/questions/3959/recommended-of-iterations-when-using-pkbdf2-sha256/3993#3993
> *
> http://security.stackexchange.com/questions/4781/do-any-security-experts-recommend-bcrypt-for-password-storage/6415
>
> *
> http://web.archive.org/web/20130407190430/http://chargen.matasano.com/chargen/2007/9/7/enough-with-the-rainbow-tables-what-you-need-to-know-about-s.html
> *
> https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2015/march/enough-with-the-salts-updates-on-secure-password-schemes/
> * http://www.tarsnap.com/scrypt.html
> * http://www.tarsnap.com/scrypt/scrypt.pdf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
