[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/834


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75917246
  
--- Diff: 
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
 ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties
+
+import groovy.io.GroovyPrintWriter
+import org.apache.commons.cli.CommandLine
+import org.apache.commons.cli.CommandLineParser
+import org.apache.commons.cli.DefaultParser
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.commons.cli.ParseException
+import org.apache.commons.codec.binary.Hex
+import org.apache.nifi.toolkit.tls.commandLine.CommandLineParseException
+import org.apache.nifi.toolkit.tls.commandLine.ExitCode
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.console.TextDevice
+import org.apache.nifi.util.console.TextDevices
+import org.bouncycastle.crypto.generators.SCrypt
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import javax.crypto.Cipher
+import java.nio.charset.StandardCharsets
+import java.security.KeyException
+import java.security.Security
+
+class ConfigEncryptionTool {
+private static final Logger logger = 
LoggerFactory.getLogger(ConfigEncryptionTool.class)
+
+public String bootstrapConfPath
+public String niFiPropertiesPath
+public String outputNiFiPropertiesPath
+public String loginIdentityProvidersPath
+
+private String keyHex
+private String password
+private NiFiProperties niFiProperties
+
+private boolean usingPassword = true
+
+private static final String HELP_ARG = "help"
+private static final String BOOTSTRAP_CONF_ARG = "bootstrapConf"
+private static final String NIFI_PROPERTIES_ARG = "niFiProperties"
+private static final String OUTPUT_NIFI_PROPERTIES_ARG = 
"outputNiFiProperties"
+private static final String KEY_ARG = "key"
+private static final String PASSWORD_ARG = "password"
+private static final String USE_KEY_ARG = "useRawKey"
+
+private static final int MIN_PASSWORD_LENGTH = 12
+
+// Strong parameters as of 12 Aug 2016
+private static final int SCRYPT_N = 2**16
+private static final int SCRYPT_R = 8
+private static final int SCRYPT_P = 1
+
+private static
+final String BOOTSTRAP_KEY_COMMENT = "# Master key in hexadecimal 
format for encrypted sensitive configuration values"
+private static final String BOOTSTRAP_KEY_PREFIX = 
"nifi.bootstrap.sensitive.key="
+private static final String JAVA_HOME = "JAVA_HOME"
+private static final String NIFI_TOOLKIT_HOME = "NIFI_TOOLKIT_HOME"
+private static final String SEP = System.lineSeparator()
+
+private static final String FOOTER = buildFooter()
+
+private static
+final String DEFAULT_DESCRIPTION = "This tool reads from a 
nifi.properties file with plain sensitive configuration values, prompts the 
user for a master key, and encrypts each value. It will replace the plain value 
with the protected value in the same file (or write to a new nifi.properties 
file if specified)."
+
+private static String buildHeader(String description = 
DEFAULT_DESCRIPTION) {
+"${SEP}${description}${SEP * 2}"
+}
+
+private static String buildFooter() {
+"${SEP}Java home: ${System.getenv(JAVA_HOME)}${SEP}NiFi Toolkit 
home: ${System.getenv(NIFI_TOOLKIT_HOME)}"
+}
+
+private final Options options;
+private final String header;
+
+
+public ConfigEncryptionTool() {
+this(DEFAULT_DESCRIPTION)
+}
+
+public ConfigEncryptionTool(String description) {
+this.header = buildHeader(description)
+

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75912621
  
--- Diff: 
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
 ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties
+
+import groovy.io.GroovyPrintWriter
+import org.apache.commons.cli.CommandLine
+import org.apache.commons.cli.CommandLineParser
+import org.apache.commons.cli.DefaultParser
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.commons.cli.ParseException
+import org.apache.commons.codec.binary.Hex
+import org.apache.nifi.toolkit.tls.commandLine.CommandLineParseException
+import org.apache.nifi.toolkit.tls.commandLine.ExitCode
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.console.TextDevice
+import org.apache.nifi.util.console.TextDevices
+import org.bouncycastle.crypto.generators.SCrypt
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import javax.crypto.Cipher
+import java.nio.charset.StandardCharsets
+import java.security.KeyException
+import java.security.Security
+
+class ConfigEncryptionTool {
+private static final Logger logger = 
LoggerFactory.getLogger(ConfigEncryptionTool.class)
+
+public String bootstrapConfPath
+public String niFiPropertiesPath
+public String outputNiFiPropertiesPath
+public String loginIdentityProvidersPath
+
+private String keyHex
+private String password
+private NiFiProperties niFiProperties
+
+private boolean usingPassword = true
+
+private static final String HELP_ARG = "help"
+private static final String BOOTSTRAP_CONF_ARG = "bootstrapConf"
+private static final String NIFI_PROPERTIES_ARG = "niFiProperties"
+private static final String OUTPUT_NIFI_PROPERTIES_ARG = 
"outputNiFiProperties"
+private static final String KEY_ARG = "key"
+private static final String PASSWORD_ARG = "password"
+private static final String USE_KEY_ARG = "useRawKey"
+
+private static final int MIN_PASSWORD_LENGTH = 12
+
+// Strong parameters as of 12 Aug 2016
+private static final int SCRYPT_N = 2**16
+private static final int SCRYPT_R = 8
+private static final int SCRYPT_P = 1
+
+private static
+final String BOOTSTRAP_KEY_COMMENT = "# Master key in hexadecimal 
format for encrypted sensitive configuration values"
+private static final String BOOTSTRAP_KEY_PREFIX = 
"nifi.bootstrap.sensitive.key="
+private static final String JAVA_HOME = "JAVA_HOME"
+private static final String NIFI_TOOLKIT_HOME = "NIFI_TOOLKIT_HOME"
+private static final String SEP = System.lineSeparator()
+
+private static final String FOOTER = buildFooter()
+
+private static
+final String DEFAULT_DESCRIPTION = "This tool reads from a 
nifi.properties file with plain sensitive configuration values, prompts the 
user for a master key, and encrypts each value. It will replace the plain value 
with the protected value in the same file (or write to a new nifi.properties 
file if specified)."
+
+private static String buildHeader(String description = 
DEFAULT_DESCRIPTION) {
+"${SEP}${description}${SEP * 2}"
+}
+
+private static String buildFooter() {
+"${SEP}Java home: ${System.getenv(JAVA_HOME)}${SEP}NiFi Toolkit 
home: ${System.getenv(NIFI_TOOLKIT_HOME)}"
+}
+
+private final Options options;
+private final String header;
+
+
+public ConfigEncryptionTool() {
+this(DEFAULT_DESCRIPTION)
+}
+
+public ConfigEncryptionTool(String description) {
+this.header = buildHeader(description)
+

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75909986
  
--- Diff: 
nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy
 ---
@@ -0,0 +1,1225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties
+
+import ch.qos.logback.classic.spi.LoggingEvent
+import ch.qos.logback.core.AppenderBase
+import org.apache.commons.codec.binary.Hex
+import org.apache.nifi.toolkit.tls.commandLine.CommandLineParseException
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.console.TextDevice
+import org.apache.nifi.util.console.TextDevices
+import org.bouncycastle.crypto.generators.SCrypt
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.junit.After
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Rule
+import org.junit.Test
+import org.junit.contrib.java.lang.system.Assertion
+import org.junit.contrib.java.lang.system.ExpectedSystemExit
+import org.junit.contrib.java.lang.system.SystemOutRule
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import javax.crypto.Cipher
+import java.nio.file.Files
+import java.nio.file.attribute.PosixFilePermission
+import java.security.KeyException
+import java.security.Security
+
+@RunWith(JUnit4.class)
+class ConfigEncryptionToolTest extends GroovyTestCase {
+private static final Logger logger = 
LoggerFactory.getLogger(ConfigEncryptionToolTest.class)
+
+@Rule
+public final ExpectedSystemExit exit = ExpectedSystemExit.none()
+
+@Rule
+public final SystemOutRule systemOutRule = new 
SystemOutRule().enableLog()
+
+private static final String KEY_HEX_128 = 
"0123456789ABCDEFFEDCBA9876543210"
+private static final String KEY_HEX_256 = KEY_HEX_128 * 2
+public static final String KEY_HEX = 
isUnlimitedStrengthCryptoAvailable() ? KEY_HEX_256 : KEY_HEX_128
+private static final String PASSWORD = "thisIsABadPassword"
+
+@BeforeClass
+public static void setUpOnce() throws Exception {
+Security.addProvider(new BouncyCastleProvider())
+
+logger.metaClass.methodMissing = { String name, args ->
+logger.info("[${name?.toUpperCase()}] ${(args as List).join(" 
")}")
+}
+}
+
+@Before
+public void setUp() throws Exception {
+
+}
+
+@After
+public void tearDown() throws Exception {
+TestAppender.reset()
+}
+
+private static boolean isUnlimitedStrengthCryptoAvailable() {
+Cipher.getMaxAllowedKeyLength("AES") > 128
+}
+
+@Test
+void testShouldPrintHelpMessage() {
+// Arrange
+def flags = ["-h", "--help"]
+ConfigEncryptionTool tool = new ConfigEncryptionTool()
+
+// Act
+flags.each { String arg ->
+def msg = shouldFail(CommandLineParseException) {
+tool.parse([arg] as String[])
+}
+
+// Assert
+assert msg == null
+assert systemOutRule.getLog().contains("usage: 
org.apache.nifi.properties.ConfigEncryptionTool [")
+}
+}
+
+@Test
+void testShouldParseBootstrapConfArgument() {
+// Arrange
+def flags = ["-b", "--bootstrapConf"]
+String bootstrapPath = "src/test/resources/bootstrap.conf"
+ConfigEncryptionTool tool = new ConfigEncryptionTool()
+
+// Act
+flags.each { String arg ->
+tool.parse([arg, bootstrapPath] as String[])
+logger.info("Parsed bootstrap.conf location: 
${tool.bootstrapConfPath}")
+
+// Assert
+assert 

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75909949
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/nifi.properties
 ---
@@ -131,8 +131,10 @@ nifi.web.jetty.threads=${nifi.web.jetty.threads}
 
 # security properties #
 nifi.sensitive.props.key=
+nifi.sensitive.props.key.protected=${nifi.sensitive.props.key.protected}
--- End diff --

Thanks for catching this Bryan. It was an issue where the tool had to merge 
two instances of `Properties` together and the logic wasn't checking the right 
key set. Added a test and fixed. Empty protection values no longer stop the 
field from being protected. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75874508
  
--- Diff: 
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
 ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties
+
+import groovy.io.GroovyPrintWriter
+import org.apache.commons.cli.CommandLine
+import org.apache.commons.cli.CommandLineParser
+import org.apache.commons.cli.DefaultParser
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.commons.cli.ParseException
+import org.apache.commons.codec.binary.Hex
+import org.apache.nifi.toolkit.tls.commandLine.CommandLineParseException
+import org.apache.nifi.toolkit.tls.commandLine.ExitCode
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.console.TextDevice
+import org.apache.nifi.util.console.TextDevices
+import org.bouncycastle.crypto.generators.SCrypt
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import javax.crypto.Cipher
+import java.nio.charset.StandardCharsets
+import java.security.KeyException
+import java.security.Security
+
+class ConfigEncryptionTool {
+private static final Logger logger = 
LoggerFactory.getLogger(ConfigEncryptionTool.class)
+
+public String bootstrapConfPath
+public String niFiPropertiesPath
+public String outputNiFiPropertiesPath
+public String loginIdentityProvidersPath
+
+private String keyHex
+private String password
+private NiFiProperties niFiProperties
+
+private boolean usingPassword = true
+
+private static final String HELP_ARG = "help"
+private static final String BOOTSTRAP_CONF_ARG = "bootstrapConf"
+private static final String NIFI_PROPERTIES_ARG = "niFiProperties"
+private static final String OUTPUT_NIFI_PROPERTIES_ARG = 
"outputNiFiProperties"
+private static final String KEY_ARG = "key"
+private static final String PASSWORD_ARG = "password"
+private static final String USE_KEY_ARG = "useRawKey"
+
+private static final int MIN_PASSWORD_LENGTH = 12
+
+// Strong parameters as of 12 Aug 2016
+private static final int SCRYPT_N = 2**16
+private static final int SCRYPT_R = 8
+private static final int SCRYPT_P = 1
+
+private static
+final String BOOTSTRAP_KEY_COMMENT = "# Master key in hexadecimal 
format for encrypted sensitive configuration values"
+private static final String BOOTSTRAP_KEY_PREFIX = 
"nifi.bootstrap.sensitive.key="
+private static final String JAVA_HOME = "JAVA_HOME"
+private static final String NIFI_TOOLKIT_HOME = "NIFI_TOOLKIT_HOME"
+private static final String SEP = System.lineSeparator()
+
+private static final String FOOTER = buildFooter()
+
+private static
+final String DEFAULT_DESCRIPTION = "This tool reads from a 
nifi.properties file with plain sensitive configuration values, prompts the 
user for a master key, and encrypts each value. It will replace the plain value 
with the protected value in the same file (or write to a new nifi.properties 
file if specified)."
+
+private static String buildHeader(String description = 
DEFAULT_DESCRIPTION) {
+"${SEP}${description}${SEP * 2}"
+}
+
+private static String buildFooter() {
+"${SEP}Java home: ${System.getenv(JAVA_HOME)}${SEP}NiFi Toolkit 
home: ${System.getenv(NIFI_TOOLKIT_HOME)}"
+}
+
+private final Options options;
+private final String header;
+
+
+public ConfigEncryptionTool() {
+this(DEFAULT_DESCRIPTION)
+}
+
+public ConfigEncryptionTool(String description) {
+this.header = buildHeader(description)
+

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75871505
  
--- Diff: 
nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy
 ---
@@ -0,0 +1,1225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties
+
+import ch.qos.logback.classic.spi.LoggingEvent
+import ch.qos.logback.core.AppenderBase
+import org.apache.commons.codec.binary.Hex
+import org.apache.nifi.toolkit.tls.commandLine.CommandLineParseException
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.console.TextDevice
+import org.apache.nifi.util.console.TextDevices
+import org.bouncycastle.crypto.generators.SCrypt
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.junit.After
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Rule
+import org.junit.Test
+import org.junit.contrib.java.lang.system.Assertion
+import org.junit.contrib.java.lang.system.ExpectedSystemExit
+import org.junit.contrib.java.lang.system.SystemOutRule
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import javax.crypto.Cipher
+import java.nio.file.Files
+import java.nio.file.attribute.PosixFilePermission
+import java.security.KeyException
+import java.security.Security
+
+@RunWith(JUnit4.class)
+class ConfigEncryptionToolTest extends GroovyTestCase {
+private static final Logger logger = 
LoggerFactory.getLogger(ConfigEncryptionToolTest.class)
+
+@Rule
+public final ExpectedSystemExit exit = ExpectedSystemExit.none()
+
+@Rule
+public final SystemOutRule systemOutRule = new 
SystemOutRule().enableLog()
+
+private static final String KEY_HEX_128 = 
"0123456789ABCDEFFEDCBA9876543210"
+private static final String KEY_HEX_256 = KEY_HEX_128 * 2
+public static final String KEY_HEX = 
isUnlimitedStrengthCryptoAvailable() ? KEY_HEX_256 : KEY_HEX_128
+private static final String PASSWORD = "thisIsABadPassword"
+
+@BeforeClass
+public static void setUpOnce() throws Exception {
+Security.addProvider(new BouncyCastleProvider())
+
+logger.metaClass.methodMissing = { String name, args ->
+logger.info("[${name?.toUpperCase()}] ${(args as List).join(" 
")}")
+}
+}
+
+@Before
+public void setUp() throws Exception {
+
+}
+
+@After
+public void tearDown() throws Exception {
+TestAppender.reset()
+}
+
+private static boolean isUnlimitedStrengthCryptoAvailable() {
+Cipher.getMaxAllowedKeyLength("AES") > 128
+}
+
+@Test
+void testShouldPrintHelpMessage() {
+// Arrange
+def flags = ["-h", "--help"]
+ConfigEncryptionTool tool = new ConfigEncryptionTool()
+
+// Act
+flags.each { String arg ->
+def msg = shouldFail(CommandLineParseException) {
+tool.parse([arg] as String[])
+}
+
+// Assert
+assert msg == null
+assert systemOutRule.getLog().contains("usage: 
org.apache.nifi.properties.ConfigEncryptionTool [")
+}
+}
+
+@Test
+void testShouldParseBootstrapConfArgument() {
+// Arrange
+def flags = ["-b", "--bootstrapConf"]
+String bootstrapPath = "src/test/resources/bootstrap.conf"
+ConfigEncryptionTool tool = new ConfigEncryptionTool()
+
+// Act
+flags.each { String arg ->
+tool.parse([arg, bootstrapPath] as String[])
+logger.info("Parsed bootstrap.conf location: 
${tool.bootstrapConfPath}")
+
+// Assert
+assert 

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75870493
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/nifi.properties
 ---
@@ -131,8 +131,10 @@ nifi.web.jetty.threads=${nifi.web.jetty.threads}
 
 # security properties #
 nifi.sensitive.props.key=
+nifi.sensitive.props.key.protected=${nifi.sensitive.props.key.protected}
--- End diff --

After building this PR, in conf/nifi.properties the property 
nifi.sensitive.props.key.protected was empty. I then filled in a value for 
nifi.sensitive.props.key and when I ran the toolkit it did not protect 
nifi.sensitive.props.key, I assume because it saw 
nifi.sensitive.props.key.protected was already there? I tested it again by 
first deleting the empty protected property and that time it did protect the 
sensitive properties key.

Seems like nifi.sensitive.props.key.protected should not be here by 
default, or the toolkit should recognize an empty value and protect it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75752491
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/AESSensitivePropertyProvider.java
 ---
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties;
+
+import java.nio.charset.StandardCharsets;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.crypto.BadPaddingException;
+import javax.crypto.Cipher;
+import javax.crypto.IllegalBlockSizeException;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.IvParameterSpec;
+import javax.crypto.spec.SecretKeySpec;
+import org.apache.commons.lang3.StringUtils;
+import org.bouncycastle.util.encoders.Base64;
+import org.bouncycastle.util.encoders.DecoderException;
+import org.bouncycastle.util.encoders.EncoderException;
+import org.bouncycastle.util.encoders.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class AESSensitivePropertyProvider implements 
SensitivePropertyProvider {
+
+private static final Logger logger = 
LoggerFactory.getLogger(AESSensitivePropertyProvider.class);
+
+private static final String IMPLEMENTATION_NAME = "AES Sensitive 
Property Provider";
+private static final String IMPLEMENTATION_KEY = "aes/gcm/";
+private static final String ALGORITHM = "AES/GCM/NoPadding";
+private static final String PROVIDER = "BC";
+private static final String DELIMITER = "||"; // "|" is not a valid 
Base64 character, so ensured not to be present in cipher text
+private static final int IV_LENGTH = 12;
+private static final int MIN_CIPHER_TEXT_LENGTH = IV_LENGTH * 4 / 3 + 
DELIMITER.length() + 1;
+
+private Cipher cipher;
+private SecretKey key;
+
+public AESSensitivePropertyProvider() {
--- End diff --

There was a use case for this originally, but now the only access is from 
the constructor, so I will make this fix. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75751891
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/AESSensitivePropertyProvider.java
 ---
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties;
+
+import java.nio.charset.StandardCharsets;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.crypto.BadPaddingException;
+import javax.crypto.Cipher;
+import javax.crypto.IllegalBlockSizeException;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.IvParameterSpec;
+import javax.crypto.spec.SecretKeySpec;
+import org.apache.commons.lang3.StringUtils;
+import org.bouncycastle.util.encoders.Base64;
+import org.bouncycastle.util.encoders.DecoderException;
+import org.bouncycastle.util.encoders.EncoderException;
+import org.bouncycastle.util.encoders.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class AESSensitivePropertyProvider implements 
SensitivePropertyProvider {
+
+private static final Logger logger = 
LoggerFactory.getLogger(AESSensitivePropertyProvider.class);
+
+private static final String IMPLEMENTATION_NAME = "AES Sensitive 
Property Provider";
+private static final String IMPLEMENTATION_KEY = "aes/gcm/";
+private static final String ALGORITHM = "AES/GCM/NoPadding";
+private static final String PROVIDER = "BC";
+private static final String DELIMITER = "||"; // "|" is not a valid 
Base64 character, so ensured not to be present in cipher text
+private static final int IV_LENGTH = 12;
+private static final int MIN_CIPHER_TEXT_LENGTH = IV_LENGTH * 4 / 3 + 
DELIMITER.length() + 1;
+
+private Cipher cipher;
+private SecretKey key;
+
+public AESSensitivePropertyProvider() {
+
+}
+
+public AESSensitivePropertyProvider(String key) throws 
NoSuchPaddingException, NoSuchAlgorithmException, NoSuchProviderException {
+setKey(key);
+}
+
+public AESSensitivePropertyProvider(byte[] key) throws 
NoSuchPaddingException, NoSuchAlgorithmException, NoSuchProviderException {
+this(key == null ? "" : Hex.toHexString(key));
+}
+
+public void setKey(String keyHex) throws 
SensitivePropertyProtectionException {
+if (!isKeySet()) {
+if (keyHex == null || keyHex.length() == 0) {
--- End diff --

Will do. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75751825
  
--- Diff: nifi-commons/nifi-properties/pom.xml ---
@@ -21,4 +21,10 @@
 1.0.0-SNAPSHOT
 
 nifi-properties
+
+
+org.bouncycastle
+bcprov-jdk15on
--- End diff --

Good catch. Thanks Bryan. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75733078
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/AESSensitivePropertyProvider.java
 ---
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties;
+
+import java.nio.charset.StandardCharsets;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.crypto.BadPaddingException;
+import javax.crypto.Cipher;
+import javax.crypto.IllegalBlockSizeException;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.IvParameterSpec;
+import javax.crypto.spec.SecretKeySpec;
+import org.apache.commons.lang3.StringUtils;
+import org.bouncycastle.util.encoders.Base64;
+import org.bouncycastle.util.encoders.DecoderException;
+import org.bouncycastle.util.encoders.EncoderException;
+import org.bouncycastle.util.encoders.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class AESSensitivePropertyProvider implements 
SensitivePropertyProvider {
+
+private static final Logger logger = 
LoggerFactory.getLogger(AESSensitivePropertyProvider.class);
+
+private static final String IMPLEMENTATION_NAME = "AES Sensitive 
Property Provider";
+private static final String IMPLEMENTATION_KEY = "aes/gcm/";
+private static final String ALGORITHM = "AES/GCM/NoPadding";
+private static final String PROVIDER = "BC";
+private static final String DELIMITER = "||"; // "|" is not a valid 
Base64 character, so ensured not to be present in cipher text
+private static final int IV_LENGTH = 12;
+private static final int MIN_CIPHER_TEXT_LENGTH = IV_LENGTH * 4 / 3 + 
DELIMITER.length() + 1;
+
+private Cipher cipher;
+private SecretKey key;
+
+public AESSensitivePropertyProvider() {
--- End diff --

Is there a use case for creating an instance of this class without an 
initial key and setting the key later? 
If there is then you can ignore this comment... Was thinking it might be 
simpler to make SecretKey final and only set from the constructor, and setKey 
could be private, and then no need to check if (!isKeySet()) when 
protecting/unprotecting


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75728485
  
--- Diff: nifi-commons/nifi-properties/pom.xml ---
@@ -21,4 +21,10 @@
 1.0.0-SNAPSHOT
 
 nifi-properties
+
+
+org.bouncycastle
+bcprov-jdk15on
--- End diff --

Should this be removed since we now have nifi-properties-loader in the 
framework which has all the bouncy castle stuff?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/834

NIFI-1831 Implemented encrypted configuration capabilities

This is a "beta" PR -- there are still some edge cases I want to clean up 
but I wanted to get eyes on it because it will be a large change. Specific 
areas of focus for review:
* LICENSE
* `AESSensitivePropertyProvider` -- class which actually performs 
encryption/decryption
* `ConfigEncryptionTool` -- command-line class to make the process of 
encrypting the configuration values easy

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-1831

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/834.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #834


commit b75cdb0742fd814e0c9c7831e8e0bcbdc48c1223
Author: Andy LoPresto 
Date:   2016-06-11T04:06:55Z

NIFI-1831 Cleaned up trailing whitespace. (+1 squashed commit)
Squashed commits:
[7fc7e0a] NIFI-1831 Adapted Apache Commons Hex and Base64 encoding/decoding 
classes and implemented in nifi-properties (avoiding extraneous Maven 
dependencies).
Added sanity checks in unit tests.
Added utility join method in StringUtils.

commit f9f173099b026eb78b2fe16c6ad15ab1bdeed8d3
Author: Andy LoPresto 
Date:   2016-06-11T04:18:37Z

NIFI-1831 Added interface and first implementation of 
SensitivePropertyProvider.

commit 354596a2e3ea9cde41bd12c0ff4d145e1ab910c4
Author: Andy LoPresto 
Date:   2016-06-11T04:19:13Z

NIFI-1831 Added skeleton for regression tests and new unit tests for 
sensitive property loading.

commit 836ee1e84c2b352ee96c8831e02178d95396400c
Author: Andy LoPresto 
Date:   2016-07-14T20:31:17Z

NIFI-1831 Added initial logic in NiFiProperties to detect sensitive 
property keys and return list of sensitive properties.
Added unit tests (and resource).

commit dafa426d36e2bb86361ee8c14681addfa31677f9
Author: Andy LoPresto 
Date:   2016-07-14T20:39:17Z

NIFI-1831 Moved Base64 and Hex sanity tests into correct package.

commit eee44e107ddbecbacd8cb44b9c94fd0bbef5ec60
Author: Andy LoPresto 
Date:   2016-07-14T22:58:01Z

NIFI-1831 Added BouncyCastle dependency to nifi-properties module.

commit a0286b55eb1c1f469ae9612a51a3aadb4198f175
Author: Andy LoPresto 
Date:   2016-07-14T22:59:06Z

NIFI-1831 Added SensitivePropertyProtectionException.
Updated unprotect method contract in interface and implementation with new 
exception.
Added unprotect logic.

commit 064b2d14c088f7e984af625658011fe9310a759b
Author: Andy LoPresto 
Date:   2016-07-14T22:59:34Z

NIFI-1831 Added unit tests for sensitive property encryption/decryption.

commit 46a7b1b902672843f10758edbe763dcd35f1ffc7
Author: Andy LoPresto 
Date:   2016-07-15T19:11:50Z

NIFI-1831 Finished logic for sensitive property encryption/decryption.
Added negative/edge case unit tests for sensitive property 
encryption/decryption.

commit 2d8ec30adfb6f46e1a4302c0fe074d7b132a3cda
Author: Andy LoPresto 
Date:   2016-07-15T19:17:28Z

NIFI-1831 Excluded additional keys property from self-inclusion to avoid 
recursive/protection issue.

commit 723523059a1c8fffe90ddbf29a9eb8ac3e2ec32a
Author: Andy LoPresto 
Date:   2016-07-15T19:33:20Z

NIFI-1831 Removed unnecessary duplication of Base64 and Hex encoders as 
BouncyCastle includes implementations.

commit b81f89fa8bb20d96d4d2ff1a45d15c2271e88ad6
Author: Andy LoPresto 
Date:   2016-07-15T19:35:53Z

NIFI-1831 Removed unnecessary reference to commons-codec from NOTICE files 
as the duplicate code is removed.

commit 323756119e6841e4b6bab234f793386ea1ce13a5
Author: Andy LoPresto 
Date:   2016-07-15T19:36:57Z

NIFI-1831 Removed unnecessary nifi-properties NOTICE file as the duplicate 
code is removed.

commit 0798cd12d2712132285284209be51ac40bfd970b
Author: Andy LoPresto 
Date:   2016-07-15T21:25:32Z

NIFI-1831 Fixed tests which were not deterministic (shuffling IV and 
ciphertext could occasionally result in same value, rendering successful 
decryption and lacking expected exception).
Added license to AESSensitivePropertyProviderTest.

commit 7d9c3a8159a4d06d0a96364e87141de10a298e33
Author: Andy LoPresto 
Date:   2016-07-15T23:20:59Z

NIFI-1831 Added logic to retrieve protected keys and their protection 
mechanism from properties.
Added logic to determine percentage of sensitive keys currently protected.
Added unit tests.
Added failing unit test for