exceptionfactory commented on code in PR #6273:
URL: https://github.com/apache/nifi/pull/6273#discussion_r941394985
##########
nifi-commons/nifi-property-protection-factory/src/main/java/org/apache/nifi/properties/scheme/PropertyProtectionScheme.java:
##########
@@ -19,7 +19,7 @@
/**
* Property Protection Schemes supported as arguments for encryption commands
should not have direct references
*/
-enum PropertyProtectionScheme implements ProtectionScheme {
+public enum PropertyProtectionScheme implements ProtectionScheme {
Review Comment:
This enum was intentionally package-private to encapsulate resolution of the
scheme. As indicated in the enum values, there are different ways of referring
to the scheme. With the introduction of a new command, this seems like a good
opportunity to change the argument contract to use the scheme values, instead
of the enum names.
##########
nifi-toolkit/nifi-property-encryptor-tool/pom.xml:
##########
@@ -0,0 +1,129 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <parent>
+ <artifactId>nifi</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.18.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>nifi-property-encryptor-tool</artifactId>
+
+ <properties>
+ <maven.compiler.source>8</maven.compiler.source>
+ <maven.compiler.target>8</maven.compiler.target>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>info.picocli</groupId>
+ <artifactId>picocli</artifactId>
+ <version>4.6.3</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-property-protection-factory</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-properties-loader</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi.registry</groupId>
+ <artifactId>nifi-registry-properties</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi.registry</groupId>
+ <artifactId>nifi-registry-properties-loader</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ <version>${org.slf4j.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-simple</artifactId>
+ <version>${org.slf4j.version}</version>
+ </dependency>
+ </dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <configuration>
+ <archive>
+ <manifest>
+
<mainClass>org.apache.nifi.util.console.PropertyEncryptorCLI</mainClass>
+ </manifest>
+ </archive>
+ <descriptorRefs>
+ <descriptorRef>jar-with-dependencies</descriptorRef>
+ </descriptorRefs>
+ </configuration>
+ <executions>
+ <execution>
+ <id>make-assembly</id>
+ <phase>package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>appassembler-maven-plugin</artifactId>
+ <version>2.1.0</version>
+ <configuration>
+ <programs>
+ <program>
+
<mainClass>org.apache.nifi.util.console.PropertyEncryptorCLI</mainClass>
+ <id>property-encryptor</id>
+ </program>
+ </programs>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <!-- annotationProcessorPaths requires maven-compiler-plugin
version 3.5 or higher -->
Review Comment:
Is this comment necessary? The Maven compiler plugin version should be set
in the root configuration.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/PropertyEncryptorMain.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi;
+
+import org.apache.nifi.properties.AbstractBootstrapPropertiesLoader;
+import org.apache.nifi.properties.BootstrapProperties;
+import org.apache.nifi.properties.MutableBootstrapProperties;
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.StandardSensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.PropertyProtectionScheme;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+import
org.apache.nifi.registry.properties.util.NiFiRegistryBootstrapPropertiesLoader;
+import org.apache.nifi.serde.PropertiesWriter;
+import org.apache.nifi.util.NiFiBootstrapPropertiesLoader;
+import org.apache.nifi.util.crypto.CryptographyUtils;
+import org.apache.nifi.xml.XmlDecryptor;
+import org.apache.nifi.xml.XmlEncryptor;
+import org.apache.nifi.util.file.ConfigurationFileResolver;
+import org.apache.nifi.util.file.FileUtilities;
+import org.apache.nifi.properties.ApplicationProperties;
+import org.apache.nifi.properties.NiFiPropertiesLoader;
+import org.apache.nifi.properties.PropertiesLoader;
+import org.apache.nifi.registry.properties.NiFiRegistryPropertiesLoader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Path;
+import java.util.List;
+
+public class PropertyEncryptorMain {
+
+ private static final Logger logger =
LoggerFactory.getLogger(PropertyEncryptorMain.class);
+ private final AbstractBootstrapPropertiesLoader bootstrapLoader;
+ private final PropertiesLoader<ApplicationProperties> propertiesLoader;
+ private final ConfigurationFileResolver fileResolver;
+ private final List<File> configurationFiles;
+ private String hexKey;
+ private final Path confDirectory;
+
+ public PropertyEncryptorMain(final Path baseDirectory, final String
passphrase) throws PropertyEncryptorException {
+ confDirectory =
FileUtilities.resolveAbsoluteConfDirectory(baseDirectory);
+ try {
+ bootstrapLoader = getBootstrapPropertiesLoader(confDirectory);
+ fileResolver = new ConfigurationFileResolver(confDirectory);
+ final File applicationProperties =
FileUtilities.resolvePropertiesFile(confDirectory);
+ propertiesLoader = getPropertiesLoader(confDirectory);
+ configurationFiles =
fileResolver.resolveConfigurationFilesFromApplicationProperties(propertiesLoader.load(applicationProperties));
+ hexKey = getKeyHex(confDirectory, passphrase);
+ } catch (final Exception e) {
+ throw new PropertyEncryptorException("Failed to run property
encryptor", e);
+ }
+ }
+
+ private AbstractBootstrapPropertiesLoader
getBootstrapPropertiesLoader(final Path baseDirectory) {
+ if (FileUtilities.isNiFiRegistryConfDirectory(baseDirectory)) {
+ return new NiFiRegistryBootstrapPropertiesLoader();
+ } else if (FileUtilities.isNiFiConfDirectory(baseDirectory)) {
+ return new NiFiBootstrapPropertiesLoader();
+ } else {
+ throw new PropertyEncryptorException("The base directory provided
does not contain a recognized bootstrap.conf file");
+ }
+ }
+
+ private PropertiesLoader<ApplicationProperties> getPropertiesLoader(final
Path baseDirectory) {
+ if (FileUtilities.isNiFiRegistryConfDirectory(baseDirectory)) {
+ return new NiFiRegistryPropertiesLoader();
+ } else if (FileUtilities.isNiFiConfDirectory(baseDirectory)) {
+ return new NiFiPropertiesLoader();
+ } else {
+ throw new PropertyEncryptorException("The base directory provided
does not contain a recognized .properties file");
+ }
+ }
+
+ /**
+ * @param baseDirectory The base directory of a NiFi / NiFi Registry
installation that should be encrypted
+ */
+ public int encryptConfigurationFiles(final Path baseDirectory, final
PropertyProtectionScheme scheme) {
+ XmlEncryptor encryptor = getXmlEncryptor(scheme);
+ try {
+ encryptConfigurationFiles(configurationFiles, encryptor);
+ outputKeyToBootstrap();
+ logger.info("The Property Encryptor successfully encrypted
configuration files in the [{}] directory with the scheme [{}]", baseDirectory,
scheme.getPath());
+ return 0;
+ } catch (Exception e) {
+ logger.error("The Property Encryptor failed to encrypt
configuration files in the [{}] directory with the scheme [{}]", baseDirectory,
scheme.getPath(), e);
+ return 1;
+ }
+ }
+
+ private void outputKeyToBootstrap() throws IOException {
+ final File bootstrapFile =
bootstrapLoader.getBootstrapFileWithinConfDirectory(confDirectory);
+ File tempBootstrapFile = FileUtilities.getTemporaryOutputFile("tmp",
bootstrapFile);
Review Comment:
The reference to `tmp` should be promoted to a static variable and reused.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/util/console/ConfigSubcommand.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.util.console;
+
+import org.apache.nifi.PropertyEncryptorMain;
+import org.apache.nifi.properties.scheme.PropertyProtectionScheme;
+import org.apache.nifi.util.console.utils.BaseCommandParameters;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
[email protected](name = "config",
+ description = "Operate on application configs",
+ usageHelpWidth=140
+)
+class ConfigSubcommand extends BaseCommandParameters implements Runnable {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ConfigSubcommand.class);
+
+ @CommandLine.ParentCommand
+ private DefaultCLIOptions parent;
+
+ @CommandLine.Parameters(description="The encryption scheme to use, from
one of the following schemes: [@|bold ${COMPLETION-CANDIDATES}|@]")
+ PropertyProtectionScheme scheme;
Review Comment:
As mentioned on the enum, this is a good opportunity to revisit the
supported arguments. Right now, the supported arguments are different than the
values used to reference encrypted properties, so it would be helpful to have a
unified approach.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/util/console/PropertyEncryptorTranslateForCLI.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.util.console;
+
+import picocli.CommandLine;
+
[email protected](name = "translate-for-nifi-cli", description =
"@|bold,fg(blue) Translates|@ the nifi.properties file to a format suitable for
use with the NiFi CLI tool")
+class PropertyEncryptorTranslateForCLI implements Runnable {
Review Comment:
Are there applicable NiFi CLI commands for encryption and decryption? It
seems like this may be unnecessary.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/util/crypto/CryptographyUtils.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.util.crypto;
+
+import org.apache.commons.codec.binary.Hex;
+import org.bouncycastle.crypto.generators.SCrypt;
+
+import javax.crypto.Cipher;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyException;
+import java.security.NoSuchAlgorithmException;
+import java.util.Arrays;
+import java.util.List;
+
+public class CryptographyUtils {
+
+ private static final String NIFI_SCRYPT_SALT = "NIFI_SCRYPT_SALT";
+ private static final int DEFAULT_MIN_PASSPHRASE_LENGTH = 12;
+
+ // Strong parameters as of 12 Aug 2016
+ private static final int SCRYPT_N = (int) Math.pow(2, 16);
+ private static final int SCRYPT_R = 8;
+ private static final int SCRYPT_P = 1;
Review Comment:
It seems the like scrypt details should be abstracted in some kind of
interface, perhaps a `SaltGenerator`?
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/xml/XmlCryptoParser.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.xml;
+
+import org.apache.nifi.properties.SensitivePropertyProvider;
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+
+import javax.xml.namespace.QName;
+import javax.xml.stream.XMLEventFactory;
+import javax.xml.stream.XMLEventReader;
+import javax.xml.stream.XMLEventWriter;
+import javax.xml.stream.XMLInputFactory;
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.events.Attribute;
+import javax.xml.stream.events.Characters;
+import javax.xml.stream.events.StartElement;
+import javax.xml.stream.events.XMLEvent;
+import java.io.FileNotFoundException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.Objects;
+
+public abstract class XmlCryptoParser {
Review Comment:
Similar to other components, defining an interface would provide a clear
contract, as opposed to an abstract base class.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/test/java/org/apache/nifi/encryptor/XmlEncryptorTest.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.encryptor;
+
+import org.apache.nifi.util.crypto.CryptographyUtils;
+import org.apache.nifi.util.file.FileUtilities;
+import org.apache.nifi.properties.StandardSensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+import org.apache.nifi.properties.scheme.StandardProtectionScheme;
+import org.apache.nifi.xml.XmlDecryptor;
+import org.apache.nifi.xml.XmlEncryptor;
+import org.junit.jupiter.api.Test;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.nio.file.Files;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+class XmlEncryptorTest {
+
+ private static final String KEY_HEX_128 =
"0123456789ABCDEFFEDCBA9876543210";
+ private static final String KEY_HEX_256 = KEY_HEX_128 + KEY_HEX_128;
+ static final ProtectionScheme DEFAULT_PROTECTION_SCHEME = new
StandardProtectionScheme("aes/gcm");
+ public static final String KEY_HEX =
CryptographyUtils.isUnlimitedStrengthCryptoAvailable() ? KEY_HEX_256 :
KEY_HEX_128;
Review Comment:
This can be refactored to just use AES-256 since Java 8 Update 171 and later
do not limit key sizes.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/xml/XmlDecryptor.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.xml;
+
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+import javax.xml.namespace.QName;
+import javax.xml.stream.XMLEventFactory;
+import javax.xml.stream.XMLEventReader;
+import javax.xml.stream.XMLInputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.events.Characters;
+import javax.xml.stream.events.StartElement;
+import javax.xml.stream.events.XMLEvent;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.Objects;
+
+public class XmlDecryptor extends XmlCryptoParser {
+
+ protected static final String ENCRYPTION_NONE = "none";
+ protected static final String PROPERTY_ELEMENT = "property";
+ protected static final String ENCRYPTION_ATTRIBUTE_NAME = "encryption";
+
+ public XmlDecryptor(final SensitivePropertyProviderFactory
providerFactory, final ProtectionScheme scheme) {
+ super(providerFactory, scheme);
+ this.providerFactory = providerFactory;
+ }
+
+ public void decrypt(final InputStream encryptedXmlContent, final
OutputStream decryptedOutputStream) {
+ cryptographicXmlOperation(encryptedXmlContent, decryptedOutputStream);
+ }
+
+ @Override
+ protected StartElement updateStartElementEncryptionAttribute(XMLEvent
xmlEvent) {
+ return convertToDecryptedElement(xmlEvent);
+ }
+
+ @Override
+ protected Characters cryptoOperationOnCharacters(XMLEvent xmlEvent, String
groupIdentifier, final String propertyName) {
+ return decryptElementCharacters(xmlEvent, groupIdentifier,
propertyName);
+ }
+
+ /**
+ * Decrypt the XMLEvent element characters/value, which should contain an
encrypted value
+ * @param xmlEvent The encrypted Characters event to be decrypted
+ * @return The decrypted Characters event
+ */
+ private Characters decryptElementCharacters(final XMLEvent xmlEvent, final
String groupIdentifier, final String propertyName) {
+ final XMLEventFactory eventFactory = XMLEventFactory.newInstance();
+ final String encryptedCharacters =
xmlEvent.asCharacters().getData().trim();
+ String decryptedCharacters =
cryptoProvider.unprotect(encryptedCharacters,
providerFactory.getPropertyContext(groupIdentifier, propertyName));
+ return eventFactory.createCharacters(decryptedCharacters);
+ }
+
+ /**
+ * Takes a StartElement and updates the 'encrypted' attribute to empty
string to remove the encryption method/scheme
+ * @param xmlEvent The opening/start XMLEvent for an encrypted property
+ * @return The updated element to be written to XML file
+ */
+ private StartElement convertToDecryptedElement(final XMLEvent xmlEvent) {
+ if (isEncryptedElement(xmlEvent)) {
+ return updateElementAttribute(xmlEvent, ENCRYPTION_ATTRIBUTE_NAME,
ENCRYPTION_NONE);
+ } else {
+ throw new XmlCryptoException(String.format("Failed to update the
element's [%s] attribute when decrypting the element value",
ENCRYPTION_ATTRIBUTE_NAME));
+ }
+ }
+
+ public boolean isEncryptedElement(final XMLEvent xmlEvent) {
+ return xmlEvent.isStartElement()
+ &&
xmlEvent.asStartElement().getName().toString().equals(PROPERTY_ELEMENT)
+ && elementHasEncryptionAttribute(xmlEvent.asStartElement());
+ }
+
+ private boolean elementHasEncryptionAttribute(final StartElement xmlEvent)
{
+ return xmlElementHasAttribute(xmlEvent, ENCRYPTION_ATTRIBUTE_NAME);
+ }
+
+ private boolean xmlElementHasAttribute(final StartElement xmlEvent, final
String attributeName) {
+ return !Objects.isNull(xmlEvent.getAttributeByName(new
QName(attributeName)));
+ }
+
+ public XMLEventReader getXMLReader(final InputStream fileStream) throws
XMLStreamException {
+ XMLInputFactory xmlInputFactory = XMLInputFactory.newFactory();
+
xmlInputFactory.setProperty(XMLInputFactory.IS_SUPPORTING_EXTERNAL_ENTITIES,
false);
+ xmlInputFactory.setProperty(XMLInputFactory.SUPPORT_DTD, false);
+
+ return xmlInputFactory.createXMLEventReader(fileStream);
+ }
Review Comment:
This should be refactored to use components from `nifi-xml-processing`.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/serde/PropertiesWriter.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.serde;
+
+import org.apache.nifi.properties.ReadableProperties;
+
+import java.io.BufferedReader;
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.util.Set;
+
+public class PropertiesWriter {
Review Comment:
Recommend declaring an interface and creating a standard implementation,
even if it is just one method.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/PropertyEncryptorMain.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi;
+
+import org.apache.nifi.properties.AbstractBootstrapPropertiesLoader;
+import org.apache.nifi.properties.BootstrapProperties;
+import org.apache.nifi.properties.MutableBootstrapProperties;
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.StandardSensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.PropertyProtectionScheme;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+import
org.apache.nifi.registry.properties.util.NiFiRegistryBootstrapPropertiesLoader;
+import org.apache.nifi.serde.PropertiesWriter;
+import org.apache.nifi.util.NiFiBootstrapPropertiesLoader;
+import org.apache.nifi.util.crypto.CryptographyUtils;
+import org.apache.nifi.xml.XmlDecryptor;
+import org.apache.nifi.xml.XmlEncryptor;
+import org.apache.nifi.util.file.ConfigurationFileResolver;
+import org.apache.nifi.util.file.FileUtilities;
+import org.apache.nifi.properties.ApplicationProperties;
+import org.apache.nifi.properties.NiFiPropertiesLoader;
+import org.apache.nifi.properties.PropertiesLoader;
+import org.apache.nifi.registry.properties.NiFiRegistryPropertiesLoader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Path;
+import java.util.List;
+
+public class PropertyEncryptorMain {
+
+ private static final Logger logger =
LoggerFactory.getLogger(PropertyEncryptorMain.class);
+ private final AbstractBootstrapPropertiesLoader bootstrapLoader;
+ private final PropertiesLoader<ApplicationProperties> propertiesLoader;
+ private final ConfigurationFileResolver fileResolver;
+ private final List<File> configurationFiles;
+ private String hexKey;
+ private final Path confDirectory;
+
+ public PropertyEncryptorMain(final Path baseDirectory, final String
passphrase) throws PropertyEncryptorException {
+ confDirectory =
FileUtilities.resolveAbsoluteConfDirectory(baseDirectory);
+ try {
+ bootstrapLoader = getBootstrapPropertiesLoader(confDirectory);
+ fileResolver = new ConfigurationFileResolver(confDirectory);
+ final File applicationProperties =
FileUtilities.resolvePropertiesFile(confDirectory);
+ propertiesLoader = getPropertiesLoader(confDirectory);
+ configurationFiles =
fileResolver.resolveConfigurationFilesFromApplicationProperties(propertiesLoader.load(applicationProperties));
+ hexKey = getKeyHex(confDirectory, passphrase);
+ } catch (final Exception e) {
+ throw new PropertyEncryptorException("Failed to run property
encryptor", e);
+ }
+ }
+
+ private AbstractBootstrapPropertiesLoader
getBootstrapPropertiesLoader(final Path baseDirectory) {
+ if (FileUtilities.isNiFiRegistryConfDirectory(baseDirectory)) {
+ return new NiFiRegistryBootstrapPropertiesLoader();
+ } else if (FileUtilities.isNiFiConfDirectory(baseDirectory)) {
+ return new NiFiBootstrapPropertiesLoader();
+ } else {
+ throw new PropertyEncryptorException("The base directory provided
does not contain a recognized bootstrap.conf file");
+ }
+ }
+
+ private PropertiesLoader<ApplicationProperties> getPropertiesLoader(final
Path baseDirectory) {
+ if (FileUtilities.isNiFiRegistryConfDirectory(baseDirectory)) {
+ return new NiFiRegistryPropertiesLoader();
+ } else if (FileUtilities.isNiFiConfDirectory(baseDirectory)) {
+ return new NiFiPropertiesLoader();
+ } else {
+ throw new PropertyEncryptorException("The base directory provided
does not contain a recognized .properties file");
Review Comment:
It would be helpful to include the `baseDirectory` in the message.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/PropertyEncryptorMain.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi;
+
+import org.apache.nifi.properties.AbstractBootstrapPropertiesLoader;
+import org.apache.nifi.properties.BootstrapProperties;
+import org.apache.nifi.properties.MutableBootstrapProperties;
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.StandardSensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.PropertyProtectionScheme;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+import
org.apache.nifi.registry.properties.util.NiFiRegistryBootstrapPropertiesLoader;
+import org.apache.nifi.serde.PropertiesWriter;
+import org.apache.nifi.util.NiFiBootstrapPropertiesLoader;
+import org.apache.nifi.util.crypto.CryptographyUtils;
+import org.apache.nifi.xml.XmlDecryptor;
+import org.apache.nifi.xml.XmlEncryptor;
+import org.apache.nifi.util.file.ConfigurationFileResolver;
+import org.apache.nifi.util.file.FileUtilities;
+import org.apache.nifi.properties.ApplicationProperties;
+import org.apache.nifi.properties.NiFiPropertiesLoader;
+import org.apache.nifi.properties.PropertiesLoader;
+import org.apache.nifi.registry.properties.NiFiRegistryPropertiesLoader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Path;
+import java.util.List;
+
+public class PropertyEncryptorMain {
+
+ private static final Logger logger =
LoggerFactory.getLogger(PropertyEncryptorMain.class);
+ private final AbstractBootstrapPropertiesLoader bootstrapLoader;
+ private final PropertiesLoader<ApplicationProperties> propertiesLoader;
+ private final ConfigurationFileResolver fileResolver;
+ private final List<File> configurationFiles;
+ private String hexKey;
+ private final Path confDirectory;
+
+ public PropertyEncryptorMain(final Path baseDirectory, final String
passphrase) throws PropertyEncryptorException {
+ confDirectory =
FileUtilities.resolveAbsoluteConfDirectory(baseDirectory);
+ try {
+ bootstrapLoader = getBootstrapPropertiesLoader(confDirectory);
+ fileResolver = new ConfigurationFileResolver(confDirectory);
+ final File applicationProperties =
FileUtilities.resolvePropertiesFile(confDirectory);
+ propertiesLoader = getPropertiesLoader(confDirectory);
+ configurationFiles =
fileResolver.resolveConfigurationFilesFromApplicationProperties(propertiesLoader.load(applicationProperties));
+ hexKey = getKeyHex(confDirectory, passphrase);
+ } catch (final Exception e) {
+ throw new PropertyEncryptorException("Failed to run property
encryptor", e);
+ }
+ }
+
+ private AbstractBootstrapPropertiesLoader
getBootstrapPropertiesLoader(final Path baseDirectory) {
+ if (FileUtilities.isNiFiRegistryConfDirectory(baseDirectory)) {
+ return new NiFiRegistryBootstrapPropertiesLoader();
+ } else if (FileUtilities.isNiFiConfDirectory(baseDirectory)) {
+ return new NiFiBootstrapPropertiesLoader();
+ } else {
+ throw new PropertyEncryptorException("The base directory provided
does not contain a recognized bootstrap.conf file");
+ }
+ }
+
+ private PropertiesLoader<ApplicationProperties> getPropertiesLoader(final
Path baseDirectory) {
+ if (FileUtilities.isNiFiRegistryConfDirectory(baseDirectory)) {
+ return new NiFiRegistryPropertiesLoader();
+ } else if (FileUtilities.isNiFiConfDirectory(baseDirectory)) {
+ return new NiFiPropertiesLoader();
+ } else {
+ throw new PropertyEncryptorException("The base directory provided
does not contain a recognized .properties file");
+ }
+ }
+
+ /**
+ * @param baseDirectory The base directory of a NiFi / NiFi Registry
installation that should be encrypted
+ */
+ public int encryptConfigurationFiles(final Path baseDirectory, final
PropertyProtectionScheme scheme) {
+ XmlEncryptor encryptor = getXmlEncryptor(scheme);
+ try {
+ encryptConfigurationFiles(configurationFiles, encryptor);
+ outputKeyToBootstrap();
+ logger.info("The Property Encryptor successfully encrypted
configuration files in the [{}] directory with the scheme [{}]", baseDirectory,
scheme.getPath());
+ return 0;
+ } catch (Exception e) {
+ logger.error("The Property Encryptor failed to encrypt
configuration files in the [{}] directory with the scheme [{}]", baseDirectory,
scheme.getPath(), e);
+ return 1;
+ }
+ }
+
+ private void outputKeyToBootstrap() throws IOException {
+ final File bootstrapFile =
bootstrapLoader.getBootstrapFileWithinConfDirectory(confDirectory);
+ File tempBootstrapFile = FileUtilities.getTemporaryOutputFile("tmp",
bootstrapFile);
+ final MutableBootstrapProperties bootstrapProperties =
bootstrapLoader.loadMutableBootstrapProperties(bootstrapFile.getPath());
+
bootstrapProperties.setProperty(BootstrapProperties.BootstrapPropertyKey.SENSITIVE_KEY.getKey(),
hexKey);
+ PropertiesWriter.writePropertiesFile(new
FileInputStream(bootstrapFile), new FileOutputStream(tempBootstrapFile),
bootstrapProperties);
+ logger.info("Output the bootstrap key to {}", tempBootstrapFile);
+ }
+
+ private int encryptConfigurationFiles(final List<File> configurationFiles,
final XmlEncryptor encryptor) throws IOException {
+ for (final File configurationFile : configurationFiles) {
+ File temp = FileUtilities.getTemporaryOutputFile("tmp",
configurationFile);
+ try (InputStream inputStream = new
FileInputStream(configurationFile);
+ FileOutputStream outputStream = new FileOutputStream(temp)) {
+ encryptor.encrypt(inputStream, outputStream);
+ logger.info("Successfully encrypted [{}]",
configurationFile.getAbsolutePath());
+ } catch (Exception e) {
+ throw new PropertyEncryptorException(String.format("Failed to
encrypt configuration file: [%s]", configurationFile.getAbsolutePath()), e);
+ }
+
+ //Files.copy(temp.toPath(), configurationFile.toPath());
+ }
+ return 0;
+ }
+
+ public int migrateConfigurationFiles(final File baseDirectory) {
+ logger.info("Not yet implemented.");
+ return 0;
+ }
+
+ public int encryptFlowDefinition(final File baseDirectory) {
+ logger.info("Not yet implemented.");
+ return 0;
+ }
+
+ private XmlEncryptor getXmlEncryptor(final PropertyProtectionScheme
scheme) {
+ final SensitivePropertyProviderFactory providerFactory =
StandardSensitivePropertyProviderFactory.withKey(hexKey);
+ return new XmlEncryptor(providerFactory, scheme);
+ }
+
+ private XmlDecryptor getXmlDecryptor(final
SensitivePropertyProviderFactory providerFactory, final ProtectionScheme
scheme) {
+ return new XmlDecryptor(providerFactory, scheme);
+ }
+
+ private String getKeyHex(final Path confDirectory, final String
passphrase) {
Review Comment:
This is a good opportunity to name this method more clearly than the current
tool. Perhaps `getEncodedRootKey`?
##########
nifi-toolkit/nifi-property-encryptor-tool/pom.xml:
##########
@@ -0,0 +1,129 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <parent>
+ <artifactId>nifi</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.18.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>nifi-property-encryptor-tool</artifactId>
+
+ <properties>
+ <maven.compiler.source>8</maven.compiler.source>
+ <maven.compiler.target>8</maven.compiler.target>
+ </properties>
Review Comment:
These properties are not necessary and should be removed.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/util/crypto/CryptographyUtils.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.util.crypto;
+
+import org.apache.commons.codec.binary.Hex;
+import org.bouncycastle.crypto.generators.SCrypt;
+
+import javax.crypto.Cipher;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyException;
+import java.security.NoSuchAlgorithmException;
+import java.util.Arrays;
+import java.util.List;
+
+public class CryptographyUtils {
+
+ private static final String NIFI_SCRYPT_SALT = "NIFI_SCRYPT_SALT";
+ private static final int DEFAULT_MIN_PASSPHRASE_LENGTH = 12;
+
+ // Strong parameters as of 12 Aug 2016
+ private static final int SCRYPT_N = (int) Math.pow(2, 16);
+ private static final int SCRYPT_R = 8;
+ private static final int SCRYPT_P = 1;
+
+ public static String deriveKeyFromPassphrase(final String passphrase)
throws KeyException, NoSuchAlgorithmException {
+ return deriveKeyFromPassphrase(passphrase,
DEFAULT_MIN_PASSPHRASE_LENGTH);
+ }
+
+ public static boolean isUnlimitedStrengthCryptoAvailable() {
+ try {
+ return Cipher.getMaxAllowedKeyLength("AES") > 128;
+ } catch (NoSuchAlgorithmException e) {
+ return false;
+ }
+ }
Review Comment:
This method is no longer necessary for versions of Java 8 after Update 171,
so it can be removed.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/util/console/ConfigSubcommand.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.util.console;
+
+import org.apache.nifi.PropertyEncryptorMain;
+import org.apache.nifi.properties.scheme.PropertyProtectionScheme;
+import org.apache.nifi.util.console.utils.BaseCommandParameters;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
[email protected](name = "config",
+ description = "Operate on application configs",
+ usageHelpWidth=140
+)
+class ConfigSubcommand extends BaseCommandParameters implements Runnable {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ConfigSubcommand.class);
+
+ @CommandLine.ParentCommand
+ private DefaultCLIOptions parent;
+
+ @CommandLine.Parameters(description="The encryption scheme to use, from
one of the following schemes: [@|bold ${COMPLETION-CANDIDATES}|@]")
+ PropertyProtectionScheme scheme;
+
+ @Override
+ public void run() {
+ final PropertyEncryptorMain propertyEncryptorMain = new
PropertyEncryptorMain(baseDirectory, passphrase);
Review Comment:
The instantiation and invocation of `PropertyEncryptorMain` calls into
question the approach. Instead of having that `Main` class, it seems like more
of the methods should be implemented in this class, or broken into their
component parts.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/PropertyEncryptorMain.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi;
+
+import org.apache.nifi.properties.AbstractBootstrapPropertiesLoader;
+import org.apache.nifi.properties.BootstrapProperties;
+import org.apache.nifi.properties.MutableBootstrapProperties;
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.StandardSensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.PropertyProtectionScheme;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+import
org.apache.nifi.registry.properties.util.NiFiRegistryBootstrapPropertiesLoader;
+import org.apache.nifi.serde.PropertiesWriter;
+import org.apache.nifi.util.NiFiBootstrapPropertiesLoader;
+import org.apache.nifi.util.crypto.CryptographyUtils;
+import org.apache.nifi.xml.XmlDecryptor;
+import org.apache.nifi.xml.XmlEncryptor;
+import org.apache.nifi.util.file.ConfigurationFileResolver;
+import org.apache.nifi.util.file.FileUtilities;
+import org.apache.nifi.properties.ApplicationProperties;
+import org.apache.nifi.properties.NiFiPropertiesLoader;
+import org.apache.nifi.properties.PropertiesLoader;
+import org.apache.nifi.registry.properties.NiFiRegistryPropertiesLoader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Path;
+import java.util.List;
+
+public class PropertyEncryptorMain {
Review Comment:
As a general convention, recommend naming this `PropertyEncryptorCommand`.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/util/crypto/CryptographyUtils.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.util.crypto;
+
+import org.apache.commons.codec.binary.Hex;
+import org.bouncycastle.crypto.generators.SCrypt;
+
+import javax.crypto.Cipher;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyException;
+import java.security.NoSuchAlgorithmException;
+import java.util.Arrays;
+import java.util.List;
+
+public class CryptographyUtils {
Review Comment:
Rather than reintroducing a generalized utility class, it would be better to
define one or more interface for specific purposes. In this case, a method for
returning a derived key seems like the right interface contract.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/PropertyEncryptorMain.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi;
+
+import org.apache.nifi.properties.AbstractBootstrapPropertiesLoader;
+import org.apache.nifi.properties.BootstrapProperties;
+import org.apache.nifi.properties.MutableBootstrapProperties;
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.StandardSensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.PropertyProtectionScheme;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+import
org.apache.nifi.registry.properties.util.NiFiRegistryBootstrapPropertiesLoader;
+import org.apache.nifi.serde.PropertiesWriter;
+import org.apache.nifi.util.NiFiBootstrapPropertiesLoader;
+import org.apache.nifi.util.crypto.CryptographyUtils;
+import org.apache.nifi.xml.XmlDecryptor;
+import org.apache.nifi.xml.XmlEncryptor;
+import org.apache.nifi.util.file.ConfigurationFileResolver;
+import org.apache.nifi.util.file.FileUtilities;
+import org.apache.nifi.properties.ApplicationProperties;
+import org.apache.nifi.properties.NiFiPropertiesLoader;
+import org.apache.nifi.properties.PropertiesLoader;
+import org.apache.nifi.registry.properties.NiFiRegistryPropertiesLoader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Path;
+import java.util.List;
+
+public class PropertyEncryptorMain {
+
+ private static final Logger logger =
LoggerFactory.getLogger(PropertyEncryptorMain.class);
+ private final AbstractBootstrapPropertiesLoader bootstrapLoader;
+ private final PropertiesLoader<ApplicationProperties> propertiesLoader;
+ private final ConfigurationFileResolver fileResolver;
+ private final List<File> configurationFiles;
+ private String hexKey;
+ private final Path confDirectory;
+
+ public PropertyEncryptorMain(final Path baseDirectory, final String
passphrase) throws PropertyEncryptorException {
+ confDirectory =
FileUtilities.resolveAbsoluteConfDirectory(baseDirectory);
+ try {
+ bootstrapLoader = getBootstrapPropertiesLoader(confDirectory);
+ fileResolver = new ConfigurationFileResolver(confDirectory);
+ final File applicationProperties =
FileUtilities.resolvePropertiesFile(confDirectory);
+ propertiesLoader = getPropertiesLoader(confDirectory);
+ configurationFiles =
fileResolver.resolveConfigurationFilesFromApplicationProperties(propertiesLoader.load(applicationProperties));
+ hexKey = getKeyHex(confDirectory, passphrase);
+ } catch (final Exception e) {
+ throw new PropertyEncryptorException("Failed to run property
encryptor", e);
+ }
+ }
+
+ private AbstractBootstrapPropertiesLoader
getBootstrapPropertiesLoader(final Path baseDirectory) {
+ if (FileUtilities.isNiFiRegistryConfDirectory(baseDirectory)) {
+ return new NiFiRegistryBootstrapPropertiesLoader();
+ } else if (FileUtilities.isNiFiConfDirectory(baseDirectory)) {
+ return new NiFiBootstrapPropertiesLoader();
+ } else {
+ throw new PropertyEncryptorException("The base directory provided
does not contain a recognized bootstrap.conf file");
+ }
+ }
+
+ private PropertiesLoader<ApplicationProperties> getPropertiesLoader(final
Path baseDirectory) {
+ if (FileUtilities.isNiFiRegistryConfDirectory(baseDirectory)) {
+ return new NiFiRegistryPropertiesLoader();
+ } else if (FileUtilities.isNiFiConfDirectory(baseDirectory)) {
+ return new NiFiPropertiesLoader();
+ } else {
+ throw new PropertyEncryptorException("The base directory provided
does not contain a recognized .properties file");
+ }
+ }
+
+ /**
+ * @param baseDirectory The base directory of a NiFi / NiFi Registry
installation that should be encrypted
+ */
+ public int encryptConfigurationFiles(final Path baseDirectory, final
PropertyProtectionScheme scheme) {
Review Comment:
Returning an `int` from this method does not provide a very clear indication
of the status. Recommend at least using an enum, and incorporating a numeric
status if needed.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/util/crypto/CryptographyUtils.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.util.crypto;
+
+import org.apache.commons.codec.binary.Hex;
+import org.bouncycastle.crypto.generators.SCrypt;
+
+import javax.crypto.Cipher;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyException;
+import java.security.NoSuchAlgorithmException;
+import java.util.Arrays;
+import java.util.List;
+
+public class CryptographyUtils {
+
+ private static final String NIFI_SCRYPT_SALT = "NIFI_SCRYPT_SALT";
+ private static final int DEFAULT_MIN_PASSPHRASE_LENGTH = 12;
+
+ // Strong parameters as of 12 Aug 2016
+ private static final int SCRYPT_N = (int) Math.pow(2, 16);
+ private static final int SCRYPT_R = 8;
+ private static final int SCRYPT_P = 1;
+
+ public static String deriveKeyFromPassphrase(final String passphrase)
throws KeyException, NoSuchAlgorithmException {
+ return deriveKeyFromPassphrase(passphrase,
DEFAULT_MIN_PASSPHRASE_LENGTH);
+ }
+
+ public static boolean isUnlimitedStrengthCryptoAvailable() {
+ try {
+ return Cipher.getMaxAllowedKeyLength("AES") > 128;
+ } catch (NoSuchAlgorithmException e) {
+ return false;
+ }
+ }
+
+ private static String deriveKeyFromPassphrase(final String passphrase,
final int minPassphraseLength) throws KeyException, NoSuchAlgorithmException {
+ final String trimmedPassphrase = passphrase.trim();
+ if (trimmedPassphrase.length() < minPassphraseLength) {
+ throw new KeyException(String.format("Cannot derive key from
empty/short passphrase -- passphrase must be at least %d characters",
DEFAULT_MIN_PASSPHRASE_LENGTH));
+ }
+
+ // Generate a 128 bit salt
+ byte[] salt = generateScryptSalt();
+ int keyLengthInBytes =
getValidKeyLengths().stream().max(Integer::compare).get() / 8;
+ byte[] derivedKeyBytes =
SCrypt.generate(passphrase.getBytes(StandardCharsets.UTF_8), salt, SCRYPT_N,
SCRYPT_R, SCRYPT_P, keyLengthInBytes);
+ return Hex.encodeHexString(derivedKeyBytes).toUpperCase();
Review Comment:
This seems like an opportunity to introduce support for a different
algorithm. It is probably unnecessary to support the same level of
configuration as currently supported for the Sensitive Properties Algorithm,
but perhaps standardizing on PBKDF2 would be better, as long as there is a way
to maintain compatibility with current implementations.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/util/file/ConfigurationFileResolver.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.util.file;
+
+import org.apache.nifi.properties.ApplicationProperties;
+import org.apache.nifi.registry.properties.NiFiRegistryProperties;
+import org.apache.nifi.util.NiFiProperties;
+
+import java.io.File;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Resolve configuration files that need to be encrypted from a given
ApplicationProperties
+ */
+public class ConfigurationFileResolver {
Review Comment:
As noted with other classes, this looks like a good opportunity to define an
interface and then provide a standard implementation.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/PropertyEncryptorMain.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi;
+
+import org.apache.nifi.properties.AbstractBootstrapPropertiesLoader;
+import org.apache.nifi.properties.BootstrapProperties;
+import org.apache.nifi.properties.MutableBootstrapProperties;
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.StandardSensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.PropertyProtectionScheme;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+import
org.apache.nifi.registry.properties.util.NiFiRegistryBootstrapPropertiesLoader;
+import org.apache.nifi.serde.PropertiesWriter;
+import org.apache.nifi.util.NiFiBootstrapPropertiesLoader;
+import org.apache.nifi.util.crypto.CryptographyUtils;
+import org.apache.nifi.xml.XmlDecryptor;
+import org.apache.nifi.xml.XmlEncryptor;
+import org.apache.nifi.util.file.ConfigurationFileResolver;
+import org.apache.nifi.util.file.FileUtilities;
+import org.apache.nifi.properties.ApplicationProperties;
+import org.apache.nifi.properties.NiFiPropertiesLoader;
+import org.apache.nifi.properties.PropertiesLoader;
+import org.apache.nifi.registry.properties.NiFiRegistryPropertiesLoader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Path;
+import java.util.List;
+
+public class PropertyEncryptorMain {
+
+ private static final Logger logger =
LoggerFactory.getLogger(PropertyEncryptorMain.class);
+ private final AbstractBootstrapPropertiesLoader bootstrapLoader;
+ private final PropertiesLoader<ApplicationProperties> propertiesLoader;
+ private final ConfigurationFileResolver fileResolver;
+ private final List<File> configurationFiles;
+ private String hexKey;
+ private final Path confDirectory;
+
+ public PropertyEncryptorMain(final Path baseDirectory, final String
passphrase) throws PropertyEncryptorException {
+ confDirectory =
FileUtilities.resolveAbsoluteConfDirectory(baseDirectory);
+ try {
+ bootstrapLoader = getBootstrapPropertiesLoader(confDirectory);
+ fileResolver = new ConfigurationFileResolver(confDirectory);
+ final File applicationProperties =
FileUtilities.resolvePropertiesFile(confDirectory);
+ propertiesLoader = getPropertiesLoader(confDirectory);
+ configurationFiles =
fileResolver.resolveConfigurationFilesFromApplicationProperties(propertiesLoader.load(applicationProperties));
+ hexKey = getKeyHex(confDirectory, passphrase);
+ } catch (final Exception e) {
+ throw new PropertyEncryptorException("Failed to run property
encryptor", e);
+ }
+ }
+
+ private AbstractBootstrapPropertiesLoader
getBootstrapPropertiesLoader(final Path baseDirectory) {
+ if (FileUtilities.isNiFiRegistryConfDirectory(baseDirectory)) {
+ return new NiFiRegistryBootstrapPropertiesLoader();
+ } else if (FileUtilities.isNiFiConfDirectory(baseDirectory)) {
+ return new NiFiBootstrapPropertiesLoader();
+ } else {
+ throw new PropertyEncryptorException("The base directory provided
does not contain a recognized bootstrap.conf file");
+ }
+ }
+
+ private PropertiesLoader<ApplicationProperties> getPropertiesLoader(final
Path baseDirectory) {
+ if (FileUtilities.isNiFiRegistryConfDirectory(baseDirectory)) {
+ return new NiFiRegistryPropertiesLoader();
+ } else if (FileUtilities.isNiFiConfDirectory(baseDirectory)) {
+ return new NiFiPropertiesLoader();
+ } else {
+ throw new PropertyEncryptorException("The base directory provided
does not contain a recognized .properties file");
+ }
+ }
+
+ /**
+ * @param baseDirectory The base directory of a NiFi / NiFi Registry
installation that should be encrypted
+ */
+ public int encryptConfigurationFiles(final Path baseDirectory, final
PropertyProtectionScheme scheme) {
+ XmlEncryptor encryptor = getXmlEncryptor(scheme);
+ try {
+ encryptConfigurationFiles(configurationFiles, encryptor);
+ outputKeyToBootstrap();
+ logger.info("The Property Encryptor successfully encrypted
configuration files in the [{}] directory with the scheme [{}]", baseDirectory,
scheme.getPath());
+ return 0;
+ } catch (Exception e) {
+ logger.error("The Property Encryptor failed to encrypt
configuration files in the [{}] directory with the scheme [{}]", baseDirectory,
scheme.getPath(), e);
+ return 1;
+ }
+ }
+
+ private void outputKeyToBootstrap() throws IOException {
+ final File bootstrapFile =
bootstrapLoader.getBootstrapFileWithinConfDirectory(confDirectory);
+ File tempBootstrapFile = FileUtilities.getTemporaryOutputFile("tmp",
bootstrapFile);
+ final MutableBootstrapProperties bootstrapProperties =
bootstrapLoader.loadMutableBootstrapProperties(bootstrapFile.getPath());
+
bootstrapProperties.setProperty(BootstrapProperties.BootstrapPropertyKey.SENSITIVE_KEY.getKey(),
hexKey);
+ PropertiesWriter.writePropertiesFile(new
FileInputStream(bootstrapFile), new FileOutputStream(tempBootstrapFile),
bootstrapProperties);
Review Comment:
It looks like these streams should be created in a try-with-resources block.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/xml/XmlCryptoException.java:
##########
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.xml;
+
+public class XmlCryptoException extends RuntimeException {
Review Comment:
This seems a bit more specific than necessary, perhaps `CryptoException`?
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/util/file/FileUtilities.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.util.file;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Arrays;
+
+public class FileUtilities {
Review Comment:
This class name seems a bit too generic. Perhaps ConfigurationFileUtils?
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/xml/XmlCryptoParser.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.xml;
+
+import org.apache.nifi.properties.SensitivePropertyProvider;
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+
+import javax.xml.namespace.QName;
+import javax.xml.stream.XMLEventFactory;
+import javax.xml.stream.XMLEventReader;
+import javax.xml.stream.XMLEventWriter;
+import javax.xml.stream.XMLInputFactory;
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.events.Attribute;
+import javax.xml.stream.events.Characters;
+import javax.xml.stream.events.StartElement;
+import javax.xml.stream.events.XMLEvent;
+import java.io.FileNotFoundException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.Objects;
+
+public abstract class XmlCryptoParser {
+
+ protected static final String ENCRYPTION_ATTRIBUTE_NAME = "encryption";
+ protected static final String PARENT_IDENTIFIER = "identifier";
+ protected static final String PROPERTY_ELEMENT = "property";
+
+ protected final SensitivePropertyProvider cryptoProvider;
+ protected SensitivePropertyProviderFactory providerFactory;
+
+ public XmlCryptoParser(final SensitivePropertyProviderFactory
providerFactory, final ProtectionScheme scheme) {
+ this.providerFactory = providerFactory;
+ cryptoProvider = providerFactory.getProvider(scheme);
+ }
+
+ protected void cryptographicXmlOperation(final InputStream
encryptedXmlContent, final OutputStream decryptedOutputStream) {
+ final XMLOutputFactory factory = XMLOutputFactory.newInstance();
+ factory.setProperty("com.ctc.wstx.outputValidateStructure", false);
+
+ try {
+ final XMLEventReader eventReader =
getXMLReader(encryptedXmlContent);
+ final XMLEventWriter xmlWriter =
factory.createXMLEventWriter(decryptedOutputStream);
+ String groupIdentifier = "";
+
+ while(eventReader.hasNext()) {
+ XMLEvent event = eventReader.nextEvent();
+
+ if (isGroupIdentifier(event)) {
+ groupIdentifier =
getGroupIdentifier(eventReader.nextEvent());
+ }
+
+ if (isSensitiveElement(event)) {
+
xmlWriter.add(updateStartElementEncryptionAttribute(event));
+
xmlWriter.add(cryptoOperationOnCharacters(eventReader.nextEvent(),
groupIdentifier, getPropertyName(event)));
+ } else {
+ try {
+ xmlWriter.add(event);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed operation on XML
content", e);
Review Comment:
It would be helpful to throw a more specific type of exception.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/xml/XmlCryptoParser.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.xml;
+
+import org.apache.nifi.properties.SensitivePropertyProvider;
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+
+import javax.xml.namespace.QName;
+import javax.xml.stream.XMLEventFactory;
+import javax.xml.stream.XMLEventReader;
+import javax.xml.stream.XMLEventWriter;
+import javax.xml.stream.XMLInputFactory;
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.events.Attribute;
+import javax.xml.stream.events.Characters;
+import javax.xml.stream.events.StartElement;
+import javax.xml.stream.events.XMLEvent;
+import java.io.FileNotFoundException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.Objects;
+
+public abstract class XmlCryptoParser {
+
+ protected static final String ENCRYPTION_ATTRIBUTE_NAME = "encryption";
+ protected static final String PARENT_IDENTIFIER = "identifier";
+ protected static final String PROPERTY_ELEMENT = "property";
+
+ protected final SensitivePropertyProvider cryptoProvider;
+ protected SensitivePropertyProviderFactory providerFactory;
+
+ public XmlCryptoParser(final SensitivePropertyProviderFactory
providerFactory, final ProtectionScheme scheme) {
+ this.providerFactory = providerFactory;
+ cryptoProvider = providerFactory.getProvider(scheme);
+ }
+
+ protected void cryptographicXmlOperation(final InputStream
encryptedXmlContent, final OutputStream decryptedOutputStream) {
+ final XMLOutputFactory factory = XMLOutputFactory.newInstance();
+ factory.setProperty("com.ctc.wstx.outputValidateStructure", false);
Review Comment:
This property presumes the use of Woodstox, which is not clearly implied
elsewhere. It looks like this should be removed and re-evaluated.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/xml/XmlCryptoParser.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.xml;
+
+import org.apache.nifi.properties.SensitivePropertyProvider;
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+
+import javax.xml.namespace.QName;
+import javax.xml.stream.XMLEventFactory;
+import javax.xml.stream.XMLEventReader;
+import javax.xml.stream.XMLEventWriter;
+import javax.xml.stream.XMLInputFactory;
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.events.Attribute;
+import javax.xml.stream.events.Characters;
+import javax.xml.stream.events.StartElement;
+import javax.xml.stream.events.XMLEvent;
+import java.io.FileNotFoundException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.Objects;
+
+public abstract class XmlCryptoParser {
+
+ protected static final String ENCRYPTION_ATTRIBUTE_NAME = "encryption";
+ protected static final String PARENT_IDENTIFIER = "identifier";
+ protected static final String PROPERTY_ELEMENT = "property";
+
+ protected final SensitivePropertyProvider cryptoProvider;
+ protected SensitivePropertyProviderFactory providerFactory;
+
+ public XmlCryptoParser(final SensitivePropertyProviderFactory
providerFactory, final ProtectionScheme scheme) {
+ this.providerFactory = providerFactory;
+ cryptoProvider = providerFactory.getProvider(scheme);
+ }
+
+ protected void cryptographicXmlOperation(final InputStream
encryptedXmlContent, final OutputStream decryptedOutputStream) {
+ final XMLOutputFactory factory = XMLOutputFactory.newInstance();
+ factory.setProperty("com.ctc.wstx.outputValidateStructure", false);
+
+ try {
+ final XMLEventReader eventReader =
getXMLReader(encryptedXmlContent);
+ final XMLEventWriter xmlWriter =
factory.createXMLEventWriter(decryptedOutputStream);
+ String groupIdentifier = "";
+
+ while(eventReader.hasNext()) {
+ XMLEvent event = eventReader.nextEvent();
+
+ if (isGroupIdentifier(event)) {
+ groupIdentifier =
getGroupIdentifier(eventReader.nextEvent());
+ }
+
+ if (isSensitiveElement(event)) {
+
xmlWriter.add(updateStartElementEncryptionAttribute(event));
+
xmlWriter.add(cryptoOperationOnCharacters(eventReader.nextEvent(),
groupIdentifier, getPropertyName(event)));
+ } else {
+ try {
+ xmlWriter.add(event);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed operation on XML
content", e);
+ }
+ }
+ }
+
+ eventReader.close();
+ xmlWriter.flush();
+ xmlWriter.close();
+ } catch (Exception e) {
+ throw new RuntimeException("Failed operation on XML content", e);
+ }
+ }
+
+ /**
+ * Update the StartElement 'encryption' attribute for a sensitive value to
add or remove the respective encryption details eg. encryption="aes/gcm/128"
+ * @param xmlEvent A 'sensitive' StartElement that contains the
'encryption' tag attribute
+ * @return The updated StartElement
+ */
+ protected abstract StartElement
updateStartElementEncryptionAttribute(final XMLEvent xmlEvent);
+
+ /**
+ * Perform an encrypt or decrypt cryptographic operation on a Characters
element
+ * @param xmlEvent A Characters XmlEvent
+ * @param groupIdentifier The XML <identifier/> tag
+ * @return The Characters XmlEvent that has been updated by the
cryptographic operation
+ */
+ protected abstract Characters cryptoOperationOnCharacters(final XMLEvent
xmlEvent, final String groupIdentifier, final String propertyName);
+
+ private String getGroupIdentifier(final XMLEvent xmlEvent) {
+ if (xmlEvent.isCharacters()) {
+ return xmlEvent.asCharacters().getData();
+ } else {
+ return "";
+ }
+ }
+
+ protected String getPropertyName(final XMLEvent xmlEvent) {
+ return xmlEvent.asStartElement().getName().toString();
+ }
+
+ protected boolean isGroupIdentifier(final XMLEvent xmlEvent) {
+ return xmlEvent.isStartElement()
+ &&
xmlEvent.asStartElement().getName().toString().equals(PARENT_IDENTIFIER);
+ }
+
+ protected boolean isSensitiveElement(final XMLEvent xmlEvent) {
+ return xmlEvent.isStartElement()
+ &&
xmlEvent.asStartElement().getName().toString().equals(PROPERTY_ELEMENT)
+ && elementHasEncryptionAttribute(xmlEvent.asStartElement());
+ }
+
+ protected XMLEventReader getXMLReader(final InputStream fileStream) throws
XMLStreamException, FileNotFoundException {
+ XMLInputFactory xmlInputFactory = XMLInputFactory.newFactory();
+
xmlInputFactory.setProperty(XMLInputFactory.IS_SUPPORTING_EXTERNAL_ENTITIES,
false);
+ xmlInputFactory.setProperty(XMLInputFactory.SUPPORT_DTD, false);
+
+ return xmlInputFactory.createXMLEventReader(fileStream);
+ }
Review Comment:
This should be replaced with usage of XML components from
`nifi-xml-processing` to avoid the possibility of external entity parsing.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/test/resources/login-identity-providers-populated-encrypted.xml:
##########
@@ -0,0 +1,102 @@
+<?xml version='1.0' encoding='UTF-8' standalone='yes'?><!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+--><!--
+ This file lists the login identity providers to use when running securely.
In order
+ to use a specific provider it must be configured here and it's identifier
+ must be specified in the nifi.properties file.
+--><loginIdentityProviders>
+ <!--
+ Identity Provider for users logging in with username/password against
an LDAP server.
+
+ 'Authentication Strategy' - How the connection to the LDAP server is
authenticated. Possible
+ values are ANONYMOUS, SIMPLE, or START_TLS.
+
+ 'Manager DN' - The DN of the manager that is used to bind to the LDAP
server to search for users.
+ 'Manager Password' - The password of the manager that is used to bind
to the LDAP server to
+ search for users.
+
+ 'TLS - Keystore' - Path to the Keystore that is used when connecting
to LDAP using START_TLS.
+ 'TLS - Keystore Password' - Password for the Keystore that is used
when connecting to LDAP
+ using START_TLS.
+ 'TLS - Keystore Type' - Type of the Keystore that is used when
connecting to LDAP using
+ START_TLS (i.e. JKS or PKCS12).
+ 'TLS - Truststore' - Path to the Truststore that is used when
connecting to LDAP using START_TLS.
+ 'TLS - Truststore Password' - Password for the Truststore that is used
when connecting to
+ LDAP using START_TLS.
+ 'TLS - Truststore Type' - Type of the Truststore that is used when
connecting to LDAP using
+ START_TLS (i.e. JKS or PKCS12).
+ 'TLS - Client Auth' - Client authentication policy when connecting to
LDAP using START_TLS.
+ Possible values are REQUIRED, WANT, NONE.
+ 'TLS - Protocol' - Protocol to use when connecting to LDAP using
START_TLS. (i.e. TLS,
+ TLSv1.1, TLSv1.2, etc).
+ 'TLS - Shutdown Gracefully' - Specifies whether the TLS should be shut
down gracefully
+ before the target context is closed. Defaults to false.
+
+ 'Referral Strategy' - Strategy for handling referrals. Possible values
are FOLLOW, IGNORE, THROW.
+ 'Connect Timeout' - Duration of connect timeout. (i.e. 10 secs).
+ 'Read Timeout' - Duration of read timeout. (i.e. 10 secs).
+
+ 'Url' - Url of the LDAP servier (i.e. ldap://<hostname>:<port>).
+ 'User Search Base' - Base DN for searching for users (i.e.
CN=Users,DC=example,DC=com).
+ 'User Search Filter' - Filter for searching for users against the
'User Search Base'.
+ (i.e. sAMAccountName={0}). The user specified name is inserted
into '{0}'.
+
+ 'Authentication Expiration' - The duration of how long the user
authentication is valid
+ for. If the user never logs out, they will be required to log back
in following
+ this duration.
+ -->
Review Comment:
The file contents should be reduced to the minimum, so comments like this
can be removed.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/xml/XmlCryptoParser.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.xml;
+
+import org.apache.nifi.properties.SensitivePropertyProvider;
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+
+import javax.xml.namespace.QName;
+import javax.xml.stream.XMLEventFactory;
+import javax.xml.stream.XMLEventReader;
+import javax.xml.stream.XMLEventWriter;
+import javax.xml.stream.XMLInputFactory;
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.events.Attribute;
+import javax.xml.stream.events.Characters;
+import javax.xml.stream.events.StartElement;
+import javax.xml.stream.events.XMLEvent;
+import java.io.FileNotFoundException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.Objects;
+
+public abstract class XmlCryptoParser {
+
+ protected static final String ENCRYPTION_ATTRIBUTE_NAME = "encryption";
+ protected static final String PARENT_IDENTIFIER = "identifier";
+ protected static final String PROPERTY_ELEMENT = "property";
+
+ protected final SensitivePropertyProvider cryptoProvider;
+ protected SensitivePropertyProviderFactory providerFactory;
+
+ public XmlCryptoParser(final SensitivePropertyProviderFactory
providerFactory, final ProtectionScheme scheme) {
+ this.providerFactory = providerFactory;
+ cryptoProvider = providerFactory.getProvider(scheme);
+ }
+
+ protected void cryptographicXmlOperation(final InputStream
encryptedXmlContent, final OutputStream decryptedOutputStream) {
+ final XMLOutputFactory factory = XMLOutputFactory.newInstance();
+ factory.setProperty("com.ctc.wstx.outputValidateStructure", false);
+
+ try {
+ final XMLEventReader eventReader =
getXMLReader(encryptedXmlContent);
+ final XMLEventWriter xmlWriter =
factory.createXMLEventWriter(decryptedOutputStream);
+ String groupIdentifier = "";
+
+ while(eventReader.hasNext()) {
+ XMLEvent event = eventReader.nextEvent();
+
+ if (isGroupIdentifier(event)) {
+ groupIdentifier =
getGroupIdentifier(eventReader.nextEvent());
+ }
+
+ if (isSensitiveElement(event)) {
+
xmlWriter.add(updateStartElementEncryptionAttribute(event));
+
xmlWriter.add(cryptoOperationOnCharacters(eventReader.nextEvent(),
groupIdentifier, getPropertyName(event)));
+ } else {
+ try {
+ xmlWriter.add(event);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed operation on XML
content", e);
+ }
+ }
+ }
+
+ eventReader.close();
+ xmlWriter.flush();
+ xmlWriter.close();
+ } catch (Exception e) {
+ throw new RuntimeException("Failed operation on XML content", e);
+ }
+ }
+
+ /**
+ * Update the StartElement 'encryption' attribute for a sensitive value to
add or remove the respective encryption details eg. encryption="aes/gcm/128"
+ * @param xmlEvent A 'sensitive' StartElement that contains the
'encryption' tag attribute
+ * @return The updated StartElement
+ */
+ protected abstract StartElement
updateStartElementEncryptionAttribute(final XMLEvent xmlEvent);
+
+ /**
+ * Perform an encrypt or decrypt cryptographic operation on a Characters
element
+ * @param xmlEvent A Characters XmlEvent
+ * @param groupIdentifier The XML <identifier/> tag
+ * @return The Characters XmlEvent that has been updated by the
cryptographic operation
+ */
+ protected abstract Characters cryptoOperationOnCharacters(final XMLEvent
xmlEvent, final String groupIdentifier, final String propertyName);
+
+ private String getGroupIdentifier(final XMLEvent xmlEvent) {
+ if (xmlEvent.isCharacters()) {
+ return xmlEvent.asCharacters().getData();
+ } else {
+ return "";
+ }
+ }
+
+ protected String getPropertyName(final XMLEvent xmlEvent) {
+ return xmlEvent.asStartElement().getName().toString();
+ }
+
+ protected boolean isGroupIdentifier(final XMLEvent xmlEvent) {
+ return xmlEvent.isStartElement()
+ &&
xmlEvent.asStartElement().getName().toString().equals(PARENT_IDENTIFIER);
+ }
+
+ protected boolean isSensitiveElement(final XMLEvent xmlEvent) {
+ return xmlEvent.isStartElement()
+ &&
xmlEvent.asStartElement().getName().toString().equals(PROPERTY_ELEMENT)
+ && elementHasEncryptionAttribute(xmlEvent.asStartElement());
+ }
+
+ protected XMLEventReader getXMLReader(final InputStream fileStream) throws
XMLStreamException, FileNotFoundException {
+ XMLInputFactory xmlInputFactory = XMLInputFactory.newFactory();
+
xmlInputFactory.setProperty(XMLInputFactory.IS_SUPPORTING_EXTERNAL_ENTITIES,
false);
+ xmlInputFactory.setProperty(XMLInputFactory.SUPPORT_DTD, false);
+
+ return xmlInputFactory.createXMLEventReader(fileStream);
+ }
+
+ private boolean elementHasEncryptionAttribute(final StartElement xmlEvent)
{
+ return xmlElementHasAttribute(xmlEvent, ENCRYPTION_ATTRIBUTE_NAME);
+ }
+
+ private boolean xmlElementHasAttribute(final StartElement xmlEvent, final
String attributeName) {
+ return !Objects.isNull(xmlEvent.getAttributeByName(new
QName(attributeName)));
+ }
+
+ protected StartElement updateElementAttribute(final XMLEvent xmlEvent,
final String attributeName, final String attributeValue) {
Review Comment:
The general order of methods should be public, protected, then private.
##########
nifi-toolkit/nifi-property-encryptor-tool/src/test/resources/login-identity-providers-populated-unecrypted.xml:
##########
@@ -0,0 +1,102 @@
+<?xml version='1.0' encoding='UTF-8' standalone='yes'?><!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+--><!--
+ This file lists the login identity providers to use when running securely.
In order
+ to use a specific provider it must be configured here and it's identifier
+ must be specified in the nifi.properties file.
+--><loginIdentityProviders>
+ <!--
+ Identity Provider for users logging in with username/password against
an LDAP server.
+
+ 'Authentication Strategy' - How the connection to the LDAP server is
authenticated. Possible
+ values are ANONYMOUS, SIMPLE, or START_TLS.
+
+ 'Manager DN' - The DN of the manager that is used to bind to the LDAP
server to search for users.
+ 'Manager Password' - The password of the manager that is used to bind
to the LDAP server to
+ search for users.
+
+ 'TLS - Keystore' - Path to the Keystore that is used when connecting
to LDAP using START_TLS.
+ 'TLS - Keystore Password' - Password for the Keystore that is used
when connecting to LDAP
+ using START_TLS.
+ 'TLS - Keystore Type' - Type of the Keystore that is used when
connecting to LDAP using
+ START_TLS (i.e. JKS or PKCS12).
+ 'TLS - Truststore' - Path to the Truststore that is used when
connecting to LDAP using START_TLS.
+ 'TLS - Truststore Password' - Password for the Truststore that is used
when connecting to
+ LDAP using START_TLS.
+ 'TLS - Truststore Type' - Type of the Truststore that is used when
connecting to LDAP using
+ START_TLS (i.e. JKS or PKCS12).
+ 'TLS - Client Auth' - Client authentication policy when connecting to
LDAP using START_TLS.
+ Possible values are REQUIRED, WANT, NONE.
+ 'TLS - Protocol' - Protocol to use when connecting to LDAP using
START_TLS. (i.e. TLS,
+ TLSv1.1, TLSv1.2, etc).
+ 'TLS - Shutdown Gracefully' - Specifies whether the TLS should be shut
down gracefully
+ before the target context is closed. Defaults to false.
+
+ 'Referral Strategy' - Strategy for handling referrals. Possible values
are FOLLOW, IGNORE, THROW.
+ 'Connect Timeout' - Duration of connect timeout. (i.e. 10 secs).
+ 'Read Timeout' - Duration of read timeout. (i.e. 10 secs).
+
+ 'Url' - Url of the LDAP servier (i.e. ldap://<hostname>:<port>).
+ 'User Search Base' - Base DN for searching for users (i.e.
CN=Users,DC=example,DC=com).
+ 'User Search Filter' - Filter for searching for users against the
'User Search Base'.
+ (i.e. sAMAccountName={0}). The user specified name is inserted
into '{0}'.
+
+ 'Authentication Expiration' - The duration of how long the user
authentication is valid
+ for. If the user never logs out, they will be required to log back
in following
+ this duration.
+ -->
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
