NIFI-4701 Add authorizers.xml support to toolkit. Adds authorizers.xml to the files understood by the encrypt-config tool in the NiFi Toolkit. If enabled, then the sensitive properties for LdapUserGroupProvider in authorizers.xml will be encrypted. Also fixes a bug wherein encrypt-config replaces multiple XML nodes in login-indentity-providers.xml when LdapProvider is not the first provider listed in the file. Enable properties in authorizers.xml to be encrypted by the master key.
This closes #2350. Signed-off-by: Andy LoPresto <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/482f3719 Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/482f3719 Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/482f3719 Branch: refs/heads/master Commit: 482f371958a96b33791c3f6cc4c26efd47c43169 Parents: c91d998 Author: Kevin Doran <[email protected]> Authored: Sun Dec 17 11:40:06 2017 -0500 Committer: Andy LoPresto <[email protected]> Committed: Sun Dec 31 17:41:04 2017 -0500 ---------------------------------------------------------------------- .../src/main/asciidoc/administration-guide.adoc | 55 +- .../nifi-framework/nifi-authorizer/pom.xml | 4 + .../authorization/AuthorizerFactoryBean.java | 78 +- .../src/main/xsd/authorizers.xsd | 1 + .../AuthorizerFactoryBeanTest.groovy | 120 ++ .../nifi/properties/ConfigEncryptionTool.groovy | 293 ++++- .../properties/ConfigEncryptionToolTest.groovy | 1095 ++++++++++++++++-- .../test/resources/authorizers-commented.xml | 309 +++++ .../resources/authorizers-populated-empty.xml | 309 +++++ ...uthorizers-populated-encrypted-multiline.xml | 314 +++++ ...rs-populated-encrypted-multiple-per-line.xml | 305 +++++ .../authorizers-populated-encrypted.xml | 309 +++++ .../authorizers-populated-multiline.xml | 315 +++++ .../authorizers-populated-multiple-per-line.xml | 305 +++++ .../resources/authorizers-populated-renamed.xml | 309 +++++ .../test/resources/authorizers-populated.xml | 309 +++++ ...-providers-populated-with-many-providers.xml | 122 ++ 17 files changed, 4386 insertions(+), 166 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/nifi/blob/482f3719/nifi-docs/src/main/asciidoc/administration-guide.adoc ---------------------------------------------------------------------- diff --git a/nifi-docs/src/main/asciidoc/administration-guide.adoc b/nifi-docs/src/main/asciidoc/administration-guide.adoc index 37fe45e..6bbf8a2 100644 --- a/nifi-docs/src/main/asciidoc/administration-guide.adoc +++ b/nifi-docs/src/main/asciidoc/administration-guide.adoc @@ -1455,25 +1455,27 @@ The default encryption algorithm utilized is AES/GCM 128/256-bit. 128-bit is use You can use the following command line options with the `encrypt-config` tool: - * `-A`,`--newFlowAlgorithm <arg>` The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz - * `-b`,`--bootstrapConf <arg>` The bootstrap.conf file to persist master key - * `-e`,`--oldKey <arg>` The old raw hexadecimal key to use during key migration - * `-f`,`--flowXml <arg>` The flow.xml.gz file currently protected with old password (will be overwritten) - * `-g`,`--outputFlowXml <arg>` The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz) * `-h`,`--help` Prints this usage message - * `-i`,`--outputLoginIdentityProviders <arg>` The destination login-identity-providers.xml file containing protected config values (will not modify input login-identity-providers.xml) - * `-k`,`--key <arg>` The raw hexadecimal key to use to encrypt the sensitive properties - * `-l`,`--loginIdentityProviders <arg>` The login-identity-providers.xml file containing unprotected config values (will be overwritten) - * `-m`,`--migrate` If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with a new key + * `-v`,`--verbose` Sets verbose mode (default false) * `-n`,`--niFiProperties <arg>` The nifi.properties file containing unprotected config values (will be overwritten) + * `-l`,`--loginIdentityProviders <arg>` The login-identity-providers.xml file containing unprotected config values (will be overwritten) + * `-a`,`--authorizers <arg>` The authorizers.xml file containing unprotected config values (will be overwritten) + * `-f`,`--flowXml <arg>` The flow.xml.gz file currently protected with old password (will be overwritten) + * `-b`,`--bootstrapConf <arg>` The bootstrap.conf file to persist master key * `-o`,`--outputNiFiProperties <arg>` The destination nifi.properties file containing protected config values (will not modify input nifi.properties) + * `-i`,`--outputLoginIdentityProviders <arg>` The destination login-identity-providers.xml file containing protected config values (will not modify input login-identity-providers.xml) + * `-u`,`--outputAuthorizers <arg>` The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml) + * `-g`,`--outputFlowXml <arg>` The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz) + * `-k`,`--key <arg>` The raw hexadecimal key to use to encrypt the sensitive properties + * `-e`,`--oldKey <arg>` The old raw hexadecimal key to use during key migration * `-p`,`--password <arg>` The password from which to derive the key to use to encrypt the sensitive properties - * `-P`,`--newFlowProvider <arg>` The security provider to use to encrypt the sensitive processor properties in flow.xml.gz - * `-r`,`--useRawKey` If provided, the secure console will prompt for the raw key value in hexadecimal form - * `-s`,`--propsKey <arg>` The password or key to use to encrypt the sensitive processor properties in flow.xml.gz - * `-v`,`--verbose` Sets verbose mode (default false) * `-w`,`--oldPassword <arg>` The old password from which to derive the key during migration + * `-r`,`--useRawKey` If provided, the secure console will prompt for the raw key value in hexadecimal form + * `-m`,`--migrate` If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with a new key * `-x`,`--encryptFlowXmlOnly` If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/or login-identity-providers.xml files will not be modified + * `-s`,`--propsKey <arg>` The password or key to use to encrypt the sensitive processor properties in flow.xml.gz + * `-A`,`--newFlowAlgorithm <arg>` The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz + * `-P`,`--newFlowProvider <arg>` The security provider to use to encrypt the sensitive processor properties in flow.xml.gz As an example of how the tool works, assume that you have installed the tool on a machine supporting 256-bit encryption and with the following existing values in the 'nifi.properties' file: @@ -1534,11 +1536,13 @@ Sensitive configuration values are encrypted by the tool by default, however you If the 'nifi.properties' file already has valid protected values, those property values are not modified by the tool. -When applied to 'login-identity-providers.xml', the property elements are updated with an `encryption` attribute: +When applied to 'login-identity-providers.xml' and 'authorizers.xml', the property elements are updated with an `encryption` attribute: + +Example of protected login-identity-providers.xml: ---- -<!-- LDAP Provider --> -<provider> + <!-- LDAP Provider --> + <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">START_TLS</property> @@ -1547,10 +1551,27 @@ When applied to 'login-identity-providers.xml', the property elements are update <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password" encryption="aes/gcm/128">Uah59TWX+Ru5GY5p||B44RT/LJtC08QWA5ehQf01JxIpf0qSJUzug25UwkF5a50g</property> <property name="TLS - Keystore Type"></property> - ... + ... </provider> ---- +Example of protected authorizers.xml: + +--- + <!-- LDAP User Group Provider --> + <userGroupProvider> + <identifier>ldap-user-group-provider</identifier> + <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> + <property name="Authentication Strategy">START_TLS</property> + <property name="Manager DN">someuser</property> + <property name="Manager Password" encryption="aes/gcm/128">q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA</property> + <property name="TLS - Keystore"></property> + <property name="TLS - Keystore Password" encryption="aes/gcm/128">Uah59TWX+Ru5GY5p||B44RT/LJtC08QWA5ehQf01JxIpf0qSJUzug25UwkF5a50g</property> + <property name="TLS - Keystore Type"></property> + ... + </userGroupProvider> +--- + [encrypt_config_property_migration] === Sensitive Property Key Migration http://git-wip-us.apache.org/repos/asf/nifi/blob/482f3719/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/pom.xml ---------------------------------------------------------------------- diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/pom.xml b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/pom.xml index 32ca5cf..672d9fc 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/pom.xml +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/pom.xml @@ -91,6 +91,10 @@ <groupId>org.apache.nifi</groupId> <artifactId>nifi-security-utils</artifactId> </dependency> + <dependency> + <groupId>org.apache.nifi</groupId> + <artifactId>nifi-properties-loader</artifactId> + </dependency> </dependencies> </project> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/nifi/blob/482f3719/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/main/java/org/apache/nifi/authorization/AuthorizerFactoryBean.java ---------------------------------------------------------------------- diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/main/java/org/apache/nifi/authorization/AuthorizerFactoryBean.java b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/main/java/org/apache/nifi/authorization/AuthorizerFactoryBean.java index 746c0ed..7a5617c 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/main/java/org/apache/nifi/authorization/AuthorizerFactoryBean.java +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/main/java/org/apache/nifi/authorization/AuthorizerFactoryBean.java @@ -16,25 +16,6 @@ */ package org.apache.nifi.authorization; -import java.io.File; -import java.lang.reflect.Constructor; -import java.lang.reflect.Field; -import java.lang.reflect.InvocationTargetException; -import java.lang.reflect.Method; -import java.net.URL; -import java.net.URLClassLoader; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import javax.xml.XMLConstants; -import javax.xml.bind.JAXBContext; -import javax.xml.bind.JAXBElement; -import javax.xml.bind.JAXBException; -import javax.xml.bind.Unmarshaller; -import javax.xml.stream.XMLStreamReader; -import javax.xml.transform.stream.StreamSource; -import javax.xml.validation.Schema; -import javax.xml.validation.SchemaFactory; import org.apache.commons.lang3.StringUtils; import org.apache.nifi.authorization.annotation.AuthorizerContext; import org.apache.nifi.authorization.exception.AuthorizationAccessException; @@ -44,6 +25,11 @@ import org.apache.nifi.authorization.generated.Authorizers; import org.apache.nifi.authorization.generated.Property; import org.apache.nifi.bundle.Bundle; import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.properties.AESSensitivePropertyProviderFactory; +import org.apache.nifi.properties.NiFiPropertiesLoader; +import org.apache.nifi.properties.SensitivePropertyProtectionException; +import org.apache.nifi.properties.SensitivePropertyProvider; +import org.apache.nifi.properties.SensitivePropertyProviderFactory; import org.apache.nifi.security.xml.XmlUtils; import org.apache.nifi.util.NiFiProperties; import org.apache.nifi.util.file.classloader.ClassLoaderUtils; @@ -53,6 +39,27 @@ import org.springframework.beans.factory.DisposableBean; import org.springframework.beans.factory.FactoryBean; import org.xml.sax.SAXException; +import javax.xml.XMLConstants; +import javax.xml.bind.JAXBContext; +import javax.xml.bind.JAXBElement; +import javax.xml.bind.JAXBException; +import javax.xml.bind.Unmarshaller; +import javax.xml.stream.XMLStreamReader; +import javax.xml.transform.stream.StreamSource; +import javax.xml.validation.Schema; +import javax.xml.validation.SchemaFactory; +import java.io.File; +import java.io.IOException; +import java.lang.reflect.Constructor; +import java.lang.reflect.Field; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + /** * Factory bean for loading the configured authorizer. */ @@ -63,6 +70,9 @@ public class AuthorizerFactoryBean implements FactoryBean, DisposableBean, UserG private static final String JAXB_GENERATED_PATH = "org.apache.nifi.authorization.generated"; private static final JAXBContext JAXB_CONTEXT = initializeJaxbContext(); + private static SensitivePropertyProviderFactory SENSITIVE_PROPERTY_PROVIDER_FACTORY; + private static SensitivePropertyProvider SENSITIVE_PROPERTY_PROVIDER; + /** * Load the JAXBContext. */ @@ -335,8 +345,14 @@ public class AuthorizerFactoryBean implements FactoryBean, DisposableBean, UserG final Map<String, String> authorizerProperties = new HashMap<>(); for (final Property property : properties) { - authorizerProperties.put(property.getName(), property.getValue()); + if (!StringUtils.isBlank(property.getEncryption())) { + String decryptedValue = decryptValue(property.getValue(), property.getEncryption()); + authorizerProperties.put(property.getName(), decryptedValue); + } else { + authorizerProperties.put(property.getName(), property.getValue()); + } } + return new StandardAuthorizerConfigurationContext(identifier, authorizerProperties); } @@ -428,6 +444,28 @@ public class AuthorizerFactoryBean implements FactoryBean, DisposableBean, UserG }; } + private String decryptValue(String cipherText, String encryptionScheme) throws SensitivePropertyProtectionException { + initializeSensitivePropertyProvider(encryptionScheme); + return SENSITIVE_PROPERTY_PROVIDER.unprotect(cipherText); + } + + private static void initializeSensitivePropertyProvider(String encryptionScheme) throws SensitivePropertyProtectionException { + if (SENSITIVE_PROPERTY_PROVIDER == null || !SENSITIVE_PROPERTY_PROVIDER.getIdentifierKey().equalsIgnoreCase(encryptionScheme)) { + try { + String keyHex = getMasterKey(); + SENSITIVE_PROPERTY_PROVIDER_FACTORY = new AESSensitivePropertyProviderFactory(keyHex); + SENSITIVE_PROPERTY_PROVIDER = SENSITIVE_PROPERTY_PROVIDER_FACTORY.getProvider(); + } catch (IOException e) { + logger.error("Error extracting master key from bootstrap.conf for login identity provider decryption", e); + throw new SensitivePropertyProtectionException("Could not read master key from bootstrap.conf"); + } + } + } + + private static String getMasterKey() throws IOException { + return NiFiPropertiesLoader.extractKeyFromBootstrapFile(); + } + @Override public Class getObjectType() { return Authorizer.class; http://git-wip-us.apache.org/repos/asf/nifi/blob/482f3719/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/main/xsd/authorizers.xsd ---------------------------------------------------------------------- diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/main/xsd/authorizers.xsd b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/main/xsd/authorizers.xsd index 41963fd..f20bf95 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/main/xsd/authorizers.xsd +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/main/xsd/authorizers.xsd @@ -47,6 +47,7 @@ <xs:simpleContent> <xs:extension base="xs:string"> <xs:attribute name="name" type="NonEmptyStringType"></xs:attribute> + <xs:attribute name="encryption" type="xs:string"/> </xs:extension> </xs:simpleContent> </xs:complexType> http://git-wip-us.apache.org/repos/asf/nifi/blob/482f3719/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/test/groovy/org/apache/nifi/authorization/AuthorizerFactoryBeanTest.groovy ---------------------------------------------------------------------- diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/test/groovy/org/apache/nifi/authorization/AuthorizerFactoryBeanTest.groovy b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/test/groovy/org/apache/nifi/authorization/AuthorizerFactoryBeanTest.groovy new file mode 100644 index 0000000..dccc46c --- /dev/null +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-authorizer/src/test/groovy/org/apache/nifi/authorization/AuthorizerFactoryBeanTest.groovy @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.authorization + +import org.apache.nifi.authorization.generated.Property +import org.apache.nifi.properties.AESSensitivePropertyProvider +import org.bouncycastle.jce.provider.BouncyCastleProvider +import org.junit.After +import org.junit.AfterClass +import org.junit.Before +import org.junit.BeforeClass +import org.junit.Test +import org.junit.runner.RunWith +import org.junit.runners.JUnit4 +import org.slf4j.Logger +import org.slf4j.LoggerFactory + +import javax.crypto.Cipher +import java.security.Security + +@RunWith(JUnit4.class) +class AuthorizerFactoryBeanTest extends GroovyTestCase { + private static final Logger logger = LoggerFactory.getLogger(AuthorizerFactoryBeanTest.class) + + // These blocks configure the constant values depending on JCE policies of the machine running the tests + private static final String KEY_HEX_128 = "0123456789ABCDEFFEDCBA9876543210" + private static final String KEY_HEX_256 = KEY_HEX_128 * 2 + public static final String KEY_HEX = isUnlimitedStrengthCryptoAvailable() ? KEY_HEX_256 : KEY_HEX_128 + + private static final String CIPHER_TEXT_128 = "6pqdM1urBEPHtj+L||ds0Z7RpqOA2321c/+7iPMfxDrqmH5Qx6UwQG0eIYB//3Ng" + private static final String CIPHER_TEXT_256 = "TepMCD7v3LAMF0KX||ydSRWPRl1/JXgTsZtfzCnDXu7a0lTLysjPL2I06EPUCHzw" + public static final String CIPHER_TEXT = isUnlimitedStrengthCryptoAvailable() ? CIPHER_TEXT_256 : CIPHER_TEXT_128 + + private static final String ENCRYPTION_SCHEME_128 = "aes/gcm/128" + private static final String ENCRYPTION_SCHEME_256 = "aes/gcm/256" + public static + final String ENCRYPTION_SCHEME = isUnlimitedStrengthCryptoAvailable() ? ENCRYPTION_SCHEME_256 : ENCRYPTION_SCHEME_128 + + private static final String PASSWORD = "thisIsABadPassword" + + @BeforeClass + public static void setUpOnce() throws Exception { + Security.addProvider(new BouncyCastleProvider()) + + logger.metaClass.methodMissing = { String name, args -> + logger.info("[${name?.toUpperCase()}] ${(args as List).join(" ")}") + } + } + + @AfterClass + public static void tearDownOnce() throws Exception { + } + + @Before + public void setUp() throws Exception { + AuthorizerFactoryBean.SENSITIVE_PROPERTY_PROVIDER = new AESSensitivePropertyProvider(KEY_HEX) + } + + @After + public void tearDown() throws Exception { + AuthorizerFactoryBean.SENSITIVE_PROPERTY_PROVIDER = null + AuthorizerFactoryBean.SENSITIVE_PROPERTY_PROVIDER_FACTORY = null + } + + private static boolean isUnlimitedStrengthCryptoAvailable() { + Cipher.getMaxAllowedKeyLength("AES") > 128 + } + + private static int getKeyLength(String keyHex = KEY_HEX) { + keyHex?.size() * 4 + } + + @Test + void testShouldDecryptValue() { + // Arrange + logger.info("Encryption scheme: ${ENCRYPTION_SCHEME}") + logger.info("Cipher text: ${CIPHER_TEXT}") + + // Act + String decrypted = new AuthorizerFactoryBean().decryptValue(CIPHER_TEXT, ENCRYPTION_SCHEME) + logger.info("Decrypted ${CIPHER_TEXT} -> ${decrypted}") + + // Assert + assert decrypted == PASSWORD + } + + @Test + void testShouldLoadEncryptedAuthorizersConfiguration() { + // Arrange + def identifier = "ldap-user-group-provider" + def managerPasswordName = "Manager Password" + Property managerPasswordProperty = new Property(name: managerPasswordName, value: CIPHER_TEXT, encryption: ENCRYPTION_SCHEME) + List<Property> properties = [managerPasswordProperty] + + logger.info("Manager Password property: ${managerPasswordProperty.dump()}") + def bean = new AuthorizerFactoryBean() + + // Act + def context = bean.loadAuthorizerConfiguration(identifier, properties) + logger.info("Loaded context: ${context.dump()}") + + // Assert + String decryptedPropertyValue = context.getProperty(managerPasswordName) + assert decryptedPropertyValue == PASSWORD + } +} http://git-wip-us.apache.org/repos/asf/nifi/blob/482f3719/nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy ---------------------------------------------------------------------- diff --git a/nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy b/nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy index 5956f53..0a1112f 100644 --- a/nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy +++ b/nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy @@ -57,6 +57,8 @@ class ConfigEncryptionTool { public String outputNiFiPropertiesPath public String loginIdentityProvidersPath public String outputLoginIdentityProvidersPath + public String authorizersPath + public String outputAuthorizersPath public String flowXmlPath public String outputFlowXmlPath @@ -73,6 +75,7 @@ class ConfigEncryptionTool { private NiFiProperties niFiProperties private String loginIdentityProviders + private String authorizers private String flowXml private boolean usingPassword = true @@ -81,6 +84,7 @@ class ConfigEncryptionTool { private boolean isVerbose = false private boolean handlingNiFiProperties = false private boolean handlingLoginIdentityProviders = false + private boolean handlingAuthorizers = false private boolean handlingFlowXml = false private boolean ignorePropertiesFiles = false @@ -88,9 +92,11 @@ class ConfigEncryptionTool { private static final String VERBOSE_ARG = "verbose" private static final String BOOTSTRAP_CONF_ARG = "bootstrapConf" private static final String NIFI_PROPERTIES_ARG = "niFiProperties" - private static final String LOGIN_IDENTITY_PROVIDERS_ARG = "loginIdentityProviders" private static final String OUTPUT_NIFI_PROPERTIES_ARG = "outputNiFiProperties" + private static final String LOGIN_IDENTITY_PROVIDERS_ARG = "loginIdentityProviders" private static final String OUTPUT_LOGIN_IDENTITY_PROVIDERS_ARG = "outputLoginIdentityProviders" + private static final String AUTHORIZERS_ARG = "authorizers" + private static final String OUTPUT_AUTHORIZERS_ARG = "outputAuthorizers" private static final String FLOW_XML_ARG = "flowXml" private static final String OUTPUT_FLOW_XML_ARG = "outputFlowXml" private static final String KEY_ARG = "key" @@ -133,9 +139,38 @@ class ConfigEncryptionTool { "plain value with the protected value in the same file (or write to a new file if " + "specified). It can also be used to migrate already-encrypted values in those " + "files or in flow.xml.gz to be encrypted with a new key." + private static final String LDAP_PROVIDER_CLASS = "org.apache.nifi.ldap.LdapProvider" - private static - final String LDAP_PROVIDER_REGEX = /<provider>[\s\S]*?<class>\s*org\.apache\.nifi\.ldap\.LdapProvider[\s\S]*?<\/provider>/ + private static final String LDAP_PROVIDER_REGEX = /(?s)<provider>(?:(?!<provider>).)*?<class>\s*org\.apache\.nifi\.ldap\.LdapProvider.*?<\/provider>/ + /* Explanation of LDAP_PROVIDER_REGEX: + * (?s) -> single-line mode (i.e., `.` in regex matches newlines) + * <provider> -> find occurrence of `<provider>` literally (case-sensitive) + * (?: ... ) -> group but do not capture submatch + * (?! ... ) -> negative lookahead + * (?:(?!<provider>).)*? -> find everything until a new `<provider>` starts. This is for not selecting multiple providers in one match + * <class> -> find occurrence of `<class>` literally (case-sensitive) + * \s* -> find any whitespace + * org\.apache\.nifi\.ldap\.LdapProvider + * -> find occurrence of `org.apache.nifi.ldap.LdapProvider` literally (case-sensitive) + * .*?</provider> -> find everything as needed up until and including occurrence of `</provider>` + */ + + private static final String LDAP_USER_GROUP_PROVIDER_CLASS = "org.apache.nifi.ldap.tenants.LdapUserGroupProvider" + private static final String LDAP_USER_GROUP_PROVIDER_REGEX = + /(?s)<userGroupProvider>(?:(?!<userGroupProvider>).)*?<class>\s*org\.apache\.nifi\.ldap\.tenants\.LdapUserGroupProvider.*?<\/userGroupProvider>/ + /* Explanation of LDAP_USER_GROUP_PROVIDER_REGEX: + * (?s) -> single-line mode (i.e., `.` in regex matches newlines) + * <userGroupProvider> -> find occurrence of `<userGroupProvider>` literally (case-sensitive) + * (?: ... ) -> group but do not capture submatch + * (?! ... ) -> negative lookahead + * (?:(?!<userGroupProvider>).)*? -> find everything until a new `<userGroupProvider>` starts. This is for not selecting multiple userGroupProviders in one match + * <class> -> find occurrence of `<class>` literally (case-sensitive) + * \s* -> find any whitespace + * org\.apache\.nifi\.ldap\.tenants\.LdapUserGroupProvider + * -> find occurrence of `org.apache.nifi.ldap.tenants.LdapUserGroupProvider` literally (case-sensitive) + * .*?</userGroupProvider> -> find everything as needed up until and including occurrence of '</userGroupProvider>' + */ + private static final String XML_DECLARATION_REGEX = /<\?xml version="1.0" encoding="UTF-8"\?>/ private static final String WRAPPED_FLOW_XML_CIPHER_TEXT_REGEX = /enc\{[a-fA-F0-9]+?\}/ @@ -154,21 +189,23 @@ class ConfigEncryptionTool { private final String header - public ConfigEncryptionTool() { + ConfigEncryptionTool() { this(DEFAULT_DESCRIPTION) } - public ConfigEncryptionTool(String description) { + ConfigEncryptionTool(String description) { this.header = buildHeader(description) this.options = new Options() options.addOption("h", HELP_ARG, false, "Prints this usage message") options.addOption("v", VERBOSE_ARG, false, "Sets verbose mode (default false)") options.addOption("n", NIFI_PROPERTIES_ARG, true, "The nifi.properties file containing unprotected config values (will be overwritten)") options.addOption("l", LOGIN_IDENTITY_PROVIDERS_ARG, true, "The login-identity-providers.xml file containing unprotected config values (will be overwritten)") + options.addOption("a", AUTHORIZERS_ARG, true, "The authorizers.xml file containing unprotected config values (will be overwritten)") options.addOption("f", FLOW_XML_ARG, true, "The flow.xml.gz file currently protected with old password (will be overwritten)") options.addOption("b", BOOTSTRAP_CONF_ARG, true, "The bootstrap.conf file to persist master key") options.addOption("o", OUTPUT_NIFI_PROPERTIES_ARG, true, "The destination nifi.properties file containing protected config values (will not modify input nifi.properties)") options.addOption("i", OUTPUT_LOGIN_IDENTITY_PROVIDERS_ARG, true, "The destination login-identity-providers.xml file containing protected config values (will not modify input login-identity-providers.xml)") + options.addOption("u", OUTPUT_AUTHORIZERS_ARG, true, "The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)") options.addOption("g", OUTPUT_FLOW_XML_ARG, true, "The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)") options.addOption("k", KEY_ARG, true, "The raw hexadecimal key to use to encrypt the sensitive properties") options.addOption("e", KEY_MIGRATION_ARG, true, "The old raw hexadecimal key to use during key migration") @@ -187,19 +224,20 @@ class ConfigEncryptionTool { * * @param errorMessage the optional error message */ - public void printUsage(String errorMessage) { + void printUsage(String errorMessage) { if (errorMessage) { System.out.println(errorMessage) System.out.println() } HelpFormatter helpFormatter = new HelpFormatter() helpFormatter.setWidth(160) + helpFormatter.setOptionComparator(null) // preserve manual ordering of options when printing instead of alphabetical helpFormatter.printHelp(ConfigEncryptionTool.class.getCanonicalName(), header, options, FOOTER, true) } protected void printUsageAndThrow(String errorMessage, ExitCode exitCode) throws CommandLineParseException { - printUsage(errorMessage); - throw new CommandLineParseException(errorMessage, exitCode); + printUsage(errorMessage) + throw new CommandLineParseException(errorMessage, exitCode) } // TODO: Refactor component steps into methods @@ -220,6 +258,7 @@ class ConfigEncryptionTool { if (commandLine.hasOption(DO_NOT_ENCRYPT_NIFI_PROPERTIES_ARG)) { handlingNiFiProperties = false handlingLoginIdentityProviders = false + handlingAuthorizers = false ignorePropertiesFiles = true } else { if (commandLine.hasOption(LOGIN_IDENTITY_PROVIDERS_ARG)) { @@ -235,6 +274,19 @@ class ConfigEncryptionTool { logger.warn("The source login-identity-providers.xml and destination login-identity-providers.xml are identical [${outputLoginIdentityProvidersPath}] so the original will be overwritten") } } + if (commandLine.hasOption(AUTHORIZERS_ARG)) { + if (isVerbose) { + logger.info("Handling encryption of authorizers.xml") + } + authorizersPath = commandLine.getOptionValue(AUTHORIZERS_ARG) + outputAuthorizersPath = commandLine.getOptionValue(OUTPUT_AUTHORIZERS_ARG, authorizersPath) + handlingAuthorizers = true + + if (authorizersPath == outputAuthorizersPath) { + // TODO: Add confirmation pause and provide -y flag to offer no-interaction mode? + logger.warn("The source authorizers.xml and destination authorizers.xml are identical [${outputAuthorizersPath}] so the original will be overwritten") + } + } } // This needs to occur even if the nifi.properties won't be encrypted @@ -275,17 +327,28 @@ class ConfigEncryptionTool { } if (isVerbose) { - logger.info(" bootstrap.conf: \t${bootstrapConfPath}") - logger.info("(src) nifi.properties: \t${niFiPropertiesPath}") - logger.info("(dest) nifi.properties: \t${outputNiFiPropertiesPath}") - logger.info("(src) login-identity-providers.xml: \t${loginIdentityProvidersPath}") - logger.info("(dest) login-identity-providers.xml: \t${outputLoginIdentityProvidersPath}") - logger.info("(src) flow.xml.gz: \t\t\t\t\t${flowXmlPath}") - logger.info("(dest) flow.xml.gz: \t\t\t\t\t${outputFlowXmlPath}") + logger.info(" bootstrap.conf: ${bootstrapConfPath}") + logger.info("(src) nifi.properties: ${niFiPropertiesPath}") + logger.info("(dest) nifi.properties: ${outputNiFiPropertiesPath}") + logger.info("(src) login-identity-providers.xml: ${loginIdentityProvidersPath}") + logger.info("(dest) login-identity-providers.xml: ${outputLoginIdentityProvidersPath}") + logger.info("(src) authorizers.xml: ${authorizersPath}") + logger.info("(dest) authorizers.xml: ${outputAuthorizersPath}") + logger.info("(src) flow.xml.gz: ${flowXmlPath}") + logger.info("(dest) flow.xml.gz: ${outputFlowXmlPath}") } - if (!commandLine.hasOption(NIFI_PROPERTIES_ARG) && !commandLine.hasOption(LOGIN_IDENTITY_PROVIDERS_ARG) && !commandLine.hasOption(DO_NOT_ENCRYPT_NIFI_PROPERTIES_ARG)) { - printUsageAndThrow("One or both of '-n'/'--${NIFI_PROPERTIES_ARG}' or '-l'/'--${LOGIN_IDENTITY_PROVIDERS_ARG}' must be provided unless '-x'/--'${DO_NOT_ENCRYPT_NIFI_PROPERTIES_ARG}' is specified", ExitCode.INVALID_ARGS) + if (!commandLine.hasOption(NIFI_PROPERTIES_ARG) + && !commandLine.hasOption(LOGIN_IDENTITY_PROVIDERS_ARG) + && !commandLine.hasOption(AUTHORIZERS_ARG) + && !commandLine.hasOption(DO_NOT_ENCRYPT_NIFI_PROPERTIES_ARG) + ) { + printUsageAndThrow("One or more of [" + + "'-n'/'--${NIFI_PROPERTIES_ARG}', " + + "'-l'/'--${LOGIN_IDENTITY_PROVIDERS_ARG}', " + + "'-a'/'--${AUTHORIZERS_ARG}'" + + "] must be provided unless " + + "'-x'/--'${DO_NOT_ENCRYPT_NIFI_PROPERTIES_ARG}' is specified", ExitCode.INVALID_ARGS) } if (commandLine.hasOption(MIGRATION_ARG)) { @@ -416,7 +479,7 @@ class ConfigEncryptionTool { * * @return 128 , [192, 256] */ - public static List<Integer> getValidKeyLengths() { + static List<Integer> getValidKeyLengths() { Cipher.getMaxAllowedKeyLength("AES") > 128 ? [128, 192, 256] : [128] } @@ -457,19 +520,47 @@ class ConfigEncryptionTool { File loginIdentityProvidersFile if (loginIdentityProvidersPath && (loginIdentityProvidersFile = new File(loginIdentityProvidersPath)).exists()) { try { - String xmlContent = loginIdentityProvidersFile.text List<String> lines = loginIdentityProvidersFile.readLines() - logger.info("Loaded LoginIdentityProviders content (${lines.size()} lines)") + String xmlContent = lines.join("\n") + logger.info("Loaded login identity providers content (${lines.size()} lines)") String decryptedXmlContent = decryptLoginIdentityProviders(xmlContent, existingKeyHex) return decryptedXmlContent } catch (RuntimeException e) { if (isVerbose) { logger.error("Encountered an error", e) } - throw new IOException("Cannot load LoginIdentityProviders from [${loginIdentityProvidersPath}]", e) + throw new IOException("Cannot load login identity providers from [${loginIdentityProvidersPath}]", e) } } else { - printUsageAndThrow("Cannot load LoginIdentityProviders from [${loginIdentityProvidersPath}]", ExitCode.ERROR_READING_NIFI_PROPERTIES) + printUsageAndThrow("Cannot load login identity providers from [${loginIdentityProvidersPath}]", ExitCode.ERROR_READING_NIFI_PROPERTIES) + } + } + + /** + * Loads the authorizers configuration from the provided file path. + * + * @param existingKeyHex the key used to encrypt the configs (defaults to the current key) + * + * @return the file content + * @throw IOException if the authorizers.xml file cannot be read + */ + private String loadAuthorizers(String existingKeyHex = keyHex) throws IOException { + File authorizersFile + if (authorizersPath && (authorizersFile = new File(authorizersPath)).exists()) { + try { + List<String> lines = authorizersFile.readLines() + String xmlContent = lines.join("\n") + logger.info("Loaded authorizers content (${lines.size()} lines)") + String decryptedXmlContent = decryptAuthorizers(xmlContent, existingKeyHex) + return decryptedXmlContent + } catch (RuntimeException e) { + if (isVerbose) { + logger.error("Encountered an error", e) + } + throw new IOException("Cannot load authorizers from [${authorizersPath}]", e) + } + } else { + printUsageAndThrow("Cannot load authorizers from [${authorizersPath}]", ExitCode.ERROR_READING_NIFI_PROPERTIES) } } @@ -480,8 +571,7 @@ class ConfigEncryptionTool { * @throw IOException if the flow.xml.gz file cannot be read */ private String loadFlowXml() throws IOException { - File flowXmlFile - if (flowXmlPath && (flowXmlFile = new File(flowXmlPath)).exists()) { + if (flowXmlPath && (new File(flowXmlPath)).exists()) { try { new FileInputStream(flowXmlPath).withCloseable { new GZIPInputStream(it).withCloseable { @@ -730,6 +820,43 @@ class ConfigEncryptionTool { } } + String decryptAuthorizers(String encryptedXml, String existingKeyHex = keyHex) { + AESSensitivePropertyProvider sensitivePropertyProvider = new AESSensitivePropertyProvider(existingKeyHex) + + try { + def doc = new XmlSlurper().parseText(encryptedXml) + // Find the provider element by class even if it has been renamed + def passwords = doc.userGroupProvider.find { it.'class' as String == LDAP_USER_GROUP_PROVIDER_CLASS }.property.findAll { + it.@name =~ "Password" && it.@encryption =~ "aes/gcm/\\d{3}" + } + + if (passwords.isEmpty()) { + if (isVerbose) { + logger.info("No encrypted password property elements found in authorizers.xml") + } + return encryptedXml + } + + passwords.each { password -> + // TODO: Capture the raw password, and only display it in the log if the decrypted value is different (to avoid possibly printing an incorrectly provided plaintext password) + if (isVerbose) { + logger.info("Attempting to decrypt ${password.text()}") + } + String decryptedValue = sensitivePropertyProvider.unprotect(password.text().trim()) + password.replaceNode { + property(name: password.@name, encryption: "none", decryptedValue) + } + } + + // Does not preserve whitespace formatting or comments + String updatedXml = XmlUtil.serialize(doc) + logger.info("Updated XML content: ${updatedXml}") + updatedXml + } catch (Exception e) { + printUsageAndThrow("Cannot decrypt authorizers XML content", ExitCode.SERVICE_ERROR) + } + } + String encryptLoginIdentityProviders(String plainXml, String newKeyHex = keyHex) { AESSensitivePropertyProvider sensitivePropertyProvider = new AESSensitivePropertyProvider(newKeyHex) @@ -772,6 +899,48 @@ class ConfigEncryptionTool { } } + String encryptAuthorizers(String plainXml, String newKeyHex = keyHex) { + AESSensitivePropertyProvider sensitivePropertyProvider = new AESSensitivePropertyProvider(newKeyHex) + + // TODO: Switch to XmlParser & XmlNodePrinter to maintain "empty" element structure + try { + def doc = new XmlSlurper().parseText(plainXml) + // Find the provider element by class even if it has been renamed + def passwords = doc.userGroupProvider.find { it.'class' as String == LDAP_USER_GROUP_PROVIDER_CLASS } + .property.findAll { + // Only operate on un-encrypted passwords + it.@name =~ "Password" && (it.@encryption == "none" || it.@encryption == "") && it.text() + } + + if (passwords.isEmpty()) { + if (isVerbose) { + logger.info("No unencrypted password property elements found in authorizers.xml") + } + return plainXml + } + + passwords.each { password -> + if (isVerbose) { + logger.info("Attempting to encrypt ${password.name()}") + } + String encryptedValue = sensitivePropertyProvider.protect(password.text().trim()) + password.replaceNode { + property(name: password.@name, encryption: sensitivePropertyProvider.identifierKey, encryptedValue) + } + } + + // Does not preserve whitespace formatting or comments + String updatedXml = XmlUtil.serialize(doc) + logger.info("Updated XML content: ${updatedXml}") + updatedXml + } catch (Exception e) { + if (isVerbose) { + logger.error("Encountered exception", e) + } + printUsageAndThrow("Cannot encrypt authorizers XML content", ExitCode.SERVICE_ERROR) + } + } + /** * Accepts a {@link NiFiProperties} instance, iterates over all non-empty sensitive properties which are not already marked as protected, encrypts them using the master key, and updates the property with the protected value. Additionally, adds a new sibling property {@code x.y.z.protected=aes/gcm/{128,256}} for each indicating the encryption scheme used. * @@ -907,6 +1076,8 @@ class ConfigEncryptionTool { if (loginIdentityProvidersFile.exists() && loginIdentityProvidersFile.canRead()) { // Instead of just writing the XML content to a file, this method attempts to maintain the structure of the original file and preserves comments updatedXmlContent = serializeLoginIdentityProvidersAndPreserveFormat(loginIdentityProviders, loginIdentityProvidersFile).join("\n") + } else { + updatedXmlContent = loginIdentityProviders } // Write the updated values back to the file @@ -922,6 +1093,41 @@ class ConfigEncryptionTool { } /** + * Writes the contents of the authorizers configuration file with encrypted values to the output {@code authorizers.xml} file. + * + * @throw IOException if there is a problem reading or writing the authorizers.xml file + */ + private void writeAuthorizers() throws IOException { + if (!outputAuthorizersPath) { + throw new IllegalArgumentException("Cannot write encrypted properties to empty authorizers.xml path") + } + + File outputAuthorizersFile = new File(outputAuthorizersPath) + + if (isSafeToWrite(outputAuthorizersFile)) { + try { + String updatedXmlContent + File authorizersFile = new File(authorizersPath) + if (authorizersFile.exists() && authorizersFile.canRead()) { + // Instead of just writing the XML content to a file, this method attempts to maintain the structure of the original file and preserves comments + updatedXmlContent = serializeAuthorizersAndPreserveFormat(authorizers, authorizersFile).join("\n") + } else { + updatedXmlContent = authorizers + } + + // Write the updated values back to the file + outputAuthorizersFile.text = updatedXmlContent + } catch (IOException e) { + def msg = "Encountered an exception updating the authorizers.xml file with the encrypted values" + logger.error(msg, e) + throw e + } + } else { + throw new IOException("The authorizers.xml file at ${outputAuthorizersPath} must be writable by the user running this tool") + } + } + + /** * Writes the contents of the {@link NiFiProperties} instance with encrypted values to the output {@code nifi.properties} file. * * @throw IOException if there is a problem reading or writing the nifi.properties file @@ -1016,7 +1222,30 @@ class ConfigEncryptionTool { throw new SAXException("No ldap-provider element found") } } catch (SAXException e) { - logger.error("No provider element with class org.apache.nifi.ldap.LdapProvider found in XML content; the file could be empty or the element may be missing or commented out") + logger.error("No provider element with class {} found in XML content; " + + "the file could be empty or the element may be missing or commented out", LDAP_PROVIDER_CLASS) + return fileContents.split("\n") + } + } + + static List<String> serializeAuthorizersAndPreserveFormat(String xmlContent, File originalAuthorizersFile) { + // Find the provider element of the new XML in the file contents + String fileContents = originalAuthorizersFile.text + try { + def parsedXml = new XmlSlurper().parseText(xmlContent) + def provider = parsedXml.userGroupProvider.find { it.'class' as String == LDAP_USER_GROUP_PROVIDER_CLASS } + if (provider) { + def serializedProvider = new XmlUtil().serialize(provider) + // Remove XML declaration from top + serializedProvider = serializedProvider.replaceFirst(XML_DECLARATION_REGEX, "") + fileContents = fileContents.replaceFirst(LDAP_USER_GROUP_PROVIDER_REGEX, serializedProvider) + return fileContents.split("\n") + } else { + throw new SAXException("No ldap-user-group-provider element found") + } + } catch (SAXException e) { + logger.error("No provider element with class {} found in XML content; " + + "the file could be empty or the element may be missing or commented out", LDAP_USER_GROUP_PROVIDER_CLASS) return fileContents.split("\n") } } @@ -1087,7 +1316,7 @@ class ConfigEncryptionTool { * * @param args the command-line arguments */ - public static void main(String[] args) { + static void main(String[] args) { Security.addProvider(new BouncyCastleProvider()) ConfigEncryptionTool tool = new ConfigEncryptionTool() @@ -1157,6 +1386,15 @@ class ConfigEncryptionTool { tool.loginIdentityProviders = tool.encryptLoginIdentityProviders(tool.loginIdentityProviders) } + if (tool.handlingAuthorizers) { + try { + tool.authorizers = tool.loadAuthorizers(existingKeyHex) + } catch (Exception e) { + tool.printUsageAndThrow("Cannot migrate key if no previous encryption occurred", ExitCode.ERROR_INCORRECT_NUMBER_OF_PASSWORDS) + } + tool.authorizers = tool.encryptAuthorizers(tool.authorizers) + } + if (tool.handlingFlowXml) { try { tool.flowXml = tool.loadFlowXml() @@ -1238,6 +1476,9 @@ class ConfigEncryptionTool { if (tool.handlingLoginIdentityProviders) { tool.writeLoginIdentityProviders() } + if (tool.handlingAuthorizers) { + tool.writeAuthorizers() + } } } catch (Exception e) { if (tool.isVerbose) {
