exceptionfactory commented on a change in pull request #4863:
URL: https://github.com/apache/nifi/pull/4863#discussion_r590388525
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/utils/AzureStorageUtils.java
##########
@@ -143,6 +144,54 @@
.required(false)
.build();
+ public static final PropertyDescriptor ACCOUNT_NAME_SECRET = new
PropertyDescriptor.Builder()
+ .name("storage-account-name-secret")
+ .displayName("Storage Account Name Secret Name")
+ .description("Storage Account Name Secret Name")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
Review comment:
With the introduction of sensitive Parameters, it is not good practice
to support expression language for sensitive properties. Inserting sensitive
property values in FlowFile attributes does not provide encryption by default
when storing provenance records. Although other Azure components indicated
support for expression language in FlowFile attributes, it was not actually
implemented, so PR #4843 removed support. Is there a particular reason for
support expression language in these sensitive properties?
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/cosmos/document/AbstractCosmosDBClientService.java
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.services.azure.cosmos.document;
+
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosException;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.processors.azure.cosmos.document.AzureCosmosDBUtils;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+@Tags({"azure", "cosmos", "document", "service"})
+@CapabilityDescription(
+ "Provides a controller service that configures a connection to Cosmos
DB (Core SQL API) " +
+ " and provides access to that connection to other Cosmos
DB-related components."
+)
+public abstract class AbstractCosmosDBClientService
+ extends AbstractControllerService
+ implements AzureCosmosDBConnectionService {
+
+ protected CosmosClient cosmosClient;
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.cosmosClient != null) {
+ try {
+ cosmosClient.close();
+ } catch(CosmosException e) {
+ getLogger().error("Closing CosmosClient Failed: " +
e.getMessage(), e);
+ } finally {
+ this.cosmosClient = null;
+ }
+ }
+ }
+
+ protected void createCosmosClient(final String uri, final String
accessKey, final String selectedConsistency){
+ final ConsistencyLevel cLevel;
+
+ switch(selectedConsistency) {
+ case AzureCosmosDBUtils.CONSISTENCY_STRONG:
+ cLevel = ConsistencyLevel.STRONG;
Review comment:
It looks like there is an extra space separating the value assignment in
this line.
```suggestion
cLevel = ConsistencyLevel.STRONG;
```
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/utils/AzureStorageUtils.java
##########
@@ -143,6 +144,54 @@
.required(false)
.build();
+ public static final PropertyDescriptor ACCOUNT_NAME_SECRET = new
PropertyDescriptor.Builder()
+ .name("storage-account-name-secret")
+ .displayName("Storage Account Name Secret Name")
+ .description("Storage Account Name Secret Name")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .sensitive(true)
+ .build();
+
+ public static final PropertyDescriptor ACCOUNT_KEY_SECRET = new
PropertyDescriptor.Builder()
+ .name("storage-account-key-secret")
+ .displayName("Storage Account Key Secret Name")
+ .description("Storage Account Key Secret Name")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(false)
+ .sensitive(true)
+ .build();
+
+ public static final PropertyDescriptor ACCOUNT_SAS_TOKEN_SECRET = new
PropertyDescriptor.Builder()
+ .name("storage-sas-token")
+ .displayName("SAS Token Secret Name")
+ .description("SAS Token Secret Name")
+ .required(false)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor KEYVAULT_CONNECTION_SERVICE = new
PropertyDescriptor.Builder()
+ .name("azure-keyvault-connection-service")
+ .displayName("KeyVault Connection Service")
+ .description("If configured, the controller service used to obtain
the secrets from KeyVault")
+ .required(true)
+ .identifiesControllerService(AzureKeyVaultConnectionService.class)
+ .build();
+
+ public static final PropertyDescriptor ADLS_ENDPOINT_SUFFIX = new
PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AzureStorageUtils.ENDPOINT_SUFFIX)
+ .displayName("Endpoint Suffix")
+ .description(
+ "Storage accounts in public Azure always use a common FQDN
suffix. " +
+ "Override this endpoint suffix with a different suffix in
certain circumstances (like Azure Stack or non-public Azure regions).")
+ .required(true)
+ .defaultValue("dfs.core.windows.net")
+ .build();
Review comment:
Should this property descriptor have some kind of validator, such as the
NON_BLANK_VALIDATOR, or is blank valid?
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/keyvault/AzureKeyVaultClientService.java
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.services.azure.keyvault;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.util.StringUtils;
+
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.identity.ClientSecretCredential;
+import com.azure.identity.ClientSecretCredentialBuilder;
+import com.azure.security.keyvault.secrets.SecretClient;
+import com.azure.security.keyvault.secrets.SecretClientBuilder;
+
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"azure", "keyvault", "credential", "service", "secure"})
+@CapabilityDescription(
+ "Provides a controller service that configures a connection to Azure
Key Vault" +
+ " and provides access to that connection to Azure Key Vault
components."
+)
+public class AzureKeyVaultClientService
+ extends AbstractControllerService
+ implements AzureKeyVaultConnectionService {
+
+ private String keyVaultName;
+ private String servicePrincipalClientID;
+ private String servicePrincipalClientSecret;
+ private String tenantID;
+ private String endPointSuffix;
+ private Boolean useManagedIdentity;
+ private SecretClient keyVaultSecretClient;
+ private ComponentLog logger;
+ private LoadingCache<String, String> secretCache;
+
+ public String getKeyVaultName() {
+ return this.keyVaultName;
+ }
+
+ public String getServicePrincipalClientID() {
+ return this.servicePrincipalClientID;
+ }
+
+ public String getServicePrincipalClientSecret() {
+ return this.servicePrincipalClientSecret;
+ }
+
+ public String getTenantID() {
+ return this.tenantID;
+ }
+
+ public String getEndPointSuffix() {
+ return this.endPointSuffix;
+ }
+
+ public Boolean getUseManagedIdentity() {
Review comment:
Are the above public get methods necessary? In general, public methods
on Controller Service implementations that do not implement interface methods
should be avoided.
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/keyvault/AzureKeyVaultConnectionService.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.services.azure.keyvault;
+
+import org.apache.nifi.controller.ControllerService;
+import com.azure.security.keyvault.secrets.SecretClient;
+
Review comment:
It would be helpful to add some basic comments about the purpose and
capabilities of this interface.
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/keyvault/AzureKeyVaultConnectionService.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.services.azure.keyvault;
+
+import org.apache.nifi.controller.ControllerService;
+import com.azure.security.keyvault.secrets.SecretClient;
+
+public interface AzureKeyVaultConnectionService extends ControllerService {
+
+ SecretClient getKeyVaultSecretClient();
+
+ String getSecret(String secretName);
+
+ String getSecretFromKeyVault(String secretName);
Review comment:
Is it necessary to have a separate method with basically the same
signature as `getSecret`? The implementation purpose seems to be avoiding
checking the cache, but it seems better to simplify the interface and have the
implementation drive the usage.
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/cosmos/document/AzureCosmosSecureDBClientService.java
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.services.azure.cosmos.document;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.cosmos.document.AzureCosmosDBUtils;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+import org.apache.nifi.services.azure.keyvault.AzureKeyVaultConnectionService;
+import org.apache.nifi.util.StringUtils;
+
+@Tags({"azure", "cosmos", "document", "service", "secure"})
+@CapabilityDescription(
+ "Provides a controller service that configures a connection to Cosmos
DB (Core SQL API) " +
+ " and provides access to that connection to other Cosmos
DB-related components."
+)
+public class AzureCosmosSecureDBClientService
+ extends AbstractCosmosDBClientService
+ implements AzureCosmosDBConnectionService {
+
+ private String uriSecret;
+ private String accessKeySecret;
+ private String consistencyLevel;
+ private AzureKeyVaultConnectionService keyVaultClientService;
+
+ public static final PropertyDescriptor URI_SECRET = new
PropertyDescriptor.Builder()
+ .name("azure-cosmos-db-uri-secret")
+ .displayName("Cosmos DB URI Secret Name")
+ .description("Cosmos DB URI Secret Name")
+ .required(true)
+ .addValidator(StandardValidators.URI_VALIDATOR)
+ .sensitive(true)
+ .build();
+
+ public static final PropertyDescriptor DB_ACCESS_KEY_SECRET = new
PropertyDescriptor.Builder()
+ .name("azure-cosmos-db-key-secret")
+ .displayName("Cosmos DB Access Key Secret Name")
+ .description("Cosmos DB Access Key Secret Name")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .sensitive(true)
+ .build();
+
+ static final PropertyDescriptor KEYVAULT_CONNECTION_SERVICE = new
PropertyDescriptor.Builder()
+ .name("azure-keyvault-connection-service")
+ .displayName("KeyVault Connection Service")
+ .description("If configured, the controller service used to obtain
the secrets from KeyVault")
+ .required(true)
+ .identifiesControllerService(AzureKeyVaultConnectionService.class)
+ .build();
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) {
+ this.uriSecret = context.getProperty(URI_SECRET).getValue();
+ this.accessKeySecret =
context.getProperty(DB_ACCESS_KEY_SECRET).getValue();
+ this.consistencyLevel = context.getProperty(
+ AzureCosmosDBUtils.CONSISTENCY).getValue();
+ this.keyVaultClientService = context.getProperty(
+ KEYVAULT_CONNECTION_SERVICE
+ ).asControllerService(AzureKeyVaultConnectionService.class);
+
+ createCosmosClient(
+ getURI(),
+ getAccessKey(),
+ getConsistencyLevel()
+ );
+ }
+
+ static List<PropertyDescriptor> descriptors = new ArrayList<>();
+
+ static {
+ descriptors.add(KEYVAULT_CONNECTION_SERVICE);
+ descriptors.add(URI_SECRET);
+ descriptors.add(DB_ACCESS_KEY_SECRET);
+ descriptors.add(AzureCosmosDBUtils.CONSISTENCY);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public String getURI() {
+ if (keyVaultClientService == null) {
+ throw new IllegalArgumentException(String.format(
+ "Cannot get '%s'.",
KEYVAULT_CONNECTION_SERVICE.getDisplayName()));
+ }
+ if (StringUtils.isBlank(uriSecret)) {
Review comment:
Is it possible for `uriSecret` to be blank in light of the property
validation and custom validate implementation?
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/keyvault/AzureKeyVaultClientService.java
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.services.azure.keyvault;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.util.StringUtils;
+
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.identity.ClientSecretCredential;
+import com.azure.identity.ClientSecretCredentialBuilder;
+import com.azure.security.keyvault.secrets.SecretClient;
+import com.azure.security.keyvault.secrets.SecretClientBuilder;
+
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"azure", "keyvault", "credential", "service", "secure"})
+@CapabilityDescription(
+ "Provides a controller service that configures a connection to Azure
Key Vault" +
+ " and provides access to that connection to Azure Key Vault
components."
+)
+public class AzureKeyVaultClientService
+ extends AbstractControllerService
+ implements AzureKeyVaultConnectionService {
+
+ private String keyVaultName;
+ private String servicePrincipalClientID;
+ private String servicePrincipalClientSecret;
+ private String tenantID;
+ private String endPointSuffix;
+ private Boolean useManagedIdentity;
+ private SecretClient keyVaultSecretClient;
+ private ComponentLog logger;
+ private LoadingCache<String, String> secretCache;
+
+ public String getKeyVaultName() {
+ return this.keyVaultName;
+ }
+
+ public String getServicePrincipalClientID() {
+ return this.servicePrincipalClientID;
+ }
+
+ public String getServicePrincipalClientSecret() {
+ return this.servicePrincipalClientSecret;
+ }
+
+ public String getTenantID() {
+ return this.tenantID;
+ }
+
+ public String getEndPointSuffix() {
+ return this.endPointSuffix;
+ }
+
+ public Boolean getUseManagedIdentity() {
+ return this.useManagedIdentity;
+ }
+
+ @Override
+ public SecretClient getKeyVaultSecretClient() {
+ return this.keyVaultSecretClient;
+ }
+
+ @Override
+ public String getSecretFromKeyVault(String secretName) {
+ return this.keyVaultSecretClient.getSecret(secretName).getValue();
+ }
+
+ @Override
+ public String getSecret(String secretName) {
+ if (secretCache != null) {
+ try {
+ return secretCache.get(secretName);
+ } catch (final Exception e) {
+ logger.error("Failed to get secret '"+ secretName +"' from
cache", e);
+ }
+ }
+ return getSecretFromKeyVault(secretName);
+ }
+
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) {
+ logger = getLogger();
+ this.keyVaultName = context.getProperty(
+ AzureKeyVaultUtils.KEYVAULT_NAME).getValue();
+ this.servicePrincipalClientID = context.getProperty(
+ AzureKeyVaultUtils.SP_CLIENT_ID).getValue();
+ this.servicePrincipalClientSecret = context.getProperty(
+ AzureKeyVaultUtils.SP_CLIENT_SECRET).getValue();
+ this.tenantID = context.getProperty(
+ AzureKeyVaultUtils.TENANT_ID).getValue();
+ this.endPointSuffix = context.getProperty(
+ AzureKeyVaultUtils.ENDPOINT_SUFFIX).getValue();
+ this.useManagedIdentity = context.getProperty(
+ AzureKeyVaultUtils.USE_MANAGED_IDENTITY).asBoolean();
+
+ createKeyVaultSecretClient();
+
+ final Integer cacheSize =
context.getProperty(AzureKeyVaultUtils.CACHE_SIZE).asInteger();
+ final Long cacheTTL = context.getProperty(
+ AzureKeyVaultUtils.CACHE_TTL_AFTER_WRITE
+ ).asTimePeriod(TimeUnit.SECONDS);
+
+ if (cacheSize > 0) {
+ CacheBuilder<Object, Object> cacheBuilder =
CacheBuilder.newBuilder().maximumSize(cacheSize);
+ if (cacheTTL > 0) {
+ cacheBuilder = cacheBuilder.expireAfterWrite(cacheTTL,
TimeUnit.SECONDS);
+ }
+
+ logger.info(String.format(
+ "Secret cache enabled with cacheSize: %d and cacheTTL: %d
secs",
+ cacheSize, cacheTTL));
+ secretCache = cacheBuilder.build(
+ new CacheLoader<String, String>() {
+ @Override
+ public String load(String secretName) throws Exception
{
+ return getSecretFromKeyVault(secretName);
+ }
+ });
+ } else {
+ secretCache = null;
+ logger.info("Secret cache disabled because cache size is set to
0");
+ }
+ }
+
+ static List<PropertyDescriptor> descriptors = new ArrayList<>();
+
+ static {
+ descriptors.add(AzureKeyVaultUtils.KEYVAULT_NAME);
+ descriptors.add(AzureKeyVaultUtils.SP_CLIENT_ID);
+ descriptors.add(AzureKeyVaultUtils.SP_CLIENT_SECRET);
+ descriptors.add(AzureKeyVaultUtils.TENANT_ID);
+ descriptors.add(AzureKeyVaultUtils.ENDPOINT_SUFFIX);
+ descriptors.add(AzureKeyVaultUtils.USE_MANAGED_IDENTITY);
+ descriptors.add(AzureKeyVaultUtils.CACHE_SIZE);
+ descriptors.add(AzureKeyVaultUtils.CACHE_TTL_AFTER_WRITE);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ protected Collection<ValidationResult> customValidate(ValidationContext
validationContext) {
+ final List<ValidationResult> results = new ArrayList<>();
+
+ final String keyVaultName = validationContext.getProperty(
+ AzureKeyVaultUtils.KEYVAULT_NAME).getValue();
+ final String clientID = validationContext.getProperty(
+ AzureKeyVaultUtils.SP_CLIENT_ID).getValue();
+ final String clientSecret = validationContext.getProperty(
+ AzureKeyVaultUtils.SP_CLIENT_SECRET).getValue();
+ final String tenantID = validationContext.getProperty(
+ AzureKeyVaultUtils.TENANT_ID).getValue();
+ final Boolean useManagedIdentity = validationContext.getProperty(
+ AzureKeyVaultUtils.USE_MANAGED_IDENTITY).asBoolean();
+
+ if (StringUtils.isBlank(keyVaultName)) {
+ results.add(new ValidationResult.Builder()
+ .subject(this.getClass().getSimpleName())
+ .valid(false)
+
.explanation(AzureKeyVaultUtils.KEYVAULT_NAME.getDisplayName() +" is required")
+ .build());
+ } else if (useManagedIdentity && (StringUtils.isNotBlank(clientID)
+ || StringUtils.isNotBlank(clientSecret))) {
+ results.add(new ValidationResult.Builder()
+ .subject(this.getClass().getSimpleName())
+ .valid(false)
+ .explanation("if " +
AzureKeyVaultUtils.USE_MANAGED_IDENTITY.getDisplayName()
+ + " is used then " +
AzureKeyVaultUtils.SP_CLIENT_ID.getDisplayName()
+ + ", " +
AzureKeyVaultUtils.SP_CLIENT_SECRET.getDisplayName()
+ + ", " +
AzureKeyVaultUtils.TENANT_ID.getDisplayName()
+ + " should be blank.")
+ .build());
+ } else if (!useManagedIdentity && (StringUtils.isBlank(clientID)
Review comment:
PR #4843 introduced support for Service Principals and NIFI-8277
proposes support for Client Certificate credentials. This method and
`createKeyValueSecretClient()` would benefit from some refactoring along with
the existing credential handling to avoid duplication and ensure that that new
credential implementations would be supported.
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/cosmos/document/AbstractCosmosDBClientService.java
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.services.azure.cosmos.document;
+
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosException;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.processors.azure.cosmos.document.AzureCosmosDBUtils;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+@Tags({"azure", "cosmos", "document", "service"})
+@CapabilityDescription(
+ "Provides a controller service that configures a connection to Cosmos
DB (Core SQL API) " +
+ " and provides access to that connection to other Cosmos
DB-related components."
+)
+public abstract class AbstractCosmosDBClientService
+ extends AbstractControllerService
+ implements AzureCosmosDBConnectionService {
+
+ protected CosmosClient cosmosClient;
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.cosmosClient != null) {
+ try {
+ cosmosClient.close();
+ } catch(CosmosException e) {
+ getLogger().error("Closing CosmosClient Failed: " +
e.getMessage(), e);
+ } finally {
+ this.cosmosClient = null;
+ }
+ }
+ }
+
+ protected void createCosmosClient(final String uri, final String
accessKey, final String selectedConsistency){
+ final ConsistencyLevel cLevel;
+
+ switch(selectedConsistency) {
+ case AzureCosmosDBUtils.CONSISTENCY_STRONG:
+ cLevel = ConsistencyLevel.STRONG;
+ break;
+ case AzureCosmosDBUtils.CONSISTENCY_CONSISTENT_PREFIX:
+ cLevel = ConsistencyLevel.CONSISTENT_PREFIX;
+ break;
+ case AzureCosmosDBUtils.CONSISTENCY_BOUNDED_STALENESS:
+ cLevel = ConsistencyLevel.BOUNDED_STALENESS;
+ break;
+ case AzureCosmosDBUtils.CONSISTENCY_EVENTUAL:
+ cLevel = ConsistencyLevel.EVENTUAL;
+ break;
+ default:
+ cLevel = ConsistencyLevel.SESSION;
+ }
+ this.cosmosClient = new CosmosClientBuilder()
+ .endpoint(uri)
+ .key(accessKey)
+ .consistencyLevel(cLevel)
+ .buildClient();
+ }
+
+ @Override
+ public CosmosClient getCosmosClient() {
+ return this.cosmosClient;
+ }
+
+ public void setCosmosClient(CosmosClient client) {
Review comment:
Is this method used only for testing? If so, it would be better to
override the `cosmosClient` directly instead of having a public set method.
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/cosmos/document/AzureCosmosSecureDBClientService.java
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.services.azure.cosmos.document;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.cosmos.document.AzureCosmosDBUtils;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+import org.apache.nifi.services.azure.keyvault.AzureKeyVaultConnectionService;
+import org.apache.nifi.util.StringUtils;
+
+@Tags({"azure", "cosmos", "document", "service", "secure"})
+@CapabilityDescription(
+ "Provides a controller service that configures a connection to Cosmos
DB (Core SQL API) " +
+ " and provides access to that connection to other Cosmos
DB-related components."
+)
+public class AzureCosmosSecureDBClientService
+ extends AbstractCosmosDBClientService
+ implements AzureCosmosDBConnectionService {
+
+ private String uriSecret;
+ private String accessKeySecret;
+ private String consistencyLevel;
+ private AzureKeyVaultConnectionService keyVaultClientService;
+
+ public static final PropertyDescriptor URI_SECRET = new
PropertyDescriptor.Builder()
+ .name("azure-cosmos-db-uri-secret")
+ .displayName("Cosmos DB URI Secret Name")
+ .description("Cosmos DB URI Secret Name")
+ .required(true)
+ .addValidator(StandardValidators.URI_VALIDATOR)
+ .sensitive(true)
+ .build();
+
+ public static final PropertyDescriptor DB_ACCESS_KEY_SECRET = new
PropertyDescriptor.Builder()
+ .name("azure-cosmos-db-key-secret")
+ .displayName("Cosmos DB Access Key Secret Name")
+ .description("Cosmos DB Access Key Secret Name")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .sensitive(true)
+ .build();
+
+ static final PropertyDescriptor KEYVAULT_CONNECTION_SERVICE = new
PropertyDescriptor.Builder()
+ .name("azure-keyvault-connection-service")
+ .displayName("KeyVault Connection Service")
+ .description("If configured, the controller service used to obtain
the secrets from KeyVault")
+ .required(true)
+ .identifiesControllerService(AzureKeyVaultConnectionService.class)
+ .build();
+ @OnEnabled
Review comment:
It would be helpful to add a newline above this line for readability.
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/storage/AzureStorageSecureCredentialsControllerService.java
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.storage;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.microsoft.azure.storage.StorageCredentials;
+import com.microsoft.azure.storage.StorageCredentialsAccountAndKey;
+import com.microsoft.azure.storage.StorageCredentialsSharedAccessSignature;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
+import org.apache.nifi.services.azure.keyvault.AzureKeyVaultConnectionService;
+
+/**
+ * Implementation of AbstractControllerService interface
+ *
+ * @see AbstractControllerService
+ */
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob", "queue",
"credentials" })
+@CapabilityDescription("Defines credentials for Azure Storage processors. " +
+ "Uses Account Name with Account Key or Account Name with SAS Token.")
+public class AzureStorageSecureCredentialsControllerService
+ extends AbstractControllerService
+ implements AzureStorageCredentialsService {
+
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(Arrays.asList(
+ AzureStorageUtils.KEYVAULT_CONNECTION_SERVICE,
+ AzureStorageUtils.ACCOUNT_NAME_SECRET,
+ AzureStorageUtils.ACCOUNT_KEY_SECRET,
+ AzureStorageUtils.ACCOUNT_SAS_TOKEN_SECRET,
+ AzureStorageUtils.ENDPOINT_SUFFIX));
+
+ private ConfigurationContext context;
+ private ComponentLog logger;
Review comment:
Is there a reason for assigning logger as opposed to calling
`getLogger()` when needed?
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/cosmos/document/AzureCosmosSecureDBClientService.java
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.services.azure.cosmos.document;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.cosmos.document.AzureCosmosDBUtils;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+import org.apache.nifi.services.azure.keyvault.AzureKeyVaultConnectionService;
+import org.apache.nifi.util.StringUtils;
+
+@Tags({"azure", "cosmos", "document", "service", "secure"})
+@CapabilityDescription(
+ "Provides a controller service that configures a connection to Cosmos
DB (Core SQL API) " +
+ " and provides access to that connection to other Cosmos
DB-related components."
+)
+public class AzureCosmosSecureDBClientService
+ extends AbstractCosmosDBClientService
+ implements AzureCosmosDBConnectionService {
+
+ private String uriSecret;
+ private String accessKeySecret;
+ private String consistencyLevel;
+ private AzureKeyVaultConnectionService keyVaultClientService;
+
+ public static final PropertyDescriptor URI_SECRET = new
PropertyDescriptor.Builder()
+ .name("azure-cosmos-db-uri-secret")
+ .displayName("Cosmos DB URI Secret Name")
+ .description("Cosmos DB URI Secret Name")
+ .required(true)
+ .addValidator(StandardValidators.URI_VALIDATOR)
+ .sensitive(true)
+ .build();
+
+ public static final PropertyDescriptor DB_ACCESS_KEY_SECRET = new
PropertyDescriptor.Builder()
+ .name("azure-cosmos-db-key-secret")
+ .displayName("Cosmos DB Access Key Secret Name")
+ .description("Cosmos DB Access Key Secret Name")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .sensitive(true)
+ .build();
+
+ static final PropertyDescriptor KEYVAULT_CONNECTION_SERVICE = new
PropertyDescriptor.Builder()
+ .name("azure-keyvault-connection-service")
+ .displayName("KeyVault Connection Service")
+ .description("If configured, the controller service used to obtain
the secrets from KeyVault")
+ .required(true)
+ .identifiesControllerService(AzureKeyVaultConnectionService.class)
+ .build();
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) {
+ this.uriSecret = context.getProperty(URI_SECRET).getValue();
+ this.accessKeySecret =
context.getProperty(DB_ACCESS_KEY_SECRET).getValue();
+ this.consistencyLevel = context.getProperty(
+ AzureCosmosDBUtils.CONSISTENCY).getValue();
+ this.keyVaultClientService = context.getProperty(
+ KEYVAULT_CONNECTION_SERVICE
+ ).asControllerService(AzureKeyVaultConnectionService.class);
+
+ createCosmosClient(
+ getURI(),
+ getAccessKey(),
+ getConsistencyLevel()
+ );
+ }
+
+ static List<PropertyDescriptor> descriptors = new ArrayList<>();
Review comment:
Following standard conventions, static variables and initializers should
be declared before non-static methods.
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/keyvault/AzureKeyVaultConnectionService.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.services.azure.keyvault;
+
+import org.apache.nifi.controller.ControllerService;
+import com.azure.security.keyvault.secrets.SecretClient;
+
+public interface AzureKeyVaultConnectionService extends ControllerService {
+
+ SecretClient getKeyVaultSecretClient();
Review comment:
Is it necessary to expose the SecretClient as a interface method? The
interface could be simplified to something more generic if it is not necessary
to make the SecretClient available to consumers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
