exceptionfactory commented on code in PR #5369:
URL: https://github.com/apache/nifi/pull/5369#discussion_r955359353
##########
nifi-api/src/main/java/org/apache/nifi/flow/VersionedParameterContext.java:
##########
@@ -25,6 +25,9 @@ public class VersionedParameterContext extends
VersionedComponent {
private Set<VersionedParameter> parameters;
private List<String> inheritedParameterContexts;
private String description;
+ private String parameterProvider;
+ private String parameterGroupName;
+ private Boolean isSynchronized;
Review Comment:
Recommend naming this property `synchronized` to align with Bean property
naming conventions.
##########
nifi-mock/src/main/java/org/apache/nifi/util/MockParameterProviderInitializationContext.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.util;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerServiceLookup;
+import org.apache.nifi.controller.NodeTypeProvider;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.parameter.ParameterProviderInitializationContext;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+public class MockParameterProviderInitializationContext extends
MockControllerServiceLookup implements ParameterProviderInitializationContext,
ControllerServiceLookup {
+
+ private final String identifier;
+ private final String name;
+ private final Map<PropertyDescriptor, String> properties = new HashMap<>();
+ private final ComponentLog logger;
+
+ public MockParameterProviderInitializationContext(final String identifier,
final String name, final ComponentLog logger) {
+ this.identifier = identifier;
+ this.name = name;
+ this.logger = logger;
+ }
+
+ @Override
+ public String getIdentifier() {
+ return identifier;
+ }
+
+ @Override
+ public String getName() {
+ return name;
+ }
+
+ public void setProperty(final String propertyName, final String value) {
+ setProperty(new
PropertyDescriptor.Builder().name(propertyName).build(), value);
+ }
+
+ public void setProperty(final PropertyDescriptor propertyName, final
String value) {
+ this.properties.put(propertyName, value);
+ }
+
+ public void setProperties(final Map<PropertyDescriptor, String>
properties) {
+ this.properties.clear();
+ this.properties.putAll(properties);
+ }
+
+ @Override
+ public NodeTypeProvider getNodeTypeProvider() {
+ return null;
+ }
+
+ @Override
+ public ComponentLog getLogger() {
+ return logger;
+ }
+
+ @Override
+ public String getKerberosServicePrincipal() {
+ return null; //this needs to be wired in.
Review Comment:
These comments are not quite clear, perhaps they should be removed, or
expanded explaining expected implementation.
##########
nifi-api/src/main/java/org/apache/nifi/parameter/ParameterGroupConfiguration.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parameter;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * User-provided configuration for a group of parameters fetched from a
ParameterProvider.
+ */
+public class ParameterGroupConfiguration implements
Comparable<ParameterGroupConfiguration> {
+
+ private final String groupName;
+
+ private final String parameterContextName;
+
+ private final Map<String, ParameterSensitivity> parameterSensitivities;
+
+ private final Boolean isSynchronized;
+
+ /**
+ * Creates a named group of parameter names.
+ * @param groupName The parameter group name
+ * @param parameterContextName The parameter context name to which
parameters will be applied
+ * @param parameterSensitivities A map from parameter name to desired
sensitivity. Any parameter not included in this map will not be included
+ * when applied to the parameter context.
+ * @param isSynchronized If true, indicates that a ParameterContext should
be created if not already existing, or updated if existing
+ */
+ public ParameterGroupConfiguration(final String groupName, final String
parameterContextName, final Map<String, ParameterSensitivity>
parameterSensitivities,
+ final Boolean isSynchronized) {
Review Comment:
```suggestion
final Boolean synchronized) {
```
##########
nifi-api/src/main/java/org/apache/nifi/parameter/ParameterGroupConfiguration.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parameter;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * User-provided configuration for a group of parameters fetched from a
ParameterProvider.
+ */
+public class ParameterGroupConfiguration implements
Comparable<ParameterGroupConfiguration> {
+
+ private final String groupName;
+
+ private final String parameterContextName;
+
+ private final Map<String, ParameterSensitivity> parameterSensitivities;
+
+ private final Boolean isSynchronized;
Review Comment:
Recommend renaming to `synchronized`. Should this property be nullable, or
should it a primitive and default to `false`?
##########
.java-version:
##########
@@ -0,0 +1 @@
+1.8
Review Comment:
This file should be removed.
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ParameterGroupConfigurationEntity.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.api.entity;
+
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.parameter.ParameterSensitivity;
+
+import javax.xml.bind.annotation.XmlType;
+import java.util.Map;
+
+/**
+ * Entity encapsulating the configuration for a single parameter group.
+ */
+@XmlType(name = "parameterGroupConfiguration")
+public class ParameterGroupConfigurationEntity extends Entity implements
Comparable<ParameterGroupConfigurationEntity> {
+
+ private String groupName;
+ private String parameterContextName;
+ private Boolean isSynchronized;
Review Comment:
```suggestion
private Boolean synchronized;
```
##########
nifi-api/src/main/java/org/apache/nifi/parameter/ParameterProviderInitializationContext.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parameter;
+
+import org.apache.nifi.controller.NodeTypeProvider;
+import org.apache.nifi.kerberos.KerberosContext;
+import org.apache.nifi.logging.ComponentLog;
+
+public interface ParameterProviderInitializationContext extends
KerberosContext {
Review Comment:
Is it necessary to extend `KerberosContext`?
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ParameterProviderConfigurationDTO.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.api.dto;
+
+import io.swagger.annotations.ApiModelProperty;
+
+import javax.xml.bind.annotation.XmlType;
+
+@XmlType(name = "parameterProviderConfiguration")
+public class ParameterProviderConfigurationDTO {
+ private String parameterProviderId;
+ private String parameterProviderName;
+ private String parameterGroupName;
+ private Boolean isSynchronized;
Review Comment:
```suggestion
private Boolean synchronized;
```
##########
nifi-api/src/main/java/org/apache/nifi/parameter/ParameterProvider.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parameter;
+
+import org.apache.nifi.annotation.lifecycle.OnConfigurationRestored;
+import org.apache.nifi.components.ConfigurableComponent;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * Defines a provider that is responsible for fetching from an external source
Parameters with
+ * which a ParameterContext can be populated.
+ *
+ * <p>
+ * <code>ParameterProvider</code>s are discovered using Java's
+ * <code>ServiceLoader</code> mechanism. As a result, all implementations must
+ * follow these rules:
Review Comment:
With internal changes to bundle management and loading, it follows
ServiceLoader conventions, but does not actually use the ServiceLoader
implementation, so it would be best to adjust the wording accordingly.
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/parameter/StandardParameterProviderNode.java:
##########
@@ -0,0 +1,609 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.parameter;
+
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.documentation.DeprecationNotice;
+import org.apache.nifi.authorization.Resource;
+import org.apache.nifi.authorization.resource.Authorizable;
+import org.apache.nifi.authorization.resource.ResourceFactory;
+import org.apache.nifi.authorization.resource.ResourceType;
+import org.apache.nifi.bundle.Bundle;
+import org.apache.nifi.bundle.BundleCoordinate;
+import org.apache.nifi.components.ClassloaderIsolationKeyProvider;
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.ConfigVerificationResult.Outcome;
+import org.apache.nifi.components.ConfigurableComponent;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.validation.ValidationStatus;
+import org.apache.nifi.components.validation.ValidationTrigger;
+import org.apache.nifi.controller.AbstractComponentNode;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceLookup;
+import org.apache.nifi.controller.LoggableComponent;
+import org.apache.nifi.controller.ParameterProviderNode;
+import org.apache.nifi.controller.ParametersApplication;
+import org.apache.nifi.controller.ReloadComponent;
+import org.apache.nifi.controller.TerminationAwareLogger;
+import org.apache.nifi.controller.ValidationContextFactory;
+import org.apache.nifi.controller.service.ControllerServiceProvider;
+import org.apache.nifi.controller.service.StandardConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.nar.ExtensionManager;
+import org.apache.nifi.nar.InstanceClassLoader;
+import org.apache.nifi.nar.NarCloseable;
+import org.apache.nifi.parameter.Parameter;
+import org.apache.nifi.parameter.ParameterContext;
+import org.apache.nifi.parameter.ParameterDescriptor;
+import org.apache.nifi.parameter.ParameterGroup;
+import org.apache.nifi.parameter.ParameterGroupConfiguration;
+import org.apache.nifi.parameter.ParameterLookup;
+import org.apache.nifi.parameter.ParameterProvider;
+import org.apache.nifi.parameter.ParameterSensitivity;
+import org.apache.nifi.parameter.VerifiableParameterProvider;
+import org.apache.nifi.registry.ComponentVariableRegistry;
+import org.apache.nifi.util.CharacterFilterUtils;
+import org.apache.nifi.util.FormatUtils;
+import org.apache.nifi.util.file.classloader.ClassLoaderUtils;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Function;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class StandardParameterProviderNode extends AbstractComponentNode
implements ParameterProviderNode {
+
+ private static final Pattern PARAMETER_NAME_PATTERN =
Pattern.compile("^[a-zA-Z0-9-_. ]+$");
+
+ private final AtomicReference<ParameterProviderDetails>
parameterProviderRef;
+ private final ControllerServiceLookup serviceLookup;
+
+ private final ReadWriteLock rwLock = new ReentrantReadWriteLock();
+ private final Lock readLock = rwLock.readLock();
+ private final Lock writeLock = rwLock.writeLock();
+
+ private final Set<ParameterContext> referencingParameterContexts;
+
+ private final List<ParameterGroup> fetchedParameterGroups = new
ArrayList<>();
+
+ private volatile String comment;
+
+ private final Authorizable parentAuthorizable;
+
+ public StandardParameterProviderNode(final
LoggableComponent<ParameterProvider> parameterProvider, final String id, final
Authorizable parentAuthorizable,
+ final ControllerServiceProvider
controllerServiceProvider, final ValidationContextFactory
validationContextFactory,
+ final ComponentVariableRegistry
variableRegistry, final ReloadComponent reloadComponent, final ExtensionManager
extensionManager,
+ final ValidationTrigger
validationTrigger) {
+
+ this(parameterProvider, id, parentAuthorizable,
controllerServiceProvider, validationContextFactory,
+ parameterProvider.getComponent().getClass().getSimpleName(),
parameterProvider.getComponent().getClass().getCanonicalName(),
+ variableRegistry, reloadComponent, extensionManager,
validationTrigger, false);
+ }
+
+ public StandardParameterProviderNode(final
LoggableComponent<ParameterProvider> parameterProvider, final String id, final
Authorizable parentAuthorizable,
+ final ControllerServiceProvider
controllerServiceProvider, final ValidationContextFactory
validationContextFactory,
+ final String componentType, final
String canonicalClassName, final ComponentVariableRegistry variableRegistry,
+ final ReloadComponent
reloadComponent, final ExtensionManager extensionManager, final
ValidationTrigger validationTrigger, final boolean isExtensionMissing) {
+ super(id, validationContextFactory, controllerServiceProvider,
componentType, canonicalClassName, variableRegistry, reloadComponent,
+ extensionManager, validationTrigger, isExtensionMissing);
+ this.parameterProviderRef = new AtomicReference<>(new
ParameterProviderDetails(parameterProvider));
+ this.serviceLookup = controllerServiceProvider;
+ this.referencingParameterContexts = new HashSet<>();
+ this.parentAuthorizable = parentAuthorizable;
+ }
+
+ @Override
+ public Authorizable getParentAuthorizable() {
+ return parentAuthorizable;
+ }
+
+ @Override
+ public Resource getResource() {
+ return
ResourceFactory.getComponentResource(ResourceType.ParameterProvider,
getIdentifier(), getName());
+ }
+
+ @Override
+ public boolean isRestricted() {
+ return
getParameterProvider().getClass().isAnnotationPresent(Restricted.class);
+ }
+
+ @Override
+ public Class<?> getComponentClass() {
+ return getParameterProvider().getClass();
+ }
+
+ @Override
+ public boolean isDeprecated() {
+ return
getParameterProvider().getClass().isAnnotationPresent(DeprecationNotice.class);
+ }
+
+ @Override
+ protected ParameterContext getParameterContext() {
+ return null;
+ }
+ @Override
+ public ConfigurableComponent getComponent() {
+ return parameterProviderRef.get().getParameterProvider();
+ }
+
+ @Override
+ public BundleCoordinate getBundleCoordinate() {
+ return parameterProviderRef.get().getBundleCoordinate();
+ }
+
+ @Override
+ public TerminationAwareLogger getLogger() {
+ return parameterProviderRef.get().getComponentLog();
+ }
+
+ @Override
+ public ParameterProvider getParameterProvider() {
+ return parameterProviderRef.get().getParameterProvider();
+ }
+
+ @Override
+ public void setParameterProvider(final
LoggableComponent<ParameterProvider> parameterProvider) {
+ this.parameterProviderRef.set(new
ParameterProviderDetails(parameterProvider));
+ }
+
+ @Override
+ public void reload(final Set<URL> additionalUrls) throws
ParameterProviderInstantiationException {
+ final String additionalResourcesFingerprint =
ClassLoaderUtils.generateAdditionalUrlsFingerprint(additionalUrls,
determineClasloaderIsolationKey());
+ setAdditionalResourcesFingerprint(additionalResourcesFingerprint);
+ getReloadComponent().reload(this, getCanonicalClassName(),
getBundleCoordinate(), additionalUrls);
+ }
+
+ @Override
+ public boolean isValidationNecessary() {
+ return true;
+ }
+
+ @Override
+ public ConfigurationContext getConfigurationContext() {
+ return new StandardConfigurationContext(this, serviceLookup, null,
getVariableRegistry());
+ }
+
+ @Override
+ public void verifyModifiable() throws IllegalStateException {
+
+ }
+
+ @Override
+ public String getComments() {
+ return comment;
+ }
+
+ @Override
+ public void setComments(final String comment) {
+ this.comment =
CharacterFilterUtils.filterInvalidXmlCharacters(comment);
+ }
+
+ @Override
+ public void verifyCanClearState() {
+
+ }
+
+ @Override
+ public String toString() {
+ return "ParameterProvider[id=" + getIdentifier() + "]";
+ }
+
+ @Override
+ public String getProcessGroupIdentifier() {
+ return null;
+ }
+
+ @Override
+ public ParameterLookup getParameterLookup() {
+ return ParameterLookup.EMPTY;
+ }
+
+ @Override
+ public Set<ParameterContext> getReferences() {
+ readLock.lock();
+ try {
+ return Collections.unmodifiableSet(referencingParameterContexts);
+ } finally {
+ readLock.unlock();
+ }
+ }
+
+ @Override
+ public void addReference(final ParameterContext reference) {
+ writeLock.lock();
+ try {
+ referencingParameterContexts.add(reference);
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+ @Override
+ public void removeReference(final ParameterContext reference) {
+ writeLock.lock();
+ try {
+ referencingParameterContexts.remove(reference);
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+ @Override
+ protected List<ValidationResult> validateConfig() {
+ return Collections.emptyList();
+ }
+
+ @Override
+ public void verifyCanFetchParameters() {
+ final ValidationStatus validationStatus = performValidation();
+ if (validationStatus != ValidationStatus.VALID) {
+ throw new IllegalStateException(String.format("Parameter Provider
[%s] cannot fetch parameters while validation state is %s",
+ getIdentifier(), validationStatus));
+ }
+ }
+
+ @Override
+ public void fetchParameters() {
+ final ParameterProvider parameterProvider =
parameterProviderRef.get().getParameterProvider();
+ final ConfigurationContext configurationContext =
getConfigurationContext();
+ List<ParameterGroup> fetchedParameterGroups;
+ try (final NarCloseable narCloseable =
NarCloseable.withComponentNarLoader(getExtensionManager(),
parameterProvider.getClass(), parameterProvider.getIdentifier())) {
+ fetchedParameterGroups =
parameterProvider.fetchParameters(configurationContext);
+ } catch (final IOException e) {
+ throw new RuntimeException(String.format("Error fetching
parameters for Parameter Provider [%s]", getName()), e);
+ }
+
+ if (fetchedParameterGroups == null ||
fetchedParameterGroups.isEmpty()) {
+ return;
+ }
+
+ final Set<String> parameterGroupNames = new HashSet<>();
+
+ writeLock.lock();
+ try {
+ this.fetchedParameterGroups.clear();
+ for (final ParameterGroup group : fetchedParameterGroups) {
+ final String groupName = group.getGroupName();
+ if (parameterGroupNames.contains(groupName)) {
+ throw new IllegalStateException(String.format("Parameter
group [%s] is provided twice, which is not allowed", groupName));
+ }
+ final Collection<Parameter> parameters = group.getParameters();
+
+ if (parameters == null) {
+ continue;
+ }
+ final List<Parameter> validParameters = new ArrayList<>();
+ final Set<String> parameterNames = new HashSet<>();
+ for (final Parameter parameter : parameters) {
+ final ParameterDescriptor descriptor =
parameter.getDescriptor();
+ if (descriptor == null) {
+ throw new IllegalStateException("All fetched
parameters require a ParameterDescriptor");
+ }
+ final String parameterName = descriptor.getName();
+ if (parameterNames.contains(parameterName)) {
+ throw new
IllegalStateException(String.format("Parameter [%s] is provided in group [%s]
twice, which is not allowed",
+ parameterName, groupName));
+ }
+
+ if (parameter.getValue() == null) {
+ getLogger().warn("Skipping parameter [{}], which is
missing a value", new Object[] {parameterName});
+ continue;
+ }
+
+ if
(PARAMETER_NAME_PATTERN.matcher(parameter.getDescriptor().getName()).matches())
{
+ validParameters.add(parameter);
+
parameterNames.add(parameter.getDescriptor().getName());
+ } else {
+ getLogger().warn("Skipping parameter [{}}], whose name
has invalid characters. Only alpha-numeric characters (a-z, A-Z, 0-9), hyphens
(-), underscores (_), " +
+ "periods (.), and spaces ( ) are accepted.",
new Object[] {parameterName});
Review Comment:
```suggestion
getLogger().warn("Skipping parameter [{}], whose
name has invalid characters. Only alpha-numeric characters (a-z, A-Z, 0-9),
hyphens (-), underscores (_), " +
"periods (.), and spaces ( ) are accepted.",
parameterName);
```
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/FlowFromDOMFactory.java:
##########
@@ -148,6 +151,21 @@ public static ReportingTaskDTO getReportingTask(final
Element element, final Pro
return dto;
}
+ public static ParameterProviderDTO getParameterProvider(final Element
element, final PropertyEncryptor encryptor, final FlowEncodingVersion
flowEncodingVersion) {
+ final ParameterProviderDTO dto = new ParameterProviderDTO();
+
+ dto.setId(getString(element, "id"));
+ dto.setName(getString(element, "name"));
+ dto.setComments(getString(element, "comment"));
Review Comment:
Should these be `comments` instead of `comment`?
```suggestion
dto.setComments(getString(element, "comments"));
```
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/parameter/StandardParameterProviderNode.java:
##########
@@ -0,0 +1,609 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.parameter;
+
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.documentation.DeprecationNotice;
+import org.apache.nifi.authorization.Resource;
+import org.apache.nifi.authorization.resource.Authorizable;
+import org.apache.nifi.authorization.resource.ResourceFactory;
+import org.apache.nifi.authorization.resource.ResourceType;
+import org.apache.nifi.bundle.Bundle;
+import org.apache.nifi.bundle.BundleCoordinate;
+import org.apache.nifi.components.ClassloaderIsolationKeyProvider;
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.ConfigVerificationResult.Outcome;
+import org.apache.nifi.components.ConfigurableComponent;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.validation.ValidationStatus;
+import org.apache.nifi.components.validation.ValidationTrigger;
+import org.apache.nifi.controller.AbstractComponentNode;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceLookup;
+import org.apache.nifi.controller.LoggableComponent;
+import org.apache.nifi.controller.ParameterProviderNode;
+import org.apache.nifi.controller.ParametersApplication;
+import org.apache.nifi.controller.ReloadComponent;
+import org.apache.nifi.controller.TerminationAwareLogger;
+import org.apache.nifi.controller.ValidationContextFactory;
+import org.apache.nifi.controller.service.ControllerServiceProvider;
+import org.apache.nifi.controller.service.StandardConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.nar.ExtensionManager;
+import org.apache.nifi.nar.InstanceClassLoader;
+import org.apache.nifi.nar.NarCloseable;
+import org.apache.nifi.parameter.Parameter;
+import org.apache.nifi.parameter.ParameterContext;
+import org.apache.nifi.parameter.ParameterDescriptor;
+import org.apache.nifi.parameter.ParameterGroup;
+import org.apache.nifi.parameter.ParameterGroupConfiguration;
+import org.apache.nifi.parameter.ParameterLookup;
+import org.apache.nifi.parameter.ParameterProvider;
+import org.apache.nifi.parameter.ParameterSensitivity;
+import org.apache.nifi.parameter.VerifiableParameterProvider;
+import org.apache.nifi.registry.ComponentVariableRegistry;
+import org.apache.nifi.util.CharacterFilterUtils;
+import org.apache.nifi.util.FormatUtils;
+import org.apache.nifi.util.file.classloader.ClassLoaderUtils;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Function;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class StandardParameterProviderNode extends AbstractComponentNode
implements ParameterProviderNode {
+
+ private static final Pattern PARAMETER_NAME_PATTERN =
Pattern.compile("^[a-zA-Z0-9-_. ]+$");
+
+ private final AtomicReference<ParameterProviderDetails>
parameterProviderRef;
+ private final ControllerServiceLookup serviceLookup;
+
+ private final ReadWriteLock rwLock = new ReentrantReadWriteLock();
+ private final Lock readLock = rwLock.readLock();
+ private final Lock writeLock = rwLock.writeLock();
+
+ private final Set<ParameterContext> referencingParameterContexts;
+
+ private final List<ParameterGroup> fetchedParameterGroups = new
ArrayList<>();
+
+ private volatile String comment;
+
+ private final Authorizable parentAuthorizable;
+
+ public StandardParameterProviderNode(final
LoggableComponent<ParameterProvider> parameterProvider, final String id, final
Authorizable parentAuthorizable,
+ final ControllerServiceProvider
controllerServiceProvider, final ValidationContextFactory
validationContextFactory,
+ final ComponentVariableRegistry
variableRegistry, final ReloadComponent reloadComponent, final ExtensionManager
extensionManager,
+ final ValidationTrigger
validationTrigger) {
+
+ this(parameterProvider, id, parentAuthorizable,
controllerServiceProvider, validationContextFactory,
+ parameterProvider.getComponent().getClass().getSimpleName(),
parameterProvider.getComponent().getClass().getCanonicalName(),
+ variableRegistry, reloadComponent, extensionManager,
validationTrigger, false);
+ }
+
+ public StandardParameterProviderNode(final
LoggableComponent<ParameterProvider> parameterProvider, final String id, final
Authorizable parentAuthorizable,
+ final ControllerServiceProvider
controllerServiceProvider, final ValidationContextFactory
validationContextFactory,
+ final String componentType, final
String canonicalClassName, final ComponentVariableRegistry variableRegistry,
+ final ReloadComponent
reloadComponent, final ExtensionManager extensionManager, final
ValidationTrigger validationTrigger, final boolean isExtensionMissing) {
+ super(id, validationContextFactory, controllerServiceProvider,
componentType, canonicalClassName, variableRegistry, reloadComponent,
+ extensionManager, validationTrigger, isExtensionMissing);
+ this.parameterProviderRef = new AtomicReference<>(new
ParameterProviderDetails(parameterProvider));
+ this.serviceLookup = controllerServiceProvider;
+ this.referencingParameterContexts = new HashSet<>();
+ this.parentAuthorizable = parentAuthorizable;
+ }
+
+ @Override
+ public Authorizable getParentAuthorizable() {
+ return parentAuthorizable;
+ }
+
+ @Override
+ public Resource getResource() {
+ return
ResourceFactory.getComponentResource(ResourceType.ParameterProvider,
getIdentifier(), getName());
+ }
+
+ @Override
+ public boolean isRestricted() {
+ return
getParameterProvider().getClass().isAnnotationPresent(Restricted.class);
+ }
+
+ @Override
+ public Class<?> getComponentClass() {
+ return getParameterProvider().getClass();
+ }
+
+ @Override
+ public boolean isDeprecated() {
+ return
getParameterProvider().getClass().isAnnotationPresent(DeprecationNotice.class);
+ }
+
+ @Override
+ protected ParameterContext getParameterContext() {
+ return null;
+ }
+ @Override
+ public ConfigurableComponent getComponent() {
+ return parameterProviderRef.get().getParameterProvider();
+ }
+
+ @Override
+ public BundleCoordinate getBundleCoordinate() {
+ return parameterProviderRef.get().getBundleCoordinate();
+ }
+
+ @Override
+ public TerminationAwareLogger getLogger() {
+ return parameterProviderRef.get().getComponentLog();
+ }
+
+ @Override
+ public ParameterProvider getParameterProvider() {
+ return parameterProviderRef.get().getParameterProvider();
+ }
+
+ @Override
+ public void setParameterProvider(final
LoggableComponent<ParameterProvider> parameterProvider) {
+ this.parameterProviderRef.set(new
ParameterProviderDetails(parameterProvider));
+ }
+
+ @Override
+ public void reload(final Set<URL> additionalUrls) throws
ParameterProviderInstantiationException {
+ final String additionalResourcesFingerprint =
ClassLoaderUtils.generateAdditionalUrlsFingerprint(additionalUrls,
determineClasloaderIsolationKey());
+ setAdditionalResourcesFingerprint(additionalResourcesFingerprint);
+ getReloadComponent().reload(this, getCanonicalClassName(),
getBundleCoordinate(), additionalUrls);
+ }
+
+ @Override
+ public boolean isValidationNecessary() {
+ return true;
+ }
+
+ @Override
+ public ConfigurationContext getConfigurationContext() {
+ return new StandardConfigurationContext(this, serviceLookup, null,
getVariableRegistry());
+ }
+
+ @Override
+ public void verifyModifiable() throws IllegalStateException {
+
+ }
+
+ @Override
+ public String getComments() {
+ return comment;
+ }
+
+ @Override
+ public void setComments(final String comment) {
+ this.comment =
CharacterFilterUtils.filterInvalidXmlCharacters(comment);
+ }
+
+ @Override
+ public void verifyCanClearState() {
+
+ }
+
+ @Override
+ public String toString() {
+ return "ParameterProvider[id=" + getIdentifier() + "]";
+ }
+
+ @Override
+ public String getProcessGroupIdentifier() {
+ return null;
+ }
+
+ @Override
+ public ParameterLookup getParameterLookup() {
+ return ParameterLookup.EMPTY;
+ }
+
+ @Override
+ public Set<ParameterContext> getReferences() {
+ readLock.lock();
+ try {
+ return Collections.unmodifiableSet(referencingParameterContexts);
+ } finally {
+ readLock.unlock();
+ }
+ }
+
+ @Override
+ public void addReference(final ParameterContext reference) {
+ writeLock.lock();
+ try {
+ referencingParameterContexts.add(reference);
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+ @Override
+ public void removeReference(final ParameterContext reference) {
+ writeLock.lock();
+ try {
+ referencingParameterContexts.remove(reference);
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+ @Override
+ protected List<ValidationResult> validateConfig() {
+ return Collections.emptyList();
+ }
+
+ @Override
+ public void verifyCanFetchParameters() {
+ final ValidationStatus validationStatus = performValidation();
+ if (validationStatus != ValidationStatus.VALID) {
+ throw new IllegalStateException(String.format("Parameter Provider
[%s] cannot fetch parameters while validation state is %s",
+ getIdentifier(), validationStatus));
+ }
+ }
+
+ @Override
+ public void fetchParameters() {
+ final ParameterProvider parameterProvider =
parameterProviderRef.get().getParameterProvider();
+ final ConfigurationContext configurationContext =
getConfigurationContext();
+ List<ParameterGroup> fetchedParameterGroups;
+ try (final NarCloseable narCloseable =
NarCloseable.withComponentNarLoader(getExtensionManager(),
parameterProvider.getClass(), parameterProvider.getIdentifier())) {
+ fetchedParameterGroups =
parameterProvider.fetchParameters(configurationContext);
+ } catch (final IOException e) {
+ throw new RuntimeException(String.format("Error fetching
parameters for Parameter Provider [%s]", getName()), e);
+ }
+
+ if (fetchedParameterGroups == null ||
fetchedParameterGroups.isEmpty()) {
+ return;
+ }
+
+ final Set<String> parameterGroupNames = new HashSet<>();
+
+ writeLock.lock();
+ try {
+ this.fetchedParameterGroups.clear();
+ for (final ParameterGroup group : fetchedParameterGroups) {
+ final String groupName = group.getGroupName();
+ if (parameterGroupNames.contains(groupName)) {
+ throw new IllegalStateException(String.format("Parameter
group [%s] is provided twice, which is not allowed", groupName));
+ }
+ final Collection<Parameter> parameters = group.getParameters();
+
+ if (parameters == null) {
+ continue;
+ }
+ final List<Parameter> validParameters = new ArrayList<>();
+ final Set<String> parameterNames = new HashSet<>();
+ for (final Parameter parameter : parameters) {
+ final ParameterDescriptor descriptor =
parameter.getDescriptor();
+ if (descriptor == null) {
+ throw new IllegalStateException("All fetched
parameters require a ParameterDescriptor");
+ }
+ final String parameterName = descriptor.getName();
+ if (parameterNames.contains(parameterName)) {
+ throw new
IllegalStateException(String.format("Parameter [%s] is provided in group [%s]
twice, which is not allowed",
+ parameterName, groupName));
+ }
+
+ if (parameter.getValue() == null) {
+ getLogger().warn("Skipping parameter [{}], which is
missing a value", new Object[] {parameterName});
+ continue;
+ }
+
+ if
(PARAMETER_NAME_PATTERN.matcher(parameter.getDescriptor().getName()).matches())
{
+ validParameters.add(parameter);
+
parameterNames.add(parameter.getDescriptor().getName());
+ } else {
+ getLogger().warn("Skipping parameter [{}}], whose name
has invalid characters. Only alpha-numeric characters (a-z, A-Z, 0-9), hyphens
(-), underscores (_), " +
+ "periods (.), and spaces ( ) are accepted.",
new Object[] {parameterName});
+ }
+ }
+ this.fetchedParameterGroups.add(new ParameterGroup(groupName,
toProvidedParameters(validParameters)));
+ parameterGroupNames.add(groupName);
+ }
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+ @Override
+ public void verifyCanApplyParameters(final
Collection<ParameterGroupConfiguration> parameterGroupConfigurations) {
+ if (fetchedParameterGroups.isEmpty()) {
+ return;
+ }
+ readLock.lock();
+ try {
+ final Map<String, ParameterGroupConfiguration>
parameterGroupConfigurationMap = parameterGroupConfigurations.stream()
+
.collect(Collectors.toMap(ParameterGroupConfiguration::getParameterContextName,
Function.identity()));
+ for (final ParameterContext reference : getReferences()) {
+ final ParameterGroupConfiguration groupConfiguration =
parameterGroupConfigurationMap.get(reference.getName());
+ if (groupConfiguration == null) {
+ continue;
+ }
+
reference.verifyCanSetParameters(getFetchedParameterUpdateMap(reference,
groupConfiguration));
+ }
+ } finally {
+ readLock.unlock();
+ }
+ }
+
+ @Override
+ public void verifyCanDelete() {
+ if (!getReferences().isEmpty()) {
+ final String contextNameList = getReferences().stream()
+ .map(ParameterContext::getName)
+ .collect(Collectors.joining(", "));
+ throw new IllegalStateException(String.format("Cannot delete
Parameter Provider [%s] while it is referenced by Contexts: [%s]",
getIdentifier(), contextNameList));
+ }
+ }
+
+ @Override
+ public List<ConfigVerificationResult> verifyConfiguration(final
ConfigurationContext context, final ComponentLog logger, final ExtensionManager
extensionManager) {
+ final List<ConfigVerificationResult> results = new ArrayList<>();
+
+ try {
+ final long startNanos = System.nanoTime();
+ // Call super's verifyConfig, which will perform component
validation
+ results.addAll(super.verifyConfig(context.getProperties(),
context.getAnnotationData(), null));
+ final long validationComplete = System.nanoTime();
+
+ // If any invalid outcomes from validation, we do not want to
perform additional verification, because we only run additional verification
when the component is valid.
+ // This is done in order to make it much simpler to develop these
verifications, since the developer doesn't have to worry about whether or not
the given values are valid.
+ if (!results.isEmpty() && results.stream().anyMatch(result ->
result.getOutcome() == Outcome.FAILED)) {
+ return results;
+ }
+
+ final ParameterProvider parameterProvider = getParameterProvider();
+ if (parameterProvider instanceof VerifiableParameterProvider) {
+ logger.debug("{} is a VerifiableParameterProvider. Will
perform full verification of configuration.", this);
+ final VerifiableParameterProvider verifiable =
(VerifiableParameterProvider) parameterProvider;
+
+ // Check if the given configuration requires a different
classloader than the current configuration
+ final boolean classpathDifferent =
isClasspathDifferent(context.getProperties());
+
+ if (classpathDifferent) {
+ // Create a classloader for the given configuration and
use that to verify the component's configuration
+ final Bundle bundle =
extensionManager.getBundle(getBundleCoordinate());
+ final Set<URL> classpathUrls =
getAdditionalClasspathResources(context.getProperties().keySet(), descriptor ->
context.getProperty(descriptor).getValue());
+
+ final String classloaderIsolationKey =
getClassLoaderIsolationKey(context);
+
+ final ClassLoader currentClassLoader =
Thread.currentThread().getContextClassLoader();
+ try (final InstanceClassLoader detectedClassLoader =
extensionManager.createInstanceClassLoader(getComponentType(), getIdentifier(),
bundle, classpathUrls, false,
+ classloaderIsolationKey)) {
+
Thread.currentThread().setContextClassLoader(detectedClassLoader);
+ results.addAll(verifiable.verify(context, logger));
+ } finally {
+
Thread.currentThread().setContextClassLoader(currentClassLoader);
+ }
+ } else {
+ // Verify the configuration, using the component's
classloader
+ try (final NarCloseable narCloseable =
NarCloseable.withComponentNarLoader(extensionManager,
parameterProvider.getClass(), getIdentifier())) {
+ results.addAll(verifiable.verify(context, logger));
+ }
+ }
+
+ final long validationNanos = validationComplete - startNanos;
+ final long verificationNanos = System.nanoTime() -
validationComplete;
+ logger.debug("{} completed full configuration validation in {}
plus {} for validation",
+ this, FormatUtils.formatNanos(verificationNanos,
false), FormatUtils.formatNanos(validationNanos, false));
+ } else {
+ logger.debug("{} is not a VerifiableParameterProvider, so will
not perform full verification of configuration. Validation took {}", this,
+ FormatUtils.formatNanos(validationComplete -
startNanos, false));
+ }
+ } catch (final Throwable t) {
+ logger.error("Failed to perform verification of Parameter
Provider's configuration for {}", this, t);
+
+ results.add(new ConfigVerificationResult.Builder()
+ .outcome(Outcome.FAILED)
+ .verificationStepName("Perform Verification")
+ .explanation("Encountered unexpected failure when
attempting to perform verification: " + t)
+ .build());
+ }
+
+ return results;
+ }
+
+ /**
+ * Using the existing parameters from the ParameterContext and the fetched
parameters in the provider, constructs
+ * a map from Parameter name to updated Parameter (or null if the
parameter has been removed in the fetch).
+ * @param parameterContext A ParameterContext
+ * @param parameterGroupConfiguration The configuration for the fetched
parameter group
+ * @return A map from name to Parameter (or null if parameter should be
removed)
+ */
+ private Map<String, Parameter> getFetchedParameterUpdateMap(final
ParameterContext parameterContext, ParameterGroupConfiguration
parameterGroupConfiguration) {
+ final Map<String, Parameter> parameterUpdateMap = new HashMap<>();
+
+ final ParameterGroup parameterGroup = fetchedParameterGroups.stream()
+ .filter(group ->
parameterContext.getParameterProviderConfiguration().getParameterGroupName().equals(group.getGroupName()))
+ .findFirst()
+ .orElse(null);
+
+ if (parameterGroup == null) {
+ return parameterUpdateMap;
+ }
+
+ // Get a filtered list of the parameters with their sensitivity set
based on group configuration
+ final List<Parameter> configuredParameters =
configureParameters(parameterGroup.getParameters(),
parameterGroupConfiguration);
+
+ final Map<ParameterDescriptor, Parameter> fetchedParameterMap =
configuredParameters.stream()
+ .collect(Collectors.toMap(Parameter::getDescriptor,
Function.identity()));
+
+ final Map<ParameterDescriptor, Parameter> currentParameters =
parameterContext.getParameters();
+ // Find parameters that were removed
+ currentParameters.keySet().forEach(descriptor -> {
+ if (!fetchedParameterMap.containsKey(descriptor)) {
+ parameterUpdateMap.put(descriptor.getName(), null);
+ }
+ });
+ // Add all changed and new parameters
+ for (final Map.Entry<ParameterDescriptor, Parameter> entry :
fetchedParameterMap.entrySet()) {
+ final ParameterDescriptor descriptor = entry.getKey();
+ final Parameter fetchedParameter = entry.getValue();
+ final Parameter currentParameter =
currentParameters.get(descriptor);
+ if (currentParameter == null) {
+ // Add if it's a new parameter
+ parameterUpdateMap.put(descriptor.getName(), fetchedParameter);
+ } else {
+ final boolean isSensitivityChanged =
currentParameter.getDescriptor().isSensitive() !=
fetchedParameter.getDescriptor().isSensitive();
+ if (!Objects.equals(currentParameter.getValue(),
fetchedParameter.getValue()) || isSensitivityChanged) {
+ // Also add if it's an existing parameter that has a
changed value
+ parameterUpdateMap.put(descriptor.getName(),
fetchedParameter);
+
+ if (isSensitivityChanged) {
+ final ParameterSensitivity currentSensitivity =
currentParameter.getDescriptor().isSensitive() ? ParameterSensitivity.SENSITIVE
: ParameterSensitivity.NON_SENSITIVE;
+ final ParameterSensitivity fetchedSensitivity =
fetchedParameter.getDescriptor().isSensitive() ? ParameterSensitivity.SENSITIVE
: ParameterSensitivity.NON_SENSITIVE;
+ getLogger().info("Parameter [{}] is being changed from
{} to {}", new Object[] { descriptor.getName(),
+ currentSensitivity.getName(),
fetchedSensitivity.getName()});
Review Comment:
```suggestion
getLogger().info("Parameter [{}] sensitivity is
being changed from {} to {}", descriptor.getName(),
currentSensitivity.getName(),
fetchedSensitivity.getName());
```
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/StandardFlowSerializer.java:
##########
@@ -656,6 +673,20 @@ public static void addReportingTask(final Element element,
final ReportingTaskNo
element.appendChild(taskElement);
}
+ public static void addParameterProvider(final Element element, final
ParameterProviderNode providerNode, final PropertyEncryptor encryptor) {
+ final Element taskElement =
element.getOwnerDocument().createElement("parameterProvider");
+ addTextElement(taskElement, "id", providerNode.getIdentifier());
+ addTextElement(taskElement, "name", providerNode.getName());
+ addTextElement(taskElement, "comment", providerNode.getComments());
Review Comment:
```suggestion
addTextElement(taskElement, "comments", providerNode.getComments());
```
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/resources/FlowConfiguration.xsd:
##########
@@ -69,6 +71,9 @@
<xs:element name="name" type="NonEmptyStringType" />
<xs:element name="description" type="xs:string" minOccurs="0"
maxOccurs="1" />
<xs:element name="inheritedParameterContextId" type="xs:string"
minOccurs="0" maxOccurs="unbounded" />
+ <xs:element name="parameterProviderId" type="xs:string"
minOccurs="0" maxOccurs="1" />
+ <xs:element name="parameterGroupName" type="xs:string"
minOccurs="0" maxOccurs="1" />
+ <xs:element name="isSynchronized" type="xs:boolean" minOccurs="0"
maxOccurs="1" />
Review Comment:
```suggestion
<xs:element name="synchronized" type="xs:boolean" minOccurs="0"
maxOccurs="1" />
```
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/META-INF/services/org.apache.nifi.parameter.ParameterProvider:
##########
@@ -0,0 +1,15 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+org.apache.nifi.parameter.mock.DummyParameterProvider
Review Comment:
Noting this for adjustment along with the class name.
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ParameterProviderDTO.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.api.dto;
+
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.web.api.entity.AffectedComponentEntity;
+import
org.apache.nifi.web.api.entity.ParameterProviderReferencingComponentEntity;
+import org.apache.nifi.web.api.entity.ParameterGroupConfigurationEntity;
+
+import javax.xml.bind.annotation.XmlType;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Component that is capable of providing parameters to NiFi from an external
source
+ */
+@XmlType(name = "parameterProvider")
+public class ParameterProviderDTO extends ComponentDTO {
+ public static final String VALID = "VALID";
+ public static final String INVALID = "INVALID";
+ public static final String VALIDATING = "VALIDATING";
+
+ private String name;
+ private String type;
+ private BundleDTO bundle;
+ private String comments;
+ private Boolean persistsState;
+ private Boolean restricted;
+ private Boolean deprecated;
+ private Boolean isExtensionMissing;
Review Comment:
```suggestion
private Boolean extensionMissing;
```
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/parameter/StandardParameterProviderNode.java:
##########
@@ -0,0 +1,609 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.parameter;
+
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.documentation.DeprecationNotice;
+import org.apache.nifi.authorization.Resource;
+import org.apache.nifi.authorization.resource.Authorizable;
+import org.apache.nifi.authorization.resource.ResourceFactory;
+import org.apache.nifi.authorization.resource.ResourceType;
+import org.apache.nifi.bundle.Bundle;
+import org.apache.nifi.bundle.BundleCoordinate;
+import org.apache.nifi.components.ClassloaderIsolationKeyProvider;
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.ConfigVerificationResult.Outcome;
+import org.apache.nifi.components.ConfigurableComponent;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.validation.ValidationStatus;
+import org.apache.nifi.components.validation.ValidationTrigger;
+import org.apache.nifi.controller.AbstractComponentNode;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceLookup;
+import org.apache.nifi.controller.LoggableComponent;
+import org.apache.nifi.controller.ParameterProviderNode;
+import org.apache.nifi.controller.ParametersApplication;
+import org.apache.nifi.controller.ReloadComponent;
+import org.apache.nifi.controller.TerminationAwareLogger;
+import org.apache.nifi.controller.ValidationContextFactory;
+import org.apache.nifi.controller.service.ControllerServiceProvider;
+import org.apache.nifi.controller.service.StandardConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.nar.ExtensionManager;
+import org.apache.nifi.nar.InstanceClassLoader;
+import org.apache.nifi.nar.NarCloseable;
+import org.apache.nifi.parameter.Parameter;
+import org.apache.nifi.parameter.ParameterContext;
+import org.apache.nifi.parameter.ParameterDescriptor;
+import org.apache.nifi.parameter.ParameterGroup;
+import org.apache.nifi.parameter.ParameterGroupConfiguration;
+import org.apache.nifi.parameter.ParameterLookup;
+import org.apache.nifi.parameter.ParameterProvider;
+import org.apache.nifi.parameter.ParameterSensitivity;
+import org.apache.nifi.parameter.VerifiableParameterProvider;
+import org.apache.nifi.registry.ComponentVariableRegistry;
+import org.apache.nifi.util.CharacterFilterUtils;
+import org.apache.nifi.util.FormatUtils;
+import org.apache.nifi.util.file.classloader.ClassLoaderUtils;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Function;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class StandardParameterProviderNode extends AbstractComponentNode
implements ParameterProviderNode {
+
+ private static final Pattern PARAMETER_NAME_PATTERN =
Pattern.compile("^[a-zA-Z0-9-_. ]+$");
+
+ private final AtomicReference<ParameterProviderDetails>
parameterProviderRef;
+ private final ControllerServiceLookup serviceLookup;
+
+ private final ReadWriteLock rwLock = new ReentrantReadWriteLock();
+ private final Lock readLock = rwLock.readLock();
+ private final Lock writeLock = rwLock.writeLock();
+
+ private final Set<ParameterContext> referencingParameterContexts;
+
+ private final List<ParameterGroup> fetchedParameterGroups = new
ArrayList<>();
+
+ private volatile String comment;
+
+ private final Authorizable parentAuthorizable;
+
+ public StandardParameterProviderNode(final
LoggableComponent<ParameterProvider> parameterProvider, final String id, final
Authorizable parentAuthorizable,
+ final ControllerServiceProvider
controllerServiceProvider, final ValidationContextFactory
validationContextFactory,
+ final ComponentVariableRegistry
variableRegistry, final ReloadComponent reloadComponent, final ExtensionManager
extensionManager,
+ final ValidationTrigger
validationTrigger) {
+
+ this(parameterProvider, id, parentAuthorizable,
controllerServiceProvider, validationContextFactory,
+ parameterProvider.getComponent().getClass().getSimpleName(),
parameterProvider.getComponent().getClass().getCanonicalName(),
+ variableRegistry, reloadComponent, extensionManager,
validationTrigger, false);
+ }
+
+ public StandardParameterProviderNode(final
LoggableComponent<ParameterProvider> parameterProvider, final String id, final
Authorizable parentAuthorizable,
+ final ControllerServiceProvider
controllerServiceProvider, final ValidationContextFactory
validationContextFactory,
+ final String componentType, final
String canonicalClassName, final ComponentVariableRegistry variableRegistry,
+ final ReloadComponent
reloadComponent, final ExtensionManager extensionManager, final
ValidationTrigger validationTrigger, final boolean isExtensionMissing) {
+ super(id, validationContextFactory, controllerServiceProvider,
componentType, canonicalClassName, variableRegistry, reloadComponent,
+ extensionManager, validationTrigger, isExtensionMissing);
+ this.parameterProviderRef = new AtomicReference<>(new
ParameterProviderDetails(parameterProvider));
+ this.serviceLookup = controllerServiceProvider;
+ this.referencingParameterContexts = new HashSet<>();
+ this.parentAuthorizable = parentAuthorizable;
+ }
+
+ @Override
+ public Authorizable getParentAuthorizable() {
+ return parentAuthorizable;
+ }
+
+ @Override
+ public Resource getResource() {
+ return
ResourceFactory.getComponentResource(ResourceType.ParameterProvider,
getIdentifier(), getName());
+ }
+
+ @Override
+ public boolean isRestricted() {
+ return
getParameterProvider().getClass().isAnnotationPresent(Restricted.class);
+ }
+
+ @Override
+ public Class<?> getComponentClass() {
+ return getParameterProvider().getClass();
+ }
+
+ @Override
+ public boolean isDeprecated() {
+ return
getParameterProvider().getClass().isAnnotationPresent(DeprecationNotice.class);
+ }
+
+ @Override
+ protected ParameterContext getParameterContext() {
+ return null;
+ }
+ @Override
+ public ConfigurableComponent getComponent() {
+ return parameterProviderRef.get().getParameterProvider();
+ }
+
+ @Override
+ public BundleCoordinate getBundleCoordinate() {
+ return parameterProviderRef.get().getBundleCoordinate();
+ }
+
+ @Override
+ public TerminationAwareLogger getLogger() {
+ return parameterProviderRef.get().getComponentLog();
+ }
+
+ @Override
+ public ParameterProvider getParameterProvider() {
+ return parameterProviderRef.get().getParameterProvider();
+ }
+
+ @Override
+ public void setParameterProvider(final
LoggableComponent<ParameterProvider> parameterProvider) {
+ this.parameterProviderRef.set(new
ParameterProviderDetails(parameterProvider));
+ }
+
+ @Override
+ public void reload(final Set<URL> additionalUrls) throws
ParameterProviderInstantiationException {
+ final String additionalResourcesFingerprint =
ClassLoaderUtils.generateAdditionalUrlsFingerprint(additionalUrls,
determineClasloaderIsolationKey());
+ setAdditionalResourcesFingerprint(additionalResourcesFingerprint);
+ getReloadComponent().reload(this, getCanonicalClassName(),
getBundleCoordinate(), additionalUrls);
+ }
+
+ @Override
+ public boolean isValidationNecessary() {
+ return true;
+ }
+
+ @Override
+ public ConfigurationContext getConfigurationContext() {
+ return new StandardConfigurationContext(this, serviceLookup, null,
getVariableRegistry());
+ }
+
+ @Override
+ public void verifyModifiable() throws IllegalStateException {
+
+ }
+
+ @Override
+ public String getComments() {
+ return comment;
+ }
+
+ @Override
+ public void setComments(final String comment) {
+ this.comment =
CharacterFilterUtils.filterInvalidXmlCharacters(comment);
+ }
+
+ @Override
+ public void verifyCanClearState() {
+
+ }
+
+ @Override
+ public String toString() {
+ return "ParameterProvider[id=" + getIdentifier() + "]";
+ }
+
+ @Override
+ public String getProcessGroupIdentifier() {
+ return null;
+ }
+
+ @Override
+ public ParameterLookup getParameterLookup() {
+ return ParameterLookup.EMPTY;
+ }
+
+ @Override
+ public Set<ParameterContext> getReferences() {
+ readLock.lock();
+ try {
+ return Collections.unmodifiableSet(referencingParameterContexts);
+ } finally {
+ readLock.unlock();
+ }
+ }
+
+ @Override
+ public void addReference(final ParameterContext reference) {
+ writeLock.lock();
+ try {
+ referencingParameterContexts.add(reference);
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+ @Override
+ public void removeReference(final ParameterContext reference) {
+ writeLock.lock();
+ try {
+ referencingParameterContexts.remove(reference);
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+ @Override
+ protected List<ValidationResult> validateConfig() {
+ return Collections.emptyList();
+ }
+
+ @Override
+ public void verifyCanFetchParameters() {
+ final ValidationStatus validationStatus = performValidation();
+ if (validationStatus != ValidationStatus.VALID) {
+ throw new IllegalStateException(String.format("Parameter Provider
[%s] cannot fetch parameters while validation state is %s",
+ getIdentifier(), validationStatus));
+ }
+ }
+
+ @Override
+ public void fetchParameters() {
+ final ParameterProvider parameterProvider =
parameterProviderRef.get().getParameterProvider();
+ final ConfigurationContext configurationContext =
getConfigurationContext();
+ List<ParameterGroup> fetchedParameterGroups;
+ try (final NarCloseable narCloseable =
NarCloseable.withComponentNarLoader(getExtensionManager(),
parameterProvider.getClass(), parameterProvider.getIdentifier())) {
+ fetchedParameterGroups =
parameterProvider.fetchParameters(configurationContext);
+ } catch (final IOException e) {
+ throw new RuntimeException(String.format("Error fetching
parameters for Parameter Provider [%s]", getName()), e);
+ }
+
+ if (fetchedParameterGroups == null ||
fetchedParameterGroups.isEmpty()) {
+ return;
+ }
+
+ final Set<String> parameterGroupNames = new HashSet<>();
+
+ writeLock.lock();
+ try {
+ this.fetchedParameterGroups.clear();
+ for (final ParameterGroup group : fetchedParameterGroups) {
+ final String groupName = group.getGroupName();
+ if (parameterGroupNames.contains(groupName)) {
+ throw new IllegalStateException(String.format("Parameter
group [%s] is provided twice, which is not allowed", groupName));
+ }
+ final Collection<Parameter> parameters = group.getParameters();
+
+ if (parameters == null) {
+ continue;
+ }
+ final List<Parameter> validParameters = new ArrayList<>();
+ final Set<String> parameterNames = new HashSet<>();
+ for (final Parameter parameter : parameters) {
+ final ParameterDescriptor descriptor =
parameter.getDescriptor();
+ if (descriptor == null) {
+ throw new IllegalStateException("All fetched
parameters require a ParameterDescriptor");
+ }
+ final String parameterName = descriptor.getName();
+ if (parameterNames.contains(parameterName)) {
+ throw new
IllegalStateException(String.format("Parameter [%s] is provided in group [%s]
twice, which is not allowed",
+ parameterName, groupName));
+ }
+
+ if (parameter.getValue() == null) {
+ getLogger().warn("Skipping parameter [{}], which is
missing a value", new Object[] {parameterName});
Review Comment:
```suggestion
getLogger().warn("Skipping parameter [{}], which is
missing a value", parameterName);
```
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractComponentNode.java:
##########
@@ -1129,17 +1129,21 @@ public Optional<Parameter> getParameter(final String
parameterName) {
// There is an update to the parameter. We want to return the
previous value of the Parameter.
final ParameterDescriptor parameterDescriptor;
+ final boolean isProvided;
Review Comment:
```suggestion
final boolean provided;
```
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/resources/FlowConfiguration.xsd:
##########
@@ -534,6 +540,25 @@
</xs:sequence>
</xs:complexType>
+ <xs:complexType name="ParameterProvidersType">
+ <xs:sequence>
+ <xs:element name="parameterProvider" type="ParameterProviderType"
minOccurs="0" maxOccurs="unbounded" />
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="ParameterProviderType">
+ <xs:sequence>
+ <xs:element name="id" type="NonEmptyStringType" />
+ <xs:element name="name" type="NonEmptyStringType" />
+ <xs:element name="comment" type="xs:string" />
Review Comment:
```suggestion
<xs:element name="comments" type="xs:string" />
```
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardReloadComponent.java:
##########
@@ -209,4 +211,52 @@ public void reload(final ReportingTaskNode existingNode,
final String newType, f
flowController.getValidationTrigger().triggerAsync(existingNode);
}
+ @Override
+ public void reload(final ParameterProviderNode existingNode, final String
newType, final BundleCoordinate bundleCoordinate, final Set<URL> additionalUrls)
+ throws ParameterProviderInstantiationException {
+ if (existingNode == null) {
+ throw new IllegalStateException("Existing ParameterProviderNode
cannot be null");
+ }
+
+ final String id = existingNode.getParameterProvider().getIdentifier();
+
+ // ghost components will have a null logger
+ if (existingNode.getLogger() != null) {
+ existingNode.getLogger().debug("Reloading component {} to type {}
from bundle {}", new Object[]{id, newType, bundleCoordinate});
Review Comment:
```suggestion
existingNode.getLogger().debug("Reloading component {} to type
{} from bundle {}", id, newType, bundleCoordinate);
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-parameter-providers/src/main/java/org/apache/nifi/parameter/FileParameterProvider.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parameter;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.LimitingInputStream;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+@Tags({"file"})
+@CapabilityDescription("Fetches parameters from files. Parameter groups are
indicated by a set of directories, and files within the directories map to
parameter names. " +
+ "The content of the file becomes the parameter value.")
+
+@Restricted(
+ restrictions = {
+ @Restriction(
+ requiredPermission =
RequiredPermission.READ_FILESYSTEM,
+ explanation = "Provides operator the ability to read
from any file that NiFi has access to.")
+ }
+)
+public class FileParameterProvider extends AbstractParameterProvider
implements VerifiableParameterProvider {
+ private static final int MAX_SIZE_LIMIT = 8096;
+ private static final Charset DEFAULT_CHARSET = StandardCharsets.UTF_8;
+
+ private enum ParameterValueEncoding {
+ BASE64,
+ PLAINTEXT
+ }
+
+ private static final AllowableValue BASE64_ENCODING = new
AllowableValue("base64", "Base64", "File content is Base64-encoded, " +
+ "and will be decoded before providing the value as a Parameter.");
+ private static final AllowableValue PLAIN_TEXT = new
AllowableValue("plaintext", "Plain text", "File content is not encoded, " +
+ "and will be provided directly as a Parameter value.");
+
+ public static final PropertyDescriptor PARAMETER_GROUP_DIRECTORIES = new
PropertyDescriptor.Builder()
+ .name("parameter-group-directories")
+ .displayName("Parameter Group Directories")
+ .description("A comma-separated list of directory absolute paths
that will map to named parameter groups. Each directory that contains " +
+ "files will map to a parameter group, named after the
innermost directory in the path. Files inside the directory will map to " +
+ "parameter names, whose values are the content of each
respective file.")
+ .addValidator(new MultiDirectoryExistsValidator())
+ .required(true)
+ .build();
+ public static final PropertyDescriptor PARAMETER_VALUE_BYTE_LIMIT = new
PropertyDescriptor.Builder()
+ .name("parameter-value-byte-limit")
+ .displayName("Parameter Value Byte Limit")
+ .description("The maximum byte size of a parameter value. Since
parameter values are pulled from the contents of files, this is a safeguard
that can " +
+ "prevent memory issues if large files are included.")
+ .addValidator(StandardValidators.createDataSizeBoundsValidator(1,
MAX_SIZE_LIMIT))
+ .defaultValue("256 B")
+ .required(true)
+ .build();
+ public static final PropertyDescriptor PARAMETER_VALUE_ENCODING = new
PropertyDescriptor.Builder()
+ .name("parameter-value-encoding")
+ .displayName("Parameter Value Encoding")
+ .description("Indicates how parameter values are encoded inside
Parameter files.")
+ .allowableValues(BASE64_ENCODING, PLAIN_TEXT)
+ .defaultValue(BASE64_ENCODING.getValue())
+ .required(true)
+ .build();
+
+ private List<PropertyDescriptor> properties;
+
+ @Override
+ protected void init(final ParameterProviderInitializationContext config) {
+ final List<PropertyDescriptor> properties = new ArrayList<>();
+ properties.add(PARAMETER_GROUP_DIRECTORIES);
+ properties.add(PARAMETER_VALUE_BYTE_LIMIT);
+ properties.add(PARAMETER_VALUE_ENCODING);
+
+ this.properties = Collections.unmodifiableList(properties);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return properties;
+ }
+
+ @Override
+ public List<ParameterGroup> fetchParameters(final ConfigurationContext
context) {
+
+ final List<ParameterGroup> parameterGroups = new ArrayList<>();
+
+ final Collection<File> groupDirectories = getDirectories(context,
PARAMETER_GROUP_DIRECTORIES);
+ groupDirectories.forEach(directory -> {
+ parameterGroups.add(getParameterGroup(context, directory,
directory.getName()));
+ });
+ final AtomicInteger groupedParameterCount = new AtomicInteger(0);
+ final Collection<String> groupNames = new HashSet<>();
+ parameterGroups.forEach(group -> {
+ groupedParameterCount.addAndGet(group.getParameters().size());
+ groupNames.add(group.getGroupName());
+ });
+ getLogger().info("Fetched {} parameters. Group names: {}",
groupedParameterCount.get(), groupNames);
+ return parameterGroups;
+ }
+
+ @Override
+ public List<ConfigVerificationResult> verify(final ConfigurationContext
context, final ComponentLog verificationLogger) {
+ final List<ConfigVerificationResult> results = new ArrayList<>();
+
+ try {
+ final List<ParameterGroup> parameterGroups =
fetchParameters(context);
+ final Set<String> parameterGroupNames =
parameterGroups.stream().map(ParameterGroup::getGroupName).collect(Collectors.toSet());
+ final long parameterCount = parameterGroups.stream()
+ .flatMap(group -> group.getParameters().stream())
+ .count();
+ results.add(new ConfigVerificationResult.Builder()
+ .outcome(ConfigVerificationResult.Outcome.SUCCESSFUL)
+ .verificationStepName("Fetch Parameters")
+ .explanation(String.format("Fetched %s files as
parameters.", parameterCount))
+ .build());
+ } catch (final IllegalArgumentException e) {
+ verificationLogger.error("Failed to fetch parameters", e);
+ results.add(new ConfigVerificationResult.Builder()
+ .outcome(ConfigVerificationResult.Outcome.FAILED)
+ .verificationStepName("Fetch Parameters")
+ .explanation("Failed to fetch parameters: " +
e.getMessage())
+ .build());
+ }
+ return results;
+ }
+
+ private String getParameterValue(final String rawValue, final
ParameterValueEncoding encoding) {
+ if (ParameterValueEncoding.BASE64 == encoding) {
+ return new
String(Base64.getDecoder().decode(rawValue.getBytes(DEFAULT_CHARSET)),
DEFAULT_CHARSET);
+ }
+ return rawValue;
+ }
+
+ private Collection<File> getDirectories(final ConfigurationContext
context, final PropertyDescriptor descriptor) {
+ return context.getProperty(descriptor).isSet()
+ ?
Arrays.stream(context.getProperty(descriptor).getValue().split(",")).map(String::trim).map(File::new).collect(Collectors.toSet())
+ : Collections.emptySet();
+ }
+
+ private ParameterGroup getParameterGroup(final ConfigurationContext
context, final File directory, final String groupName) {
+ final int parameterSizeLimit =
context.getProperty(PARAMETER_VALUE_BYTE_LIMIT).asDataSize(DataUnit.B).intValue();
+ final ParameterValueEncoding parameterEncoding =
ParameterValueEncoding.valueOf(context.getProperty(PARAMETER_VALUE_ENCODING).getValue().toUpperCase());
+
+ final File[] files = directory.listFiles();
+
+ final List<Parameter> parameters = new ArrayList<>();
+ for (final File file : files) {
+ if (file.isDirectory() || file.isHidden()) {
+ continue;
+ }
+
+ final String parameterName = file.getName();
+
+ try (final InputStream in = new BufferedInputStream(new
LimitingInputStream(new FileInputStream(file), parameterSizeLimit))) {
+ final String rawValue = IOUtils.toString(in,
Charset.defaultCharset()).trim();
+ final String parameterValue = getParameterValue(rawValue,
parameterEncoding);
+
+ if (parameterValue.length() >= parameterSizeLimit) {
+ getLogger().warn("Parameter {} may be truncated at {}
bytes", parameterName, parameterValue.length());
+ }
+
+ final ParameterDescriptor parameterDescriptor = new
ParameterDescriptor.Builder().name(parameterName).build();
+ parameters.add(new Parameter(parameterDescriptor,
parameterValue, null, true));
+ } catch (final IOException e) {
+ getLogger().error("Failed to read file [{}]", new Object[] {
file }, e);
Review Comment:
Is it necessary to log an error and throw an exception, or can this log be
removed?
```suggestion
getLogger().error("Failed to read file [{}]", file, e);
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-parameter-providers/src/test/java/org/apache/nifi/parameter/TestEnvironmentVariableParameterProvider.java:
##########
@@ -0,0 +1,96 @@
+package org.apache.nifi.parameter;/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.MockComponentLog;
+import org.apache.nifi.util.MockConfigurationContext;
+import org.apache.nifi.util.MockParameterProviderInitializationContext;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThrows;
Review Comment:
As a new test, this should be switched to JUnit 5
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/parameter/mock/DummyParameterProvider.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parameter.mock;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.parameter.AbstractParameterProvider;
+import org.apache.nifi.parameter.Parameter;
+import org.apache.nifi.parameter.ParameterDescriptor;
+import org.apache.nifi.parameter.ParameterGroup;
+import org.apache.nifi.parameter.ParameterProvider;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class DummyParameterProvider extends AbstractParameterProvider
implements ParameterProvider {
Review Comment:
Recommend renaming to `PlaceholderParameterProvider` or some other option.
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedFlowSynchronizer.java:
##########
@@ -195,6 +199,14 @@ public synchronized void sync(final FlowController
controller, final DataFlow pr
verifyNoConnectionsWithDataRemoved(existingDataFlow, proposedFlow,
controller, flowComparison);
synchronizeFlow(controller, existingDataFlow, proposedFlow,
affectedComponents);
+
+ for (final ParameterProviderNode parameterProviderNode :
flowManager.getAllParameterProviders()) {
+ try {
+ parameterProviderNode.fetchParameters();
+ } catch (final Exception e) {
+ logger.warn("Failed to fetch parameters for provider {}:
{}", new Object[] { parameterProviderNode.getName(), e.getMessage() }, e);
Review Comment:
```suggestion
logger.warn("Failed to fetch parameters for provider {}:
{}", parameterProviderNode.getName(), e.getMessage(), e);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
