lordgamez commented on code in PR #1895:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1895#discussion_r1915173455
##########
CONFIGURE.md:
##########
@@ -204,46 +206,170 @@ An example for using parameters in a JSON configuration
file:
An example for using parameters in a YAML configuration file:
```yaml
- MiNiFi Config Version: 3
- Flow Controller:
- name: MiNiFi Flow
- Parameter Contexts:
- - id: 235e6b47-ea22-45cd-a472-545801db98e6
- name: common-parameter-context
- description: Common parameter context
- Parameters:
- - name: common_timeout
- description: 'Common timeout seconds'
- sensitive: false
- value: 30
- - id: 804e6b47-ea22-45cd-a472-545801db98e6
- name: root-process-group-context
- description: Root process group parameter context
- Parameters:
- - name: tail_base_dir
- description: 'Base dir of tailed files'
- sensitive: false
- value: /tmp/tail/file/path
- Inherited Parameter Contexts:
- - common-parameter-context
- Processors:
- - name: Tail test_file1.log
- id: 83b58f9f-e661-4634-96fb-0e82b92becdf
- class: org.apache.nifi.minifi.processors.TailFile
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1000 ms
- Properties:
- File to Tail: "#{tail_base_dir}/test_file1.log"
- - name: Tail test_file2.log
- id: 8a772a10-7c34-48e7-b152-b1a32c5db83e
- class: org.apache.nifi.minifi.processors.TailFile
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1000 ms
- Properties:
- File to Tail: "#{tail_base_dir}/test_file2.log"
- Parameter Context Name: root-process-group-context
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Parameter Contexts:
+ - id: 235e6b47-ea22-45cd-a472-545801db98e6
+ name: common-parameter-context
+ description: Common parameter context
+ Parameters:
+ - name: common_timeout
+ description: 'Common timeout seconds'
+ sensitive: false
+ value: 30
+ - id: 804e6b47-ea22-45cd-a472-545801db98e6
+ name: root-process-group-context
+ description: Root process group parameter context
+ Parameters:
+ - name: tail_base_dir
+ description: 'Base dir of tailed files'
+ sensitive: false
+ value: /tmp/tail/file/path
+ Inherited Parameter Contexts:
+ - common-parameter-context
+Processors:
+- name: Tail test_file1.log
+ id: 83b58f9f-e661-4634-96fb-0e82b92becdf
+ class: org.apache.nifi.minifi.processors.TailFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1000 ms
+ Properties:
+ File to Tail: "#{tail_base_dir}/test_file1.log"
+- name: Tail test_file2.log
+ id: 8a772a10-7c34-48e7-b152-b1a32c5db83e
+ class: org.apache.nifi.minifi.processors.TailFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1000 ms
+ Properties:
+ File to Tail: "#{tail_base_dir}/test_file2.log"
+Parameter Context Name: root-process-group-context
+```
+
+### Parameter Providers
+
+Parameter contexts can be generated by Parameter Providers. Parameter
Providers can be added to the flow configuration, after which parameter
contexts and parameters generated by these providers can be referenced in the
properties. The parameter contexts generated are persisted in the flow
configuration file and are only regenerated on MiNiFi C++ restart if the
context is removed from the flow configuration. Other parameter contexts can be
also inherited from provider generated parameter contexts.
+
+There are two properties that can be set for all parameter providers for
selecting which properties should be sensitive parameters:
+
+- `Sensitive Parameter Scope`: This property can be set to `none`, `selected`
or `all`. If set to `All`, all parameters generated by the provider will be
marked as sensitive. If set to `none`, all parameters generated by the provider
will be marked as non-sensitive. If set to `selected`, the `Sensitive Parameter
List` property should be set to a list of parameter names that should be marked
as sensitive.
+- `Sensitive Parameter List`: This property should be set to a comma-separated
list of parameter names that should be marked as sensitive. This property is
only used if the `Sensitive Parameter Scope` property is set to `selected`.
+
+An example for using parameter providers in a JSON configuration file:
+
+```json
+{
+ "parameterProviders": [
+ {
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "EnvironmentVariableParameterProvider",
+ "type": "EnvironmentVariableParameterProvider",
+ "properties": {
+ "Parameter Group Name":
"environment-variable-parameter-context",
+ "Environment Variable Inclusion Strategy": "Regular
Expression",
+ "Include Environment Variables": "INPUT_.*"
+ }
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "identifier": "00000000-0000-0000-0000-000000000001",
+ "name": "MyProcessor",
+ "type": "org.apache.nifi.processors.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "3 sec",
+ "properties": {
+ "Input Directory": "#{INPUT_DIR}"
+ }
+ }
+ ],
+ "parameterContextName": "environment-variable-parameter-context"
+ }
+}
+```
+
+The same example in YAML configuration file:
+
+```yaml
+MiNiFi Config Version: 3
+Flow Controller:
+name: MiNiFi Flow
+Parameter Providers:
+ - id: d26ee5f5-0192-1000-0482-4e333725e089
+ name: EnvironmentVariableParameterProvider
+ type: EnvironmentVariableParameterProvider
+ Properties:
+ Parameter Group Name: environment-variable-parameter-context
+ Environment Variable Inclusion Strategy: Regular Expression
+ Include Environment Variables: INPUT_.*
Review Comment:
Updated in
https://github.com/apache/nifi-minifi-cpp/pull/1895/commits/59232bff0daaff05f1416b7ab45db3545fc8d651
##########
CONFIGURE.md:
##########
@@ -204,46 +206,170 @@ An example for using parameters in a JSON configuration
file:
An example for using parameters in a YAML configuration file:
```yaml
- MiNiFi Config Version: 3
- Flow Controller:
- name: MiNiFi Flow
- Parameter Contexts:
- - id: 235e6b47-ea22-45cd-a472-545801db98e6
- name: common-parameter-context
- description: Common parameter context
- Parameters:
- - name: common_timeout
- description: 'Common timeout seconds'
- sensitive: false
- value: 30
- - id: 804e6b47-ea22-45cd-a472-545801db98e6
- name: root-process-group-context
- description: Root process group parameter context
- Parameters:
- - name: tail_base_dir
- description: 'Base dir of tailed files'
- sensitive: false
- value: /tmp/tail/file/path
- Inherited Parameter Contexts:
- - common-parameter-context
- Processors:
- - name: Tail test_file1.log
- id: 83b58f9f-e661-4634-96fb-0e82b92becdf
- class: org.apache.nifi.minifi.processors.TailFile
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1000 ms
- Properties:
- File to Tail: "#{tail_base_dir}/test_file1.log"
- - name: Tail test_file2.log
- id: 8a772a10-7c34-48e7-b152-b1a32c5db83e
- class: org.apache.nifi.minifi.processors.TailFile
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1000 ms
- Properties:
- File to Tail: "#{tail_base_dir}/test_file2.log"
- Parameter Context Name: root-process-group-context
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Parameter Contexts:
+ - id: 235e6b47-ea22-45cd-a472-545801db98e6
+ name: common-parameter-context
+ description: Common parameter context
+ Parameters:
+ - name: common_timeout
+ description: 'Common timeout seconds'
+ sensitive: false
+ value: 30
+ - id: 804e6b47-ea22-45cd-a472-545801db98e6
+ name: root-process-group-context
+ description: Root process group parameter context
+ Parameters:
+ - name: tail_base_dir
+ description: 'Base dir of tailed files'
+ sensitive: false
+ value: /tmp/tail/file/path
+ Inherited Parameter Contexts:
+ - common-parameter-context
+Processors:
+- name: Tail test_file1.log
+ id: 83b58f9f-e661-4634-96fb-0e82b92becdf
+ class: org.apache.nifi.minifi.processors.TailFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1000 ms
+ Properties:
+ File to Tail: "#{tail_base_dir}/test_file1.log"
+- name: Tail test_file2.log
+ id: 8a772a10-7c34-48e7-b152-b1a32c5db83e
+ class: org.apache.nifi.minifi.processors.TailFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1000 ms
+ Properties:
+ File to Tail: "#{tail_base_dir}/test_file2.log"
+Parameter Context Name: root-process-group-context
+```
+
+### Parameter Providers
+
+Parameter contexts can be generated by Parameter Providers. Parameter
Providers can be added to the flow configuration, after which parameter
contexts and parameters generated by these providers can be referenced in the
properties. The parameter contexts generated are persisted in the flow
configuration file and are only regenerated on MiNiFi C++ restart if the
context is removed from the flow configuration. Other parameter contexts can be
also inherited from provider generated parameter contexts.
+
+There are two properties that can be set for all parameter providers for
selecting which properties should be sensitive parameters:
+
+- `Sensitive Parameter Scope`: This property can be set to `none`, `selected`
or `all`. If set to `All`, all parameters generated by the provider will be
marked as sensitive. If set to `none`, all parameters generated by the provider
will be marked as non-sensitive. If set to `selected`, the `Sensitive Parameter
List` property should be set to a list of parameter names that should be marked
as sensitive.
+- `Sensitive Parameter List`: This property should be set to a comma-separated
list of parameter names that should be marked as sensitive. This property is
only used if the `Sensitive Parameter Scope` property is set to `selected`.
+
+An example for using parameter providers in a JSON configuration file:
+
+```json
+{
+ "parameterProviders": [
+ {
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "EnvironmentVariableParameterProvider",
+ "type": "EnvironmentVariableParameterProvider",
+ "properties": {
+ "Parameter Group Name":
"environment-variable-parameter-context",
+ "Environment Variable Inclusion Strategy": "Regular
Expression",
+ "Include Environment Variables": "INPUT_.*"
+ }
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "identifier": "00000000-0000-0000-0000-000000000001",
+ "name": "MyProcessor",
+ "type": "org.apache.nifi.processors.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "3 sec",
+ "properties": {
+ "Input Directory": "#{INPUT_DIR}"
+ }
+ }
+ ],
+ "parameterContextName": "environment-variable-parameter-context"
+ }
+}
+```
+
+The same example in YAML configuration file:
+
+```yaml
+MiNiFi Config Version: 3
+Flow Controller:
+name: MiNiFi Flow
+Parameter Providers:
+ - id: d26ee5f5-0192-1000-0482-4e333725e089
+ name: EnvironmentVariableParameterProvider
+ type: EnvironmentVariableParameterProvider
+ Properties:
+ Parameter Group Name: environment-variable-parameter-context
+ Environment Variable Inclusion Strategy: Regular Expression
+ Include Environment Variables: INPUT_.*
+Processors:
+ - name: MyProcessor
+ id: 00000000-0000-0000-0000-000000000001
+ class: org.apache.nifi.processors.GetFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 3 sec
+ Properties:
+ Input Directory: "#{INPUT_DIR}"
+Parameter Context Name: environment-variable-parameter-context
```
+In the above example, the `EnvironmentVariableParameterProvider` is used to
generate a parameter context with the name
`environment-variable-parameter-context` that includes all environment
variables starting with `INPUT_`. The generated parameter context is assigned
to the root process group and the `INPUT_DIR` environment variable is used in
the `Input Directory` property of the `MyProcessor` processor which is a
generated parameter in the `environment-variable-parameter-context` parameter
context.
+
+After the parameter contexts are generated successfully, the parameter
contexts are persisted in the flow configuration file, which looks like this
for the above example:
Review Comment:
Actually this is the general parameter provider behavior, not environment
variable parameter provider specific. Before the implementation we discussed
what would the best behavior, with the options that we could either regenerate
the context on every agent restart, only regenerate the contexts if they are
not present in the flow configuration, or have an option in the parameter
providers to regenerate the contexts or not on restart. At that point we
decided that we should only generate the contexts if they are not present,
otherwise the user should delete them from the flow configuration if they want
it regenerated. If you prefer we can reopen the discussion about this and
adjust the behavior accordingly.
##########
PARAMETER_PROVIDERS.md:
##########
Review Comment:
Created a Jira ticket for this issue:
https://issues.apache.org/jira/browse/MINIFICPP-2508
##########
libminifi/src/core/FlowConfiguration.cpp:
##########
@@ -179,4 +181,22 @@ std::shared_ptr<core::controller::ControllerServiceNode>
FlowConfiguration::crea
return controllerServicesNode;
}
+std::unique_ptr<core::ParameterProvider>
FlowConfiguration::createParameterProvider(const std::string &class_name, const
std::string &full_class_name, const utils::Identifier& uuid) {
+ auto ptr =
core::ClassLoader::getDefaultClassLoader().instantiate(class_name, uuid);
+ if (ptr == nullptr) {
+ ptr =
core::ClassLoader::getDefaultClassLoader().instantiate(full_class_name, uuid);
+ }
+ if (ptr == nullptr) {
+ return nullptr;
+ }
+
+ auto returnPtr =
utils::dynamic_unique_cast<core::ParameterProvider>(std::move(ptr));
+ if (!returnPtr) {
+ throw std::runtime_error("Invalid return from the classloader");
Review Comment:
Updated in
https://github.com/apache/nifi-minifi-cpp/pull/1895/commits/59232bff0daaff05f1416b7ab45db3545fc8d651
##########
libminifi/src/core/ParameterProvider.cpp:
##########
@@ -0,0 +1,71 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "core/ParameterProvider.h"
+
+namespace org::apache::nifi::minifi::core {
+
+ParameterProviderConfig ParameterProvider::readParameterProviderConfig() const
{
+ ParameterProviderConfig config;
+
+ auto sensitive_parameter_scope_str = getProperty(SensitiveParameterScope);
+ if (!sensitive_parameter_scope_str) {
+ throw ParameterException("Sensitive Parameter Scope is required");
+ }
+ auto sensitive_parameter_scope =
magic_enum::enum_cast<SensitiveParameterScopeOptions>(*sensitive_parameter_scope_str);
+ if (!sensitive_parameter_scope) {
+ throw ParameterException("Sensitive Parameter Scope has invalid value: '"
+ *sensitive_parameter_scope_str + "'");
+ }
+ config.sensitive_parameter_scope = sensitive_parameter_scope.value();
+
+ if (config.sensitive_parameter_scope ==
SensitiveParameterScopeOptions::selected) {
+ if (auto sensitive_parameter_list = getProperty(SensitiveParameterList)) {
Review Comment:
Good point, updated in
https://github.com/apache/nifi-minifi-cpp/pull/1895/commits/59232bff0daaff05f1416b7ab45db3545fc8d651
##########
libminifi/src/core/ParameterProvider.cpp:
##########
@@ -0,0 +1,71 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "core/ParameterProvider.h"
+
+namespace org::apache::nifi::minifi::core {
+
+ParameterProviderConfig ParameterProvider::readParameterProviderConfig() const
{
+ ParameterProviderConfig config;
+
+ auto sensitive_parameter_scope_str = getProperty(SensitiveParameterScope);
+ if (!sensitive_parameter_scope_str) {
+ throw ParameterException("Sensitive Parameter Scope is required");
+ }
+ auto sensitive_parameter_scope =
magic_enum::enum_cast<SensitiveParameterScopeOptions>(*sensitive_parameter_scope_str);
+ if (!sensitive_parameter_scope) {
+ throw ParameterException("Sensitive Parameter Scope has invalid value: '"
+ *sensitive_parameter_scope_str + "'");
+ }
+ config.sensitive_parameter_scope = sensitive_parameter_scope.value();
+
+ if (config.sensitive_parameter_scope ==
SensitiveParameterScopeOptions::selected) {
+ if (auto sensitive_parameter_list = getProperty(SensitiveParameterList)) {
+ for (const auto& sensitive_parameter :
minifi::utils::string::split(*sensitive_parameter_list, ",")) {
Review Comment:
Updated in
https://github.com/apache/nifi-minifi-cpp/pull/1895/commits/59232bff0daaff05f1416b7ab45db3545fc8d651
##########
libminifi/src/core/ParameterProvider.cpp:
##########
@@ -0,0 +1,71 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "core/ParameterProvider.h"
+
+namespace org::apache::nifi::minifi::core {
+
+ParameterProviderConfig ParameterProvider::readParameterProviderConfig() const
{
+ ParameterProviderConfig config;
+
+ auto sensitive_parameter_scope_str = getProperty(SensitiveParameterScope);
+ if (!sensitive_parameter_scope_str) {
+ throw ParameterException("Sensitive Parameter Scope is required");
+ }
+ auto sensitive_parameter_scope =
magic_enum::enum_cast<SensitiveParameterScopeOptions>(*sensitive_parameter_scope_str);
+ if (!sensitive_parameter_scope) {
+ throw ParameterException("Sensitive Parameter Scope has invalid value: '"
+ *sensitive_parameter_scope_str + "'");
+ }
+ config.sensitive_parameter_scope = sensitive_parameter_scope.value();
+
+ if (config.sensitive_parameter_scope ==
SensitiveParameterScopeOptions::selected) {
+ if (auto sensitive_parameter_list = getProperty(SensitiveParameterList)) {
+ for (const auto& sensitive_parameter :
minifi::utils::string::split(*sensitive_parameter_list, ",")) {
+ config.sensitive_parameters.insert(sensitive_parameter);
+ }
+ }
+ }
+
+ return config;
+}
+
+std::vector<gsl::not_null<std::unique_ptr<ParameterContext>>>
ParameterProvider::createParameterContexts() {
+ auto config = readParameterProviderConfig();
+
+ auto parameter_groups = buildParameterGroups();
+ std::vector<gsl::not_null<std::unique_ptr<ParameterContext>>> result;
+ for (const auto& parameter_group : parameter_groups) {
+ auto parameter_context =
std::make_unique<ParameterContext>(parameter_group.name);
+ parameter_context->setParameterProvider(getUUIDStr());
+ for (const auto& [name, value] : parameter_group.parameters) {
+ bool sensitive = config.sensitive_parameter_scope ==
SensitiveParameterScopeOptions::all ||
+ (config.sensitive_parameter_scope ==
SensitiveParameterScopeOptions::selected &&
config.sensitive_parameters.contains(name));
+ parameter_context->addParameter(Parameter{
+ .name = name,
+ .description = "",
+ .sensitive = sensitive,
+ .provided = true,
+ .value = value
+ });
Review Comment:
Good idea, updated in
https://github.com/apache/nifi-minifi-cpp/pull/1895/commits/59232bff0daaff05f1416b7ab45db3545fc8d651
##########
libminifi/src/core/flow/StructuredConfiguration.cpp:
##########
@@ -191,23 +192,80 @@ void
StructuredConfiguration::parseParameterContexts(const Node& parameter_conte
uuid = id;
auto parameter_context = std::make_unique<ParameterContext>(name, uuid);
parameter_context->setDescription(getOptionalField(parameter_context_node,
schema_.description, ""));
+
parameter_context->setParameterProvider(getOptionalField(parameter_context_node,
schema_.parameter_provider, ""));
for (const auto& parameter_node :
parameter_context_node[schema_.parameters]) {
checkRequiredField(parameter_node, schema_.name);
checkRequiredField(parameter_node, schema_.value);
checkRequiredField(parameter_node, schema_.sensitive);
auto parameter_name = parameter_node[schema_.name].getString().value();
auto parameter_value = parameter_node[schema_.value].getString().value();
auto sensitive = parameter_node[schema_.sensitive].getBool().value();
+ auto provided =
parameter_node[schema_.provided].getBool().value_or(false);
auto parameter_description = getOptionalField(parameter_node,
schema_.description, "");
if (sensitive) {
parameter_value =
utils::crypto::property_encryption::decrypt(parameter_value,
sensitive_values_encryptor_);
}
- parameter_context->addParameter(Parameter{parameter_name,
parameter_description, sensitive, parameter_value});
+ parameter_context->addParameter(Parameter{
+ .name = parameter_name,
+ .description = parameter_description,
+ .sensitive = sensitive,
+ .provided = provided,
+ .value = parameter_value});
}
parameter_contexts_.emplace(name,
gsl::make_not_null(std::move(parameter_context)));
}
+}
+
+void StructuredConfiguration::parseParameterProvidersNode(const Node&
parameter_providers_node) {
+ if (!parameter_providers_node || !parameter_providers_node.isSequence()) {
Review Comment:
I would keep it this way to be consistent with the handling of the other
sequence nodes like the controller services. Maybe we can add a throw for this
issue for all of them later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
