fgerlits commented on code in PR #1895:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1895#discussion_r1910224295
##########
CONFIGURE.md:
##########
@@ -204,46 +206,170 @@ An example for using parameters in a JSON configuration
file:
An example for using parameters in a YAML configuration file:
```yaml
- MiNiFi Config Version: 3
- Flow Controller:
- name: MiNiFi Flow
- Parameter Contexts:
- - id: 235e6b47-ea22-45cd-a472-545801db98e6
- name: common-parameter-context
- description: Common parameter context
- Parameters:
- - name: common_timeout
- description: 'Common timeout seconds'
- sensitive: false
- value: 30
- - id: 804e6b47-ea22-45cd-a472-545801db98e6
- name: root-process-group-context
- description: Root process group parameter context
- Parameters:
- - name: tail_base_dir
- description: 'Base dir of tailed files'
- sensitive: false
- value: /tmp/tail/file/path
- Inherited Parameter Contexts:
- - common-parameter-context
- Processors:
- - name: Tail test_file1.log
- id: 83b58f9f-e661-4634-96fb-0e82b92becdf
- class: org.apache.nifi.minifi.processors.TailFile
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1000 ms
- Properties:
- File to Tail: "#{tail_base_dir}/test_file1.log"
- - name: Tail test_file2.log
- id: 8a772a10-7c34-48e7-b152-b1a32c5db83e
- class: org.apache.nifi.minifi.processors.TailFile
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1000 ms
- Properties:
- File to Tail: "#{tail_base_dir}/test_file2.log"
- Parameter Context Name: root-process-group-context
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Parameter Contexts:
+ - id: 235e6b47-ea22-45cd-a472-545801db98e6
+ name: common-parameter-context
+ description: Common parameter context
+ Parameters:
+ - name: common_timeout
+ description: 'Common timeout seconds'
+ sensitive: false
+ value: 30
+ - id: 804e6b47-ea22-45cd-a472-545801db98e6
+ name: root-process-group-context
+ description: Root process group parameter context
+ Parameters:
+ - name: tail_base_dir
+ description: 'Base dir of tailed files'
+ sensitive: false
+ value: /tmp/tail/file/path
+ Inherited Parameter Contexts:
+ - common-parameter-context
+Processors:
+- name: Tail test_file1.log
+ id: 83b58f9f-e661-4634-96fb-0e82b92becdf
+ class: org.apache.nifi.minifi.processors.TailFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1000 ms
+ Properties:
+ File to Tail: "#{tail_base_dir}/test_file1.log"
+- name: Tail test_file2.log
+ id: 8a772a10-7c34-48e7-b152-b1a32c5db83e
+ class: org.apache.nifi.minifi.processors.TailFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1000 ms
+ Properties:
+ File to Tail: "#{tail_base_dir}/test_file2.log"
+Parameter Context Name: root-process-group-context
+```
+
+### Parameter Providers
+
+Parameter contexts can be generated by Parameter Providers. Parameter
Providers can be added to the flow configuration, after which parameter
contexts and parameters generated by these providers can be referenced in the
properties. The parameter contexts generated are persisted in the flow
configuration file and are only regenerated on MiNiFi C++ restart if the
context is removed from the flow configuration. Other parameter contexts can be
also inherited from provider generated parameter contexts.
+
+There are two properties that can be set for all parameter providers for
selecting which properties should be sensitive parameters:
+
+- `Sensitive Parameter Scope`: This property can be set to `none`, `selected`
or `all`. If set to `All`, all parameters generated by the provider will be
marked as sensitive. If set to `none`, all parameters generated by the provider
will be marked as non-sensitive. If set to `selected`, the `Sensitive Parameter
List` property should be set to a list of parameter names that should be marked
as sensitive.
+- `Sensitive Parameter List`: This property should be set to a comma-separated
list of parameter names that should be marked as sensitive. This property is
only used if the `Sensitive Parameter Scope` property is set to `selected`.
+
+An example for using parameter providers in a JSON configuration file:
+
+```json
+{
+ "parameterProviders": [
+ {
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "EnvironmentVariableParameterProvider",
+ "type": "EnvironmentVariableParameterProvider",
+ "properties": {
+ "Parameter Group Name":
"environment-variable-parameter-context",
+ "Environment Variable Inclusion Strategy": "Regular
Expression",
+ "Include Environment Variables": "INPUT_.*"
+ }
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "identifier": "00000000-0000-0000-0000-000000000001",
+ "name": "MyProcessor",
+ "type": "org.apache.nifi.processors.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "3 sec",
+ "properties": {
+ "Input Directory": "#{INPUT_DIR}"
+ }
+ }
+ ],
+ "parameterContextName": "environment-variable-parameter-context"
+ }
+}
+```
+
+The same example in YAML configuration file:
+
+```yaml
+MiNiFi Config Version: 3
+Flow Controller:
+name: MiNiFi Flow
+Parameter Providers:
+ - id: d26ee5f5-0192-1000-0482-4e333725e089
+ name: EnvironmentVariableParameterProvider
+ type: EnvironmentVariableParameterProvider
+ Properties:
+ Parameter Group Name: environment-variable-parameter-context
+ Environment Variable Inclusion Strategy: Regular Expression
+ Include Environment Variables: INPUT_.*
Review Comment:
these need to be indented:
```suggestion
Properties:
Parameter Group Name: environment-variable-parameter-context
Environment Variable Inclusion Strategy: Regular Expression
Include Environment Variables: INPUT_.*
```
(also in line 314 below)
##########
CONFIGURE.md:
##########
@@ -204,46 +206,170 @@ An example for using parameters in a JSON configuration
file:
An example for using parameters in a YAML configuration file:
```yaml
- MiNiFi Config Version: 3
- Flow Controller:
- name: MiNiFi Flow
- Parameter Contexts:
- - id: 235e6b47-ea22-45cd-a472-545801db98e6
- name: common-parameter-context
- description: Common parameter context
- Parameters:
- - name: common_timeout
- description: 'Common timeout seconds'
- sensitive: false
- value: 30
- - id: 804e6b47-ea22-45cd-a472-545801db98e6
- name: root-process-group-context
- description: Root process group parameter context
- Parameters:
- - name: tail_base_dir
- description: 'Base dir of tailed files'
- sensitive: false
- value: /tmp/tail/file/path
- Inherited Parameter Contexts:
- - common-parameter-context
- Processors:
- - name: Tail test_file1.log
- id: 83b58f9f-e661-4634-96fb-0e82b92becdf
- class: org.apache.nifi.minifi.processors.TailFile
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1000 ms
- Properties:
- File to Tail: "#{tail_base_dir}/test_file1.log"
- - name: Tail test_file2.log
- id: 8a772a10-7c34-48e7-b152-b1a32c5db83e
- class: org.apache.nifi.minifi.processors.TailFile
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1000 ms
- Properties:
- File to Tail: "#{tail_base_dir}/test_file2.log"
- Parameter Context Name: root-process-group-context
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Parameter Contexts:
+ - id: 235e6b47-ea22-45cd-a472-545801db98e6
+ name: common-parameter-context
+ description: Common parameter context
+ Parameters:
+ - name: common_timeout
+ description: 'Common timeout seconds'
+ sensitive: false
+ value: 30
+ - id: 804e6b47-ea22-45cd-a472-545801db98e6
+ name: root-process-group-context
+ description: Root process group parameter context
+ Parameters:
+ - name: tail_base_dir
+ description: 'Base dir of tailed files'
+ sensitive: false
+ value: /tmp/tail/file/path
+ Inherited Parameter Contexts:
+ - common-parameter-context
+Processors:
+- name: Tail test_file1.log
+ id: 83b58f9f-e661-4634-96fb-0e82b92becdf
+ class: org.apache.nifi.minifi.processors.TailFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1000 ms
+ Properties:
+ File to Tail: "#{tail_base_dir}/test_file1.log"
+- name: Tail test_file2.log
+ id: 8a772a10-7c34-48e7-b152-b1a32c5db83e
+ class: org.apache.nifi.minifi.processors.TailFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1000 ms
+ Properties:
+ File to Tail: "#{tail_base_dir}/test_file2.log"
+Parameter Context Name: root-process-group-context
+```
+
+### Parameter Providers
+
+Parameter contexts can be generated by Parameter Providers. Parameter
Providers can be added to the flow configuration, after which parameter
contexts and parameters generated by these providers can be referenced in the
properties. The parameter contexts generated are persisted in the flow
configuration file and are only regenerated on MiNiFi C++ restart if the
context is removed from the flow configuration. Other parameter contexts can be
also inherited from provider generated parameter contexts.
+
+There are two properties that can be set for all parameter providers for
selecting which properties should be sensitive parameters:
+
+- `Sensitive Parameter Scope`: This property can be set to `none`, `selected`
or `all`. If set to `All`, all parameters generated by the provider will be
marked as sensitive. If set to `none`, all parameters generated by the provider
will be marked as non-sensitive. If set to `selected`, the `Sensitive Parameter
List` property should be set to a list of parameter names that should be marked
as sensitive.
+- `Sensitive Parameter List`: This property should be set to a comma-separated
list of parameter names that should be marked as sensitive. This property is
only used if the `Sensitive Parameter Scope` property is set to `selected`.
+
+An example for using parameter providers in a JSON configuration file:
+
+```json
+{
+ "parameterProviders": [
+ {
+ "identifier": "d26ee5f5-0192-1000-0482-4e333725e089",
+ "name": "EnvironmentVariableParameterProvider",
+ "type": "EnvironmentVariableParameterProvider",
+ "properties": {
+ "Parameter Group Name":
"environment-variable-parameter-context",
+ "Environment Variable Inclusion Strategy": "Regular
Expression",
+ "Include Environment Variables": "INPUT_.*"
+ }
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "identifier": "00000000-0000-0000-0000-000000000001",
+ "name": "MyProcessor",
+ "type": "org.apache.nifi.processors.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "3 sec",
+ "properties": {
+ "Input Directory": "#{INPUT_DIR}"
+ }
+ }
+ ],
+ "parameterContextName": "environment-variable-parameter-context"
+ }
+}
+```
+
+The same example in YAML configuration file:
+
+```yaml
+MiNiFi Config Version: 3
+Flow Controller:
+name: MiNiFi Flow
+Parameter Providers:
+ - id: d26ee5f5-0192-1000-0482-4e333725e089
+ name: EnvironmentVariableParameterProvider
+ type: EnvironmentVariableParameterProvider
+ Properties:
+ Parameter Group Name: environment-variable-parameter-context
+ Environment Variable Inclusion Strategy: Regular Expression
+ Include Environment Variables: INPUT_.*
+Processors:
+ - name: MyProcessor
+ id: 00000000-0000-0000-0000-000000000001
+ class: org.apache.nifi.processors.GetFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 3 sec
+ Properties:
+ Input Directory: "#{INPUT_DIR}"
+Parameter Context Name: environment-variable-parameter-context
```
+In the above example, the `EnvironmentVariableParameterProvider` is used to
generate a parameter context with the name
`environment-variable-parameter-context` that includes all environment
variables starting with `INPUT_`. The generated parameter context is assigned
to the root process group and the `INPUT_DIR` environment variable is used in
the `Input Directory` property of the `MyProcessor` processor which is a
generated parameter in the `environment-variable-parameter-context` parameter
context.
+
+After the parameter contexts are generated successfully, the parameter
contexts are persisted in the flow configuration file, which looks like this
for the above example:
Review Comment:
As a user, this would surprise me: I would have expected the environment
variable to be re-read on agent restart. What is the reason for doing it this
way?
##########
PARAMETER_PROVIDERS.md:
##########
Review Comment:
We can do this in a later PR, but it would be nice if
`PARAMETER_PROVIDERS.md` was auto-generated by `minifi --docs` like
`CONTROLLERS.md` and `PROCESSORS.md` are.
##########
libminifi/src/core/ParameterProvider.cpp:
##########
@@ -0,0 +1,71 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "core/ParameterProvider.h"
+
+namespace org::apache::nifi::minifi::core {
+
+ParameterProviderConfig ParameterProvider::readParameterProviderConfig() const
{
+ ParameterProviderConfig config;
+
+ auto sensitive_parameter_scope_str = getProperty(SensitiveParameterScope);
+ if (!sensitive_parameter_scope_str) {
+ throw ParameterException("Sensitive Parameter Scope is required");
+ }
+ auto sensitive_parameter_scope =
magic_enum::enum_cast<SensitiveParameterScopeOptions>(*sensitive_parameter_scope_str);
+ if (!sensitive_parameter_scope) {
+ throw ParameterException("Sensitive Parameter Scope has invalid value: '"
+ *sensitive_parameter_scope_str + "'");
+ }
+ config.sensitive_parameter_scope = sensitive_parameter_scope.value();
+
+ if (config.sensitive_parameter_scope ==
SensitiveParameterScopeOptions::selected) {
+ if (auto sensitive_parameter_list = getProperty(SensitiveParameterList)) {
+ for (const auto& sensitive_parameter :
minifi::utils::string::split(*sensitive_parameter_list, ",")) {
Review Comment:
we could use `splitAndTrim` or `splitAndTrimRemovingEmpty` instead of `split`
##########
libminifi/src/core/flow/StructuredConfiguration.cpp:
##########
@@ -191,23 +192,80 @@ void
StructuredConfiguration::parseParameterContexts(const Node& parameter_conte
uuid = id;
auto parameter_context = std::make_unique<ParameterContext>(name, uuid);
parameter_context->setDescription(getOptionalField(parameter_context_node,
schema_.description, ""));
+
parameter_context->setParameterProvider(getOptionalField(parameter_context_node,
schema_.parameter_provider, ""));
for (const auto& parameter_node :
parameter_context_node[schema_.parameters]) {
checkRequiredField(parameter_node, schema_.name);
checkRequiredField(parameter_node, schema_.value);
checkRequiredField(parameter_node, schema_.sensitive);
auto parameter_name = parameter_node[schema_.name].getString().value();
auto parameter_value = parameter_node[schema_.value].getString().value();
auto sensitive = parameter_node[schema_.sensitive].getBool().value();
+ auto provided =
parameter_node[schema_.provided].getBool().value_or(false);
auto parameter_description = getOptionalField(parameter_node,
schema_.description, "");
if (sensitive) {
parameter_value =
utils::crypto::property_encryption::decrypt(parameter_value,
sensitive_values_encryptor_);
}
- parameter_context->addParameter(Parameter{parameter_name,
parameter_description, sensitive, parameter_value});
+ parameter_context->addParameter(Parameter{
+ .name = parameter_name,
+ .description = parameter_description,
+ .sensitive = sensitive,
+ .provided = provided,
+ .value = parameter_value});
}
parameter_contexts_.emplace(name,
gsl::make_not_null(std::move(parameter_context)));
}
+}
+
+void StructuredConfiguration::parseParameterProvidersNode(const Node&
parameter_providers_node) {
+ if (!parameter_providers_node || !parameter_providers_node.isSequence()) {
Review Comment:
`!parameter_providers_node.isSequence()` sounds like a bad error; should we
throw? Or are we confident it will never happen?
##########
libminifi/test/unit/JsonFlowSerializerTests.cpp:
##########
@@ -731,19 +732,173 @@ TEST_CASE("The encrypted flow configuration cannot be
decrypted with an incorrec
core::flow::AdaptiveConfiguration
json_configuration_before{configuration_context};
const auto schema = core::flow::FlowSchema::getNiFiFlowJson();
- const auto config_json_with_nifi_schema =
utils::string::join_pack(config_json_with_nifi_schema_part_1,
config_json_with_nifi_schema_part_2);
+ const auto config_json_with_nifi_schema =
minifi::utils::string::join_pack(config_json_with_nifi_schema_part_1,
config_json_with_nifi_schema_part_2);
const auto process_group_before =
json_configuration_before.getRootFromPayload(config_json_with_nifi_schema);
REQUIRE(process_group_before);
rapidjson::Document doc;
rapidjson::ParseResult res = doc.Parse(config_json_with_nifi_schema.data(),
config_json_with_nifi_schema.size());
REQUIRE(res);
const auto flow_serializer = core::json::JsonFlowSerializer{std::move(doc)};
- std::string config_json_encrypted =
flow_serializer.serialize(*process_group_before, schema, encryption_provider,
{});
+ std::string config_json_encrypted =
flow_serializer.serialize(*process_group_before, schema, encryption_provider,
{}, {});
- const utils::crypto::Bytes different_secret_key =
utils::string::from_hex("ea55b7d0edc22280c9547e4d89712b3fae74f96d82f240a004fb9fbd0640eec7");
- configuration_context.sensitive_values_encryptor =
utils::crypto::EncryptionProvider{different_secret_key};
+ const minifi::utils::crypto::Bytes different_secret_key =
minifi::utils::string::from_hex("ea55b7d0edc22280c9547e4d89712b3fae74f96d82f240a004fb9fbd0640eec7");
+ configuration_context.sensitive_values_encryptor =
minifi::utils::crypto::EncryptionProvider{different_secret_key};
core::flow::AdaptiveConfiguration
json_configuration_after{configuration_context};
-
REQUIRE_THROWS_AS(json_configuration_after.getRootFromPayload(config_json_encrypted),
utils::crypto::EncryptionError);
+
REQUIRE_THROWS_AS(json_configuration_after.getRootFromPayload(config_json_encrypted),
minifi::utils::crypto::EncryptionError);
}
+
+TEST_CASE("Parameter provider generated parameter context is serialized
correctly") {
Review Comment:
These two test cases are identical except for the value of
`config_json_with_nifi_schema`, aren't they? If they are, then I think it would
be better to merge them into one. This could be done with `GENERATE` or
`SECTION` or an ad-hoc helper function.
(Same comment about the YAML tests.)
##########
libminifi/src/parameter-providers/EnvironmentVariableParameterProvider.cpp:
##########
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "parameter-providers/EnvironmentVariableParameterProvider.h"
+#include "core/Resource.h"
+#include "utils/Environment.h"
+#include "Exception.h"
+#include "range/v3/view/filter.hpp"
+#include "utils/RegexUtils.h"
+
+namespace org::apache::nifi::minifi::parameter_providers {
+
+std::string EnvironmentVariableParameterProvider::readParameterGroupName()
const {
+ auto parameter_group_name = getProperty(ParameterGroupName);
+ if (!parameter_group_name || parameter_group_name.value().empty()) {
+ throw core::ParameterException("Parameter Group Name is required");
+ }
+ logger_->log_debug("Building Parameter Context with name: {}",
parameter_group_name.value());
+ return parameter_group_name.value();
+}
+
+EnvironmentVariableInclusionStrategyOptions
EnvironmentVariableParameterProvider::readEnvironmentVariableInclusionStrategy()
const {
+ auto env_variable_inclusion_strategy_str =
getProperty(EnvironmentVariableInclusionStrategy);
+ if (!env_variable_inclusion_strategy_str) {
+ throw core::ParameterException("Environment Variable Inclusion Strategy is
required");
+ }
+ auto env_variable_inclusion_strategy =
magic_enum::enum_cast<EnvironmentVariableInclusionStrategyOptions>(*env_variable_inclusion_strategy_str);
+ if (!env_variable_inclusion_strategy) {
+ throw core::ParameterException("Environment Variable Inclusion Strategy
has invalid value: '" + *env_variable_inclusion_strategy_str + "'");
+ }
+ logger_->log_debug("Environment Variable Inclusion Strategy: {}",
*env_variable_inclusion_strategy_str);
+ return env_variable_inclusion_strategy.value();
+}
+
+void
EnvironmentVariableParameterProvider::filterEnvironmentVariablesByCommaSeparatedList(std::unordered_map<std::string,
std::string>& environment_variables) const {
+ std::unordered_set<std::string> included_environment_variables;
+ if (auto incuded_environment_variables_str =
getProperty(IncludeEnvironmentVariables)) {
+ for (const auto& included_environment_variable :
minifi::utils::string::split(*incuded_environment_variables_str, ",")) {
Review Comment:
I would use `splitAndTrimRemovingEmpty` here, too.
##########
libminifi/src/core/ParameterProvider.cpp:
##########
@@ -0,0 +1,71 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "core/ParameterProvider.h"
+
+namespace org::apache::nifi::minifi::core {
+
+ParameterProviderConfig ParameterProvider::readParameterProviderConfig() const
{
+ ParameterProviderConfig config;
+
+ auto sensitive_parameter_scope_str = getProperty(SensitiveParameterScope);
+ if (!sensitive_parameter_scope_str) {
+ throw ParameterException("Sensitive Parameter Scope is required");
+ }
+ auto sensitive_parameter_scope =
magic_enum::enum_cast<SensitiveParameterScopeOptions>(*sensitive_parameter_scope_str);
+ if (!sensitive_parameter_scope) {
+ throw ParameterException("Sensitive Parameter Scope has invalid value: '"
+ *sensitive_parameter_scope_str + "'");
+ }
+ config.sensitive_parameter_scope = sensitive_parameter_scope.value();
+
+ if (config.sensitive_parameter_scope ==
SensitiveParameterScopeOptions::selected) {
+ if (auto sensitive_parameter_list = getProperty(SensitiveParameterList)) {
Review Comment:
We should probably throw if the parameter scope is 'selected' but 'Sensitive
Parameter List' is not given, as that is likely to be an error. (Also, the doc
string says "If 'selected' is chosen, the 'Sensitive Parameter List' property
must be set.")
##########
libminifi/src/core/FlowConfiguration.cpp:
##########
@@ -179,4 +181,22 @@ std::shared_ptr<core::controller::ControllerServiceNode>
FlowConfiguration::crea
return controllerServicesNode;
}
+std::unique_ptr<core::ParameterProvider>
FlowConfiguration::createParameterProvider(const std::string &class_name, const
std::string &full_class_name, const utils::Identifier& uuid) {
+ auto ptr =
core::ClassLoader::getDefaultClassLoader().instantiate(class_name, uuid);
+ if (ptr == nullptr) {
+ ptr =
core::ClassLoader::getDefaultClassLoader().instantiate(full_class_name, uuid);
+ }
+ if (ptr == nullptr) {
+ return nullptr;
+ }
+
+ auto returnPtr =
utils::dynamic_unique_cast<core::ParameterProvider>(std::move(ptr));
+ if (!returnPtr) {
+ throw std::runtime_error("Invalid return from the classloader");
Review Comment:
This error message could be improved, e.g. "Invalid parameter provider type:
{class_name} is not a subclass of ParameterProvider".
##########
libminifi/src/core/ParameterProvider.cpp:
##########
@@ -0,0 +1,71 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "core/ParameterProvider.h"
+
+namespace org::apache::nifi::minifi::core {
+
+ParameterProviderConfig ParameterProvider::readParameterProviderConfig() const
{
+ ParameterProviderConfig config;
+
+ auto sensitive_parameter_scope_str = getProperty(SensitiveParameterScope);
+ if (!sensitive_parameter_scope_str) {
+ throw ParameterException("Sensitive Parameter Scope is required");
+ }
+ auto sensitive_parameter_scope =
magic_enum::enum_cast<SensitiveParameterScopeOptions>(*sensitive_parameter_scope_str);
+ if (!sensitive_parameter_scope) {
+ throw ParameterException("Sensitive Parameter Scope has invalid value: '"
+ *sensitive_parameter_scope_str + "'");
+ }
+ config.sensitive_parameter_scope = sensitive_parameter_scope.value();
+
+ if (config.sensitive_parameter_scope ==
SensitiveParameterScopeOptions::selected) {
+ if (auto sensitive_parameter_list = getProperty(SensitiveParameterList)) {
+ for (const auto& sensitive_parameter :
minifi::utils::string::split(*sensitive_parameter_list, ",")) {
+ config.sensitive_parameters.insert(sensitive_parameter);
+ }
+ }
+ }
+
+ return config;
+}
+
+std::vector<gsl::not_null<std::unique_ptr<ParameterContext>>>
ParameterProvider::createParameterContexts() {
+ auto config = readParameterProviderConfig();
+
+ auto parameter_groups = buildParameterGroups();
+ std::vector<gsl::not_null<std::unique_ptr<ParameterContext>>> result;
+ for (const auto& parameter_group : parameter_groups) {
+ auto parameter_context =
std::make_unique<ParameterContext>(parameter_group.name);
+ parameter_context->setParameterProvider(getUUIDStr());
+ for (const auto& [name, value] : parameter_group.parameters) {
+ bool sensitive = config.sensitive_parameter_scope ==
SensitiveParameterScopeOptions::all ||
+ (config.sensitive_parameter_scope ==
SensitiveParameterScopeOptions::selected &&
config.sensitive_parameters.contains(name));
+ parameter_context->addParameter(Parameter{
+ .name = name,
+ .description = "",
+ .sensitive = sensitive,
+ .provided = true,
+ .value = value
+ });
Review Comment:
I would move the logic for determining if the property is sensitive to
inside the `ParameterProviderConfig` class:
```suggestion
parameter_context->addParameter(Parameter{
.name = name,
.description = "",
.sensitive = config.isSensitive(name),
.provided = true,
.value = value
});
```
##########
libminifi/src/core/flow/StructuredConfiguration.cpp:
##########
@@ -191,23 +192,80 @@ void
StructuredConfiguration::parseParameterContexts(const Node& parameter_conte
uuid = id;
auto parameter_context = std::make_unique<ParameterContext>(name, uuid);
parameter_context->setDescription(getOptionalField(parameter_context_node,
schema_.description, ""));
+
parameter_context->setParameterProvider(getOptionalField(parameter_context_node,
schema_.parameter_provider, ""));
for (const auto& parameter_node :
parameter_context_node[schema_.parameters]) {
checkRequiredField(parameter_node, schema_.name);
checkRequiredField(parameter_node, schema_.value);
checkRequiredField(parameter_node, schema_.sensitive);
auto parameter_name = parameter_node[schema_.name].getString().value();
auto parameter_value = parameter_node[schema_.value].getString().value();
auto sensitive = parameter_node[schema_.sensitive].getBool().value();
+ auto provided =
parameter_node[schema_.provided].getBool().value_or(false);
auto parameter_description = getOptionalField(parameter_node,
schema_.description, "");
if (sensitive) {
parameter_value =
utils::crypto::property_encryption::decrypt(parameter_value,
sensitive_values_encryptor_);
}
- parameter_context->addParameter(Parameter{parameter_name,
parameter_description, sensitive, parameter_value});
+ parameter_context->addParameter(Parameter{
+ .name = parameter_name,
+ .description = parameter_description,
+ .sensitive = sensitive,
+ .provided = provided,
+ .value = parameter_value});
}
parameter_contexts_.emplace(name,
gsl::make_not_null(std::move(parameter_context)));
}
+}
+
+void StructuredConfiguration::parseParameterProvidersNode(const Node&
parameter_providers_node) {
+ if (!parameter_providers_node || !parameter_providers_node.isSequence()) {
+ return;
+ }
+ for (const auto& parameter_provider_node : parameter_providers_node) {
+ checkRequiredField(parameter_provider_node, schema_.name);
+ auto type = getRequiredField(parameter_provider_node, schema_.type);
+ logger_->log_debug("Using type {} for parameter provider node", type);
+
+ std::string fullType = type;
+ auto lastOfIdx = type.find_last_of('.');
+ if (lastOfIdx != std::string::npos) {
+ lastOfIdx++; // if a value is found, increment to move beyond the .
+ type = type.substr(lastOfIdx);
+ }
Review Comment:
We do this 3 times in this file now; also there are almost identical
snippets in `AbstractProcessor.h` and `HTTPProtocol.cpp`; it may be time to
pull it out to a utility function, e.g.
`utils::string::partAfterLastOccurrenceOf`.
##########
libminifi/src/utils/Environment.cpp:
##########
@@ -27,9 +27,16 @@
#include <mutex>
#include <vector>
#include <iostream>
+#include <unordered_map>
#include "utils/gsl.h"
+// Apple doesn't provide the environ global variable
+#if defined(__APPLE__) && !defined(environ)
+#include <crt_externs.h>
+#define environ (*_NSGetEnviron())
Review Comment:
Does this have to be a macro, or could we do something like `static auto
environ = *_NSGetEnviron()`? (with `constexpr`, if possible)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
