[
https://issues.apache.org/jira/browse/MINIFICPP-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi updated MINIFICPP-1288:
------------------------------------
Description:
*Background:*
MiNiFi flows are represented by yaml configuration files. When these are
published to the FlowController in MiNiFi, it handles this as a “reconfigure
and restart”. The current logic is:
# Backup the old process group (in memory representation of the configuration).
# Stop the FlowController.
# Try to build up a new process group based on the new configuration.
# If we fail during the construction of the process group nodes we should keep
the backup, otherwise we let the new flow run and consider it validated.
This is not awful, but expects the checks implemented in terms of processor
construction-time checks and separated cheks on the connection network. [[The
implementation is also not very
robust]|https://github.com/apache/nifi-minifi-cpp/blob/rel/minifi-cpp-0.7.0/libminifi/src/core/ProcessGroup.cpp#L343-L359],
so there is not much extra value in keeping it.
*Proposal:*
We should try and validate the configuration files before trying to reload
using them. Not only would this make us fail earlier, but would have the checks
a bit more decoupled from the processors. We already send processor
configuratin requirement data via the heartbeat to EFM, so we have every piece
of information needed to statically validate the configuration files.
This way the new logic should simplify to:
# Validate the configuration
# If it is valid load it, otherwise load it and any subsequent error should be
considered runtime exception.
As this would potentially break configuration-file backward compatibilty, all
the changes should be implemented behind a feature flag (build-time).
was:
*Background:*
MiNiFi flows are represented by yaml configuration files. When these are
published to the FlowController in MiNiFi, it handles this as a “reconfigure
and restart”. The current logic is:
# Backup the old process group (in memory representation of the configuration).
# Stop the FlowController.
# Try to build up a new process group based on the new configuration.
# If we fail during the construction of the process group nodes we should keep
the backup, otherwise we let the new flow run and consider it validated.
This is not awful, but expects the checks implemented in terms of processor
construction-time checks and separated cheks on the connection network.[ [The
implementation is also not very
robust]|https://github.com/apache/nifi-minifi-cpp/blob/rel/minifi-cpp-0.7.0/libminifi/src/core/ProcessGroup.cpp#L343-L359],
so there is not much extra value in keeping it.
*Proposal:*
We should try and validate the configuration files before trying to reload
using them. Not only would this make us fail earlier, but would have the checks
a bit more decoupled from the processors. We already send processor
configuratin requirement data via the heartbeat to EFM, so we have every piece
of information needed to statically validate the configuration files.
This way the new logic should simplify to:
# Validate the configuration
# If it is valid load it, otherwise load it and any subsequent error should be
considered runtime exception.
As this would potentially break configuration-file backward compatibilty, all
the changes should be implemented behind a feature flag (build-time).
> Implement static flow configuration validation
> ----------------------------------------------
>
> Key: MINIFICPP-1288
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1288
> Project: Apache NiFi MiNiFi C++
> Issue Type: Task
> Reporter: Adam Hunyadi
> Assignee: Adam Hunyadi
> Priority: Major
>
> *Background:*
> MiNiFi flows are represented by yaml configuration files. When these are
> published to the FlowController in MiNiFi, it handles this as a “reconfigure
> and restart”. The current logic is:
> # Backup the old process group (in memory representation of the
> configuration).
> # Stop the FlowController.
> # Try to build up a new process group based on the new configuration.
> # If we fail during the construction of the process group nodes we should
> keep the backup, otherwise we let the new flow run and consider it validated.
> This is not awful, but expects the checks implemented in terms of processor
> construction-time checks and separated cheks on the connection network. [[The
> implementation is also not very
> robust]|https://github.com/apache/nifi-minifi-cpp/blob/rel/minifi-cpp-0.7.0/libminifi/src/core/ProcessGroup.cpp#L343-L359],
> so there is not much extra value in keeping it.
> *Proposal:*
> We should try and validate the configuration files before trying to reload
> using them. Not only would this make us fail earlier, but would have the
> checks a bit more decoupled from the processors. We already send processor
> configuratin requirement data via the heartbeat to EFM, so we have every
> piece of information needed to statically validate the configuration files.
> This way the new logic should simplify to:
> # Validate the configuration
> # If it is valid load it, otherwise load it and any subsequent error should
> be considered runtime exception.
> As this would potentially break configuration-file backward compatibilty, all
> the changes should be implemented behind a feature flag (build-time).
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
