Pierre Villard created NIP-21:
---------------------------------
Summary: Flow Definition Export/Import with Components State
Key: NIP-21
URL: https://issues.apache.org/jira/browse/NIP-21
Project: NiFi Improvement Proposal
Issue Type: New Feature
Reporter: Pierre Villard
Assignee: Pierre Villard
*Motivation*
It can happen that it is necessary, for many reasons, to move a flow definition
from one NiFi environment to another. When that happens, we currently don't
have any way to easily move the state (if any) of the components included in
the flow definition. This can make things particularly annoying when it means
re-ingesting a lot of previously ingested data.
We always provided some workarounds: some components expose properties allowing
to set an initial state, or it could be possible to temporarily modify the flow
to "skip" the data until components get back to the same state as in the source
environment. But these workarounds are error-prone, time-consuming, and not
always available for every component.
It would make things much easier if we expose the possibility to export the
flow definition and include component state in it. And, when importing such a
flow definition, properly initialize the component state with the information
from the JSON file.
This feature is scoped exclusively to the JSON-based "Download flow definition"
/ "Upload flow definition" workflow. It does not apply to flow versioning via a
registry client (registry snapshots are meant to be portable flow definitions,
not point-in-time runtime snapshots, and including state in a registry would
not make sense).
*NiFi API Changes*
The changes to the NiFi API are limited to two additions and are fully backward
compatible.
* A new class *VersionedComponentState* :
{code:java}
public class VersionedComponentState {
// The cluster-scoped state of the component, or null if the component has
no cluster state.
private Map<String, String> clusterState;
// The local-scoped state of the component for each node, keyed by ordinal
node index
// ("0", "1", ...), or null if the component has no local state.
// In a standalone instance, there is a single entry with key "0".
private Map<String, Map<String, String>> localNodeStates;
}
{code}
* A new optional field *componentState* of type *VersionedComponentState* on
the existing *VersionedConfigurableExtension* class:
{code:java}
private VersionedComponentState componentState;
@Schema(description = "The state of the component, included when the flow
definition is exported with state")
public VersionedComponentState getComponentState() { return componentState; }
public void setComponentState(VersionedComponentState componentState) {
this.componentState = componentState; }
{code}
When not exporting with state, this field is null and omitted from the
serialized JSON. When a NiFi instance that does not yet support this feature
receives a flow definition containing component state, Jackson will silently
ignore the unknown *componentState* field. This ensures full backward
compatibility.
*REST API Changes*
The existing download endpoint *GET /process-groups/\{id}/download* gains an
additional optional query parameter:
- includeComponentState (boolean, default false): When set to true, the
exported flow definition will include the state of all stateful processors and
controller services in the process group (recursively).
No new endpoints are required. The existing upload/import endpoint handles the
new componentState field automatically.
*Behavioral Constraints*
* Components must be stopped for export with state: When
includeComponentState=true, all processors in the process group must be stopped
and all controller services must be disabled. If any component is running, the
request is rejected with a 409 Conflict. This guarantees state consistency at
the time of export.
* This feature never applies to registry versioning: The componentState field
is only populated during JSON export. The flow mapper used for registry
versioning does not include state. On the import side, componentState is only
acted upon if non-null (flow snapshots from a registry will never have this
field, so no state initialization occurs).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
