[
https://issues.apache.org/jira/browse/NIP-21?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18061401#comment-18061401
]
Matt Burgess commented on NIP-21:
---------------------------------
If this doesn't affect Registry, how would users leverage the Registry to move
flows (with state) between environments? Would they need to export the canvas
to a Flow Definition then upload/import to the new environment? Or am I
misunderstanding?
And just to be clear, this is for the state the components store, not the state
of the component (Running, Stopped, etc.). Should go without saying but I
mention it for completeness :)
> Flow Definition Export/Import with Components State
> ---------------------------------------------------
>
> Key: NIP-21
> URL: https://issues.apache.org/jira/browse/NIP-21
> Project: NiFi Improvement Proposal
> Issue Type: New Feature
> Reporter: Pierre Villard
> Assignee: Pierre Villard
> Priority: Major
>
> *Motivation*
> It can happen that it is necessary, for many reasons, to move a flow
> definition from one NiFi environment to another. When that happens, we
> currently don't have any way to easily move the state (if any) of the
> components included in the flow definition. This can make things particularly
> annoying when it means re-ingesting a lot of previously ingested data.
> We always provided some workarounds: some components expose properties
> allowing to set an initial state, or it could be possible to temporarily
> modify the flow to "skip" the data until components get back to the same
> state as in the source environment. But these workarounds are error-prone,
> time-consuming, and not always available for every component.
> It would make things much easier if we expose the possibility to export the
> flow definition and include component state in it. And, when importing such a
> flow definition, properly initialize the component state with the information
> from the JSON file.
> This feature is scoped exclusively to the JSON-based "Download flow
> definition" / "Upload flow definition" workflow. It does not apply to flow
> versioning via a registry client (registry snapshots are meant to be portable
> flow definitions, not point-in-time runtime snapshots, and including state in
> a registry would not make sense).
> *NiFi API Changes*
> The changes to the NiFi API are limited to two additions and are fully
> backward compatible.
> * A new class *VersionedComponentState* :
> {code:java}
> public class VersionedComponentState {
> // The cluster-scoped state of the component, or null if the component
> has no cluster state.
> private Map<String, String> clusterState;
> // The local-scoped state of the component for each node, keyed by
> ordinal node index
> // ("0", "1", ...), or null if the component has no local state.
> // In a standalone instance, there is a single entry with key "0".
> private Map<String, Map<String, String>> localNodeStates;
> }
> {code}
> * A new optional field *componentState* of type *VersionedComponentState* on
> the existing *VersionedConfigurableExtension* class:
> {code:java}
> private VersionedComponentState componentState;
> @Schema(description = "The state of the component, included when the flow
> definition is exported with state")
> public VersionedComponentState getComponentState() { return componentState; }
> public void setComponentState(VersionedComponentState componentState) {
> this.componentState = componentState; }
> {code}
> When not exporting with state, this field is null and omitted from the
> serialized JSON. When a NiFi instance that does not yet support this feature
> receives a flow definition containing component state, Jackson will silently
> ignore the unknown *componentState* field. This ensures full backward
> compatibility.
> *REST API Changes*
> The existing download endpoint *GET /process-groups/\{id}/download* gains an
> additional optional query parameter:
> - includeComponentState (boolean, default false): When set to true, the
> exported flow definition will include the state of all stateful processors
> and controller services in the process group (recursively).
> No new endpoints are required. The existing upload/import endpoint handles
> the new componentState field automatically.
> *Behavioral Constraints*
> * Components must be stopped for export with state: When
> includeComponentState=true, all processors in the process group must be
> stopped and all controller services must be disabled. If any component is
> running, the request is rejected with a 409 Conflict. This guarantees state
> consistency at the time of export.
> * This feature never applies to registry versioning: The componentState
> field is only populated during JSON export. The flow mapper used for registry
> versioning does not include state. On the import side, componentState is only
> acted upon if non-null (flow snapshots from a registry will never have this
> field, so no state initialization occurs).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
