Bob Paulin created NIFI-16477:
---------------------------------
Summary: Implement Flow Evolution Drain Hooks and Flow persistence
Key: NIFI-16477
URL: https://issues.apache.org/jira/browse/NIFI-16477
Project: Apache NiFi
Issue Type: Improvement
Reporter: Bob Paulin
Assignee: Bob Paulin
Framework changes needed for NIP-49
h3. *Persistence*
VersionedComponentFlowMapper.mapConnector writes:
* {{{}managedParameterContext{}}}, {{{}managedFlowBundle{}}}, and
{{activeFlowConfiguration}} from one immutable in-memory
{{{}ConnectorFlowSnapshot{}}}, never by combining values from different
lifecycle phases. The record is a complete Connector flow-persistence value
that can be prepared, published as authoritative, or retained for rollback.
* {{managedProcessGroup}} from that snapshot, except while the connector is in
Troubleshooting, when it is written from the live managed group.
* {{bundle}} from the loaded Connector coordinate and {{managedFlowBundle}}
from the snapshot. While an upgrade is pending, these fields differ.
The authoritative snapshot is set after every successful apply, after an
upgraded snapshot is committed, and when a persisted authoritative flow is
loaded. {{getAuthoritativeFlowSnapshot()}} names the currently committed
instance; prepared and rollback snapshots use the same
{{ConnectorFlowSnapshot}} value type. It is captured on entering
troubleshooting only when none is held. While troubleshooting, the persisted
managed group is the live edits and the snapshot bundle stays the version that
built the flow. On exiting troubleshooting, the snapshot is refreshed from the
installed Active flow, including when an upgrade was pending.
h3. *Detection*
During synchronization, the framework resolves the snapshot bundle from
{{{}managedFlowBundle{}}}, or infers it once from the persisted {{bundle}} when
a managed group exists, including a group persisted while Troubleshooting. The
inferred value is frozen before the loaded version can change. No value is
inferred when there is no managed group.
Troubleshooting is restored from {{managedProcessGroup}} and defers evolution
until the Connector exits troubleshooting. The same detection rule applies
whenever the loaded version changes.
h3. *Replacing the flow*
Each node waits for the {{prepareForFlowEvolution}} future before applying the
working configuration and preparing the incoming snapshot. A timeout or
cancellation cancels the future, retains the authoritative pre-upgrade
snapshot, and leaves the Connector stopped and pending. After cluster-wide
commit authorization, every node atomically publishes the incoming snapshot and
persists its own flow, and START waits for cluster-wide persistence completion
before proceeding to running. Publishing the new snapshot makes its
managed-flow bundle equal the loaded bundle, so pending becomes false without
clearing a stored flag.
If preparing or committing the upgraded flow fails, the outgoing snapshot
remains authoritative or is restored, and the node stays stopped and pending.
The existing flow-update verification remains as a backstop for a FlowFile that
arrives after the check, or for an {{applyUpdate}} that installs something
other than {{getActiveFlow}} on the working context.
*Exiting troubleshooting while an upgrade is pending*
{{prepareForFlowEvolution}} is not run. Every queue in the managed flow,
including child groups, must be empty. The exit is rejected while any FlowFiles
remain, and the Connector stays in Troubleshooting with the upgrade still
pending. When the queues are empty, the same synchronous exit used otherwise
installs {{{}getActiveFlow{}}}, captures the snapshot with the loaded bundle,
and leaves the Connector {{STOPPED}} and no longer pending. This covers a
connector that was in troubleshooting when the runtime was upgraded, and a
connector whose drain was cancelled so the user could enter troubleshooting
from {{{}STOPPED{}}}.
The hook is skipped because the empty-queue check already makes the replacement
safe, and the shipped hooks only drain. Draining on exit would start components
the user added and stopped, against a flow whose structure the hook cannot rely
on after those edits. The user can empty the queues before exiting, and the
exit stays a synchronous verify-then-apply across the cluster.
h2. *New configuration*
nifi.components.connectors.flow.evolution.timeout bounds
prepareForFlowEvolution. The default is 30 mins.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
