[ 
https://issues.apache.org/jira/browse/NIFI-16477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne resolved NIFI-16477.
-------------------------------
    Fix Version/s: 2.13.0
       Resolution: Fixed

> Implement Flow Evolution Drain Hooks and Flow persistence
> ---------------------------------------------------------
>
>                 Key: NIFI-16477
>                 URL: https://issues.apache.org/jira/browse/NIFI-16477
>             Project: Apache NiFi
>          Issue Type: Improvement
>            Reporter: Bob Paulin
>            Assignee: Bob Paulin
>            Priority: Major
>             Fix For: 2.13.0
>
>          Time Spent: 14h 10m
>  Remaining Estimate: 0h
>
> Framework changes needed for NIP-49
>  
> h3. *Persistence*
> VersionedComponentFlowMapper.mapConnector writes:
>  * {{{}managedParameterContext{}}}, {{{}managedFlowBundle{}}}, and 
> {{activeFlowConfiguration}} from one immutable in-memory 
> {{{}ConnectorFlowSnapshot{}}}, never by combining values from different 
> lifecycle phases. The record is a complete Connector flow-persistence value 
> that can be prepared, published as authoritative, or retained for rollback.
>  * {{managedProcessGroup}} from that snapshot, except while the connector is 
> in Troubleshooting, when it is written from the live managed group.
>  * {{bundle}} from the loaded Connector coordinate and {{managedFlowBundle}} 
> from the snapshot. While an upgrade is pending, these fields differ.
> The authoritative snapshot is set after every successful apply, after an 
> upgraded snapshot is committed, and when a persisted authoritative flow is 
> loaded. {{getAuthoritativeFlowSnapshot()}} names the currently committed 
> instance; prepared and rollback snapshots use the same 
> {{ConnectorFlowSnapshot}} value type. It is captured on entering 
> troubleshooting only when none is held. While troubleshooting, the persisted 
> managed group is the live edits and the snapshot bundle stays the version 
> that built the flow. On exiting troubleshooting, the snapshot is refreshed 
> from the installed Active flow, including when an upgrade was pending.
> h3. *Detection*
> During synchronization, the framework resolves the snapshot bundle from 
> {{{}managedFlowBundle{}}}, or infers it once from the persisted {{bundle}} 
> when a managed group exists, including a group persisted while 
> Troubleshooting. The inferred value is frozen before the loaded version can 
> change. No value is inferred when there is no managed group.
> Troubleshooting is restored from {{managedProcessGroup}} and defers evolution 
> until the Connector exits troubleshooting. The same detection rule applies 
> whenever the loaded version changes.
> h3. *Replacing the flow*
> Each node waits for the {{prepareForFlowEvolution}} future before applying 
> the working configuration and preparing the incoming snapshot. A timeout or 
> cancellation cancels the future, retains the authoritative pre-upgrade 
> snapshot, and leaves the Connector stopped and pending. After cluster-wide 
> commit authorization, every node atomically publishes the incoming snapshot 
> and persists its own flow, and START waits for cluster-wide persistence 
> completion before proceeding to running. Publishing the new snapshot makes 
> its managed-flow bundle equal the loaded bundle, so pending becomes false 
> without clearing a stored flag.
> If preparing or committing the upgraded flow fails, the outgoing snapshot 
> remains authoritative or is restored, and the node stays stopped and pending. 
> The existing flow-update verification remains as a backstop for a FlowFile 
> that arrives after the check, or for an {{applyUpdate}} that installs 
> something other than {{getActiveFlow}} on the working context.
> *Exiting troubleshooting while an upgrade is pending*
> {{prepareForFlowEvolution}} is not run. Every queue in the managed flow, 
> including child groups, must be empty. The exit is rejected while any 
> FlowFiles remain, and the Connector stays in Troubleshooting with the upgrade 
> still pending. When the queues are empty, the same synchronous exit used 
> otherwise installs {{{}getActiveFlow{}}}, captures the snapshot with the 
> loaded bundle, and leaves the Connector {{STOPPED}} and no longer pending. 
> This covers a connector that was in troubleshooting when the runtime was 
> upgraded, and a connector whose drain was cancelled so the user could enter 
> troubleshooting from {{{}STOPPED{}}}.
> The hook is skipped because the empty-queue check already makes the 
> replacement safe, and the shipped hooks only drain. Draining on exit would 
> start components the user added and stopped, against a flow whose structure 
> the hook cannot rely on after those edits. The user can empty the queues 
> before exiting, and the exit stays a synchronous verify-then-apply across the 
> cluster.
> h2. *New configuration*
> nifi.components.connectors.flow.evolution.timeout bounds 
> prepareForFlowEvolution. The default is 30 mins.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to