[
https://issues.apache.org/jira/browse/NIP-51?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIP-51:
--------------------------
Description:
h2. Motivation
There are several times when a user needs all of the data that is already
queued in a Process Group to finish processing, without letting any new data in:
* *Moving a flow to another NiFi instance.* NiFi can now export a flow along
with the state of its components and import it elsewhere with that state
(NIFI-15671). Queued FlowFiles are not part of the export, so the queues need
to be empty first.
* *Upgrading a Versioned Flow.* A new version may change or remove the
components that hold queued data. Emptying the group first means that the data
is never processed by a flow that it was not built for.
* *Suspending a flow for a period of time.* Letting queued data finish first
means it is not left waiting until the flow is started again.
Today the user has to find and stop every component that brings data into the
group, make sure everything else is running, and then watch the queues until
they are empty. Furthermore, it can take a long time for some processors such
as MergeContent to complete because we have to wait for the Max Bin Age, etc.
Recent additions already allow stopping all source components. This is an
extension of that capability, effectively.
NiFi already supports draining a Connector. This NIP proposes the same ability
for any Process Group on the canvas.
h2. Proposed Solution
A user can choose "Drain" for a Process Group. NiFi then:
* Stops the source Processors (Processors with no incoming connections) in the
group and all of its child groups, and the Input Ports of the drained group
itself.
* Starts all other Processors, Output Ports and Remote Process Groups, and
enables the Controller Services that they reference.
* Reports progress as the number and size of FlowFiles still queued, compared
to when the drain began.
* Lists the components that keep the drain from finishing because they are
invalid or disabled and have data queued.
The user can also cancel a drain at any time. While a drain is in progress,
users cannot start the source components or stop the other components in the
group. Drains are not remembered across a restart.
Some Processors may take a very long time to pass FlowFiles along, or may never
do so, once new data stops arriving. MergeContent and MergeRecord configured
without a Max Bin Age are good examples. NiFi will notify Processors when a
drain begins and ends, so that they can process their data more aggressively
during a drain.
h2. NiFi API Changes
Two new lifecycle annotations, {{@OnDrainInitiated}} and
{{{}@OnDrainCompleted{}}}, for Processors and Controller Services that need to
behave differently while their Process Group is being drained. Methods may take
no arguments, a {{ProcessContext}} (Processors), or a {{ConfigurationContext}}
(Controller Services). This change is purely additive. Components that do not
use the annotations are not affected.
{{TestRunner}} in nifi-mock gains {{initiateDrain()}} and {{completeDrain()}}
so that extension developers can test this behavior.
h2. REST API Changes
New asynchronous endpoints will be added for creating, fetching, and deleting a
drian.
Beginning a drain requires WRITE permission on the Process Group and on
everything in it. In a cluster, all nodes must be connected.
{{ProcessGroupDTO}} gains a read-only {{drainRequestId}} field.
h2. UI Changes
The Process Group context menu gains "Drain". A dialog shows a progress bar and
a Cancel button. It also lists any blocking components with the reason they are
blocking.
was:
h2. Motivation
There are several times when a user needs all of the data that is already
queued in a Process Group to finish processing, without letting any new data in:
* *Moving a flow to another NiFi instance.* NiFi can now export a flow along
with the state of its components and import it elsewhere with that state
(NIFI-15671). Queued FlowFiles are not part of the export, so the queues need
to be empty first.
* *Upgrading a Versioned Flow.* A new version may change or remove the
components that hold queued data. Emptying the group first means that the data
is never processed by a flow that it was not built for.
* *Suspending a flow for a period of time.* Letting queued data finish first
means it is not left waiting until the flow is started again.
Today the user has to find and stop every component that brings data into the
group, make sure everything else is running, and then watch the queues until
they are empty. Furthermore, it can take a long time for some processors such
as MergeContent to complete because we have to wait for the Max Bin Age, etc.
Recent additions already allow stopping all source components. This is not
intended to replace that but to complement it, as there are reasons for
stopping the source components other than necessarily a full drain.
NiFi already supports draining a Connector. This NIP proposes the same ability
for any Process Group on the canvas.
h2. Proposed Solution
A user can choose "Drain" for a Process Group. NiFi then:
* Stops the source Processors (Processors with no incoming connections) in the
group and all of its child groups, and the Input Ports of the drained group
itself.
* Starts all other Processors, Output Ports and Remote Process Groups, and
enables the Controller Services that they reference.
* Reports progress as the number and size of FlowFiles still queued, compared
to when the drain began.
* Lists the components that keep the drain from finishing because they are
invalid or disabled and have data queued.
The user can also cancel a drain at any time. While a drain is in progress,
users cannot start the source components or stop the other components in the
group. Drains are not remembered across a restart.
Some Processors may take a very long time to pass FlowFiles along, or may never
do so, once new data stops arriving. MergeContent and MergeRecord configured
without a Max Bin Age are good examples. NiFi will notify Processors when a
drain begins and ends, so that they can process their data more aggressively
during a drain.
h2. NiFi API Changes
Two new lifecycle annotations, {{@OnDrainInitiated}} and
{{{}@OnDrainCompleted{}}}, for Processors and Controller Services that need to
behave differently while their Process Group is being drained. Methods may take
no arguments, a {{ProcessContext}} (Processors), or a {{ConfigurationContext}}
(Controller Services). This change is purely additive. Components that do not
use the annotations are not affected.
{{TestRunner}} in nifi-mock gains {{initiateDrain()}} and {{completeDrain()}}
so that extension developers can test this behavior.
h2. REST API Changes
New asynchronous endpoints will be added for creating, fetching, and deleting a
drian.
Beginning a drain requires WRITE permission on the Process Group and on
everything in it. In a cluster, all nodes must be connected.
{{ProcessGroupDTO}} gains a read-only {{drainRequestId}} field.
h2. UI Changes
The Process Group context menu gains "Drain". A dialog shows a progress bar and
a Cancel button. It also lists any blocking components with the reason they are
blocking.
> Allow Draining of a Process Group
> ---------------------------------
>
> Key: NIP-51
> URL: https://issues.apache.org/jira/browse/NIP-51
> Project: NiFi Improvement Proposal
> Issue Type: Improvement
> Reporter: Mark Payne
> Priority: Major
>
> h2. Motivation
> There are several times when a user needs all of the data that is already
> queued in a Process Group to finish processing, without letting any new data
> in:
> * *Moving a flow to another NiFi instance.* NiFi can now export a flow along
> with the state of its components and import it elsewhere with that state
> (NIFI-15671). Queued FlowFiles are not part of the export, so the queues need
> to be empty first.
> * *Upgrading a Versioned Flow.* A new version may change or remove the
> components that hold queued data. Emptying the group first means that the
> data is never processed by a flow that it was not built for.
> * *Suspending a flow for a period of time.* Letting queued data finish first
> means it is not left waiting until the flow is started again.
> Today the user has to find and stop every component that brings data into the
> group, make sure everything else is running, and then watch the queues until
> they are empty. Furthermore, it can take a long time for some processors such
> as MergeContent to complete because we have to wait for the Max Bin Age, etc.
> Recent additions already allow stopping all source components. This is an
> extension of that capability, effectively.
> NiFi already supports draining a Connector. This NIP proposes the same
> ability for any Process Group on the canvas.
> h2. Proposed Solution
> A user can choose "Drain" for a Process Group. NiFi then:
> * Stops the source Processors (Processors with no incoming connections) in
> the group and all of its child groups, and the Input Ports of the drained
> group itself.
> * Starts all other Processors, Output Ports and Remote Process Groups, and
> enables the Controller Services that they reference.
> * Reports progress as the number and size of FlowFiles still queued,
> compared to when the drain began.
> * Lists the components that keep the drain from finishing because they are
> invalid or disabled and have data queued.
> The user can also cancel a drain at any time. While a drain is in progress,
> users cannot start the source components or stop the other components in the
> group. Drains are not remembered across a restart.
> Some Processors may take a very long time to pass FlowFiles along, or may
> never do so, once new data stops arriving. MergeContent and MergeRecord
> configured without a Max Bin Age are good examples. NiFi will notify
> Processors when a drain begins and ends, so that they can process their data
> more aggressively during a drain.
> h2. NiFi API Changes
> Two new lifecycle annotations, {{@OnDrainInitiated}} and
> {{{}@OnDrainCompleted{}}}, for Processors and Controller Services that need
> to behave differently while their Process Group is being drained. Methods may
> take no arguments, a {{ProcessContext}} (Processors), or a
> {{ConfigurationContext}} (Controller Services). This change is purely
> additive. Components that do not use the annotations are not affected.
> {{TestRunner}} in nifi-mock gains {{initiateDrain()}} and {{completeDrain()}}
> so that extension developers can test this behavior.
> h2. REST API Changes
> New asynchronous endpoints will be added for creating, fetching, and deleting
> a drian.
> Beginning a drain requires WRITE permission on the Process Group and on
> everything in it. In a cluster, all nodes must be connected.
> {{ProcessGroupDTO}} gains a read-only {{drainRequestId}} field.
> h2. UI Changes
> The Process Group context menu gains "Drain". A dialog shows a progress bar
> and a Cancel button. It also lists any blocking components with the reason
> they are blocking.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
