[
https://issues.apache.org/jira/browse/NIFI-16136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100358#comment-18100358
]
febriyansyah commented on NIFI-16136:
-------------------------------------
Hi, thank you for reporting this issue and providing the detailed reproduction
steps!
I have submitted a PR to fix this race condition. The fix modifies
StandardFlowService.offload() to await the CompletableFuture returned by
stopProcessing(). This ensures that the framework pauses and waits for the
background stop threads to fully execute the @OnStopped hooks before it
proceeds to forcefully terminate the processors.
Here is the PR: https://github.com/apache/nifi/pull/11492
> @OnStopped can be permanently skipped during offload
> ----------------------------------------------------
>
> Key: NIFI-16136
> URL: https://issues.apache.org/jira/browse/NIFI-16136
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.10.0
> Reporter: Alaksiej Ščarbaty
> Priority: Major
>
> _StandardFlowService#offload()_ stops all processors and terminates any that
> report {_}getScheduledState() == ScheduledState.STOPPED{_}, without awaiting
> the _Future_ returned by {_}stopProcessing(){_}:
>
> {code:java}
> flowManager.getRootGroup().stopProcessing(); // Future not awaited
> flowManager.getRootGroup().findAllProcessors().stream()
> .filter(pn -> pn.getScheduledState() == ScheduledState.STOPPED)
> .forEach(pn -> pn.getProcessGroup().terminateProcessor(pn)); {code}
>
> _getScheduledState()_ maps the transient _STOPPING_ state to _STOPPED_ for
> backward compatibility, so _terminateProcessor()_ can run before a
> Processor's stop sequence ({_}StandardProcessorNode.stop(){_}'s background
> runnable) has even reached the stop hook.
> _LifecycleState#terminate()_ permanently pins _activeThreadCount_ to 0.
> _@OnStopped_ isn't delayed by this race, it's skipped permanently: the stop
> _Future_ completes and the framework considers the Processor fully stopped,
> but any cleanup in _@OnStopped_ (closing connections, releasing resources,
> etc.) never runs.
> *Reproduction:*
> [testOffloadTerminateRaceSkipsOnStoppedPermanently in
> TestStandardProcessScheduler
> |https://github.com/awelless/nifi/commit/79c7ddc7a9418f1f6d643b80db6c26a68f9d5215#diff-8d6b01098138af203490bfcd481def8c309e064ec92c1d236df5b8eca3580f70R34-R833]reproduces
> {_}offload(){_}'s exact sequence (block the stop thread before
> _unschedule()_ to ensure the race happens, call _stopProcessor()_ without
> awaiting its {_}Future{_}, confirm _getScheduledState() == STOPPED_ while
> still physically {_}STOPPING{_}, call {_}terminateProcessor(){_}, assert
> _@OnStopped_ fires). It currently fails.
> *Open questions:*
> * Is it expected @OnStopped might not run when offloading happens?
> * Is it expected that we don't wait for @OnStopped during offload?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
