[
https://issues.apache.org/jira/browse/NIFI-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164171#comment-16164171
]
Koji Kawamura commented on NIFI-4155:
-------------------------------------
[~markap14] That's a good point. EnforceOrder just enforces the order of
passing through itself and can not control how subsequent flow handles
ordering. So as you mentioned, we need to use back-pressure of 1 FlowFile, or
FIFO prioritizer (I think FIFO prioritizer is recommended over back-pressure in
terms of performance).
As you stated, within a multiple node cluster, we cannot guarantee ordering by
back-pressure nor FIFO prioritizer cluster wide. User will have to use
Wait/Notify processors together to implement a distributed countdown-latch
using a shared DistributedCache.
{code}
EnforceOrder --> Wait to block only 1 FlowFile can go through --> Processors
required to run serially --> Notify to release the latch
{code}
An use-case that I can think of is, doing per FlowFile computational and time
consuming task (e.g. costly enrichment using external API or database) among
NiFi nodes concurrently before EnforceOrder, then use above EnforceOrder and
Wait/Notify part to control ordering to pass result to the final destination
which requires exact ordering. The benefit in this model is not have to
transfer FlowFiles to a single node to enforce ordering.
How do you think?
> Expand EnforceOrder capability to cluster
> -----------------------------------------
>
> Key: NIFI-4155
> URL: https://issues.apache.org/jira/browse/NIFI-4155
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
>
> Currently, EnforceOrder is able to enforce FlowFile ordering within a NiFi
> node, and it uses local managed state to track progress.
> If it is configurable which state management to use from Local and Cluster,
> EnforceOrder would be also able to enforce ordering across cluster with
> cluster scope state management. It would be useful for some use-cases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
