[
https://issues.apache.org/jira/browse/NIFI-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859432#comment-15859432
]
ASF GitHub Bot commented on NIFI-3452:
--------------------------------------
GitHub user ijokarumawak opened a pull request:
https://github.com/apache/nifi/pull/1490
NIFI-3452: Add Wait processor Wait Mode property
Ensure back-pressure is active until downstream processing completes.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ijokarumawak/nifi nifi-3452
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1490.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1490
----
commit 8767b8bf98c83e0e84e68fb53c51255fec4ab084
Author: Koji Kawamura <[email protected]>
Date: 2017-02-09T12:30:43Z
NIFI-3452: Add Wait processor Wait Mode property
Ensure back-pressure is active until downstream processing completes.
----
> Add Wait processor Wait Mode property
> -------------------------------------
>
> Key: NIFI-3452
> URL: https://issues.apache.org/jira/browse/NIFI-3452
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 1.2.0
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
> Attachments: wait-for-a-par-of-flow-to-finish.png
>
>
> NiFi back pressure is handled per relationship and as long as a relationship
> has room to receive more flow files, source processor is scheduled to run.
> However, this behavior is not ideal in some cases. For example, when there is
> very computationally expensive task and user wants to limit the number of
> FlowFiles can be processed at a given time, it's not always possible to limit
> the rate by existing RateControl nor back-pressure mechanism.
> As a more practical example, in the following flow, it's expected the GetSQS
> is triggered only when the previous FlowFile has been processed completely.
> Node 1 is parsing a flow file (indicated by the X in the connection between
> FetchS3Object and Parse). Both connections have a back-pressure threshold of
> 1, but because the object is already fetched, the first connection is empty
> and can thus be filled. This means that, if a new item becomes available in
> the queue, both of the following cases can happen with equal probability:
> {code}
> Case 1:
> ---------- ----------------- ---------
> Node 1: | GetSQS | -X-> | FetchS3Object | -X-> | Parse |
> ---------- ----------------- ---------
> ---------- ----------------- ---------
> Node 2: | GetSQS | ---> | FetchS3Object | ---> | Parse |
> ---------- ----------------- ---------
> Case 2:
> ---------- ----------------- ---------
> Node 1: | GetSQS | ---> | FetchS3Object | -X-> | Parse |
> ---------- ----------------- ---------
> ---------- ----------------- ---------
> Node 2: | GetSQS | -X-> | FetchS3Object | ---> | Parse |
> ---------- ----------------- ---------
> {code}
> To achieve that, we could improve Wait processor as follows.
> NiFi scheduler checks downstream relationship availability, when it's full,
> the processor won't be scheduled to run. In case a source processor has
> multiple outgoing relationships, and if ANY of those is full, the processor
> won't be scheduled.
> (This is how processor scheduling works with back-pressure, but can
> alter with @TriggerWhenAnyDestinationAvailable annotation. DistributeLoad is
> the only processor annotated with this)
> We could use this mechanism to keep the source processor waiting to be
> scheduled, by following flow:
> {code}
> GetSQS
> -- success --> FetchS3Object --> Parse --> Notify
> -- success --> Wait
> {code}
> To make it work as expected, we need to improve Wait so that user can choose
> how waiting FlowFile is handled, from either:
> "Route to 'wait' relationship" or "Keep in the Upstream connection".
> Currently it has only option to route to 'wait'.
> Use "Keep in the Upstream connection" Wait mode with the flow above,
> the incoming flow file in GetSQS -> Wait connection stays there until actual
> data processing finishes and Notify sends a notification signal.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
