[
https://issues.apache.org/jira/browse/NIFI-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15878230#comment-15878230
]
ASF GitHub Bot commented on NIFI-3189:
--------------------------------------
GitHub user ijokarumawak opened a pull request:
https://github.com/apache/nifi/pull/1527
NIFI-3189: ConsumeKafka & Back-pressure. ConsumeKafka_0_10
While downstream connections are full, ConsumeKafka is not scheduled to
run onTrigger, and it won't perform poll, which is important to tell
Kafka server that this client is alive. Thus, after a while in that
situation, Kafka server rebalances the client, and then if downstream
flow backs to normal and ConsumeKafka is scheduled again, the client is no
longer has a valid connection, and a warning message is logged.
This PR uses Kafka pause/resume consumer API to tell Kafka server that
the client doesn't want to consume any message but still alive. It uses
another internal thread to do so by periodically checks when was the
last time that onTrigger is executed. If it has been a while since last
onTrigger (poll), then it pauses the consumer and poll to send a
heartbeat so that the connection will be kept.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ijokarumawak/nifi nifi-3189
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1527.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1527
----
commit 79bc337a8812af41502c5a4fc046b183b72d7ee3
Author: Koji Kawamura <[email protected]>
Date: 2017-02-20T10:08:23Z
NIFI-3189: ConsumeKafka & Back-pressure.
While downstream connections are full, ConsumeKafka is not scheduled to
run onTrigger, and it won't perform poll, which is important to tell
Kafka server that this client is alive. Thus, after a while in that
situation, Kafka server rebalances the client, and then if downstream
flow backs to normal and ConsumeKafka is scheduled again, the client is no
longer has a valid connection, and a warning message is logged.
This PR uses Kafka pause/resume consumer API to tell Kafka server that
the client doesn't want to consume any message but still alive. It uses
another internal thread to do so by periodically checks when was the
last time that onTrigger is executed. If it has been a while since last
onTrigger (poll), then it pauses the consumer and poll to send a
heartbeat so that the connection will be kept.
----
> ConsumeKafka 0.9 and 0.10 can cause consumer rebalance when backpressure is
> engaged
> -----------------------------------------------------------------------------------
>
> Key: NIFI-3189
> URL: https://issues.apache.org/jira/browse/NIFI-3189
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Joseph Witt
>
> ConsumeKafka processors can alert to rebalance issues when backpressure is
> engaged on the output connection and is then freed up. This is because we're
> not doing anything with those consumers for a period of time and the kafka
> client detects this and initiates a rebalance. We should ensure that even
> when we cannot send more data due to back pressure that we at least have some
> sort of keep alive behavior with the kafka client. Or, if that isn't an
> option we should at least document the situation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
