[
https://issues.apache.org/jira/browse/NIFI-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887238#comment-15887238
]
ASF GitHub Bot commented on NIFI-3189:
--------------------------------------
Github user ijokarumawak commented on the issue:
https://github.com/apache/nifi/pull/1527
@bbende It turned out to be updating Kafka client is sufficient for 0.10.
Confirmed that ConsumeKafka_0_10 can rejoin the consumer group when it resumes
after being back-pressure engaged more than 5 minutes. For 0.9, I've added the
manual connection retaining logic using paused poll.
- For ConsumeKafka_0_10, use latest client library
Above issue has been addressed by KIP-62.
The latest Kafka consumer poll checks if the client instance is still
valid, and rejoin the group if not, before consuming messages.
- For ConsumeKafka (0.9), added manual retention logic using pause/resume
Kafka client 0.9 doesn't have background thread heartbeat, so similar
machanism is added manually.
Use Kafka pause/resume consumer API to tell Kafka server that the
client stops consuming messages but is still alive.
Another internal thread is used to perform paused poll periodically
based on the time passed since the last onTrigger(poll) is executed.
> ConsumeKafka 0.9 and 0.10 can cause consumer rebalance when backpressure is
> engaged
> -----------------------------------------------------------------------------------
>
> Key: NIFI-3189
> URL: https://issues.apache.org/jira/browse/NIFI-3189
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Joseph Witt
> Assignee: Koji Kawamura
>
> ConsumeKafka processors can alert to rebalance issues when backpressure is
> engaged on the output connection and is then freed up. This is because we're
> not doing anything with those consumers for a period of time and the kafka
> client detects this and initiates a rebalance. We should ensure that even
> when we cannot send more data due to back pressure that we at least have some
> sort of keep alive behavior with the kafka client. Or, if that isn't an
> option we should at least document the situation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
