[
https://issues.apache.org/jira/browse/KAFKA-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922375#comment-16922375
]
Xin edited comment on KAFKA-6745 at 9/6/19 11:30 AM:
-----------------------------------------------------
same problem，kafka version：1.1.0，spark streaming application
one job has 2 consumers
Received successful join group response after like session.timout.ms
was (Author: auroraxlh):
same problem，kafka version：1.1.0，spark streaming application
Received successful join group response after 5min
> kafka consumer rebalancing takes long time (from 3 secs to 5 minutes)
> ---------------------------------------------------------------------
>
> Key: KAFKA-6745
> URL: https://issues.apache.org/jira/browse/KAFKA-6745
> Project: Kafka
> Issue Type: Improvement
> Components: clients, core
> Affects Versions: 0.11.0.0
> Reporter: Ramkumar
> Priority: Major
>
> Hi, We had an HTTP service 3 nodes around Kafka 0.8 . This http service acts
> as a REST api for the publishers and consumers to use middleware intead of
> using kafka client api. Here the when the consumers rebalance is not a major
> issue.
> We wanted to upgrade to kafka 0.11 , we have updated our http services (3
> node cluster) to use new Kafka consumer API , but it takes rebalancing of
> consumer (multiple consumer under same Group) between secs to 5 mins
> (max.poll.interval.ms). Because of this time our http clients are timing out
> and do failover. This rebalancing time is major issue. It is not clear from
> the documentation ,that rebalance activity for the group takes place after
> max.poll.interval.ms or it starts after 3 secs and complete any time with in
> 5 minutes. We tried to reduce max.poll.interval.ms to 15 seconds. but this
> also triggers rebalance internally.
> Below are the other parameters we have set In our service
> max.poll.interval.ms = 30 sec
> seconds heartbeat.interval.ms = 1
> minute session.timeout.ms = 4
> minutes consumer.cache.timeout = 2 min
>
>
> below is the log
> ""2018-03-26 12:53:23,009 [qtp1404928347-11556] INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator -
> (Re-)joining group firstnetportal_001
> ""2018-03-26 12:57:52,793 [qtp1404928347-11556] INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator -
> Successfully joined group firstnetportal_001 with generation 7475
> Please let me know if there are any other application/client use http
> interace in 3 nodes with out any having this issue
>
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
