[
https://issues.apache.org/jira/browse/SPARK-51111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mars updated SPARK-51111:
-------------------------
Description:
When multiple Kafka `InputDStream` are used (number exceeding the Driver's
physical machine `Runtime.getRuntime.availableProcessors`), and the consumers
are configured with the same group id. Driver will remains stuck and log
'Request joining group due to: group is already rebalancing'.
It is more likely to occur in the k8s environment, because the cores of the
Driver is exactly the same as the cores of the physical machine
(`Runtime.getRuntime.availableProcessors`) .
was:When multiple Kafka `InputDStream` are used (number exceeding the
Driver's physical machine `Runtime.getRuntime.availableProcessors`), and the
consumers are configured with the same group id. Driver will remains stuck and
log 'Request joining group due to: group is already rebalancing'.
> Streaming job gets stuck during the startup of Driver for the consumers are
> in the rebalance
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-51111
> URL: https://issues.apache.org/jira/browse/SPARK-51111
> Project: Spark
> Issue Type: Bug
> Components: DStreams
> Affects Versions: 2.0.0
> Reporter: Mars
> Priority: Major
>
> When multiple Kafka `InputDStream` are used (number exceeding the Driver's
> physical machine `Runtime.getRuntime.availableProcessors`), and the consumers
> are configured with the same group id. Driver will remains stuck and log
> 'Request joining group due to: group is already rebalancing'.
> It is more likely to occur in the k8s environment, because the cores of the
> Driver is exactly the same as the cores of the physical machine
> (`Runtime.getRuntime.availableProcessors`) .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
