[
https://issues.apache.org/jira/browse/KAFKA-19558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18059997#comment-18059997
]
Mark Paget commented on KAFKA-19558:
------------------------------------
I guess a simple work around could be to either produce an array and then
iterate over it, or dynamically traverse. Especially as mentioned can describe
individually.
For example in Unix Shell
{code:java}
for group in `kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list`
do
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group
${group}
done {code}
Although I have not tested with 380 consumer groups and 500+ topics :)
Which Kafka version are you on now as I would think there has been lots of
improvements since 2.7.1.
> kafka-consumer-groups.sh --describe --all-groups command times out on large
> clusters with many consumer groups
> --------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-19558
> URL: https://issues.apache.org/jira/browse/KAFKA-19558
> Project: Kafka
> Issue Type: Bug
> Components: metrics, offset manager
> Affects Versions: 2.7.1
> Reporter: zhangtongr
> Assignee: Ranuga Disansa
> Priority: Blocker
>
> Description:
> When running the following command in a Kafka cluster with a large number of
> consumer groups (over 380) and topics (over 500), the
> kafka-consumer-groups.sh --describe --all-groups operation consistently times
> out and fails to return results.
> Command used:
> ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe
> --all-groups
> Observed behavior:
> The command fails with a TimeoutException, and no consumer group information
> is returned. The following stack trace is observed:
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TimeoutException:
> Call(callName=describeConsumerGroups, deadlineMs=1753170317381, tries=1,
> nextAllowedTryMs=1753170317482) timed out at 1753170317382 after 1 attempt(s)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> ...
> Caused by: org.apache.kafka.common.errors.TimeoutException:
> Call(callName=describeConsumerGroups, deadlineMs=..., tries=1, ...) timed out
> Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting
> to send the call. Call: describeConsumerGroups
> Expected behavior:
> The command should be able to return the description of all consumer groups,
> or at least fail more gracefully. Ideally, there should be:
> A way to paginate or batch the describe operation;
> Or configuration options to increase internal timeout thresholds;
> Or better recommendations for dealing with large clusters.
> Additional context:
> Manually describing individual consumer groups via --group performs as
> expected and returns data quickly.
> The issue appears to scale linearly with the number of consumer groups.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
