[ https://issues.apache.org/jira/browse/KAFKA-16028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823021#comment-17823021 ]
Luke Chen commented on KAFKA-16028: ----------------------------------- [~omersiar], it would be good if you could attach complete log in the broker side. Thanks. > AdminClient fails to describe consumer group > -------------------------------------------- > > Key: KAFKA-16028 > URL: https://issues.apache.org/jira/browse/KAFKA-16028 > Project: Kafka > Issue Type: Bug > Components: admin, clients, consumer, log > Affects Versions: 2.8.2, 3.6.1 > Reporter: Ömer Şiar Baysal > Priority: Critical > > Dear Team, > We have been investigating some quirky behavior around admin client. Here is > our conclusion: > - Due to some bug (or a feature not known by us) AdminClient (both 2.8 and > 3.6) fails to describe one of the consumer groups (with no known problems > about it) > - Pure GoLang admin client does not have the problem > (github.com/twmb/franz-go) and able to describe the consumer group. > We tried to understand what may cause the issue, first of all, the Java > client 2.8 reported, > {quote}kafka-consumer-groups --bootstrap-server broker:9092 --describe > --group 'problematic-consumer' > Error: Executing consumer group command failed due to > org.apache.kafka.common.errors.LeaderNotAvailableException: There is no > leader for this topic-partition as we are in the middle of a leadership > election. > java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.LeaderNotAvailableException: There is no > leader for this topic-partition as we are in the middle of a leadership > election. > {quote} > we waited if this is a transient error but it turned out it is not, there was > no election for the given topic > But it was not clear which topic admin client was talking about so TRACE log > revealed some more information: > {quote}[2023-12-18 10:36:38,434] DEBUG [AdminClient clientId=adminclient-1] > Sending LIST_OFFSETS request with header RequestHeader(apiKey=LIST_OFFSETS, > apiVersion=6, clientId=adminclient-1, correlationId=30) and timeout 4997 to > node 40: ListOffsetsRequestData(replicaId=-1, isolationLevel=0, > topics=[ListOffsetsTopic(name='problematic-topic', > partitions=[ListOffsetsPartition(partitionIndex=4, currentLeaderEpoch=-1, > timestamp=-1, maxNumOffsets=1), ListOffsetsPartition(partitionIndex=5, > currentLeaderEpoch=-1, timestamp=-1, maxNumOffsets=1)])]) > (org.apache.kafka.clients.NetworkClient) > [2023-12-18 10:36:38,434] TRACE [AdminClient clientId=adminclient-1] Entering > KafkaClient#poll(timeout=4997) > (org.apache.kafka.clients.admin.KafkaAdminClient) > [2023-12-18 10:36:38,435] TRACE [AdminClient clientId=adminclient-1] > KafkaClient#poll retrieved 0 response(s) > (org.apache.kafka.clients.admin.KafkaAdminClient) > [2023-12-18 10:36:38,435] TRACE [AdminClient clientId=adminclient-1] Trying > to choose nodes for [] at 1702884998435 > (org.apache.kafka.clients.admin.KafkaAdminClient) > [2023-12-18 10:36:38,435] TRACE [AdminClient clientId=adminclient-1] Entering > KafkaClient#poll(timeout=4995) > (org.apache.kafka.clients.admin.KafkaAdminClient) > Error: Executing consumer group command failed due to > org.apache.kafka.common.errors.LeaderNotAvailableException: There is no > leader for this topic-partition as we are in the middle of a leadership > election. > [2023-12-18 10:36:38,436] DEBUG [AdminClient clientId=adminclient-1] Received > LIST_OFFSETS response from node 40 for request with header > RequestHeader(apiKey=LIST_OFFSETS, apiVersion=6, clientId=adminclient-1, > correlationId=30): ListOffsetsResponseData(throttleTimeMs=0, > topics=[ListOffsetsTopicResponse(name='problematic-topic', > partitions=[ListOffsetsPartitionResponse(partitionIndex=5, errorCode=0, > oldStyleOffsets=[], timestamp=-1, offset=822516, leaderEpoch=113, > followerRestorePointObjectId=AAAAAAAAAAAAAAAAAAAAAA, > followerRestorePointEpoch=0), ListOffsetsPartitionResponse(partitionIndex=4, > errorCode=0, oldStyleOffsets=[], timestamp=-1, offset=827297, leaderEpoch=93, > followerRestorePointObjectId=AAAAAAAAAAAAAAAAAAAAAA, > followerRestorePointEpoch=0)])]) (org.apache.kafka.clients.NetworkClient) > [2023-12-18 10:36:38,436] TRACE [AdminClient clientId=adminclient-1] > KafkaClient#poll retrieved 1 response(s) > (org.apache.kafka.clients.admin.KafkaAdminClient) > [2023-12-18 10:36:38,437] TRACE [AdminClient clientId=adminclient-1] > Call(callName=listOffsets on broker 40, deadlineMs=1702885003430, tries=0, > nextAllowedTryMs=0) got response ListOffsetsResponseData(throttleTimeMs=0, > topics=[ListOffsetsTopicResponse(name='problematic-topic', > partitions=[ListOffsetsPartitionResponse(partitionIndex=5, errorCode=0, > oldStyleOffsets=[], timestamp=-1, offset=822516, leaderEpoch=113, > followerRestorePointObjectId=AAAAAAAAAAAAAAAAAAAAAA, > followerRestorePointEpoch=0), ListOffsetsPartitionResponse(partitionIndex=4, > errorCode=0, oldStyleOffsets=[], timestamp=-1, offset=827297, leaderEpoch=93, > followerRestorePointObjectId=AAAAAAAAAAAAAAAAAAAAAA, > followerRestorePointEpoch=0)])]) > (org.apache.kafka.clients.admin.KafkaAdminClient) > [2023-12-18 10:36:38,437] TRACE [AdminClient clientId=adminclient-1] Trying > to choose nodes for [] at 1702884998436 > (org.apache.kafka.clients.admin.KafkaAdminClient) > [2023-12-18 10:36:38,437] TRACE [AdminClient clientId=adminclient-1] Entering > KafkaClient#poll(timeout=299161) > (org.apache.kafka.clients.admin.KafkaAdminClient) > java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.LeaderNotAvailableException: There is no > leader for this topic-partition as we are in the middle of a leadership > election. > {quote} > adminclient version 3.6 is not returning this error, but it fails with a > timeout after retrying is exhausted. > We have also took a look into "problematic-topic", reassigned replicas to > other brokers, ran kafka-leader-election over all partitions, did not help -- This message was sent by Atlassian Jira (v8.20.10#820010)