Aaron created KAFKA-17652:
-----------------------------
Summary: Missing Kafka MBean Attributes After Rebalancing with
kafka-clients 3.7.1
Key: KAFKA-17652
URL: https://issues.apache.org/jira/browse/KAFKA-17652
Project: Kafka
Issue Type: Bug
Reporter: Aaron
We are using Java kafka-clients version 3.7.1 to consume Kafka messages. After
a consumer group rebalancing event, certain MBean attributes, such as
'records-lag' on the 'consumer-fetch-manager-metrics' MBean, become
unavailable. The 'records-lag' attribute reappears after receiving a new
message, but until then, it remains missing.
This poses an issue when using Dynatrace to monitor our application, as its
agent regularly reads MBean attributes to collect Kafka metrics. After a
rebalancing (e.g., during a rolling update), the kafka-client continuously logs
the exception "Error getting JMX attribute" for reading 'records-lag' attribute
when there are no new messages.
We have reproduced this issue locally. The rebalancing logs are shown below.
Before this log snippet, the 'records-lag' attribute was present, and
afterward, it disappeared.
{color:#505f79}{{2024-09-29T15:22:30.038+08:00 INFO 14864 --- [kafkajmx] [
Thread-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer
clientId=consumer-jmx-1, groupId=jmx] Request joining group due to: group is
already rebalancing}}
{{2024-09-29T15:22:30.038+08:00 INFO 14864 --- [kafkajmx] [ Thread-1]
k.c.c.i.ConsumerRebalanceListenerInvoker : [Consumer clientId=consumer-jmx-1,
groupId=jmx] Revoke previously assigned partitions abc-0}}
{{2024-09-29T15:22:30.038+08:00 INFO 14864 --- [kafkajmx] [ Thread-1]
o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-jmx-1,
groupId=jmx] (Re-)joining group}}
{{2024-09-29T15:22:30.038+08:00 INFO 14864 --- [kafkajmx] [ Thread-1]
o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-jmx-1,
groupId=jmx] Successfully joined group with generation
Generation\{generationId=113,
memberId='consumer-jmx-1-c5cc4f1e-16c0-4cc4-a2b4-ee8da7efad3b',
protocol='range'}}}
{{2024-09-29T15:22:30.038+08:00 INFO 14864 --- [kafkajmx] [ Thread-1]
o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-jmx-1,
groupId=jmx] Finished assignment for group at generation 113:
\{consumer-jmx-1-c5cc4f1e-16c0-4cc4-a2b4-ee8da7efad3b=Assignment(partitions=[abc-0]),
consumer-jmx-1-f2c14e7a-95a9-4af5-9b4f-fbe2c68f1c49=Assignment(partitions=[])}}}
{{2024-09-29T15:22:30.050+08:00 INFO 14864 --- [kafkajmx] [ Thread-1]
o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-jmx-1,
groupId=jmx] Successfully synced group in generation
Generation\{generationId=113,
memberId='consumer-jmx-1-c5cc4f1e-16c0-4cc4-a2b4-ee8da7efad3b',
protocol='range'}}}
{{2024-09-29T15:22:30.050+08:00 INFO 14864 --- [kafkajmx] [ Thread-1]
o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-jmx-1,
groupId=jmx] Notifying assignor about the new Assignment(partitions=[abc-0])}}
{{2024-09-29T15:22:30.050+08:00 INFO 14864 --- [kafkajmx] [ Thread-1]
k.c.c.i.ConsumerRebalanceListenerInvoker : [Consumer clientId=consumer-jmx-1,
groupId=jmx] Adding newly assigned partitions: abc-0}}{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
