Andrew Flower created KAFKA-21570:
-------------------------------------
Summary: Task-level connector-type JMX attribute returns a
ConnectorType enum, breaking remote JMX clients
Key: KAFKA-21570
URL: https://issues.apache.org/jira/browse/KAFKA-21570
Project: Kafka
Issue Type: Bug
Components: connect
Affects Versions: 4.2.1, 4.1.0
Reporter: Andrew Flower
KAFKA-18988 (4.1.0) added a {{connector-type}} attribute to
{{kafka.connect:type=connector-task-metrics}}. Its value is the
{{org.apache.kafka.connect.runtime.rest.entities.ConnectorType}} enum rather
than a String:
{code:java}
// WorkerTask.java (4.1, 4.2 and trunk, line 468)
metricGroup.addValueMetric(registry.taskConnectorType, now ->
pluginsMetadata.connectorType());
{code}
Over remote JMX (RMI), attribute values are Java-serialized, so any client
without the Connect runtime on its classpath fails to read it:
{noformat}
java.rmi.UnmarshalException: error unmarshalling return; nested exception is:
java.lang.ClassNotFoundException:
org.apache.kafka.connect.runtime.rest.entities.ConnectorType (no security
manager: RMI class loader disabled)
{noformat}
Every other Connect metric is a number or a String. The connector-level
{{connector-type}} metric already returns a String ("source" / "sink"), and
{{JmxReporter}} advertises every attribute as {{double}}.
*Impact*
Generic remote JMX collectors don't ship Kafka's jars. The Datadog agent's JMX
check treats the {{UnmarshalException}} as a broken connection and drops all
metrics for the worker, not just this attribute. We lost every
{{kafka.connect}} and JVM metric after upgrading an embedded Connect worker
from 4.0.0 to 4.2.1, and on a Connect cluster running a Confluent Platform 8.x
image. Confluent support reproduced it on CP 8.4.1; CP 8.0.x is not affected.
*Steps to reproduce*
# Run a Connect 4.1+ worker with remote JMX enabled and at least one connector
task.
# Connect with a JMX client that doesn't have Kafka on its classpath (e.g.
jmxterm or JConsole).
# Read {{connector-type}} on
{{kafka.connect:type=connector-task-metrics,connector=<name>,task=<id>}}.
*Workaround*
Exclude the attribute in the JMX collector, or collect only named attributes.
*Fix*
Return {{ConnectorType.toString()}}, which yields "source", "sink" or
"unknown", matching the connector-level metric:
https://github.com/apache/kafka/pull/24249
{code:java}
metricGroup.addValueMetric(registry.taskConnectorType, now ->
pluginsMetadata.connectorType().toString());
{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)