Sairam Polavarapu created KAFKA-6779:
----------------------------------------
Summary: Kafka connect JMX metrics returns inconsistent metrics
while running multiple connectors
Key: KAFKA-6779
URL: https://issues.apache.org/jira/browse/KAFKA-6779
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 1.0.1, 1.1.0, 1.0.0
Environment: Linux
Reporter: Sairam Polavarapu
JMX Kafka Connect metrics is returning different values for different runs. We
are running the Kafka connect in distributed mode on a 3 node cluster which
allowed us the submit the connectors using rest API calls. One of the use
cases, where we are using MySQL Source Connector (Confluent) and Amazon s3 Sink
connector (Confluent) as pipeline and need to get the counts of data received
by source and delivered to sink. Both the connectors are submitted one after
the other and the scenario what we have observed is when a new connector is
submitted using the rest call the existing connector metrics are zeroes and
restarted. This is a known case when the actual connectors are restarted but in
our case the source connector was not restarted. We have observed the similar
behavior when using Kafka 1.0.0, 1.0.1 and even after the fixes released in
Kafka 1.1.0 still the issue persists. And also on a side note when the Metrics
were tested on the local machine with Kafka connect in distributed mode on a
single instance the counts are matching between the source and sink. This issue
is only on how JMX is fetching the data. The physical data that is being
handled through the connectors don't have any issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
