[jira] [Commented] (KAFKA-6504) Connect: Some per-task-metrics not working
[ https://issues.apache.org/jira/browse/KAFKA-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355785#comment-16355785 ] Robert Yokota commented on KAFKA-6504: -- [~steff1193], it's not just the metric name change. The PR also changes calls to metricGroup.metrics().sensor() with metricGroup.sensor(). This ensures that sensors names will be prepended with the metric group ID to ensure they are not shared across metric groups. > Connect: Some per-task-metrics not working > -- > > Key: KAFKA-6504 > URL: https://issues.apache.org/jira/browse/KAFKA-6504 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 1.0.0 >Reporter: Per Steffensen >Priority: Minor > > Some Kafka-Connect-metrics seems to be wrong with respect to per-task - at > least it seems like MBean > "kafka.connect:type=source-task-metrics,connector=,task=x" > attribute "source-record-active-count" reports the same number for all x > tasks running in the same Kafka-Connect instance/JVM. E.g. if I have a > source-connector "my-connector" with 2 tasks that both run in the same > Kafka-Connect instance, but I know that only one of them actually produces > anything (and therefore can have "active source-records") both > "kafka.connect:type=source-task-metrics,connector=my-connector,task=0" and > "kafka.connect:type=source-task-metrics,connector=my-connector,task=1" goes > up (following each other). It should only go up for the one task that > actually produces something. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6504) Connect: Some per-task-metrics not working
[ https://issues.apache.org/jira/browse/KAFKA-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355562#comment-16355562 ] Randall Hauch commented on KAFKA-6504: -- [~rayokota], ping. Please see [~steff1193]'s comments. Also, [~hachikuji] this issue was not closed when the PR was merged, and there's no fix version. > Connect: Some per-task-metrics not working > -- > > Key: KAFKA-6504 > URL: https://issues.apache.org/jira/browse/KAFKA-6504 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 1.0.0 >Reporter: Per Steffensen >Priority: Minor > > Some Kafka-Connect-metrics seems to be wrong with respect to per-task - at > least it seems like MBean > "kafka.connect:type=source-task-metrics,connector=,task=x" > attribute "source-record-active-count" reports the same number for all x > tasks running in the same Kafka-Connect instance/JVM. E.g. if I have a > source-connector "my-connector" with 2 tasks that both run in the same > Kafka-Connect instance, but I know that only one of them actually produces > anything (and therefore can have "active source-records") both > "kafka.connect:type=source-task-metrics,connector=my-connector,task=0" and > "kafka.connect:type=source-task-metrics,connector=my-connector,task=1" goes > up (following each other). It should only go up for the one task that > actually produces something. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6504) Connect: Some per-task-metrics not working
[ https://issues.apache.org/jira/browse/KAFKA-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355089#comment-16355089 ] Per Steffensen commented on KAFKA-6504: --- Havnt looked that much into the patch from PR 4514, but do you expect that it fixes the problem? Besides tests, the only thing it does is to change a metric name from sink-record-active-count to source-record-active-count. But where was source-record-active-count then controlled before? Please note that my observation is not that source-record-active-count did not exist or was always 0. The observed problem was/is that it seemed to be the same value for all tasks, and that it was increasing for all tasks, even if there were only activity in one task. > Connect: Some per-task-metrics not working > -- > > Key: KAFKA-6504 > URL: https://issues.apache.org/jira/browse/KAFKA-6504 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 1.0.0 >Reporter: Per Steffensen >Priority: Minor > > Some Kafka-Connect-metrics seems to be wrong with respect to per-task - at > least it seems like MBean > "kafka.connect:type=source-task-metrics,connector=,task=x" > attribute "source-record-active-count" reports the same number for all x > tasks running in the same Kafka-Connect instance/JVM. E.g. if I have a > source-connector "my-connector" with 2 tasks that both run in the same > Kafka-Connect instance, but I know that only one of them actually produces > anything (and therefore can have "active source-records") both > "kafka.connect:type=source-task-metrics,connector=my-connector,task=0" and > "kafka.connect:type=source-task-metrics,connector=my-connector,task=1" goes > up (following each other). It should only go up for the one task that > actually produces something. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6504) Connect: Some per-task-metrics not working
[ https://issues.apache.org/jira/browse/KAFKA-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354172#comment-16354172 ] ASF GitHub Bot commented on KAFKA-6504: --- hachikuji closed pull request #4514: KAFKA-6504: Fix creation of a sensor to be specific to a metric group so it is not shared URL: https://github.com/apache/kafka/pull/4514 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java index 69614948147..4ee8ad6a808 100644 --- a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java +++ b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java @@ -677,34 +677,34 @@ public SinkTaskMetricsGroup(ConnectorTaskId id, ConnectMetrics connectMetrics) { // prevent collisions by removing any previously created metrics in this group. metricGroup.close(); -sinkRecordRead = metricGroup.metrics().sensor("sink-record-read"); +sinkRecordRead = metricGroup.sensor("sink-record-read"); sinkRecordRead.add(metricGroup.metricName(registry.sinkRecordReadRate), new Rate()); sinkRecordRead.add(metricGroup.metricName(registry.sinkRecordReadTotal), new Total()); -sinkRecordSend = metricGroup.metrics().sensor("sink-record-send"); +sinkRecordSend = metricGroup.sensor("sink-record-send"); sinkRecordSend.add(metricGroup.metricName(registry.sinkRecordSendRate), new Rate()); sinkRecordSend.add(metricGroup.metricName(registry.sinkRecordSendTotal), new Total()); -sinkRecordActiveCount = metricGroup.metrics().sensor("sink-record-active-count"); +sinkRecordActiveCount = metricGroup.sensor("sink-record-active-count"); sinkRecordActiveCount.add(metricGroup.metricName(registry.sinkRecordActiveCount), new Value()); sinkRecordActiveCount.add(metricGroup.metricName(registry.sinkRecordActiveCountMax), new Max()); sinkRecordActiveCount.add(metricGroup.metricName(registry.sinkRecordActiveCountAvg), new Avg()); -partitionCount = metricGroup.metrics().sensor("partition-count"); +partitionCount = metricGroup.sensor("partition-count"); partitionCount.add(metricGroup.metricName(registry.sinkRecordPartitionCount), new Value()); -offsetSeqNum = metricGroup.metrics().sensor("offset-seq-number"); +offsetSeqNum = metricGroup.sensor("offset-seq-number"); offsetSeqNum.add(metricGroup.metricName(registry.sinkRecordOffsetCommitSeqNum), new Value()); -offsetCompletion = metricGroup.metrics().sensor("offset-commit-completion"); +offsetCompletion = metricGroup.sensor("offset-commit-completion"); offsetCompletion.add(metricGroup.metricName(registry.sinkRecordOffsetCommitCompletionRate), new Rate()); offsetCompletion.add(metricGroup.metricName(registry.sinkRecordOffsetCommitCompletionTotal), new Total()); -offsetCompletionSkip = metricGroup.metrics().sensor("offset-commit-completion-skip"); +offsetCompletionSkip = metricGroup.sensor("offset-commit-completion-skip"); offsetCompletionSkip.add(metricGroup.metricName(registry.sinkRecordOffsetCommitSkipRate), new Rate()); offsetCompletionSkip.add(metricGroup.metricName(registry.sinkRecordOffsetCommitSkipTotal), new Total()); -putBatchTime = metricGroup.metrics().sensor("put-batch-time"); +putBatchTime = metricGroup.sensor("put-batch-time"); putBatchTime.add(metricGroup.metricName(registry.sinkRecordPutBatchTimeMax), new Max()); putBatchTime.add(metricGroup.metricName(registry.sinkRecordPutBatchTimeAvg), new Avg()); } diff --git a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java index a172cdb45f0..473e2359578 100644 --- a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java +++ b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java @@ -509,7 +509,7 @@ public SourceTaskMetricsGroup(ConnectorTaskId id, ConnectMetrics connectMetrics) pollTime.add(metricGroup.metricName(registry.sourceRecordPollBatchTimeMax), new Max()); pollTime.add(metricGroup.metricName(registry.sourceRecordPollBatchTimeAvg), n
[jira] [Commented] (KAFKA-6504) Connect: Some per-task-metrics not working
[ https://issues.apache.org/jira/browse/KAFKA-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349545#comment-16349545 ] ASF GitHub Bot commented on KAFKA-6504: --- rayokota opened a new pull request #4514: KAFKA-6504: Fix source task metric caused by copy-paste error URL: https://github.com/apache/kafka/pull/4514 This is a simple change to correct "sink-active-record-count" with "source-active-record-count" for the This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Connect: Some per-task-metrics not working > -- > > Key: KAFKA-6504 > URL: https://issues.apache.org/jira/browse/KAFKA-6504 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 1.0.0 >Reporter: Per Steffensen >Priority: Minor > > Some Kafka-Connect-metrics seems to be wrong with respect to per-task - at > least it seems like MBean > "kafka.connect:type=source-task-metrics,connector=,task=x" > attribute "source-record-active-count" reports the same number for all x > tasks running in the same Kafka-Connect instance/JVM. E.g. if I have a > source-connector "my-connector" with 2 tasks that both run in the same > Kafka-Connect instance, but I know that only one of them actually produces > anything (and therefore can have "active source-records") both > "kafka.connect:type=source-task-metrics,connector=my-connector,task=0" and > "kafka.connect:type=source-task-metrics,connector=my-connector,task=1" goes > up (following each other). It should only go up for the one task that > actually produces something. -- This message was sent by Atlassian JIRA (v7.6.3#76005)