[
https://issues.apache.org/jira/browse/FLINK-16553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Paul updated FLINK-16553:
--------------------------------
Description:
When using the Kafka universal connector, currently not all KafkaFetcher
metrics
([link|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetcherMetricsRegistry.java])
which are exposed through the KafkaConsumer are accessible within the Flink
metrics system.
Especially, all metrics which are related to topics and partitions are not
available. The KafkaConsumer internally only registers those metrics after it
has fetched some records.
Unfortunately, at the moment Flink only checks the available metrics right
after the initialization of the KafkaConsumer when no records are polled, yet.
was:
When using the Kafka universal connector, currently not all KafkaFetcher
metrics ([link|
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetcherMetricsRegistry.java]
which are exposed through the KafkaConsumer are accessible within the Flink
metrics system.
Especially, all metrics which are related to topics and partitions are not
available. The KafkaConsumer internally only registers those metrics after it
has fetched some records.
Unfortunately, at the moment Flink only checks the available metrics right
after the initialization of the KafkaConsumer when no records are polled, yet.
> KafkaFetcher topic/partition metrics
> ------------------------------------
>
> Key: FLINK-16553
> URL: https://issues.apache.org/jira/browse/FLINK-16553
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Runtime / Metrics
> Reporter: Fabian Paul
> Priority: Major
>
> When using the Kafka universal connector, currently not all KafkaFetcher
> metrics
> ([link|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetcherMetricsRegistry.java])
> which are exposed through the KafkaConsumer are accessible within the Flink
> metrics system.
> Especially, all metrics which are related to topics and partitions are not
> available. The KafkaConsumer internally only registers those metrics after it
> has fetched some records.
> Unfortunately, at the moment Flink only checks the available metrics right
> after the initialization of the KafkaConsumer when no records are polled, yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
