PatrickRen commented on a change in pull request #16838:
URL: https://github.com/apache/flink/pull/16838#discussion_r692141091
##########
File path:
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/metrics/KafkaSourceReaderMetrics.java
##########
@@ -150,6 +171,104 @@ public void recordFailedCommit() {
commitsFailed.inc();
}
+ /**
+ * Register {@link MetricNames#IO_NUM_BYTES_IN}.
+ *
+ * @param consumer Kafka consumer
+ */
+ public void registerNumBytesIn(KafkaConsumer<?, ?> consumer) {
+ try {
+ this.bytesConsumedTotalMetric =
+ KafkaConsumerMetricUtils.getKafkaConsumerMetric(
+ consumer,
+ CONSUMER_FETCH_MANAGER_GROUP,
+ BYTES_CONSUMED_TOTAL,
+ (metric) ->
!metric.metricName().tags().containsKey("topic"));
+ } catch (IllegalStateException e) {
+ LOG.warn(
+ String.format(
+ "Error when getting Kafka consumer metric \"%s\". "
+ + "I/O metric \"%s\" will not be reported.
",
+ BYTES_CONSUMED_TOTAL, MetricNames.IO_NUM_BYTES_IN),
+ e);
+ }
+ }
+
+ /**
+ * Add a partition's records-lag metric to tracking list if this partition
never appears before.
+ *
+ * <p>This method also lazily register {@link
+ * org.apache.flink.runtime.metrics.MetricNames#PENDING_RECORDS} in {@link
+ * SourceReaderMetricGroup}
+ *
+ * @param consumer Kafka consumer
+ * @param tp Topic partition
+ */
+ public void maybeAddRecordsLagMetric(KafkaConsumer<?, ?> consumer,
TopicPartition tp) {
+ // Lazily register pendingRecords
Review comment:
The value of ```pendingRecords``` can only be known after the first
```consumer.poll()```. If this metric is registered right on start, we need to
assign an initial value like 0 or -1 before the first poll. 0 might be
confusing since it also means "no lag", and -1 is kinda weird.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
