fapaul commented on a change in pull request #16838:
URL: https://github.com/apache/flink/pull/16838#discussion_r691969751
##########
File path:
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/metrics/KafkaSourceReaderMetrics.java
##########
@@ -150,6 +171,104 @@ public void recordFailedCommit() {
commitsFailed.inc();
}
+ /**
+ * Register {@link MetricNames#IO_NUM_BYTES_IN}.
+ *
+ * @param consumer Kafka consumer
+ */
+ public void registerNumBytesIn(KafkaConsumer<?, ?> consumer) {
+ try {
+ this.bytesConsumedTotalMetric =
+ KafkaConsumerMetricUtils.getKafkaConsumerMetric(
+ consumer,
+ CONSUMER_FETCH_MANAGER_GROUP,
+ BYTES_CONSUMED_TOTAL,
+ (metric) ->
!metric.metricName().tags().containsKey("topic"));
+ } catch (IllegalStateException e) {
+ LOG.warn(
+ String.format(
+ "Error when getting Kafka consumer metric \"%s\". "
+ + "I/O metric \"%s\" will not be reported.
",
+ BYTES_CONSUMED_TOTAL, MetricNames.IO_NUM_BYTES_IN),
+ e);
+ }
+ }
+
+ /**
+ * Add a partition's records-lag metric to tracking list if this partition
never appears before.
+ *
+ * <p>This method also lazily register {@link
+ * org.apache.flink.runtime.metrics.MetricNames#PENDING_RECORDS} in {@link
+ * SourceReaderMetricGroup}
+ *
+ * @param consumer Kafka consumer
+ * @param tp Topic partition
+ */
+ public void maybeAddRecordsLagMetric(KafkaConsumer<?, ?> consumer,
TopicPartition tp) {
+ // Lazily register pendingRecords
+ if (recordsLagMetrics == null) {
+ this.recordsLagMetrics = new ConcurrentHashMap<>();
+ this.sourceReaderMetricGroup.setPendingRecordsGauge(
+ () -> {
+ long pendingRecordsTotal = 0;
+ for (Metric recordsLagMetric :
this.recordsLagMetrics.values()) {
+ pendingRecordsTotal +=
+ ((Double)
recordsLagMetric.metricValue()).longValue();
Review comment:
~If I understand this code correctly we sum the lag of all assigned
partitions. What is the difference between doing so and just using
`records-lag-max`?~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
