HenryCaiHaiying commented on code in PR #17025:
URL: https://github.com/apache/iceberg/pull/17025#discussion_r3562880424
##########
kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java:
##########
@@ -133,6 +140,11 @@ void process() {
}
}
+ @Override
+ protected void recordConsumeTime(long elapsedMs) {
+ coordinatorMetrics.recordConsume(elapsedMs);
Review Comment:
I will remove the consume-available-time sensor and replace it with 2
smaller time-sensors: channel-message-read-time, channel-message-process-time
There are several steps during consumeAvailable():
1. consumer.poll() -- Kafka consumer has its own metrics covering this
portion
2. Avro deserialize the message -- covered by the new sensor
channel-message-read-time
3. Process the event/message and generate the output:
a) On Coordinator, the big time chunk is on commit(). -- We have the
commit-time metrics for this, so we don't have count it again in
channel-message-process-time (but I am OK with doing the double counting here
since commit() can be called from partial commit path as well)
b) On Worker, the big time chunk is on preparing the DataWritten and
DataComplete events/messages. -- covered by the new sensor
channel-message-process-time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
