autumnust commented on a change in pull request #2928: GOBBLIN-1087: Track and
report histogram of observed lag from Gobblin…
URL: https://github.com/apache/incubator-gobblin/pull/2928#discussion_r393982314
##########
File path:
gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractorStatsTracker.java
##########
@@ -161,14 +208,24 @@ public void resetStartFetchEpochTime(int partitionIdx) {
* @param decodeStartTime the time instant immediately before a record
decoding begins.
* @param recordSizeInBytes the size of the decoded record in bytes.
* @param logAppendTimestamp the log append time of the {@link
org.apache.gobblin.kafka.client.KafkaConsumerRecord}.
+ * @param recordCreationTimestamp the time of the {@link
org.apache.gobblin.kafka.client.KafkaConsumerRecord}.
*/
- public void onDecodeableRecord(int partitionIdx, long readStartTime, long
decodeStartTime, long recordSizeInBytes, long logAppendTimestamp) {
+ public void onDecodeableRecord(int partitionIdx, long readStartTime, long
decodeStartTime, long recordSizeInBytes, long logAppendTimestamp, long
recordCreationTimestamp) {
this.statsMap.computeIfPresent(this.partitions.get(partitionIdx), (k, v)
-> {
long currentTime = System.nanoTime();
v.processedRecordCount++;
v.partitionTotalSize += recordSizeInBytes;
v.decodeRecordTime += currentTime - decodeStartTime;
v.readRecordTime += currentTime - readStartTime;
+ if (this.observedLagHistogram != null && recordCreationTimestamp > 0) {
+ long observedLag = System.currentTimeMillis() -
recordCreationTimestamp;
Review comment:
Shall we use "latency" instead since "lag" internally refers to number of
events behind the head of stream ? Maybe you can come up with better words, it
is just my intuition of "lag" refers to something else.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
