navina commented on code in PR #9713:
URL: https://github.com/apache/pinot/pull/9713#discussion_r1186547382


##########
pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisStreamMetadataProvider.java:
##########
@@ -189,6 +192,24 @@ private boolean 
consumedEndOfShard(StreamPartitionMsgOffset startCheckpoint,
   }
 
   @Override
-  public void close() {
+  public Map<String, PartitionLagState> getCurrentPartitionLagState(
+      Map<String, ConsumerPartitionState> currentPartitionStateMap) {
+    Map<String, PartitionLagState> perPartitionLag = new HashMap<>();
+    for (Map.Entry<String, ConsumerPartitionState> entry: 
currentPartitionStateMap.entrySet()) {
+      ConsumerPartitionState partitionState = entry.getValue();
+      // Compute record-availability
+      String recordAvailabilityLag = "UNKNOWN";
+      RowMetadata lastProcessedMessageMetadata = 
partitionState.getLastProcessedRowMetadata();
+      if (lastProcessedMessageMetadata != null && 
partitionState.getLastProcessedTimeMs() > 0) {
+        long availabilityLag = partitionState.getLastProcessedTimeMs()
+            - lastProcessedMessageMetadata.getRecordIngestionTimeMs();
+        recordAvailabilityLag = String.valueOf(availabilityLag);
+      }
+      perPartitionLag.put(entry.getKey(), new 
KinesisConsumerPartitionLag(recordAvailabilityLag));

Review Comment:
   I agree that the kinesis official metric will be the most accurate one. Let 
me re-think if this approach makes sense because we now have a mechanism to 
compute ingestionDelay in the realtime engine itself based on this apprx 
arrival time. That is ideally what we should be using uniformly across all 
connectors. 
   
   > Also, we should add a javadoc here declaring what happens to lag in case 
of a reshard operation.
   
   Good point. When re-shard happens, I am guessing one or more of the 
partitions may no longer have any data being ingested and cause this lag to 
continuously increase, leading to false alarms. Is my understanding right? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to