adithyachakilam commented on PR #17491: URL: https://github.com/apache/druid/pull/17491#issuecomment-2660040468
> IIUC, we are trying to calculate the lag in terms of "how many minutes worth of records is the supervisor yet to process". This value is fairly similar to the message gap but not quite the same. The message gap is the "difference between the current timestamp of the system and the timestamp of the latest ingested record". Whereas here, I think we want the "difference between the timestamp of the latest ingested record and the timestamp of the latest record in the stream". @kfaraz I think having this approach would fail for late/early arriving data which happens very frequently with the streams that we are trying to deal. The best approach we have is to calculate the production rate and then make a best guess of how many minutes are we lagging behind. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
