navina commented on code in PR #9224:
URL: https://github.com/apache/pinot/pull/9224#discussion_r980374389
##########
pinot-spi/src/main/java/org/apache/pinot/spi/stream/RowMetadata.java:
##########
@@ -39,5 +44,27 @@ public interface RowMetadata {
* @return timestamp (epoch in milliseconds) when the row was ingested
upstream
* Long.MIN_VALUE if not available
*/
- long getIngestionTimeMs();
+ long getRecordTimestampMs();
Review Comment:
> What is the semantic that you want here?
Time at which the record was ingested into the stream.
> Time at which the record was generated upstream
This would be event time and it is possible that the event time is used as
the record timestamps (this is possible in Kafka by allowing record to use
producer's timestamp). However, from Pinot's point of view, we only care about
how soon a record became available for consumption.
> In other words, in Pinot, if I write an expression
(System.currentTimeMillis() - getRecordTimeStampMs()) what do you expect this
value to signify (independent of any underlying stream, of course)?
I want this expression `System.currentTimeMillis() - getRecordTimestampMs()`
to indicate the message availability for consumption since the time it became
available in source. It should _tend towards 0_ when the consumer is fully
caught up the upstream data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
