jugomezv commented on code in PR #9994:
URL: https://github.com/apache/pinot/pull/9994#discussion_r1053779647
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java:
##########
@@ -611,6 +616,8 @@ private boolean processStreamEvents(MessageBatch
messagesAndOffsets, long idlePi
if (_segmentLogger.isDebugEnabled()) {
_segmentLogger.debug("empty batch received - sleeping for {}ms",
idlePipeSleepTimeMillis);
}
+ // Record Pinot ingestion delay as zero since we are up-to-date and no
new events
Review Comment:
Discussed this in our online meet up: there should be a contract with Stream
providers not to provide empty batch if there are events. We currently go to
sleep when that happen and this will negatively affect performance for those
streams that return empty and do have events. We also discussed options to
handle this differently for each stream provider using a method in the stream
provider that will indicate if zeroing is valid for the specific stream
provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
