jtao15 commented on a change in pull request #7827:
URL: https://github.com/apache/pinot/pull/7827#discussion_r758681392
##########
File path:
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java
##########
@@ -439,6 +439,18 @@ private boolean validate(TableConfig tableConfig, String
taskType) {
return true;
}
+ /**
+ * Get the max valid merge bucket end time for the segment, LONG.MIN_VALUE
if the segment is not ready to merge
+ */
+ private long getMaxValidBucketEndTimeMsForSegment(SegmentZKMetadata
segmentZKMetadata, long bucketMs, long bufferMs) {
+ if (segmentZKMetadata.getStartTimeMs() / bucketMs * bucketMs + bucketMs >
System.currentTimeMillis() - bufferMs) {
Review comment:
We need the `startTimeMs` here because as long as the `first bucket <=
now - bufferMs`, it's ready to merge. These two values can be different. Say we
have a segment [10/1, 10/3], the first bucketEndTime
(`segmentZKMetadata.getStartTimeMs() / bucketMs * bucketMs + bucketMs`) will be
`10/2 00:00`, the last bucketEndTime (`(segmentZKMetadata.getEndTimeMs() /
bucketMs + 1) * bucketMs`) will be `10/4 00:00`. If `now - bufferMs` = `10/2
00:00`, the segment is ready to merge. For this segment the algorithm will use
`10/2 00:00` to calculate the delay metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
