npawar commented on a change in pull request #7605:
URL: https://github.com/apache/pinot/pull/7605#discussion_r754661298
##########
File path:
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGenerator.java
##########
@@ -317,6 +325,13 @@ private long getWatermarkMs(String realtimeTableName,
List<SegmentZKMetadata> co
realtimeToOfflineSegmentsTaskMetadata = new
RealtimeToOfflineSegmentsTaskMetadata(realtimeTableName, watermarkMs);
_clusterInfoAccessor.setRealtimeToOfflineSegmentsTaskMetadata(realtimeToOfflineSegmentsTaskMetadata);
}
+
+ _realtimeToOfflineSegmentsTaskMetadata =
realtimeToOfflineSegmentsTaskMetadata;
+
return realtimeToOfflineSegmentsTaskMetadata.getWatermarkMs();
}
+
+ private void setWatermarkMs(long watermarkMs) {
+ _realtimeToOfflineSegmentsTaskMetadata.setWatermarkMs(watermarkMs);
Review comment:
this will only set it into the RealtimeToOfflineSegmentsTaskMetadata
object. It will not get persisted into zk for the next task.
##########
File path:
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGenerator.java
##########
@@ -204,6 +205,13 @@ public String getTaskType() {
if (segmentNames.isEmpty() || skipGenerate) {
LOGGER.info("Found no eligible segments for task: {} with window [{} -
{}). Skipping task generation", taskType,
windowStartMs, windowEndMs);
+ if (!skipGenerate) {
Review comment:
can we go back to line 164 in this iteration itself, if we detect this?
So that we dont have to wait for the next scheduling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
