rohityadav1993 commented on code in PR #15863:
URL: https://github.com/apache/pinot/pull/15863#discussion_r2105698335
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompactmerge/UpsertCompactMergeTaskGenerator.java:
##########
@@ -214,15 +216,24 @@ public List<PinotTaskConfig>
generateTasks(List<TableConfig> tableConfigs) {
continue;
}
// TODO see if multiple groups of same partition can be added
- Map<String, String> configs = new
HashMap<>(getBaseTaskConfigs(tableConfig,
- groups.get(0).stream().map(x ->
x.getSegmentZKMetadata().getSegmentName()).collect(Collectors.toList())));
+
+ List<String> segmentNames =
+ groups.get(0).stream().map(x ->
x.getSegmentZKMetadata().getSegmentName()).collect(Collectors.toList());
+ //get max creation time for the segments across all servers. This will
be used as the creation time of the
+ // merge segment
+ Long maxCreationTimeMillis =
+ getMaxCreationTimeMillis(tableNameWithType, segmentNames,
serverToSegments, serverToEndpoints,
Review Comment:
Let's emit a metric when newest merging segment and compacted segment are of
UploadedRealtimeSegment. Should we also consider skipping a task where a set of
segments is selected where the newest segment is already a compacted segment so
this scenario does not happen.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/util/ServerSegmentMetadataReader.java:
##########
@@ -399,6 +400,34 @@ public ValidDocIdsBitmapResponse
getValidDocIdsBitmapFromServer(String tableName
return response;
}
+ /**
+ * Retrieves the creation time metadata for each segment of a given table
from the each servers the segment is hosted.
+ *
+ * @param tableNameWithType The name of the table with type (e.g.,
myTable_OFFLINE)
+ * @param serverToSegmentsMap A map from server instance to the list of
segments it hosts
+ * @param serverToEndpoints A BiMap from server instance to its admin
endpoint
Review Comment:
typo: BitMap*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
