kfaraz commented on code in PR #16667:
URL: https://github.com/apache/druid/pull/16667#discussion_r1675713527
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillUnusedSegmentsTask.java:
##########
@@ -300,6 +356,81 @@ private NavigableMap<DateTime, List<TaskLock>>
getNonRevokedTaskLockMap(TaskActi
return taskLockMap;
}
+ private List<DataSegment> findUnreferencedSegments(
+ List<DataSegment> unusedSegments,
+ Map<String, String> upgradedFromSegmentIds,
+ TaskActionClient taskActionClient
+ ) throws IOException
+ {
+ if (unusedSegments.isEmpty() || upgradedFromSegmentIds.isEmpty()) {
+ return unusedSegments;
+ }
+
+ final Set<String> upgradedSegmentIds = new HashSet<>();
+ for (DataSegment segment : unusedSegments) {
+ final String id = segment.getId().toString();
+ upgradedSegmentIds.add(id);
+ if (upgradedFromSegmentIds.containsKey(id)) {
+ upgradedSegmentIds.add(upgradedFromSegmentIds.get(id));
+ }
+ }
+
+ final Map<String, Set<String>> upgradedToSegmentIds;
+ try {
+ upgradedToSegmentIds = taskActionClient.submit(
+ new RetrieveUpgradedToSegmentIdsAction(getDataSource(),
upgradedSegmentIds)
+ ).getUpgradedToSegmentIds();
+ }
+ catch (Exception e) {
+ LOG.warn(
+ e,
+ "Could not retrieve UpgradedToSegmentsResponse using task
action[retrieveUpgradedToSegmentIds]."
+ + " Overlord may be on an older version."
+ );
+ return unusedSegments;
+ }
+
+ final Set<String> existingSegmentIds
+ = taskActionClient.submit(new
RetrieveSegmentsByIdAction(getDataSource(), upgradedSegmentIds))
Review Comment:
Sure, that works for me. Make sure the javadoc of the action captures
exactly what is going to be included in the map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
