AmatyaAvadhanula commented on code in PR #16667:
URL: https://github.com/apache/druid/pull/16667#discussion_r1674156880
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillUnusedSegmentsTask.java:
##########
@@ -300,6 +356,81 @@ private NavigableMap<DateTime, List<TaskLock>>
getNonRevokedTaskLockMap(TaskActi
return taskLockMap;
}
+ private List<DataSegment> findUnreferencedSegments(
+ List<DataSegment> unusedSegments,
+ Map<String, String> upgradedFromSegmentIds,
+ TaskActionClient taskActionClient
+ ) throws IOException
+ {
+ if (unusedSegments.isEmpty() || upgradedFromSegmentIds.isEmpty()) {
+ return unusedSegments;
+ }
+
+ final Set<String> upgradedSegmentIds = new HashSet<>();
+ for (DataSegment segment : unusedSegments) {
+ final String id = segment.getId().toString();
+ upgradedSegmentIds.add(id);
+ if (upgradedFromSegmentIds.containsKey(id)) {
+ upgradedSegmentIds.add(upgradedFromSegmentIds.get(id));
+ }
+ }
+
+ final Map<String, Set<String>> upgradedToSegmentIds;
+ try {
+ upgradedToSegmentIds = taskActionClient.submit(
+ new RetrieveUpgradedToSegmentIdsAction(getDataSource(),
upgradedSegmentIds)
+ ).getUpgradedToSegmentIds();
+ }
+ catch (Exception e) {
+ LOG.warn(
+ e,
+ "Could not retrieve UpgradedToSegmentsResponse using task
action[retrieveUpgradedToSegmentIds]."
+ + " Overlord may be on an older version."
+ );
+ return unusedSegments;
+ }
+
+ final Set<String> existingSegmentIds
+ = taskActionClient.submit(new
RetrieveSegmentsByIdAction(getDataSource(), upgradedSegmentIds))
Review Comment:
I have pushed changes with the new action specifications. However, the
semantics for it seem wrong to me.
I think we shouldn't include a mapping from a segment to itself in this
action for homogeneity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
