AmatyaAvadhanula commented on code in PR #16667:
URL: https://github.com/apache/druid/pull/16667#discussion_r1674160428
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillUnusedSegmentsTask.java:
##########
@@ -300,6 +356,81 @@ private NavigableMap<DateTime, List<TaskLock>>
getNonRevokedTaskLockMap(TaskActi
return taskLockMap;
}
+ private List<DataSegment> findUnreferencedSegments(
+ List<DataSegment> unusedSegments,
+ Map<String, String> upgradedFromSegmentIds,
+ TaskActionClient taskActionClient
+ ) throws IOException
+ {
+ if (unusedSegments.isEmpty() || upgradedFromSegmentIds.isEmpty()) {
+ return unusedSegments;
+ }
+
+ final Set<String> upgradedSegmentIds = new HashSet<>();
+ for (DataSegment segment : unusedSegments) {
+ final String id = segment.getId().toString();
+ upgradedSegmentIds.add(id);
+ if (upgradedFromSegmentIds.containsKey(id)) {
+ upgradedSegmentIds.add(upgradedFromSegmentIds.get(id));
+ }
+ }
+
+ final Map<String, Set<String>> upgradedToSegmentIds;
+ try {
+ upgradedToSegmentIds = taskActionClient.submit(
+ new RetrieveUpgradedToSegmentIdsAction(getDataSource(),
upgradedSegmentIds)
+ ).getUpgradedToSegmentIds();
+ }
+ catch (Exception e) {
+ LOG.warn(
+ e,
+ "Could not retrieve UpgradedToSegmentsResponse using task
action[retrieveUpgradedToSegmentIds]."
+ + " Overlord may be on an older version."
+ );
+ return unusedSegments;
+ }
+
+ final Set<String> existingSegmentIds
+ = taskActionClient.submit(new
RetrieveSegmentsByIdAction(getDataSource(), upgradedSegmentIds))
+ .stream()
+ .map(DataSegment::getId)
+ .map(SegmentId::toString)
+ .collect(Collectors.toSet());
+
+ List<DataSegment> unreferencedSegments = new ArrayList<>();
+ for (DataSegment segment : unusedSegments) {
+ final String id = segment.getId().toString();
+
+ // If the segment still exists for some reason, do not kill it
+ if (existingSegmentIds.contains(id)) {
+ break;
+ }
+
+ // If the segment is the parent of existing segments, do not kill it
+ if (upgradedToSegmentIds.containsKey(id)) {
Review Comment:
No, it cannot. Please check the early return at
https://github.com/apache/druid/pull/16667/files#diff-05dacdddeb229cca590199ba0426c69b0b07d541a83ee365729275de1613e31eR391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
