AmatyaAvadhanula commented on code in PR #16667:
URL: https://github.com/apache/druid/pull/16667#discussion_r1674185925
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillUnusedSegmentsTask.java:
##########
@@ -300,6 +356,81 @@ private NavigableMap<DateTime, List<TaskLock>>
getNonRevokedTaskLockMap(TaskActi
return taskLockMap;
}
+ private List<DataSegment> findUnreferencedSegments(
+ List<DataSegment> unusedSegments,
+ Map<String, String> upgradedFromSegmentIds,
+ TaskActionClient taskActionClient
+ ) throws IOException
+ {
+ if (unusedSegments.isEmpty() || upgradedFromSegmentIds.isEmpty()) {
+ return unusedSegments;
+ }
+
+ final Set<String> upgradedSegmentIds = new HashSet<>();
+ for (DataSegment segment : unusedSegments) {
+ final String id = segment.getId().toString();
+ upgradedSegmentIds.add(id);
+ if (upgradedFromSegmentIds.containsKey(id)) {
+ upgradedSegmentIds.add(upgradedFromSegmentIds.get(id));
+ }
+ }
+
+ final Map<String, Set<String>> upgradedToSegmentIds;
+ try {
+ upgradedToSegmentIds = taskActionClient.submit(
+ new RetrieveUpgradedToSegmentIdsAction(getDataSource(),
upgradedSegmentIds)
+ ).getUpgradedToSegmentIds();
+ }
+ catch (Exception e) {
+ LOG.warn(
+ e,
+ "Could not retrieve UpgradedToSegmentsResponse using task
action[retrieveUpgradedToSegmentIds]."
+ + " Overlord may be on an older version."
+ );
+ return unusedSegments;
+ }
+
+ final Set<String> existingSegmentIds
+ = taskActionClient.submit(new
RetrieveSegmentsByIdAction(getDataSource(), upgradedSegmentIds))
+ .stream()
+ .map(DataSegment::getId)
+ .map(SegmentId::toString)
+ .collect(Collectors.toSet());
+
+ List<DataSegment> unreferencedSegments = new ArrayList<>();
+ for (DataSegment segment : unusedSegments) {
Review Comment:
Thank you for attempting to refine this method.
I feel that the way it is written is intuitive to me at least and clearly
lists out the cases in which a segment may or may not be referenced.
Also, the suggestion you have shared seems incorrect.
Please refer to KillUnusedSegmentsTaskTest with the changes in the latest
commit.
I had made changes per your suggestion in a previous commit, found the logic
to be inaccurate and reverted to my original approach. Perhaps the suggestion
could also be tweaked to pass the test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
