abhishekrb19 commented on code in PR #19737:
URL: https://github.com/apache/druid/pull/19737#discussion_r3649556260
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillUnusedSegmentsTask.java:
##########
@@ -87,6 +85,10 @@
* <li> Filter the set of unreferenced segments using load specs from the set
of used segments. </li>
* <li> Kill the filtered set of segments from deep storage. </li>
* </ol>
+ * Note: When {@link Tasks#USE_CONCURRENT_LOCKS} is true, keep a large buffer
+ * period before killing segments after they have been marked as unused.
+ * Otherwise, there may be a potential data loss if a concurrent APPEND job
Review Comment:
I think it'd be good to document this in the kill section of the docs as
well: https://druid.apache.org/docs/latest/data-management/delete/
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillUnusedSegmentsTask.java:
##########
@@ -236,12 +238,20 @@ public TaskStatus runTask(TaskToolbox toolbox) throws
Exception
break;
}
- unusedSegments = fetchNextBatchOfUnusedSegments(toolbox, nextBatchSize);
+ unusedSegmentsPlus = fetchNextBatchOfUnusedSegments(toolbox,
nextBatchSize);
+ if (unusedSegmentsPlus.isEmpty()) {
+ // No more segments eligible for kill, do not proceed further
+ break;
+ }
// Fetch locks each time as a revokal could have occurred in between
batches
final NavigableMap<DateTime, List<TaskLock>> taskLockMap
= getNonRevokedTaskLockMap(toolbox.getTaskActionClient());
+ final Set<DataSegment> unusedSegments = unusedSegmentsPlus.stream()
+
.map(DataSegmentPlus::getDataSegment)
+
.collect(Collectors.toSet());
+
if (!TaskLocks.isLockCoversSegments(taskLockMap, unusedSegments)) {
Review Comment:
nit: perhaps we could introduce a thin wrapper for `isLockCoversSegments()`
that also accepts `DataSegmentPlus` that delegates to
isLockCoversSegments(NavigableMap<DateTime, List<TaskLock>>,
Collection<DataSegment> segments) to avoid another iteration over the set of
segments.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillUnusedSegmentsTask.java:
##########
@@ -352,7 +346,47 @@ protected List<DataSegment>
fetchNextBatchOfUnusedSegments(TaskToolbox toolbox,
nextBatchSize,
maxUsedStatusLastUpdatedTime
)
- );
+ )
+ .stream()
+ .map(segment -> new DataSegmentPlus(segment, null, null,
null, null, null, null, null))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * Fetches the parent IDs (if any) for the given unused segments.
+ *
+ * @param unusedSegments Unused segments whose parent IDs need to be fetched
+ * @return Map from segment ID to the segment ID from which
+ * it was upgraded. If an input segment was not upgraded from any other
segment,
+ * it does not have an entry in the map.
+ */
+ protected Map<String, String> fetchParentIdsForSegments(
+ TaskToolbox toolbox,
+ List<DataSegmentPlus> unusedSegments
+ )
+ {
+ try {
+ final Set<String> segmentIds = unusedSegments.stream().map(
+ s -> s.getDataSegment().getId().toString()
+ ).collect(Collectors.toSet());
Review Comment:
nit: We could compute this once in the caller and pass it to both this
function and `getKillableSegments`.
##########
server/src/main/java/org/apache/druid/indexing/overlord/IndexerMetadataStorageCoordinator.java:
##########
@@ -161,10 +161,11 @@ List<DataSegment> retrieveUnusedSegmentsForInterval(
* @param maxUpdatedTime Returned segments must have a {@code
used_status_last_updated}
* which is either null or earlier than this value.
* @param limit Maximum number of segments to return.
- *
* @return Unsorted list of unused segments that match the given parameters.
+ * The entries in the lost are required to have the {@link
DataSegmentPlus#getDataSegment()}
Review Comment:
typo:
```suggestion
* The entries in the list are required to have the {@link
DataSegmentPlus#getDataSegment()}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
