loquisgon commented on a change in pull request #12137:
URL: https://github.com/apache/druid/pull/12137#discussion_r800938276
##########
File path:
indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java
##########
@@ -941,9 +939,37 @@ private TaskStatus generateAndPublishSegments(
ingestionSchema
);
+ Set<DataSegment> tombStones = Collections.emptySet();
+ if (ingestionSchema.getIOConfig().isDropExisting()) {
+ TombstoneHelper tombstoneHelper = new
TombstoneHelper(pushed.getSegments(),
+
ingestionSchema.getDataSchema(),
+
toolbox.getTaskActionClient());
+
+ List<Interval> tombstoneIntervals =
tombstoneHelper.computeTombstoneIntervals();
+ // now find the versions for the tombstone intervals
+ Map<Interval, String> tombstonesAndVersions = new HashMap<>();
+ for (Interval interval : tombstoneIntervals) {
+ NonnullPair<Interval, String> intervalAndVersion =
+ findIntervalAndVersion(
Review comment:
@imply-cheddar At the time that I wrote that code I could not find an
easy way to use the allocators (`SegmentAllocator` implementations) since they
are typically created and used in the sub-tasks but the current code in this PR
is creating them after the sub-tasks completed (in the parallel case). But I
since thought of a potential way to integrate them in the sub-tasks (have each
sub-task create the tombstones after it completed pushing all segments, create
the tombstones there using its allocator, and then propagate the tombstones
back to the supervisor task in the task report, then have the supervisor filter
out unneeded tombstones). I will work next on using this new path, with the
allocators, since the allocator should be already prepared to deal with issues
as the one you comment above. So please hold on for a little while while I try
to implement this new path and provide an update to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
