loquisgon commented on a change in pull request #12137:
URL: https://github.com/apache/druid/pull/12137#discussion_r800938276
##########
File path:
indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java
##########
@@ -941,9 +939,37 @@ private TaskStatus generateAndPublishSegments(
ingestionSchema
);
+ Set<DataSegment> tombStones = Collections.emptySet();
+ if (ingestionSchema.getIOConfig().isDropExisting()) {
+ TombstoneHelper tombstoneHelper = new
TombstoneHelper(pushed.getSegments(),
+
ingestionSchema.getDataSchema(),
+
toolbox.getTaskActionClient());
+
+ List<Interval> tombstoneIntervals =
tombstoneHelper.computeTombstoneIntervals();
+ // now find the versions for the tombstone intervals
+ Map<Interval, String> tombstonesAndVersions = new HashMap<>();
+ for (Interval interval : tombstoneIntervals) {
+ NonnullPair<Interval, String> intervalAndVersion =
+ findIntervalAndVersion(
Review comment:
@imply-cheddar At the time that I wrote that code I could not find an
easy way to use the allocators (`SegmentAllocator` implementations) since they
are typically created and used in the sub-tasks but the current code in this PR
is creating them after the sub-tasks completed (in the parallel case). But I
since then I thought of a potential way to integrate them in the sub-tasks
(have each sub-task create the tombstones after it completed pushing all
segments, create the tombstones there using its allocator, and then propagate
the tombstones back to the supervisor task in the task report, then have the
supervisor filter out unneeded tombstones). I will work next on using this new
path, with the allocators, since the allocator should be already prepared to
deal with issues as the one you comment above. So please hold on for a little
while while I try to implement this new path and provide an update to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
