jihoonson commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r602001680
##########
File path:
indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java
##########
@@ -1447,6 +1452,7 @@ private void assertIngestionSchema(
null,
null,
null,
+ null,
null
Review comment:
> @jihoonson when it comes to judging segment handoff, isn't a
compaction task a parallel indexing task under the hood? I just tested a
compaction task submit with tuning config and the new config for handoff set to
5 minutes. The task ran and logs indicate it confirmed handoff of newly created
compacted segment.
It is the parallel index task under the hood, but [one compaction task can
run multiple parallel indexing
tasks](https://github.com/apache/druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java#L420-L447).
With the changes in this PR, each of those parallel indexing tasks will wait
for segment handoff independently. The handoff wait timeout will also apply
separately for each parallel task. I think this is OK as long as it's
documented, but what I meant is, maybe we could do this differently, such as
waiting for segments to be handed off all together after running all parallel
tasks.
> Should not allow the handoff to occur when it is a compact type task? I'm
not sure how we would do that, since the TuningConfig is not modifiable as
written and if the task(s) ran by the compact task are just parallel indexing
tasks, I'm not sure how we would tell them not to honor the handoff wait config
It seems reasonable to me to not support until we figure out what we want to
do. An easy way to do so is making a copy of tuningConfig in compaction task
but with `awaitSegmentAvailabilityTimeoutMillis` always being set to 0. We
should document that compaction tasks don't wait for handoff if you want to do
this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
