sthetland commented on a change in pull request #9360: Create splits of
multiple files for parallel indexing
URL: https://github.com/apache/druid/pull/9360#discussion_r381506006
##########
File path: docs/ingestion/native-batch.md
##########
@@ -42,11 +42,12 @@ demonstrates the "simple" (single-task) mode.
## Parallel task
The Parallel task (type `index_parallel`) is a task for parallel batch
indexing. This task only uses Druid's resource and
-doesn't depend on other external systems like Hadoop. `index_parallel` task is
a supervisor task which basically creates
-multiple worker tasks and submits them to the Overlord. Each worker task reads
input data and creates segments. Once they
-successfully generate segments for all input data, they report the generated
segment list to the supervisor task.
+doesn't depend on other external systems like Hadoop. The `index_parallel`
task is a supervisor task which orchestrates
+the whole indexing process. It splits the input data and and issues worker
tasks
+to the Overlord which actually process the assigned input split and create
segments.
+Once a worker task successfully processes all assigned input split, it reports
the generated segment list to the supervisor task.
Review comment:
It’s a little unclear to me who's doing what in this. Is the following
accurate/clearer?
“The `index_parallel` task is a supervisor task that orchestrates the
indexing process. The task splits input data for processing by Overlord worker
tasks, which process the input splits assigned to them and create segments from
the input. Once a worker task successfully processes all assigned input splits,
it reports the generated segment list to the supervisor task.”
If not, for a lighter edit, maybe just clarify that it's the worker tasks
more specifically, rather than the overlord, that is processing input splits
(if that's the case).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
