devanbenz commented on PR #27126:
URL: https://github.com/apache/datafusion/pull/27126#issuecomment-6690789008
Heh yeah it seems to have happened organically over time, there is no
specific PR that outlines a reason for the implementation:
| Date | Commit | PR / Issue | Change |
|---|---|---|---|
| 2019-09-19 | `ed485a587` | ARROW-6563 | `MergeExec` added; `execute` calls
`thread::spawn` per input partition |
| 2020-09-27 | `d6d043d93` | ARROW-9754 | `ExecutionPlan` made async;
threads become `tokio::spawn` tasks, still in `execute` |
| 2020-10-14 | `bd93ee5f2` | ARROW-10292 | Bounded channel of size
`input_partitions` ("maximize parallelism"); shape still in
use today |
| 2021-06-01 | `c794f2df5` | #379 | Spawning moved into a shared
`spawn_execution` helper; no behaviour change |
| 2023-06-06 | `39ee59aeb` | #6507 | Folded into
`RecordBatchReceiverStream::builder().run_input(..)`, the code PR #27126 defers
|
None of these commit messages calls the early start deliberate. The stated
goal was running all partitions in parallel, and starting at `execute()` was
simply where that code lived. The parallelism across different build sides was
never named as a goal either.
Let's work on brainstorming some different approaches :saluting_face:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
