andygrove opened a new pull request, #2657: URL: https://github.com/apache/datafusion-ballista/pull/2657
# Which issue does this PR close? Closes #2652. # Rationale for this change #2652 proposes an opt-in pipelined execution mode. In this mode the scheduler places every task of a job up front, and stages stream record batches to each other over Arrow Flight instead of running stage by stage through shuffle files. The goal is lower latency for interactive queries. The blocking shuffle stays the default. # What changes are included in this PR? This is a work-in-progress draft that will be pushed to as the work proceeds. The plan, the current status and the next steps are tracked in [`docs/source/contributors-guide/pipelined-execution.md`](https://github.com/andygrove/datafusion-ballista/blob/pipelined-execution/docs/source/contributors-guide/pipelined-execution.md). That page is the place to look for where things stand. Milestones: - **M0:** the `ballista.execution.mode` setting and the tracking doc. - **M1:** exchange operators, a registry, protobuf messages and a Flight action. - **M2:** executor wiring. - **M3:** scheduler work, covering the pipelined execution graph and all-or-nothing placement. - **M4:** the client reads results while the job is still running. - **M5:** a push-staged standalone cluster and end-to-end tests. - **M6:** docs and benchmarks. Once the whole feature works end to end, parts may be split out into smaller PRs. # Are there any user-facing changes? Three new session settings: `ballista.execution.mode`, `ballista.pipelined.queue_capacity` and `ballista.pipelined.registration_timeout_seconds`. The default behavior is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
