andygrove commented on code in PR #2713: URL: https://github.com/apache/datafusion-ballista/pull/2713#discussion_r4433610033
########## docs/source/user-guide/pipelined-execution.md: ########## @@ -0,0 +1,171 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Pipelined Execution (experimental) + +By default Ballista runs a job one stage at a time. Each stage writes its output +to shuffle files, and the stages that read it start once it has finished. This +is the model Apache Spark uses, and it lets a job survive failures: a lost task +is simply run again from the shuffle files it reads. + +Pipelined execution, enabled with `ballista.execution.mode = pipelined`, runs a +job the way Trino does instead. The scheduler places every task of the job at +once, and stages stream record batches to each other over Arrow Flight as they +produce them. No intermediate data is written to disk, and results start +reaching the client while the job is still running. The cost is that a failed +task fails the whole job. + +Pipelined execution is experimental. Its behavior and settings may change. + +## When to use it + +Pipelined execution suits short, latency-sensitive queries, where the time +spent scheduling stages and writing and reading shuffle files is a large part +of the total. It needs the cluster to have room for the whole job at once. + +Stay with blocking execution for: + +- long-running jobs, which a single task or executor failure would fail; +- jobs that need far more vcores than the cluster has free; +- clusters shared by many concurrent jobs, where a pipelined job may wait a + long time for its vcores to be free together. + +## Enabling it + +Pipelined jobs need push-staged scheduling, where the scheduler pushes tasks to +executors. This is the default for the `ballista-scheduler` and +`ballista-executor` binaries (`--scheduler-policy push-staged` and +`--task-scheduling-policy push-staged`). A pull-staged scheduler rejects +pipelined jobs. + +Choose the mode per session, from SQL: + +```sql +SET ballista.execution.mode = 'pipelined'; +``` + +or when building the session: + +```rust +let config = SessionConfig::new_with_ballista() Review Comment: Good idea! I'd lean toward a typed setter rather than a new constructor, to match how the other settings are exposed on `SessionConfigExt`, something like `SessionConfig::new_with_ballista().with_ballista_execution_mode(ExecutionMode::Pipelined)`. Since this PR is docs only, I filed #2729 for it and will update this snippet once it lands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
