andygrove opened a new pull request, #2713:
URL: https://github.com/apache/datafusion-ballista/pull/2713
# Which issue does this PR close?
Part of #2652. This doesn't close it.
# Rationale for this change
Pipelined execution (#2652) needs a user guide page explaining when to use
the mode, how a job runs, how much memory it takes and which settings control
it. The page was part of #2657, which turns the mode on. It is split out so the
docs can be reviewed separately from the scheduler code.
This PR is a draft until #2657 merges. Until then, `main` rejects
`ballista.execution.mode = pipelined`, so the page would describe a mode users
can't run yet.
# What changes are included in this PR?
- A new page, `docs/source/user-guide/pipelined-execution.md`. It covers:
- when pipelined execution suits a workload, and when blocking mode is the
better choice
- enabling it on a remote cluster (push-staged scheduler and executors)
and on a standalone cluster
- how a job runs: placement, the vcore reservation, streaming exchanges,
broadcast replay and failure handling
- memory use compared with blocking mode
- the settings involved and the current limitations
- An entry for the page in `docs/source/index.rst`, after the tuning guide.
# Are there any user-facing changes?
Yes, a new documentation page. There are no code changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
