NoahKusaba opened a new pull request, #2482: URL: https://github.com/apache/datafusion-ballista/pull/2482
# Rationale for this change The docs used "task slots", "concurrency levels" and "vcores" for the same thing. The executor flag is `--vcores` (`--concurrent-tasks` is deprecated), and the scheduler and config use vcores, so the docs should too. # What changes are included in this PR? - "Task slots" and "concurrency levels" become vcores in `shuffle.md`, the tuning guide heading, and the `ballista.standalone.parallelism` description (in `config.rs` and the generated `configs.md` row). - `benchmarking.md` uses `--vcores` instead of the deprecated `--concurrent-tasks`. - Fix two `upgrading/index` links in `user-personas.md`. ### Change to the user-personas contract `user-personas.md` is the contract PRs are reviewed against, and says changes to it need explicit sign-off, so this is called out on its own. Persona 2 (the Spark user) changes: - "one **task** per partition" becomes "**tasks** over partitions", and "executors with task slots" becomes "executors with vcores" (also in the root README). - "task-slot concurrency" becomes "vcore-bounded concurrency". - Adds a line on how to get Spark's model back: set `ballista.scheduler.max_partitions_per_task` to `1`. My reading is that this corrects a guarantee the page had stopped providing, rather than withdrawing one. Multi-partition tasks landed in #2038, and `max_partitions_per_task` defaults to `0`, so the scheduler packs several partitions per task by default. The capability itself is untouched. **This wants a deliberate yes or no from a maintainer.** Happy to drop the personas change and keep only the terminology. # Are there any user-facing changes? Documentation, plus one `ConfigEntry` description string in `ballista/core/src/config.rs`. Verified: `./dev/update_config_docs.sh --check` passes, and `sphinx-build -W` is clean when combined with #2476. Split out of #2438 to make it easier to review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
