andygrove opened a new pull request, #2669: URL: https://github.com/apache/datafusion-ballista/pull/2669
# Which issue does this PR close? Part of #2652. This is the first of a series of smaller PRs split out of #2657 (pipelined execution). # Rationale for this change A standalone (in-process) cluster always runs pull-staged. Pipelined execution needs push-staged scheduling, and its end-to-end tests run on standalone clusters, so the standalone cluster needs a push-staged option. That option is also useful by itself, for testing push-staged scheduling without starting separate processes. Running push-staged in standalone mode exposed a bug. The standalone scheduler advertised the hardcoded endpoint `localhost:50050` instead of the port it actually bound. Pull-staged executors never use that endpoint, but push-staged executors report task status to the endpoint each launch names. # What changes are included in this PR? - A new setting, `ballista.standalone.task_scheduling_policy`: `pull-staged` (the default, unchanged behavior) or `push-staged`. - `new_standalone_scheduler_with_builder` binds its listener before it builds the scheduler, advertises `localhost:<port>`, and uses the configured policy. - With `push-staged`, `new_standalone_executor_from_builder` takes a free port for its task-launch gRPC server and starts through `executor_server::startup`, instead of running the poll loop. - A test helper, `setup_test_cluster_with_executors`, starts a scheduler and several executors in one process. # How are these changes tested? - A config unit test covers the default, case-insensitive parsing and a rejected value. - `ballista/client/tests/standalone_push_staged.rs` runs a scan, an aggregate and a join on a push-staged standalone cluster, and on a push-staged scheduler with two executors. It compares each result with a pull-staged standalone cluster. # Are there any user-facing changes? There is one new setting, `ballista.standalone.task_scheduling_policy`, documented in the config reference. The default behavior is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
