juliomax opened a new pull request, #2894:
URL: https://github.com/apache/datafusion-ballista/pull/2894

   # Which issue does this PR close?
   
   Part of #2597 (first of three PRs; it does not close the issue).
   
   # Rationale for this change
   
   `start_executor_process_inner` in 
`ballista/executor/src/executor_process.rs` is one long
   function that mixes several unrelated startup and shutdown phases. #2597 
asks to split it
   into smaller steps so that each one can be read, tested and changed on its 
own. As described
   in the plan on the issue, this is the first PR of three: it extracts the 
phases that do not
   overlap with the open PRs touching the same function. The scheduler 
connection
   (`build_endpoint` / `connect_to_scheduler`) and `resolve_work_dir` are left 
for the next two
   PRs, once their prerequisites are merged.
   
   This is a behaviour-preserving refactor: the moved code is copied literally, 
in the same
   order, with the same log lines, errors and ordering of side effects.
   
   # What changes are included in this PR?
   
   Only `ballista/executor/src/executor_process.rs` changes. Four phases move 
out of
   `start_executor_process_inner` into their own functions, one commit per step:
   
   1. `resolve_vcores`: the "0 means use the available parallelism" rule.
   2. `spawn_shuffle_cleaner`: the periodic shuffle clean-up task and its 
shutdown handle. The
      `job_data_clean_up_interval_seconds > 0` decisions stay in 
`start_executor_process_inner`.
   3. `run_until_shutdown`: waiting for the service handlers, a stop request or 
a termination
      signal, and notifying the scheduler on failure.
   4. `resolve_memory_pool_policy`: choosing the memory pool budget, 
registering the metrics and
      wrapping the policy with the pool observer. It takes 
`Option<&ExecutorMemoryMetrics>`. (The
      plan on the issue called it `build_memory_pool_policy`; it is renamed so 
it does not read
      like the existing `memory_pool_policy(bytes, vcores)` constructor.)
   
   The first commit only adds tests and does not touch production code. They 
pin the current
   behaviour and pass on `main` before any extraction:
   
   - the scheduler connect with `scheduler_connect_timeout_seconds = 0` makes a 
single attempt;
   - the configured and the unbounded memory pool are reported to the metrics;
   - with `job_data_clean_up_interval_seconds = 0` the cleaner is not started 
and job data is
     kept on shutdown; with a positive interval job data is removed on 
shutdown. These two run a
     whole executor against a fake idle scheduler and stop it by taking its 
Flight port.
   
   Each extraction commit adds the unit tests of its own function 
(`resolve_vcores`, the cleaner,
   and both outcomes of `run_until_shutdown`).
   
   How it was tested, all on Linux, at the last commit:
   
   - The CI commands of this repository (fmt, clippy with `-D warnings`, `cargo 
check`,
     `cargo doc`, the workspace tests including the standalone and benchmark 
ones, toml, proto and
     config-docs checks, machete, RAT, prettier) pass.
   - For every new test I removed the effect it protects (mutation) and checked 
that the test
     fails, then restored the file.
   - Standalone cluster by hand, on localhost, with the binary of the last 
commit compared with the
     one built from `main`: one scheduler, one executor 
(`--job-data-clean-up-interval-seconds 1
     --job-data-ttl-seconds 1`), a `SELECT k, count(*), sum(v) ... GROUP BY k 
ORDER BY k` over a
     100000-row CSV, then `SIGTERM` on the executor. The result is the same, 
the cleaner empties the
     work dir, and the shutdown sequence in the logs is the same.
   - The two clean-up tests that start a whole executor ran 30 times each, 
alone and under heavy
     load, with no failure.
   
   Not verified: macOS builds and tests, the h2o and TPC-H/TPC-DS jobs, the 
Kubernetes chaos job,
   the Docker build, the Python bindings, CodeQL and the allowlist check. 
`SIGINT`, the unbounded
   memory pool and the automatic `--memory-pool-size` were checked only by unit 
tests, not on a
   real cluster. Under very heavy CPU load I saw the existing
   `scheduler_connect_timeout_is_honoured` and 
`scheduler_connect_timeout_bounds_a_hanging_connect`
   tests fail now and then; they are not touched by this PR.
   
   # Are there any user-facing changes?
   
   No. There is no change to the public API, configuration, logs, protocol or 
behaviour.
   
   Co-developed with Megabot, my AI-assisted development pipeline (Claude 
Sonnet 5.5; reviewed, tested and owned by me).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to