andygrove opened a new issue, #2597: URL: https://github.com/apache/datafusion-ballista/issues/2597
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** [`start_executor_process_inner` (executor_process.rs#L452-L903)](https://github.com/apache/datafusion-ballista/blob/ea1126c0649b2bc08e1e233b1972b64318821c16/ballista/executor/src/executor_process.rs#L452-L903) is a 371-line function (cognitive complexity 54) in a file with 25 commits over the last 12 months. It does, in one body: config validation, work-dir and vcore resolution, memory-pool policy selection (a self-contained `match` at about L503-L545), runtime producer and codec wiring, the scheduler connection, the shuffle-cleaner task, service startup and the shutdown sequence. The scheduler connection block (about L600-L680) also duplicates endpoint creation and override handling between the `connect_timeout == 0` branch and the retry branch. That duplication is where the timeout bug in #2593 lives. **Describe the solution you'd like** Keep `start_executor_process_inner` as a short orchestration function and extract the phases: `resolve_work_dir`, `resolve_vcores`, `build_memory_pool_policy`, `connect_to_scheduler` (one retry loop where timeout 0 means a single attempt, sharing one `build_endpoint` helper), `spawn_shuffle_cleaner`, and `run_until_shutdown` for the select-and-drain tail. Each phase can then be unit tested. **Describe alternatives you've considered** Fix only the timeout bug and leave the structure. **Additional context** Do this after the connect-timeout fix (#2593) lands. Line references are pinned to commit ea1126c0649b. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
