andygrove opened a new issue, #2224:
URL: https://github.com/apache/datafusion-ballista/issues/2224
**Describe the bug**
In push-based scheduling, an executor registers with the scheduler before
its own gRPC server has bound its listening socket. The scheduler's
registration handler dials back to the executor to verify connectivity, so if
it wins the race it gets `ConnectionRefused`, registration fails, and the
executor process exits.
The ordering is already flagged by a `TODO` in the code:
- `executor_server::startup` (`ballista/executor/src/executor_server.rs`)
logs `"... Rust Executor Grpc Server listening on ..."` and then spawns the
tonic server with `tokio::spawn(...serve_with_shutdown(addr, ...))`. The bind
happens inside that spawned task, so the log line is emitted before the socket
exists.
- Immediately afterwards, step 2 calls `register_executor`, with the
standing comment: `// TODO the executor registration should happen only after
the executor grpc server started.`
- On the scheduler side, `ExecutorManager::register_executor`
(`ballista/scheduler/src/state/executor_manager.rs`) calls `test_connectivity`,
which is a single `ExecutorGrpcClient::connect(...)` with no retry and no
backoff.
- If that connect loses the race, registration returns an error, and
`executor_process` treats it as fatal ("If there is executor registration error
during startup, return the error and stop early"), so the executor exits rather
than retrying.
**To Reproduce**
Observed in CI on the new HA chaos harness (`cargo test -p ballista-chaos
--test ha`), in `baseline_matches_local_datafusion::case_2_aqe_on`. Two
executors were started, one registered fine, the other lost the race and died,
so the harness timed out:
```
timed out waiting for 2 executors to register
```
executor-1 log:
```
INFO ballista_executor::executor_server] Ballista v54.0.0 Rust Executor
Grpc Server listening on 127.0.0.1:42635
ERROR ballista_executor::executor_server] Executor registration failed due
to: Grpc error: code: 'Internal error',
message: "Fail to do executor registration due to: Internal Ballista
error: Failed to register executor at
127.0.0.1:42635, could not connect: tonic::transport::Error(Transport,
ConnectError(ConnectError("tcp connect error",
127.0.0.1:42635, Os { code: 111, kind: ConnectionRefused, message:
"Connection refused" })))"
```
scheduler log, same second:
```
INFO ballista_scheduler::scheduler_server::grpc] Received register executor
request for ExecutorRegistration { id: "09332458-...", grpc_port: 42635, ... }
ERROR ballista_scheduler::scheduler_server::grpc] Fail to do executor
registration due to: Internal Ballista error:
Failed to register executor at 127.0.0.1:42635, could not connect: ...
ConnectionRefused ...
```
It is timing-dependent, so it reproduces rarely and mostly on loaded
machines. It is not specific to the chaos harness: any push-staged deployment
where the executor is slow to reach the bind can hit it, and the consequence
there is an executor that exits at startup instead of joining the cluster.
**Expected behavior**
An executor that is otherwise healthy should always succeed in registering.
Its gRPC listener should be accepting connections before it tells the scheduler
about itself.
**Additional context**
Two candidate fixes, not mutually exclusive:
1. Bind synchronously in `executor_server::startup` before registering, i.e.
create the `TcpListener` (or `TcpIncoming`) on the current task, then hand it
to `serve_with_incoming_shutdown` in the spawned task. This closes the race at
the source and lets the "listening on" log line tell the truth. It also removes
the existing `TODO`.
2. Give `ExecutorManager::test_connectivity` a short bounded retry (a few
hundred milliseconds) so a marginally slow executor is tolerated rather than
rejected outright.
(1) alone fixes the local case. (2) additionally helps where the executor is
on another host and is still coming up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
