andygrove opened a new issue, #2589:
URL: https://github.com/apache/datafusion-ballista/issues/2589
**Describe the bug**
The entrypoint scripts in `dev/docker/` start the Ballista binary as a child
of bash instead of `exec`ing it. For example, `executor-entrypoint.sh`:
```bash
set -e
mkdir -p /work
/root/ballista-executor "$@"
```
That leaves bash as PID 1. Bash doesn't install a SIGTERM handler, and Linux
drops signals sent to a container's PID 1 unless it has a handler for them. So
when Docker or Kubernetes stops a container, the SIGTERM never reaches the
Ballista process. The container keeps running until the grace period runs out
and it gets SIGKILLed.
The biggest impact is on executors. On SIGTERM an executor is meant to stop
taking new tasks, tell the scheduler it's stopping, and finish its running
tasks. With the published image none of that happens. The executor keeps taking
tasks until it's killed, and the scheduler only finds out when its heartbeats
stop. Every executor pod termination also waits out the full
`terminationGracePeriodSeconds`, which slows down rollouts and node drains. The
Kubernetes guide runs the images through their entrypoints (with `args:`), and
so does `docker-compose.yml` (with `command:`), so the documented deployments
are affected. The chaos harness's k8s scenarios don't catch it because they use
their own image and set `command:` to the binary directly.
The scheduler is a bit different. It doesn't handle SIGTERM at all yet
(#565), so it has nothing to drain. But with `exec` alone it would become PID 1
and still ignore SIGTERM, for the same reason as bash above.
This came up in #2577, which adds a result service image with the same
entrypoint pattern and a graceful shutdown on SIGTERM that depends on the
signal arriving.
**To Reproduce**
Here's a minimal reproduction with the same base image and entrypoint shape,
using a stand-in binary that logs when it gets SIGTERM:
```bash
mkdir sigterm-repro && cd sigterm-repro
cat > fake-service <<'EOF'
#!/bin/bash
trap 'echo "got SIGTERM"; exit 0' TERM
while true; do sleep 0.1; done
EOF
cat > entrypoint.sh <<'EOF'
#!/bin/bash
set -e
/root/fake-service "$@"
EOF
chmod +x fake-service entrypoint.sh
cat > Dockerfile <<'EOF'
FROM ubuntu:26.04
COPY fake-service /root/fake-service
COPY entrypoint.sh /root/entrypoint.sh
ENTRYPOINT ["/root/entrypoint.sh"]
EOF
docker build -t sigterm-repro .
cid=$(docker run -d sigterm-repro)
sleep 1
time docker stop -t 8 "$cid"
docker logs "$cid"
docker inspect -f '{{.State.ExitCode}}' "$cid"
```
`docker stop` takes the full 8 seconds, the logs never show `got SIGTERM`,
and the exit code is 137 (SIGKILL). Changing the last line of `entrypoint.sh`
to `exec /root/fake-service "$@"` makes the stop return straight away, with
`got SIGTERM` in the logs and exit code 0.
**Expected behavior**
`docker stop` and Kubernetes pod termination deliver SIGTERM to the Ballista
process. Executors (and the result service from #2577) shut down gracefully,
and a process without a SIGTERM handler exits promptly instead of waiting to be
SIGKILLed.
**Additional context**
Some options:
- `exec` the binary in each entrypoint, so it runs as PID 1 and gets SIGTERM
directly. That's enough for the executor, which handles SIGTERM.
- For the scheduler, `exec` alone isn't enough, because a PID 1 without a
SIGTERM handler ignores it. Either the scheduler handles SIGTERM itself (see
#565), or the images run under a small init such as `tini` that forwards
signals. I checked that a binary with no SIGTERM handler exits on SIGTERM
straight away under `docker run --init` (exit code 143), while the same binary
`exec`ed as PID 1 waits out the timeout.
- `standalone-entrypoint.sh` runs the scheduler in the background and the
executor in the foreground, so it needs a bit more than an `exec`.
Related: #2525 (rolling upgrades), #2357 (graceful executor shutdown and
Flight streams), #565 (scheduler graceful shutdown).
@villebro this came up while reviewing #2577, and it's closely tied to the
result service's graceful shutdown there. Would you be up for taking a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
