devoopsman45 opened a new pull request, #1881:
URL: https://github.com/apache/datafusion-ballista/pull/1881
## Problem
New users hit one of two silent failures when trying to run Ballista locally:
1. **`docker-compose up --build` fails with no useful error.** The
Dockerfiles only `COPY target/release/ballista-*` — they do not compile Rust
inside Docker. Without a prior `cargo build --release`, the COPY step finds
nothing and the image is broken. This prerequisite is not documented anywhere
in the deployment guides.
2. **No fast evaluation path.** The only documented path is `cargo build
--release` (~20 min cold, requires Rust + protoc). There is no way to spin up a
cluster in ~2 min to evaluate Ballista before committing to a full build.
## Solution
Three changes in one PR:
### 1. `docker-compose.quick.yml` (new file)
Pulls pre-built images from GHCR
(`ghcr.io/apache/datafusion-ballista-{scheduler,executor}:latest`). Docker is
the only prerequisite.
```shell
docker compose -f docker-compose.quick.yml up
```
Key design decisions:
- **`--advertise-flight-sql-endpoint`** on the scheduler — clients connect
only to `localhost:50050`; the scheduler proxies all Arrow Flight result
fetching internally. Without this flag, clients receive executor
Docker-internal IPs (unreachable from host), causing connection timeouts.
- **Health check + `depends_on: condition: service_healthy`** — executors
wait for the scheduler to be ready before registering, avoiding a race
condition on slower machines.
- **2 executor replicas by default** via `deploy.replicas` — demonstrates
multi-executor setup out of the box.
- Images track the latest stable release, not `main`. This is noted in the
file header.
### 2. `docs/source/user-guide/deployment/quick-start.md` (restructured)
- Two clearly labelled paths: *Evaluate (~2 min)* and *Build from source
(~20 min)*
- Comparison table at the top (prerequisites, time, terminals)
- Expected log output so users know what healthy startup looks like
- Note on the two-port executor model (Arrow Flight port + gRPC control port)
- `[!IMPORTANT]` compatibility gap warning (DataFusion / Ballista versions)
- Troubleshooting section: `protoc` missing, port conflict, executor can't
reach scheduler
### 3. `docs/source/user-guide/deployment/docker-compose.md` (fixed)
- Documents the missing `cargo build --release` prerequisite for
`docker-compose.yml`
- Explains why skipping it causes a silent failure
- Notes that no pre-built CLI image is published to GHCR (the existing doc
referenced `apache/datafusion-ballista-cli:latest` which does not exist in the
registry)
## Testing
- Started the cluster with `docker compose -f docker-compose.quick.yml up -d`
- Verified both executors registered: scheduler logs show two `Received
register executor request` entries with distinct UUIDs
- Verified `--advertise-flight-sql-endpoint` proxy: no connection-timeout
errors to Docker-internal executor IPs from the host
- Scheduler and executor health checks pass within the configured retry
window
## Checklist
- [ ] `docker-compose.quick.yml` starts cleanly on a machine with only
Docker installed
- [ ] Both executors register with the scheduler on startup
- [ ] Clients connect to `localhost:50050` only (no direct executor port
access needed)
- [ ] `quick-start.md` comparison table and both paths are accurate
- [ ] `docker-compose.md` prerequisite note is clear
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01TKhWssogfJHUnDTz4qfpiA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
