villebro opened a new pull request, #2577:
URL: https://github.com/apache/datafusion-ballista/pull/2577
## Rationale for this change
This is phase 1 of introducing the Result Service proposed in
apache/datafusion-ballista#2484. The Result Service gives clients that can't
reach executors, as is common on Kubernetes and in isolated networks, a
dedicated and independently scalable way to fetch query results. Today those
clients fetch through the scheduler's embedded proxy, which puts every result
byte on the scheduler, where it competes with planning and scales only with the
scheduler. Moving result delivery into its own stateless service keeps the
scheduler focused on coordination and lets result serving scale on its own.
## What changes are included in this PR?
- **Result Service.** A new `ballista-result-service` binary forwards each
fetch to the executor named in its ticket. Result serving moves into
`ballista_core::serving`, shared by the executor, the Result Service and the
embedded proxy. The service redeems the same `FetchPartition` tickets that
executors already accept, and clients find it through the existing
`advertise_flight_endpoint`. No client code or wire-format changes are required.
- **Graceful shutdown.** On SIGTERM the Result Service stops accepting
connections and gives in-flight result streams up to
`--graceful-shutdown-timeout-seconds` (10 by default) to finish, to reduce
interrupted fetches during rollouts and node drains.
- **Deprecation.** `--enable-embedded-flight-proxy` keeps working throughout
56.x, with a warning at startup. The 56.0.0 upgrade guide covers the migration.
- **Release and deployment.**
- The crate is published like the scheduler and executor.
- Its Docker image is published to ghcr.io with the others.
- A Kubernetes example covers drain-safe rollouts (a `preStop` sleep, a
matching grace period and a PodDisruptionBudget), an ingress, and a
NetworkPolicy.
- The chaos harness routes results through the Result Service on both its
process and k8s backends.
## Next steps
- **Flight SQL results.** Serve the Flight SQL frontend's
(apache/datafusion-ballista#2416) results from the Result Service too, in a
focused follow-up PR, together with Flight location URIs for
`advertise_flight_endpoint`.
- **Opaque result handles.** Once scheduler HA provides shared state, the
Result Service can resolve opaque handles, so clients no longer see executor
addresses and the service only dials executors it knows. Until then, the
Kubernetes example restricts what it can reach with a NetworkPolicy.
- **Authentication on the fetch path.** This applies equally to executors
and the embedded proxy today, so it's a separate discussion.
- **Removing the embedded proxy.** Planned for 57.0.0.
## Are there any user-facing changes?
Yes, all in `docs/source/upgrading/56.0.0.md`:
- the new binary and image
- the deprecation warning
- `InvalidArgument` instead of `Internal` for undecodable fetch tickets
There are no breaking API changes.
`SchedulerConfig::with_enable_embedded_flight_proxy` is deprecated.
## How was this tested?
- **Unit tests:** undecodable tickets are rejected as `InvalidArgument`, the
shutdown drain gives up at its bound, and the k8s harness reserves distinct
ports. The existing scheduler, executor, and Flight SQL test suites pass.
- **Chaos harness:** routes every scenario through the Result Service by
default, on the process backend and on the k8s backend in the `k8s-chaos`
workflow.
- **Lints:** fmt, clippy, rustdoc, taplo and prettier pass on every commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
