viirya opened a new pull request, #14:
URL: https://github.com/apache/spark-connect-gateway/pull/14
### What changes were proposed in this pull request?
Part of importing the Spark Connect Gateway reference implementation into
`apache/spark-connect-gateway` as a series of small, bottom-up PRs (tracked
in
the do-not-merge reference PR #1).
This adds the `scg-proxy` crate — the gRPC proxy core of the gateway, and the
crate that ties the previously imported layers together. It contains:
- **handler** — a `SparkConnectService` implementation that forwards every
RPC to
a backend chosen by routing + session affinity, stamping the authenticated
identity onto the forwarded request
- **dial** — a per-backend tonic `Channel` cache, so the dial cost isn't
paid per
RPC
- **outbound** — the credentials the gateway presents on the gateway→backend
hop
- **config_filter** — withholds backend-only secrets from `Config` RPC
responses
`scg-proxy` depends on `scg-routing`, `scg-auth`, `scg-genproto`,
`scg-observability`, `scg-tenant`, `scg-ratelimit`, and `scg-audit`. Its
examples
and integration tests additionally use `scg-store-memory`, `scg-store-redis`,
`scg-pool-static`, and `scg-healthcheck` as dev-dependencies — all of which
are
already in the workspace, which is why this PR lands after them.
It is imported complete: src, all integration tests (forwarding, auth, audit,
tenant, ratelimit, observability/tracing, multi-tenant e2e, clone-session,
outbound-token), and all examples (ha_smoke, drain_smoke, load, e2e_client,
fake_collector).
### Why are the changes needed?
Implements the reference gateway from the approved SPIP (SPARK-58455),
imported
one reviewable piece at a time.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
`cargo fmt --all --check`, `cargo clippy --workspace --all-targets -- -D
warnings`,
`cargo build --workspace`, and `cargo test --workspace` all pass locally: 201
tests pass, 10 Redis/testcontainer tests are `#[ignore]` so no Docker is
needed
in CI. Note `--all-targets` compiles all of this crate's examples and
integration
test binaries, which is why its four dev-dependency crates had to land first.
### Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with Claude Code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
