This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-gateway.git
The following commit(s) were added to refs/heads/main by this push:
new 3c61a47 [SPARK-60526] Use the Apache product name, fix docs that
point outside the repository, and tidy the Dockerfile
3c61a47 is described below
commit 3c61a474ea79971fa6c15c01252c11f423995666
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Tue Oct 27 13:42:55 2026 -0700
[SPARK-60526] Use the Apache product name, fix docs that point outside the
repository, and tidy the Dockerfile
### What changes were proposed in this pull request?
Cleanups found while preparing the first release. None changes runtime
behaviour.
**Product name.** Use "Apache Spark Connect Gateway" in the two user-visible
places that said "Open-source Spark Connect Gateway" -- the clap `about`
text
shown by `gateway --help` (`crates/gateway/src/main.rs`) and the Helm chart
description (`deploy/helm/scg/Chart.yaml`) -- and as the README title, which
was the bare repository name.
**Links outside the repository.** `README.md` and `docs/architecture.md`
linked to documents under `../plans/`, which is not part of this repository.
Both now point at the SPIP that proposed the project,
[SPARK-58455](https://issues.apache.org/jira/browse/SPARK-58455), which
states
the problem and links to the design document. In the README the two link
definitions for those files are replaced by one for the SPIP, and the
numbered links are renumbered in order of first use, so the text reads [1]
to
[4] from top to bottom (previously [5] appeared before [4]).
The README also called the feature set "Production-ready"; it now just lists
the features.
**Dockerfile.**
- The builder was `rust:1.82-slim` while `rust-toolchain.toml` pins 1.99.0,
so rustup installed a second toolchain on every build. It is now
`rust:1.99.0-slim-bookworm`; rustup only adds the `rustfmt` and `clippy`
components `rust-toolchain.toml` lists.
- The builder now names its Debian release, with a comment explaining why it
must match the runtime image (see below).
- `libssl-dev` and `pkg-config` are no longer installed. Nothing in the
dependency graph uses `openssl-sys` -- TLS is rustls with ring -- and
`pkg-config` was only there to locate libssl.
### Why are the changes needed?
The ASF name for the product is "Apache Spark Connect Gateway", and it is
what
users see in `--help` and in the chart.
The `../plans/` links are broken for every reader of the Apache repository
and
of a source release.
The Debian pin matters because of how glibc versions work. The runtime stage
is `distroless/cc-debian12` (bookworm), and the binary requires the glibc
symbol versions of the glibc it was *linked* against in the builder. glibc
is
only backward compatible, so if the unpinned `rust:*-slim` tag moved to a
newer
Debian, the build could produce a binary that fails to start on the runtime
image with ``version `GLIBC_x.yy' not found``. Pinning the builder to
bookworm
keeps both stages on the same release; the comment says to move them
together.
### Does this PR introduce _any_ user-facing change?
Yes, cosmetic only: `gateway --help` prints "Apache Spark Connect Gateway"
instead of "Open-source Spark Connect Gateway", and the Helm chart
description
changes the same way. No configuration, API, or behaviour change.
### How was this patch tested?
- `cargo fmt --all -- --check` and `cargo clippy --workspace --all-targets
--
-D warnings` -- clean.
- `cargo test --workspace` -- 204 passed, 0 failed.
- `gateway --help` prints the new name; `helm lint deploy/helm/scg` passes.
- The image was built for linux/amd64 from this Dockerfile without
`libssl-dev` or `pkg-config`, and the resulting image runs `gateway
--version` and `gateway --help`.
How the Dockerfile facts were established: `gateway` was built for
linux/amd64 with the pinned toolchain on bookworm. `readelf -d` shows it
needs
only `libc.so.6`, `libm.so.6`, `libgcc_s.so.1` and `ld-linux-x86-64.so.2`
(no
libssl, no libstdc++), with `GLIBC_2.34` as the highest required symbol
version against 2.36 in `distroless/cc-debian12`. The
`rust:1.99.0-slim-bookworm`
image ships `cargo`, `rustc` and `rust-std` only, which is why rustup still
adds `rustfmt` and `clippy`.
The e2e jobs build the image from this Dockerfile as well, so CI exercises
it
independently.
### Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with Claude Code.
Closes #53 from viirya/docs-01-release-cleanup.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
Dockerfile | 15 +++++++++++++--
README.md | 17 ++++++++---------
crates/gateway/src/main.rs | 2 +-
deploy/helm/scg/Chart.yaml | 2 +-
docs/architecture.md | 9 +++------
5 files changed, 26 insertions(+), 19 deletions(-)
diff --git a/Dockerfile b/Dockerfile
index f68c1b6..583f74b 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -13,10 +13,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM rust:1.82-slim AS build
+# The builder's Rust version matches rust-toolchain.toml, so the build uses the
+# image's toolchain. rustup only adds the rustfmt and clippy components that
+# rust-toolchain.toml lists, rather than installing a second toolchain.
+#
+# Its Debian release must match the runtime image below (bookworm, i.e.
+# distroless *-debian12). The binary is linked against the builder's glibc and
+# requires the symbol versions found there. glibc is only backward compatible,
+# so a builder on a newer Debian can produce a binary that fails to start on
+# the runtime image with "version `GLIBC_x.yy' not found". Move both together.
+FROM rust:1.99.0-slim-bookworm AS build
WORKDIR /src
+# protoc and the well-known-type .proto files, for crates/genproto. No OpenSSL:
+# TLS is rustls, so nothing links against libssl.
RUN apt-get update && apt-get install -y --no-install-recommends \
- protobuf-compiler libprotobuf-dev pkg-config libssl-dev && \
+ protobuf-compiler libprotobuf-dev && \
rm -rf /var/lib/apt/lists/*
COPY . .
RUN cargo build --release --bin gateway
diff --git a/README.md b/README.md
index f789474..d0ef17c 100644
--- a/README.md
+++ b/README.md
@@ -1,11 +1,11 @@
-# spark-connect-gateway
+# Apache Spark Connect Gateway
A stateless gRPC proxy that fronts a pool of [Apache Spark Connect][1]
servers, providing session affinity, multi-tenant routing, auth, and
observability features the open-source server intentionally leaves out.
> Rust workspace built with [`tonic`](https://github.com/hyperium/tonic).
-> Production-ready surface: JWT/OIDC auth, K8s service-watch discovery,
+> Features: JWT/OIDC auth, K8s service-watch discovery,
> Redis-backed multi-replica HA, per-tenant routing + rate limiting,
> structured audit logging, Prometheus metrics, OpenTelemetry tracing,
> Helm chart with sensible defaults.
@@ -30,9 +30,9 @@ observability features the open-source server intentionally
leaves out.
Open-source Spark Connect ships an excellent client-server protocol but
deliberately leaves multi-instance coordination out of scope. For anything
beyond a single Spark driver — multi-tenant platforms, HA, fleet-level
-observability — you need a layer in front of the servers. See
-[`OPEN_SOURCE_SPARK_CONNECT_GATEWAY_ANALYSIS.md`][3] in the plan repo for
-the full motivation and competitive landscape.
+observability — you need a layer in front of the servers. The SPIP that
+proposed this project, [SPARK-58455][2], explains the motivation in more
+detail and links to the design document.
## Why Rust?
@@ -41,7 +41,7 @@ the full motivation and competitive landscape.
`ExecutePlan` streams.
- `hyper` has best-in-class HTTP/2 trailing-header support, which gRPC
requires.
-- Aligns with [`Kimahriman/spark-connect-proxy`][5], the only existing OSS
+- Aligns with [`Kimahriman/spark-connect-proxy`][3], the only existing OSS
Spark-Connect-native proxy.
## Quick start
@@ -503,7 +503,6 @@ pull request you agree that your contribution is licensed
under
Apache 2.0.
[1]: https://spark.apache.org/docs/latest/spark-connect-overview.html
-[2]: ../plans/IMPLEMENTATION-PLAN-OSS-Spark-Connect-Gateway.md
-[3]: ../plans/OPEN_SOURCE_SPARK_CONNECT_GATEWAY_ANALYSIS.md
+[2]: https://issues.apache.org/jira/browse/SPARK-58455
+[3]: https://github.com/Kimahriman/spark-connect-proxy
[4]: https://github.com/apache/spark-kubernetes-operator
-[5]: https://github.com/Kimahriman/spark-connect-proxy
diff --git a/crates/gateway/src/main.rs b/crates/gateway/src/main.rs
index 5dfc42e..7ed21d6 100644
--- a/crates/gateway/src/main.rs
+++ b/crates/gateway/src/main.rs
@@ -58,7 +58,7 @@ use tonic::transport::Server;
use tracing::{info, warn};
#[derive(Parser, Debug)]
-#[command(version, about = "Open-source Spark Connect Gateway")]
+#[command(version, about = "Apache Spark Connect Gateway")]
struct Args {
/// Path to YAML config file.
#[arg(long, default_value = "config.yaml")]
diff --git a/deploy/helm/scg/Chart.yaml b/deploy/helm/scg/Chart.yaml
index dd410e2..6f4393f 100644
--- a/deploy/helm/scg/Chart.yaml
+++ b/deploy/helm/scg/Chart.yaml
@@ -16,7 +16,7 @@
apiVersion: v2
name: scg
description: |
- Open-source Spark Connect Gateway — stateless gRPC proxy in front of a
+ Apache Spark Connect Gateway — stateless gRPC proxy in front of a
pool of Apache Spark Connect servers. Provides session affinity,
multi-tenant routing, JWT/OIDC authentication, Prometheus metrics, and
OpenTelemetry trace export. Multi-replica HA via a shared Redis
diff --git a/docs/architecture.md b/docs/architecture.md
index c935d05..0e81741 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -548,9 +548,6 @@ Reading order, depending on what you're after:
| Trace an audit event to its source | `crates/audit/src/lib.rs` — one method
per event type |
| Reproduce perf numbers | `crates/proxy/examples/load.rs` — six scenarios
documented in [`perf-baseline.md`](perf-baseline.md) |
-The implementation plan that drove the build-out lives in the
-sibling repo at
-[`../plans/IMPLEMENTATION-PLAN-OSS-Spark-Connect-Gateway.md`](../../plans/IMPLEMENTATION-PLAN-OSS-Spark-Connect-Gateway.md)
-— it has the phase-by-phase task list and the rationale for
-several "why is it like this" decisions that aren't visible from
-the code alone.
+The project was proposed in the SPIP
+[SPARK-58455](https://issues.apache.org/jira/browse/SPARK-58455), which
+states the problem it solves and links to the design document.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
