viirya opened a new pull request, #26:
URL: https://github.com/apache/spark-connect-gateway/pull/26
Two CI papercuts, unrelated to any single walkthrough.
First, a race that makes the e2e jobs randomly red. Every job waited for
its pods with
kubectl wait --for=condition=ready pod -l <selector>
but that command fails immediately with "no matching resources found"
when the selector matches zero pods, and --timeout governs only how long
it waits for the condition, not for the resource to appear. A Deployment
created moments earlier, whose ReplicaSet has not yet produced a pod,
loses the race. This is what turned e2e-auth-jwt red on the PR that
added the multi-replica job, which had not touched that job at all; a
rerun passed.
All eight call sites now use `kubectl rollout status deployment/...`,
which waits for the controller instead of for a label match. Every
backend Deployment declares a readinessProbe, and a Deployment rollout
only completes once its pods pass readiness, so this is the same
guarantee without the race. Two of the jobs already used rollout status
for the gateway and Redis; this makes the rest consistent.
Second, the repo has no .dockerignore, so `docker build .` sends the
entire tree to the daemon -- including target/, which is tens of
gigabytes once anyone has built locally. CI never noticed because a
fresh checkout has no target/, but every contributor building the image
locally pays it, and at that size it looks like a hang rather than a
transfer.
Verified: with the new .dockerignore the build context drops from over
15 GB to under 1 MB, and the image still builds and its gateway binary
still runs. The exclusions deliberately keep everything the build reads
-- the crates, proto/ (genproto's build.rs panics on an empty proto
directory), Cargo.toml, Cargo.lock and rust-toolchain.toml.
---
Verified locally before pushing.
**The wait fix.** Every backend Deployment declares a `readinessProbe`, and a
Deployment rollout only completes once its pods pass readiness, so `rollout
status` carries the same guarantee as the old pod-Ready wait — it just does
not
fail when the ReplicaSet has yet to create a pod. Two jobs already used it
for
the gateway and Redis; this makes all eight sites consistent. This PR's own
CI
run exercises the new code path across all five e2e jobs.
**The `.dockerignore`.** Build context measured before and after:
| | build context |
|---|---|
| before | 15 GB+ |
| after | 999 kB |
I listed the resulting context with a throwaway probe image rather than
trusting
the size alone: all 15 crates, `proto/spark/connect/*.proto`, `Cargo.lock`,
`rust-toolchain.toml`, `LICENSE` and `NOTICE` are present, while `target/`,
`.git/`, `docs/` and `deploy/` are gone. A full image build then succeeded
and
the resulting binary runs (`gateway --help`).
`proto/` is deliberately kept: genproto's `build.rs` reads it at build time
and
panics if it is empty. Neither `.github/.licenserc.yaml` nor
`dev/.rat-excludes`
exempts `.dockerignore`, so it carries a full ASF header in the same style as
`rust-toolchain.toml`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
