viirya opened a new pull request, #42:
URL: https://github.com/apache/spark-connect-gateway/pull/42
### What changes were proposed in this pull request?
Sync the vendored Spark Connect protocol definitions to upstream `v4.2.0`:
```bash
dev/sync-protos.sh --sync --ref v4.2.0
```
`+31` lines in `relations.proto` and `+42` in `pipelines.proto`, **no
deletions** —
the drift was one-directional, so closing it is purely additive:
| File | Added |
|---|---|
| `relations.proto` | the `NearestByJoin` message, and its entry in the
`Relation.rel_type` oneof (tag 47) |
| `pipelines.proto` | the `AutoCdcFlowDetails` message, and the `import
"spark/connect/expressions.proto"` it needs |
`proto/PROVENANCE.md` and the `UPSTREAM_REF` comment in `dev/sync-protos.sh`
are
updated to describe the new state.
### Why are the changes needed?
#40 (SPARK-59856) recorded that the vendored protos were behind upstream and
added
`dev/sync-protos.sh`, but deliberately left the drift in place — re-syncing
moves
the protocol the gateway speaks, so it wanted its own review rather than
riding
along with a documentation change. This is that review.
**Why `v4.2.0`.** It is the newest *released* Spark tag. `v4.3.0` exists
only as
release candidates, and syncing to an rc would pin the gateway to a protocol
that
can still change before release. For reference, the tree is 4 files away from
`v4.3.0-rc1`, so there is another sync to make once 4.3.0 ships — this is a
step,
not an end state.
### Does this PR introduce _any_ user-facing change?
No. The added messages extend the protocol the gateway can describe; they do
not
change how it routes, authenticates, or forwards anything. No configuration,
metric, or audit-event change.
### How was this patch tested?
A compiling binding is weak evidence for a protocol change, so the
verification
goes past the build.
**Byte-identical to upstream.** Checked file by file rather than trusting the
script that wrote them — all 11 of 11 match `v4.2.0` exactly, and
`dev/sync-protos.sh` reports `identical: 11 differing: 0`.
**The new messages really are generated**, not just compiled past.
`cargo build -p scg-genproto` regenerates the bindings, and both appear in
the
generated Rust as real structs:
```rust
pub struct NearestByJoin {
#[prost(message, optional, boxed, tag = "1")]
pub left: ::core::option::Option<::prost::alloc::boxed::Box<Relation>>,
```
**The new `rel_type` variant cannot break an exhaustive match.** A new oneof
variant is the one way this change could have broken compilation somewhere
unexpected, so I checked rather than assumed: grepping the workspace outside
`crates/genproto` for `rel_type` / `RelType` returns nothing. The single
place that
does inspect message contents, `crates/proxy/src/config_filter.rs`, matches
on
`ConfigRequest.operation.op_type` — untouched here.
**End to end against a real Spark backend**, on kind, deployed through the
Helm
chart: `AnalyzePlan` returns the correct schema for a `range` plan, and
`ExecutePlan` streams 15 response messages back, with the gateway's audit log
confirming both RPCs passed through it:
```
1 "rpc":"AnalyzePlan"
1 "rpc":"ExecutePlan"
```
The client encoded its requests **from the newly synced `.proto` files**,
which is
what makes this a test of the sync rather than of the gateway alone. It also
confirms the `v4.2.0` protocol stays compatible with a Spark 4.0 backend,
which is
what the e2e jobs run.
**Nothing regressed:** `cargo test --workspace` is 204 passed, 0 failed, 10
ignored — matching `main` — with `clippy --workspace --all-targets -D
warnings` and
`cargo fmt --check` clean.
### Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with Claude Code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
