This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-gateway.git
The following commit(s) were added to refs/heads/main by this push:
new f154d51 [SPARK-59908] Sync the vendored Spark Connect protos to v4.2.0
f154d51 is described below
commit f154d512a04224a205808c8aef31087fecb3ff68
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Sat Oct 3 22:38:05 2026 -0700
[SPARK-59908] Sync the vendored Spark Connect protos to v4.2.0
### What changes were proposed in this pull request?
Sync the vendored Spark Connect protocol definitions to upstream `v4.2.0`:
```bash
dev/sync-protos.sh --sync --ref v4.2.0
```
`+31` lines in `relations.proto` and `+42` in `pipelines.proto`, **no
deletions** —
the drift was one-directional, so closing it is purely additive:
| File | Added |
|---|---|
| `relations.proto` | the `NearestByJoin` message, and its entry in the
`Relation.rel_type` oneof (tag 47) |
| `pipelines.proto` | the `AutoCdcFlowDetails` message, and the `import
"spark/connect/expressions.proto"` it needs |
`proto/PROVENANCE.md` and the `UPSTREAM_REF` comment in
`dev/sync-protos.sh` are
updated to describe the new state.
### Why are the changes needed?
#40 (SPARK-59856) recorded that the vendored protos were behind upstream
and added
`dev/sync-protos.sh`, but deliberately left the drift in place — re-syncing
moves
the protocol the gateway speaks, so it wanted its own review rather than
riding
along with a documentation change. This is that review.
**Why `v4.2.0`.** It is the newest *released* Spark tag. `v4.3.0` exists
only as
release candidates, and syncing to an rc would pin the gateway to a
protocol that
can still change before release. For reference, the tree is 4 files away
from
`v4.3.0-rc1`, so there is another sync to make once 4.3.0 ships — this is a
step,
not an end state.
### Does this PR introduce _any_ user-facing change?
No. The added messages extend the protocol the gateway can describe; they
do not
change how it routes, authenticates, or forwards anything. No configuration,
metric, or audit-event change.
### How was this patch tested?
A compiling binding is weak evidence for a protocol change, so the
verification
goes past the build.
**Byte-identical to upstream.** Checked file by file rather than trusting
the
script that wrote them — all 11 of 11 match `v4.2.0` exactly, and
`dev/sync-protos.sh` reports `identical: 11 differing: 0`.
**The new messages really are generated**, not just compiled past.
`cargo build -p scg-genproto` regenerates the bindings, and both appear in
the
generated Rust as real structs:
```rust
pub struct NearestByJoin {
#[prost(message, optional, boxed, tag = "1")]
pub left: ::core::option::Option<::prost::alloc::boxed::Box<Relation>>,
```
**The new `rel_type` variant cannot break an exhaustive match.** A new oneof
variant is the one way this change could have broken compilation somewhere
unexpected, so I checked rather than assumed: grepping the workspace outside
`crates/genproto` for `rel_type` / `RelType` returns nothing. The single
place that
does inspect message contents, `crates/proxy/src/config_filter.rs`, matches
on
`ConfigRequest.operation.op_type` — untouched here.
**End to end against a real Spark backend**, on kind, deployed through the
Helm
chart: `AnalyzePlan` returns the correct schema for a `range` plan, and
`ExecutePlan` streams 15 response messages back, with the gateway's audit
log
confirming both RPCs passed through it:
```
1 "rpc":"AnalyzePlan"
1 "rpc":"ExecutePlan"
```
The client encoded its requests **from the newly synced `.proto` files**,
which is
what makes this a test of the sync rather than of the gateway alone. It also
confirms the `v4.2.0` protocol stays compatible with a Spark 4.0 backend,
which is
what the e2e jobs run.
**Nothing regressed:** `cargo test --workspace` is 204 passed, 0 failed, 10
ignored — matching `main` — with `clippy --workspace --all-targets -D
warnings` and
`cargo fmt --check` clean.
### Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with Claude Code.
Closes #42 from viirya/proto-02-sync-4.2.0.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
dev/sync-protos.sh | 5 +--
proto/PROVENANCE.md | 68 ++++++++++++++++++++-----------------
proto/spark/connect/pipelines.proto | 42 +++++++++++++++++++++++
proto/spark/connect/relations.proto | 31 +++++++++++++++++
4 files changed, 112 insertions(+), 34 deletions(-)
diff --git a/dev/sync-protos.sh b/dev/sync-protos.sh
index 0a487a5..2b5e68b 100755
--- a/dev/sync-protos.sh
+++ b/dev/sync-protos.sh
@@ -33,8 +33,9 @@
set -euo pipefail
-# The revision proto/PROVENANCE.md records as the comparison baseline. Bump
this
-# together with that file whenever the vendored protos are re-synced.
+# The upstream revision proto/PROVENANCE.md records the vendored files as being
+# at. Bump this together with that file whenever the protos are re-synced, so
the
+# two never disagree about what proto/ contains.
UPSTREAM_REF="v4.2.0"
# Upstream location of the protocol definitions within apache/spark.
diff --git a/proto/PROVENANCE.md b/proto/PROVENANCE.md
index 7b8038d..be1c556 100644
--- a/proto/PROVENANCE.md
+++ b/proto/PROVENANCE.md
@@ -25,7 +25,7 @@ maintained here, and they are not generated: they are
checked-in schema source.
|---|---|
| Upstream project | `apache/spark` |
| Upstream path | `sql/connect/common/src/main/protobuf/spark/connect/` |
-| Baseline revision | `v4.2.0` (see *Which revision* below) |
+| Revision | `v4.2.0` — the vendored files match it exactly |
| Files vendored | all 11 `.proto` files in that directory |
| License | Apache-2.0 — the same license as this project; each file keeps its
ASF header |
@@ -36,37 +36,24 @@ the directory and fails if it is empty.
## Which revision
-The vendored copy predates the `v4.2.0` release. It matches `v4.2.0` exactly
for
-9 of the 11 files, and the files themselves refer to fields "deprecated since
-Spark 4.2+", so it was taken from the Spark 4.2 development line — but from an
-untagged commit, not from a release. No `v4.2.0-rcN` tag matches either.
+The vendored files are byte-identical to `v4.2.0`, synced by
+`dev/sync-protos.sh --sync --ref v4.2.0` (SPARK-59857). `dev/sync-protos.sh`
+with no arguments confirms this and reports `identical: 11`.
-`v4.2.0` is recorded as the baseline revision because it is the earliest
-*released* tag the vendored copy is consistent with, which makes it a
meaningful
-thing to diff against. It is not a claim that these files were taken from that
-tag.
+`v4.2.0` was chosen because it is the newest *released* Spark tag. `v4.3.0`
exists
+only as release candidates, and syncing to an rc would pin the gateway to a
+protocol that can still change before release.
-Two files differ from `v4.2.0`, in both cases because upstream has content the
-vendored copy does not:
+### History
-| File | Missing relative to `v4.2.0` |
-|---|---|
-| `spark/connect/relations.proto` | the `NearestByJoin` message and its
`Relation.rel_type` entry |
-| `spark/connect/pipelines.proto` | the `AutoCdcFlowDetails` message and its
`import "spark/connect/expressions.proto"` |
-
-Both are pure additions upstream: nothing in the vendored files was edited
-locally. In other words the vendored protos are slightly **behind** upstream,
not
-forked from it.
-
-This does not currently affect the gateway. It forwards each
-`SparkConnectService` request onward as a whole and only ever reads the
-session and identity metadata — nothing in `crates/proxy` inspects a plan's
-`rel_type` or walks the relation tree. So a request carrying one of these newer
-messages is still routed and forwarded correctly.
-
-What the gap does mean is that anything which needs to *inspect* one of those
-newer messages — a routing rule keyed on plan shape, say — would have to
re-sync
-first.
+The original vendored copy — everything before SPARK-59857 — predated the
+`v4.2.0` release. It matched `v4.2.0` for 9 of the 11 files and the files
+referenced fields "deprecated since Spark 4.2+", but no release tag matched it,
+including `v4.2.0-rc1` through `rc5`: it came from an untagged commit on the
+Spark 4.2 development line. It was behind upstream rather than forked from it,
+missing the `NearestByJoin` message in `relations.proto` and
+`AutoCdcFlowDetails` in `pipelines.proto`; the sync added both. Nothing had
been
+edited locally, then or now.
## Checking and re-syncing
@@ -100,11 +87,28 @@ gateway speaks. After a `--sync`:
A re-sync is worth reviewing on its own rather than folding into an unrelated
change, since it moves the wire protocol.
+## How protocol drift affects the gateway
+
+Useful context when deciding how urgent a re-sync is: the gateway forwards each
+`SparkConnectService` request onward whole and reads only the session and
identity
+metadata. Nothing in `crates/proxy` inspects a plan's `rel_type` or walks the
+relation tree, and nothing in the workspace matches on those enums, so a
request
+carrying a message type the vendored protos do not know about is still routed
and
+forwarded correctly.
+
+The one place that does look inside a message is
`crates/proxy/src/config_filter.rs`,
+which matches on `ConfigRequest.operation.op_type` to withhold the backend
token —
+unrelated to the relation and plan types.
+
+So falling behind upstream degrades gracefully. It becomes a real problem only
for
+something that needs to *inspect* a newer message, such as a routing rule
keyed on
+plan shape.
+
## For a release
These files are third-party content redistributed in the source release. They
are Apache-2.0 licensed and retain their upstream ASF headers, so they need no
separate entry in `LICENSE`, but the source release's `NOTICE`-side paperwork
-should account for them as sourced from `apache/spark`. That belongs with the
-`LICENSE-binary` / `NOTICE-binary` dependency-census work, which is still
-outstanding.
+should account for them as sourced from `apache/spark` at the revision recorded
+above. That belongs with the `LICENSE-binary` / `NOTICE-binary`
dependency-census
+work, which is still outstanding.
diff --git a/proto/spark/connect/pipelines.proto
b/proto/spark/connect/pipelines.proto
index 6438583..7632ec9 100644
--- a/proto/spark/connect/pipelines.proto
+++ b/proto/spark/connect/pipelines.proto
@@ -22,6 +22,7 @@ package spark.connect;
import "google/protobuf/any.proto";
import "google/protobuf/timestamp.proto";
import "spark/connect/common.proto";
+import "spark/connect/expressions.proto";
import "spark/connect/relations.proto";
import "spark/connect/types.proto";
@@ -144,6 +145,7 @@ message PipelineCommand {
oneof details {
WriteRelationFlowDetails relation_flow_details = 7;
+ AutoCdcFlowDetails auto_cdc_flow_details = 10;
google.protobuf.Any extension = 999;
}
@@ -154,6 +156,46 @@ message PipelineCommand {
optional spark.connect.Relation relation = 1;
}
+ // Details for Auto CDC flows.
+ message AutoCdcFlowDetails {
+ // The name of the CDC source to stream from.
+ optional string source = 1;
+
+ // Column(s) that uniquely identify a row in source and target data.
+ repeated Expression keys = 2;
+
+ // Expression to order the source data.
+ optional Expression sequence_by = 3;
+
+ // Delete condition for the merged operation.
+ optional Expression apply_as_deletes = 6;
+
+ // Truncate condition for the merged operation.
+ optional Expression apply_as_truncates = 7;
+
+ // Columns included in the output table.
+ repeated Expression column_list = 8;
+
+ // Columns excluded from the output table.
+ repeated Expression except_column_list = 9;
+
+ // SCD Type for target table.
+ SCDType stored_as_scd_type = 10;
+
+ // Subset of columns to ignore null in updates.
+ repeated Expression ignore_null_updates_column_list = 14;
+
+ // Subset of columns excluded from ignoring null in updates.
+ repeated Expression ignore_null_updates_except_column_list = 15;
+
+ }
+
+ // SCD Type for Auto CDC target tables.
+ enum SCDType {
+ SCD_TYPE_UNSPECIFIED = 0;
+ SCD_TYPE_1 = 1;
+ }
+
// If true, define the flow as a one-time flow, such as for backfill.
// Set to true changes the flow in two ways:
// - The flow is run one time by default. If the pipeline is ran with a
full refresh,
diff --git a/proto/spark/connect/relations.proto
b/proto/spark/connect/relations.proto
index 57c4ed7..95cc928 100644
--- a/proto/spark/connect/relations.proto
+++ b/proto/spark/connect/relations.proto
@@ -82,6 +82,7 @@ message Relation {
LateralJoin lateral_join = 44;
ChunkedCachedLocalRelation chunked_cached_local_relation = 45;
RelationChanges relation_changes = 46;
+ NearestByJoin nearest_by_join = 47;
// NA functions
NAFill fill_na = 90;
@@ -1276,3 +1277,33 @@ message LateralJoin {
// (Required) The join type.
Join.JoinType join_type = 4;
}
+
+// Relation of type [[NearestByJoin]].
+//
+// For each row on the left side, returns up to `num_results` rows from the
right side ranked
+// by `ranking_expression`.
+message NearestByJoin {
+ // (Required) Left (query) input relation.
+ Relation left = 1;
+
+ // (Required) Right (base) input relation.
+ Relation right = 2;
+
+ // (Required) Scalar expression used to rank candidate rows on the right
side.
+ Expression ranking_expression = 3;
+
+ // (Required) Maximum number of matches per left row. Must be between 1 and
100000.
+ int32 num_results = 4;
+
+ // The following three fields use `string` (not typed enums) for parity with
`AsOfJoin`,
+ // which models analogous fields the same way. Validation happens
server-side at planning time.
+
+ // (Required) The join type. Must be one of: "inner", "leftouter".
+ string join_type = 5;
+
+ // (Required) Search algorithm contract. Must be one of: "approx", "exact".
+ string mode = 6;
+
+ // (Required) Ranking direction. Must be one of: "distance", "similarity".
+ string direction = 7;
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
