[
https://issues.apache.org/jira/browse/SPARK-59908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh resolved SPARK-59908.
---------------------------------
Fix Version/s: connect-gateway-0.1.0
Resolution: Fixed
Issue resolved by pull request 42
[https://github.com/apache/spark-connect-gateway/pull/42]
> Sync the vendored Spark Connect protos to v4.2.0
> ------------------------------------------------
>
> Key: SPARK-59908
> URL: https://issues.apache.org/jira/browse/SPARK-59908
> Project: Spark
> Issue Type: Sub-task
> Components: Connect
> Affects Versions: connect-gateway-0.1.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-gateway-0.1.0
>
>
> SPARK-59856 recorded that the vendored Spark Connect protos were behind
> upstream and added dev/sync-protos.sh, but deliberately left the drift in
> place: re-syncing moves the protocol the gateway speaks, so it wanted its own
> review. This does that sync.
> dev/sync-protos.sh --sync --ref v4.2.0
> The result is +31 lines in relations.proto and +42 in pipelines.proto, with no
> deletions -- the drift was one-directional, so closing it is purely additive:
> relations.proto the NearestByJoin message, and its entry in the
> Relation.rel_type oneof (tag 47)
> pipelines.proto the AutoCdcFlowDetails message, and the import of
> spark/connect/expressions.proto it needs
> v4.2.0 was chosen because it is the newest released Spark tag. v4.3.0 exists
> only as release candidates; syncing to an rc would pin the gateway to a
> protocol that can still change before release. For reference, the tree is 4
> files away from v4.3.0-rc1, so there will be another sync to make once 4.3.0
> ships.
> Verified beyond the build, because a compiling binding is weak evidence for a
> protocol change:
> - The vendored files are byte-identical to upstream v4.2.0, all 11 of 11,
> compared file by file rather than trusting the script that wrote them.
> - cargo build -p scg-genproto regenerates the bindings, and the two new
> messages are present in the generated Rust as real structs -- not merely
> compiled past.
> - The new rel_type variant cannot break an exhaustive match: grepping the
> workspace outside crates/genproto for rel_type / RelType returns nothing.
> The one place that does inspect message contents,
> crates/proxy/src/config_filter.rs, matches on
> ConfigRequest.operation.op_type, which this change does not touch.
> - End to end against a real Spark backend on kind, deployed via the Helm
> chart: AnalyzePlan returns the correct schema for a range plan and
> ExecutePlan streams 15 response messages back, with the gateway's audit
> log
> confirming both RPCs passed through it. The client encoded its requests
> from
> the newly synced .proto files, which is what makes this a test of the sync
> rather than of the gateway alone -- and it confirms the v4.2.0 protocol
> remains compatible with a Spark 4.0 backend.
> - cargo test --workspace is 204 passed, 0 failed, 10 ignored, matching main;
> clippy --workspace --all-targets -D warnings and cargo fmt --check are
> clean.
> proto/PROVENANCE.md is updated to match the new state. The revision is now
> recorded as an exact match rather than a diff baseline, and the uncertainty
> the
> old text carried -- that the previous copy came from an untagged 4.2
> development commit that no release tag matched -- moves into a History
> section,
> since it still documents where the original snapshot came from. The
> explanation of why protocol drift degrades gracefully is kept, rewritten as
> general guidance for judging how urgent a future sync is rather than as a
> description of this particular gap. UPSTREAM_REF in dev/sync-protos.sh is
> unchanged at v4.2.0; its comment is corrected, as it now names the revision
> the
> files are at rather than one to diff against.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
