L. C. Hsieh created SPARK-59856:
-----------------------------------
Summary: Record the provenance of the vendored Spark Connect
protos and add a sync script
Key: SPARK-59856
URL: https://issues.apache.org/jira/browse/SPARK-59856
Project: Spark
Issue Type: Sub-task
Components: Connect
Affects Versions: connect-gateway-0.1.0
Reporter: L. C. Hsieh
Assignee: L. C. Hsieh
The eleven .proto files under proto/spark/connect/ are copied verbatim from
apache/spark and are the gateway's source of protocol truth -- crates/genproto
generates the Rust bindings from them at build time. Nothing in the repo records
where they came from. README calls them a "vendored read-only mirror of
upstream" but names no revision, no upstream path, and no way to re-sync.
That gap costs us three things. An Apache release manager has no answer to
"where did these come from, and under what license". Anyone who needs to follow
Spark Connect's protocol forward has no idea how far behind the vendored copy
is or which revision to sync from. And the question "have these been edited
locally since they were copied?" can only be answered by archaeology.
Establishing the answer took comparing all eleven files against upstream tags,
so this records the result rather than leaving it to be redone:
- Upstream path is sql/connect/common/src/main/protobuf/spark/connect/ in
apache/spark, all eleven files, Apache-2.0 with the upstream ASF headers
intact.
- The copy is from the Spark 4.2 development line, not a release. Nine of the
eleven files are byte-identical to v4.2.0 and the files themselves reference
fields "deprecated since Spark 4.2+", but no release tag matches --
v4.2.0-rc1
through rc5 were checked too. The doc records v4.2.0 as the baseline to diff
against and says explicitly that this is not a claim the files came from
that
tag, so the record does not look more precise than the evidence supports.
The comparison also surfaced something nobody knew: the vendored protos are
behind upstream. relations.proto is missing the NearestByJoin message and
pipelines.proto is missing AutoCdcFlowDetails, in both cases because upstream
added them after the snapshot. Nothing was edited locally -- the drift is purely
that we are behind. This PR records that; it deliberately does not fix it, since
re-syncing moves the wire protocol and deserves its own review.
The gap does not affect the gateway today. crates/proxy forwards each request
onward whole and reads only session and identity metadata; nothing there
inspects a plan's rel_type or walks the relation tree, so a request carrying one
of the newer messages still routes and forwards correctly. This was verified in
the code rather than assumed, because the claim is the reason the drift is not
urgent.
dev/sync-protos.sh makes the check repeatable. It reports drift by default and
only overwrites when asked with --sync, because overwriting these files changes
the protocol the gateway speaks and should not be a side effect of looking. It
lists the upstream directory rather than iterating over the local files, so a
file added upstream shows up as drift instead of being silently missed; a file
vendored here but absent upstream is reported as well.
Verified: check mode reports identical: 9 / differing: 2, matching the manual
comparison, and --ref v4.1.3 gives 6/5, also matching. --sync was exercised on a
throwaway copy of the tree, taking it from 9/2 to 11/0 while leaving the repo's
own protos untouched. Exit codes are 1 for an unknown ref, 2 for a bad argument,
0 for a clean check. proto/spark/ is unchanged by this PR: scg-genproto still
compiles and the full suite is 204 passed, 0 failed, 10 ignored.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
