[
https://issues.apache.org/jira/browse/SPARK-59856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59856:
-----------------------------------
Labels: pull-request-available (was: )
> Record the provenance of the vendored Spark Connect protos and add a sync
> script
> --------------------------------------------------------------------------------
>
> Key: SPARK-59856
> URL: https://issues.apache.org/jira/browse/SPARK-59856
> Project: Spark
> Issue Type: Sub-task
> Components: Connect
> Affects Versions: connect-gateway-0.1.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> The eleven .proto files under proto/spark/connect/ are copied verbatim from
> apache/spark and are the gateway's source of protocol truth -- crates/genproto
> generates the Rust bindings from them at build time. Nothing in the repo
> records
> where they came from. README calls them a "vendored read-only mirror of
> upstream" but names no revision, no upstream path, and no way to re-sync.
> That gap costs us three things. An Apache release manager has no answer to
> "where did these come from, and under what license". Anyone who needs to
> follow
> Spark Connect's protocol forward has no idea how far behind the vendored copy
> is or which revision to sync from. And the question "have these been edited
> locally since they were copied?" can only be answered by archaeology.
> Establishing the answer took comparing all eleven files against upstream tags,
> so this records the result rather than leaving it to be redone:
> - Upstream path is sql/connect/common/src/main/protobuf/spark/connect/ in
> apache/spark, all eleven files, Apache-2.0 with the upstream ASF headers
> intact.
> - The copy is from the Spark 4.2 development line, not a release. Nine of
> the
> eleven files are byte-identical to v4.2.0 and the files themselves
> reference
> fields "deprecated since Spark 4.2+", but no release tag matches --
> v4.2.0-rc1
> through rc5 were checked too. The doc records v4.2.0 as the baseline to
> diff
> against and says explicitly that this is not a claim the files came from
> that
> tag, so the record does not look more precise than the evidence supports.
> The comparison also surfaced something nobody knew: the vendored protos are
> behind upstream. relations.proto is missing the NearestByJoin message and
> pipelines.proto is missing AutoCdcFlowDetails, in both cases because upstream
> added them after the snapshot. Nothing was edited locally -- the drift is
> purely
> that we are behind. This PR records that; it deliberately does not fix it,
> since
> re-syncing moves the wire protocol and deserves its own review.
> The gap does not affect the gateway today. crates/proxy forwards each request
> onward whole and reads only session and identity metadata; nothing there
> inspects a plan's rel_type or walks the relation tree, so a request carrying
> one
> of the newer messages still routes and forwards correctly. This was verified
> in
> the code rather than assumed, because the claim is the reason the drift is not
> urgent.
> dev/sync-protos.sh makes the check repeatable. It reports drift by default and
> only overwrites when asked with --sync, because overwriting these files
> changes
> the protocol the gateway speaks and should not be a side effect of looking. It
> lists the upstream directory rather than iterating over the local files, so a
> file added upstream shows up as drift instead of being silently missed; a file
> vendored here but absent upstream is reported as well.
> Verified: check mode reports identical: 9 / differing: 2, matching the manual
> comparison, and --ref v4.1.3 gives 6/5, also matching. --sync was exercised
> on a
> throwaway copy of the tree, taking it from 9/2 to 11/0 while leaving the
> repo's
> own protos untouched. Exit codes are 1 for an unknown ref, 2 for a bad
> argument,
> 0 for a clean check. proto/spark/ is unchanged by this PR: scg-genproto still
> compiles and the full suite is 204 passed, 0 failed, 10 ignored.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
