[
https://issues.apache.org/jira/browse/SPARK-61786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61786:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Always use the Rust protobuf runtime in pyspark.sql.connect.proto, and remove
> PYSPARK_CONNECT_PROTO_IMPLEMENTATION
> ------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61786
> URL: https://issues.apache.org/jira/browse/SPARK-61786
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> pyspark.sql.connect.proto exports the messages of the drop-in's Rust protobuf
> runtime (SPARK-61755) whether or not the protobuf package is installed, and
> PYSPARK_CONNECT_PROTO_IMPLEMENTATION is removed, with the vendored generated
> *_pb2.py modules.
> Where protobuf is installed, the files of the protos are added to its default
> descriptor pool, as the generated modules add them, so that the modules that
> protoc generates of other protos that import them (a plugin's, such as Delta
> Lake's Connect client) build, and protobuf's json_format finds the types of
> an Any. The runtime gains what protobuf's own modules read of a message:
> GetOptions and CopyToProto of the descriptors, the pool of a file, service
> and method descriptors, the methods of protobuf's Any and Timestamp (a
> Timestamp field takes a datetime), HasExtension/ClearExtension/Extensions as
> upb's, and the names of the generated modules in the modules of the protos.
> example_plugins.proto is compiled into the runtime.
> A message field of a message of protobuf's own classes (of a plugin's
> generated module) takes a message of pyspark.sql.connect.proto through its
> bytes or a dict only: protobuf's messages take their own classes only, so
> their CopyFrom raises TypeError for one of the runtime's.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]