[ 
https://issues.apache.org/jira/browse/SPARK-61786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61786:
---------------------------------
    Fix Version/s: connect-rust-4.3.0

> Always use the Rust protobuf runtime in pyspark.sql.connect.proto, and remove 
> PYSPARK_CONNECT_PROTO_IMPLEMENTATION
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61786
>                 URL: https://issues.apache.org/jira/browse/SPARK-61786
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> pyspark.sql.connect.proto exports the messages of the drop-in's Rust protobuf 
> runtime (SPARK-61755) whether or not the protobuf package is installed, and 
> PYSPARK_CONNECT_PROTO_IMPLEMENTATION is removed, with the vendored generated 
> *_pb2.py modules.
> Where protobuf is installed, the files of the protos are added to its default 
> descriptor pool, as the generated modules add them, so that the modules that 
> protoc generates of other protos that import them (a plugin's, such as Delta 
> Lake's Connect client) build, and protobuf's json_format finds the types of 
> an Any. The runtime gains what protobuf's own modules read of a message: 
> GetOptions and CopyToProto of the descriptors, the pool of a file, service 
> and method descriptors, the methods of protobuf's Any and Timestamp (a 
> Timestamp field takes a datetime), HasExtension/ClearExtension/Extensions as 
> upb's, and the names of the generated modules in the modules of the protos. 
> example_plugins.proto is compiled into the runtime.
> A message field of a message of protobuf's own classes (of a plugin's 
> generated module) takes a message of pyspark.sql.connect.proto through its 
> bytes or a dict only: protobuf's messages take their own classes only, so 
> their CopyFrom raises TypeError for one of the runtime's.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to