[ 
https://issues.apache.org/jira/browse/SPARK-62047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62047:
------------------------------------

    Assignee: Hyukjin Kwon

> Pickle the Any and Timestamp messages of the drop-in's protobuf runtime, and 
> fail each request of a user context extension that is not an Any, as PySpark
> ---------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62047
>                 URL: https://issues.apache.org/jira/browse/SPARK-62047
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Two gaps of the drop-in (pyspark-client-rust) against PySpark 4.3.0:
> 1. The messages of the well-known types of the drop-in's protobuf runtime, 
> google.protobuf.Any and google.protobuf.Timestamp 
> (pyspark.sql.connect.proto._rust.Any and Timestamp, as the user context 
> extensions and the Timestamp fields give them), do not pickle: pickle.dumps 
> raises "Can't pickle <class 'pyspark.sql.connect.proto._rust.any_pb2.Any'>: 
> import of module 'pyspark.sql.connect.proto._rust.any_pb2' failed", and 
> cloudpickle raises "cannot pickle 'classmethod_descriptor' object", as their 
> modules are not in sys.modules. PySpark's (upb's) pickle of its classes of 
> google.protobuf.any_pb2 and timestamp_pb2. The runtime's modules are now in 
> sys.modules, so their pickles load where the drop-in is installed, in another 
> process too; the pickle names the runtime's module, which PySpark does not 
> have (documented).
> 2. A user context extension that is not an Any 
> (spark.client.add_threadlocal_user_context_extension(pb2.Relation()), or an 
> object that is not a message) is taken by PySpark's lists, and each request 
> of an execution, an analysis, a configuration or an interrupt raises upb's 
> TypeError when it is built ("Parameter to MergeFrom() must be instance of 
> same class: ...", "Expected a message object, but got 'x'."), of the global 
> extensions first, until it is removed. The drop-in sent the bytes of another 
> message as those of an Any (an empty Any of pb2.Relation()), and raised 
> AttributeError ('str' object has no attribute 'SerializeToString') at the add 
> of an object that is not a message. Its lists now take any object, and each 
> call that sends a request raises upb's TypeError (or the error of the 
> unpacking of an item that is not a pair) before the request is sent, as 
> PySpark's requests raise it.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to