Hyukjin Kwon created SPARK-62047:
------------------------------------

             Summary: Pickle the Any and Timestamp messages of the drop-in's 
protobuf runtime, and fail each request of a user context extension that is not 
an Any, as PySpark
                 Key: SPARK-62047
                 URL: https://issues.apache.org/jira/browse/SPARK-62047
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Two gaps of the drop-in (pyspark-client-rust) against PySpark 4.3.0:

1. The messages of the well-known types of the drop-in's protobuf runtime, 
google.protobuf.Any and google.protobuf.Timestamp 
(pyspark.sql.connect.proto._rust.Any and Timestamp, as the user context 
extensions and the Timestamp fields give them), do not pickle: pickle.dumps 
raises "Can't pickle <class 'pyspark.sql.connect.proto._rust.any_pb2.Any'>: 
import of module 'pyspark.sql.connect.proto._rust.any_pb2' failed", and 
cloudpickle raises "cannot pickle 'classmethod_descriptor' object", as their 
modules are not in sys.modules. PySpark's (upb's) pickle of its classes of 
google.protobuf.any_pb2 and timestamp_pb2. The runtime's modules are now in 
sys.modules, so their pickles load where the drop-in is installed, in another 
process too; the pickle names the runtime's module, which PySpark does not have 
(documented).

2. A user context extension that is not an Any 
(spark.client.add_threadlocal_user_context_extension(pb2.Relation()), or an 
object that is not a message) is taken by PySpark's lists, and each request of 
an execution, an analysis, a configuration or an interrupt raises upb's 
TypeError when it is built ("Parameter to MergeFrom() must be instance of same 
class: ...", "Expected a message object, but got 'x'."), of the global 
extensions first, until it is removed. The drop-in sent the bytes of another 
message as those of an Any (an empty Any of pb2.Relation()), and raised 
AttributeError ('str' object has no attribute 'SerializeToString') at the add 
of an object that is not a message. Its lists now take any object, and each 
call that sends a request raises upb's TypeError (or the error of the unpacking 
of an item that is not a pair) before the request is sent, as PySpark's 
requests raise it.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to