[
https://issues.apache.org/jira/browse/SPARK-62047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62047:
-----------------------------------
Labels: pull-request-available (was: )
> Pickle the Any and Timestamp messages of the drop-in's protobuf runtime, and
> fail each request of a user context extension that is not an Any, as PySpark
> ---------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62047
> URL: https://issues.apache.org/jira/browse/SPARK-62047
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Two gaps of the drop-in (pyspark-client-rust) against PySpark 4.3.0:
> 1. The messages of the well-known types of the drop-in's protobuf runtime,
> google.protobuf.Any and google.protobuf.Timestamp
> (pyspark.sql.connect.proto._rust.Any and Timestamp, as the user context
> extensions and the Timestamp fields give them), do not pickle: pickle.dumps
> raises "Can't pickle <class 'pyspark.sql.connect.proto._rust.any_pb2.Any'>:
> import of module 'pyspark.sql.connect.proto._rust.any_pb2' failed", and
> cloudpickle raises "cannot pickle 'classmethod_descriptor' object", as their
> modules are not in sys.modules. PySpark's (upb's) pickle of its classes of
> google.protobuf.any_pb2 and timestamp_pb2. The runtime's modules are now in
> sys.modules, so their pickles load where the drop-in is installed, in another
> process too; the pickle names the runtime's module, which PySpark does not
> have (documented).
> 2. A user context extension that is not an Any
> (spark.client.add_threadlocal_user_context_extension(pb2.Relation()), or an
> object that is not a message) is taken by PySpark's lists, and each request
> of an execution, an analysis, a configuration or an interrupt raises upb's
> TypeError when it is built ("Parameter to MergeFrom() must be instance of
> same class: ...", "Expected a message object, but got 'x'."), of the global
> extensions first, until it is removed. The drop-in sent the bytes of another
> message as those of an Any (an empty Any of pb2.Relation()), and raised
> AttributeError ('str' object has no attribute 'SerializeToString') at the add
> of an object that is not a message. Its lists now take any object, and each
> call that sends a request raises upb's TypeError (or the error of the
> unpacking of an item that is not a pair) before the request is sent, as
> PySpark's requests raise it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]