Hyukjin Kwon created SPARK-62047:
------------------------------------
Summary: Pickle the Any and Timestamp messages of the drop-in's
protobuf runtime, and fail each request of a user context extension that is not
an Any, as PySpark
Key: SPARK-62047
URL: https://issues.apache.org/jira/browse/SPARK-62047
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Two gaps of the drop-in (pyspark-client-rust) against PySpark 4.3.0:
1. The messages of the well-known types of the drop-in's protobuf runtime,
google.protobuf.Any and google.protobuf.Timestamp
(pyspark.sql.connect.proto._rust.Any and Timestamp, as the user context
extensions and the Timestamp fields give them), do not pickle: pickle.dumps
raises "Can't pickle <class 'pyspark.sql.connect.proto._rust.any_pb2.Any'>:
import of module 'pyspark.sql.connect.proto._rust.any_pb2' failed", and
cloudpickle raises "cannot pickle 'classmethod_descriptor' object", as their
modules are not in sys.modules. PySpark's (upb's) pickle of its classes of
google.protobuf.any_pb2 and timestamp_pb2. The runtime's modules are now in
sys.modules, so their pickles load where the drop-in is installed, in another
process too; the pickle names the runtime's module, which PySpark does not have
(documented).
2. A user context extension that is not an Any
(spark.client.add_threadlocal_user_context_extension(pb2.Relation()), or an
object that is not a message) is taken by PySpark's lists, and each request of
an execution, an analysis, a configuration or an interrupt raises upb's
TypeError when it is built ("Parameter to MergeFrom() must be instance of same
class: ...", "Expected a message object, but got 'x'."), of the global
extensions first, until it is removed. The drop-in sent the bytes of another
message as those of an Any (an empty Any of pb2.Relation()), and raised
AttributeError ('str' object has no attribute 'SerializeToString') at the add
of an object that is not a message. Its lists now take any object, and each
call that sends a request raises upb's TypeError (or the error of the unpacking
of an item that is not a pair) before the request is sent, as PySpark's
requests raise it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]