Hyukjin Kwon created SPARK-61964:
------------------------------------

             Summary: Import pyspark.sql.connect.proto in 
pyspark.sql.connect.client.core, so that PySpark's pickles of 
PlanObservedMetrics load in a new process
                 Key: SPARK-61964
                 URL: https://issues.apache.org/jira/browse/SPARK-61964
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Problem
PySpark's PlanObservedMetrics holds the Expression.Literal messages that the 
server sent, which
protobuf pickles as its _InternalConstructMessage of their full name 
(spark.connect.Expression.Literal),
and which so load only once the files of the protos of Spark Connect are in 
protobuf's default
pool. PySpark's pyspark.sql.connect.client.core imports 
pyspark.sql.connect.proto (core.py:90), and
a pickle of PlanObservedMetrics imports that module before it makes the 
messages, so the pickle
loads in a new process: for example one in the attrs of a pandas DataFrame of 
toPandas of an
observed DataFrame. The drop-in's pyspark.sql.connect.client.core does not 
import the package, so
in a new process such a pickle raises, with either implementation of protobuf. 
The drop-in's own
pickles, whose messages name their classes, load in both clients.

Repro
    # with pyspark-client 4.3.0
    from pyspark.sql import Observation, functions as F
    pdf = spark.range(3).observe(Observation("o"), 
F.count(F.lit(1)).alias("n")).toPandas()
    pdf.to_pickle("/tmp/frame.pkl")
    # then, in a new process of the drop-in
    import pandas as pd
    pd.read_pickle("/tmp/frame.pkl").attrs["observed_metrics"]
Actual (drop-in): KeyError: "Couldn't find message 
spark.connect.Expression.Literal" (KeyError:
'spark.connect.Expression.Literal' with 
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python).
Expected (pyspark-client 4.3.0, in a new process): [Plan observed(o=[long: 
3\n])], of pairs {'n': 3}.
The same holds for a bare pickle of PlanObservedMetrics.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to