Hyukjin Kwon created SPARK-61964:
------------------------------------
Summary: Import pyspark.sql.connect.proto in
pyspark.sql.connect.client.core, so that PySpark's pickles of
PlanObservedMetrics load in a new process
Key: SPARK-61964
URL: https://issues.apache.org/jira/browse/SPARK-61964
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Problem
PySpark's PlanObservedMetrics holds the Expression.Literal messages that the
server sent, which
protobuf pickles as its _InternalConstructMessage of their full name
(spark.connect.Expression.Literal),
and which so load only once the files of the protos of Spark Connect are in
protobuf's default
pool. PySpark's pyspark.sql.connect.client.core imports
pyspark.sql.connect.proto (core.py:90), and
a pickle of PlanObservedMetrics imports that module before it makes the
messages, so the pickle
loads in a new process: for example one in the attrs of a pandas DataFrame of
toPandas of an
observed DataFrame. The drop-in's pyspark.sql.connect.client.core does not
import the package, so
in a new process such a pickle raises, with either implementation of protobuf.
The drop-in's own
pickles, whose messages name their classes, load in both clients.
Repro
# with pyspark-client 4.3.0
from pyspark.sql import Observation, functions as F
pdf = spark.range(3).observe(Observation("o"),
F.count(F.lit(1)).alias("n")).toPandas()
pdf.to_pickle("/tmp/frame.pkl")
# then, in a new process of the drop-in
import pandas as pd
pd.read_pickle("/tmp/frame.pkl").attrs["observed_metrics"]
Actual (drop-in): KeyError: "Couldn't find message
spark.connect.Expression.Literal" (KeyError:
'spark.connect.Expression.Literal' with
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python).
Expected (pyspark-client 4.3.0, in a new process): [Plan observed(o=[long:
3\n])], of pairs {'n': 3}.
The same holds for a bare pickle of PlanObservedMetrics.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]