[
https://issues.apache.org/jira/browse/SPARK-61964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61964:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Import pyspark.sql.connect.proto in pyspark.sql.connect.client.core, so that
> PySpark's pickles of PlanObservedMetrics load in a new process
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61964
> URL: https://issues.apache.org/jira/browse/SPARK-61964
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark's PlanObservedMetrics holds the Expression.Literal messages that the
> server sent, which
> protobuf pickles as its _InternalConstructMessage of their full name
> (spark.connect.Expression.Literal),
> and which so load only once the files of the protos of Spark Connect are in
> protobuf's default
> pool. PySpark's pyspark.sql.connect.client.core imports
> pyspark.sql.connect.proto (core.py:90), and
> a pickle of PlanObservedMetrics imports that module before it makes the
> messages, so the pickle
> loads in a new process: for example one in the attrs of a pandas DataFrame of
> toPandas of an
> observed DataFrame. The drop-in's pyspark.sql.connect.client.core does not
> import the package, so
> in a new process such a pickle raises, with either implementation of
> protobuf. The drop-in's own
> pickles, whose messages name their classes, load in both clients.
> Repro
> # with pyspark-client 4.3.0
> from pyspark.sql import Observation, functions as F
> pdf = spark.range(3).observe(Observation("o"),
> F.count(F.lit(1)).alias("n")).toPandas()
> pdf.to_pickle("/tmp/frame.pkl")
> # then, in a new process of the drop-in
> import pandas as pd
> pd.read_pickle("/tmp/frame.pkl").attrs["observed_metrics"]
> Actual (drop-in): KeyError: "Couldn't find message
> spark.connect.Expression.Literal" (KeyError:
> 'spark.connect.Expression.Literal' with
> PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python).
> Expected (pyspark-client 4.3.0, in a new process): [Plan observed(o=[long:
> 3\n])], of pairs {'n': 3}.
> The same holds for a bare pickle of PlanObservedMetrics.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]