[ 
https://issues.apache.org/jira/browse/SPARK-61964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61964:
------------------------------------

    Assignee: Hyukjin Kwon

> Import pyspark.sql.connect.proto in pyspark.sql.connect.client.core, so that 
> PySpark's pickles of PlanObservedMetrics load in a new process
> -------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61964
>                 URL: https://issues.apache.org/jira/browse/SPARK-61964
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark's PlanObservedMetrics holds the Expression.Literal messages that the 
> server sent, which
> protobuf pickles as its _InternalConstructMessage of their full name 
> (spark.connect.Expression.Literal),
> and which so load only once the files of the protos of Spark Connect are in 
> protobuf's default
> pool. PySpark's pyspark.sql.connect.client.core imports 
> pyspark.sql.connect.proto (core.py:90), and
> a pickle of PlanObservedMetrics imports that module before it makes the 
> messages, so the pickle
> loads in a new process: for example one in the attrs of a pandas DataFrame of 
> toPandas of an
> observed DataFrame. The drop-in's pyspark.sql.connect.client.core does not 
> import the package, so
> in a new process such a pickle raises, with either implementation of 
> protobuf. The drop-in's own
> pickles, whose messages name their classes, load in both clients.
> Repro
>     # with pyspark-client 4.3.0
>     from pyspark.sql import Observation, functions as F
>     pdf = spark.range(3).observe(Observation("o"), 
> F.count(F.lit(1)).alias("n")).toPandas()
>     pdf.to_pickle("/tmp/frame.pkl")
>     # then, in a new process of the drop-in
>     import pandas as pd
>     pd.read_pickle("/tmp/frame.pkl").attrs["observed_metrics"]
> Actual (drop-in): KeyError: "Couldn't find message 
> spark.connect.Expression.Literal" (KeyError:
> 'spark.connect.Expression.Literal' with 
> PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python).
> Expected (pyspark-client 4.3.0, in a new process): [Plan observed(o=[long: 
> 3\n])], of pairs {'n': 3}.
> The same holds for a bare pickle of PlanObservedMetrics.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to