[ 
https://issues.apache.org/jira/browse/SPARK-62021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62021:
-----------------------------------
    Labels: pull-request-available  (was: )

> Make the relation of df._plan.plan(client) hold the relation of the 
> DataFrame, built at the call, so that it is sent after the DataFrame is gone, 
> as PySpark's
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62021
>                 URL: https://issues.apache.org/jira/browse/SPARK-62021
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Problem
> df._plan.plan(client) gives a placeholder of the relation of the DataFrame: 
> an extension whose
> value is a marker. The core replaced it, when a plan, a command or an 
> analysis that holds it was
> sent, with the relation of the DataFrame of the marker, which it looked up in 
> a registry of weak
> references (plan::placeholder_target, crates/spark-connect/src/plan.rs). So a 
> placeholder whose
> DataFrame was gone failed every request with "a placeholder of a DataFrame 
> that no longer
> exists":
> - the inline client.to_table(pb2.Plan(root=spark.sql(q)._plan.plan(client)), 
> {}), whose DataFrame
>   is freed before the call;
> - a relation that the program keeps after its DataFrame is freed;
> - a plan parsed again of its bytes.
> PySpark's LogicalPlan.plan(session) builds the relation at the call 
> (python/pyspark/sql/connect/
> plan.py), so the message stands on its own, whatever happens to the DataFrame.
> As the drop-in built the relation at the send and not at the call, the call 
> did not do what
> PySpark's build does either:
> - the PicklingError of a UDF that does not pickle was raised by the request, 
> not by plan(client);
> - the warning of a window of no partitions came from the request;
> - a UDF whose state changed after plan(client) was sent with the state of the 
> request.
> Repro (master 50594b2e, against a Spark 4.3.0 server)
>   import gc, threading
>   import pyspark.sql.connect.proto as pb2
>   from pyspark.sql import functions as F
>   client = spark.client
>   client.to_table(pb2.Plan(root=spark.sql("SELECT 1 AS 
> a")._plan.plan(client)), {})[0].to_pydict()
>   rel = spark.range(3).filter("id > 0")._plan.plan(client); gc.collect()
>   client.to_table(pb2.Plan(root=rel), {})[0].to_pydict()
>   
> client.to_table(pb2.Plan.FromString(pb2.Plan(root=rel).SerializeToString()), 
> {})[0].to_pydict()
>   lock = threading.Lock()
>   spark.range(1).select(F.udf(lambda x: lock is not None, 
> "boolean")("id"))._plan.plan(client)
>   state = ["at the call"]
>   rel = spark.range(1).select(F.udf(lambda x: state[0], 
> "string")("id").alias("s"))._plan.plan(client)
>   state[0] = "after the call"
>   client.to_table(pb2.Plan(root=rel), {})[0].to_pydict()
> Actual (drop-in at master 50594b2e)
>   - each of the first three raises PySparkValueError: a placeholder of a 
> DataFrame that no longer
>     exists;
>   - plan(client) of the unpicklable UDF returns, and the request raises the 
> PicklingError;
>   - {'s': ['after the call']}.
> Expected (pyspark-client 4.3.0)
>   - {'a': [1]}, {'id': [1, 2]} and {'id': [1, 2]};
>   - plan(client) raises PicklingError: Could not serialize object: TypeError: 
> cannot pickle
>     '_thread.lock' object, and no request is sent;
>   - {'s': ['at the call']}.
> Fix
> - The placeholder of df._plan.plan(client) holds the relation, built at the 
> call, with its
>   pickles, warnings and errors, and encoded in the value of its extension: 
> the marker, the levels
>   and the encoding of the relation. It is one level, so Python's protobuf 
> never parses the
>   relation of a deep DataFrame.
> - The core replaces the placeholder with the relation that it holds. The 
> registry by marker keeps
>   weak entries only, for the Observations of the DataFrame of a placeholder.
> - A placeholder of a marker alone stays for the plans that the core builds 
> over a DataFrame and
>   resolves while the DataFrame is alive, such as the binding's ML transforms 
> of a DataFrame of
>   another session.
> - The docs say this and state the reasons of what stays different:
>   - a placeholder in an expression's extension, or one that a plugin changes;
>   - the plan ID of DataFrame(plan, session);
>   - the retry policies of can_retry and to_state;
>   - get_tags.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to