[
https://issues.apache.org/jira/browse/SPARK-62021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-62021.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 862
[https://github.com/apache/spark-connect-rust/pull/862]
> Make the relation of df._plan.plan(client) hold the relation of the
> DataFrame, built at the call, so that it is sent after the DataFrame is gone,
> as PySpark's
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62021
> URL: https://issues.apache.org/jira/browse/SPARK-62021
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Problem
> df._plan.plan(client) gives a placeholder of the relation of the DataFrame:
> an extension whose
> value is a marker. The core replaced it, when a plan, a command or an
> analysis that holds it was
> sent, with the relation of the DataFrame of the marker, which it looked up in
> a registry of weak
> references (plan::placeholder_target, crates/spark-connect/src/plan.rs). So a
> placeholder whose
> DataFrame was gone failed every request with "a placeholder of a DataFrame
> that no longer
> exists":
> - the inline client.to_table(pb2.Plan(root=spark.sql(q)._plan.plan(client)),
> {}), whose DataFrame
> is freed before the call;
> - a relation that the program keeps after its DataFrame is freed;
> - a plan parsed again of its bytes.
> PySpark's LogicalPlan.plan(session) builds the relation at the call
> (python/pyspark/sql/connect/
> plan.py), so the message stands on its own, whatever happens to the DataFrame.
> As the drop-in built the relation at the send and not at the call, the call
> did not do what
> PySpark's build does either:
> - the PicklingError of a UDF that does not pickle was raised by the request,
> not by plan(client);
> - the warning of a window of no partitions came from the request;
> - a UDF whose state changed after plan(client) was sent with the state of the
> request.
> Repro (master 50594b2e, against a Spark 4.3.0 server)
> import gc, threading
> import pyspark.sql.connect.proto as pb2
> from pyspark.sql import functions as F
> client = spark.client
> client.to_table(pb2.Plan(root=spark.sql("SELECT 1 AS
> a")._plan.plan(client)), {})[0].to_pydict()
> rel = spark.range(3).filter("id > 0")._plan.plan(client); gc.collect()
> client.to_table(pb2.Plan(root=rel), {})[0].to_pydict()
>
> client.to_table(pb2.Plan.FromString(pb2.Plan(root=rel).SerializeToString()),
> {})[0].to_pydict()
> lock = threading.Lock()
> spark.range(1).select(F.udf(lambda x: lock is not None,
> "boolean")("id"))._plan.plan(client)
> state = ["at the call"]
> rel = spark.range(1).select(F.udf(lambda x: state[0],
> "string")("id").alias("s"))._plan.plan(client)
> state[0] = "after the call"
> client.to_table(pb2.Plan(root=rel), {})[0].to_pydict()
> Actual (drop-in at master 50594b2e)
> - each of the first three raises PySparkValueError: a placeholder of a
> DataFrame that no longer
> exists;
> - plan(client) of the unpicklable UDF returns, and the request raises the
> PicklingError;
> - {'s': ['after the call']}.
> Expected (pyspark-client 4.3.0)
> - {'a': [1]}, {'id': [1, 2]} and {'id': [1, 2]};
> - plan(client) raises PicklingError: Could not serialize object: TypeError:
> cannot pickle
> '_thread.lock' object, and no request is sent;
> - {'s': ['at the call']}.
> Fix
> - The placeholder of df._plan.plan(client) holds the relation, built at the
> call, with its
> pickles, warnings and errors, and encoded in the value of its extension:
> the marker, the levels
> and the encoding of the relation. It is one level, so Python's protobuf
> never parses the
> relation of a deep DataFrame.
> - The core replaces the placeholder with the relation that it holds. The
> registry by marker keeps
> weak entries only, for the Observations of the DataFrame of a placeholder.
> - A placeholder of a marker alone stays for the plans that the core builds
> over a DataFrame and
> resolves while the DataFrame is alive, such as the binding's ML transforms
> of a DataFrame of
> another session.
> - The docs say this and state the reasons of what stays different:
> - a placeholder in an expression's extension, or one that a plugin changes;
> - the plan ID of DataFrame(plan, session);
> - the retry policies of can_retry and to_state;
> - get_tags.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]