Hyukjin Kwon created SPARK-60960:
------------------------------------
Summary: Keep executionInfo per DataFrame, and the operation id of
every ExecutePlan error, in Spark Connect Rust as PySpark does
Key: SPARK-60960
URL: https://issues.apache.org/jira/browse/SPARK-60960
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{DataFrame.executionInfo}} is the information of the last
execution of that DataFrame, and {{None}} before one, and every error of an
{{ExecutePlan}} request carries the request's operation id. The Rust-backed
Python client differs:
* {{a = spark.range(3); a.collect(); b = spark.range(5); b.executionInfo}}
gives the information of {{a}}'s execution; PySpark gives {{None}}. The client
keeps the last execution of the session, so a DataFrame shows another one's
metrics and operation id, also across threads.
* {{spark.catalog.setCurrentDatabase("nope")}} raises an {{AnalysisException}}
whose {{operation_id}} is {{None}}; PySpark gives the request's operation id.
The catalog operations and {{writeStream.start()}} send their {{ExecutePlan}}
requests without the operation id of the reattachable path.
Keep the execution information per DataFrame, and the operation id of every
ExecutePlan error, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
