Victor Sunderland created SPARK-47445:
-----------------------------------------
Summary: Adding new 'Silent' ExplainMode
Key: SPARK-47445
URL: https://issues.apache.org/jira/browse/SPARK-47445
Project: Spark
Issue Type: Improvement
Components: Connect, Documentation, PySpark, SQL
Affects Versions: 4.0.0
Reporter: Victor Sunderland
While investigating unit test duration we found that
org.apache.spark.sql.execution.QueryExecution.explainString () takes
approximately 14% time. This method generates the string representation of the
execution plan. The string is often used for logging purposes. This is also
called for each AQE job so it can save prod execution time too. While
SPARK-44485 does exist to help optimize the prod execution time, the main
purpose of this PR is to save time during unit testing.
I've added a silent mode to ExplainMode to try and mitigate this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
