[
https://issues.apache.org/jira/browse/SPARK-61019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61019:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Print the plans of explain and StreamingQuery.explain with Python's print,
> and raise the errors of explain, in the Python client as PySpark does
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61019
> URL: https://issues.apache.org/jira/browse/SPARK-61019
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, {{df.explain(extended, mode)}} prints
> {{df._explain_string(extended, mode)}} with Python's {{print}}, so that the
> plan is written to {{sys.stdout}}, which a notebook or
> {{contextlib.redirect_stdout}} captures, and
> {{StreamingQuery.explain(extended)}} prints its plan and returns None.
> {{_explain_string}} raises {{CANNOT_SET_TOGETHER}} for both {{extended}} and
> {{mode}}, {{NOT_EXPECTED_TYPE}} for an argument that is neither a bool nor a
> string, and {{VALUE_NOT_ALLOWED}} for a mode other than 'simple', 'extended',
> 'codegen', 'cost' and 'formatted'.
> The Rust-backed Python client's {{explain}} writes the plan to the standard
> output of the process from Rust, which neither a notebook nor
> {{redirect_stdout}} captures, and {{StreamingQuery.explain}} returns the plan
> as a string. {{_explain_string}} does not exist.
> {{explain(mode="FORMATTED")}} is accepted, {{explain(mode="bogus")}} raises a
> PySparkValueError without an error class, and {{explain(True, "simple")}} and
> {{explain(1)}} raise a ValueError and a TypeError. Do as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
