This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 44db668 [SPARK-61019][CONNECT][PYTHON] Print the plans of explain and
StreamingQuery.explain with Python's print, and raise the errors of explain, as
PySpark does
44db668 is described below
commit 44db668c2217f1308ae921798ff5cc0513b3912c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 02:20:00 2026 +0900
[SPARK-61019][CONNECT][PYTHON] Print the plans of explain and
StreamingQuery.explain with Python's print, and raise the errors of explain, as
PySpark does
### What changes were proposed in this pull request?
[SPARK-61019](https://issues.apache.org/jira/browse/SPARK-61019): `explain`
and `StreamingQuery.explain` now print with Python's `print`, and `explain`
raises the errors of PySpark.
- `df.explain(extended, mode)` prints `df._explain_string(extended, mode)`
with Python's `print`. So the plan goes to `sys.stdout`, which a notebook or
`contextlib.redirect_stdout` captures.
- `StreamingQuery.explain(extended)` prints its plan and returns None.
- `_explain_string` raises `CANNOT_SET_TOGETHER` for both `extended` and
`mode`, and `NOT_EXPECTED_TYPE` for an argument that is neither a bool nor a
string. It raises `VALUE_NOT_ALLOWED` for a mode other than `"simple"`,
`"extended"`, `"codegen"`, `"cost"` and `"formatted"`, as PySpark Connect's
client checks it.
An error of a PySpark error class that the client raises is now raised with
its error class and message parameters, so that `getCondition()` and
`getMessageParameters()` return them. Examples are the `VALUE_NOT_ALLOWED` of
`explain`, and the `CANNOT_BE_EMPTY` of `spark.tvf.json_tuple(col)` without
fields. An error of a class that PySpark does not know is raised with its
message, as before.
In the Rust API, `DataFrame::explain_string` returns the plan that
`explain_mode` prints, and the mode is checked as PySpark checks it.
### Why are the changes needed?
- `explain` wrote the plan to the standard output of the process from Rust,
which neither a notebook nor `redirect_stdout` captured.
- `StreamingQuery.explain` returned the plan as a string.
- `_explain_string` did not exist.
- `explain(mode="FORMATTED")` was accepted.
- `explain(mode="bogus")`, `explain(True, "simple")` and `explain(1)`
raised a `PySparkValueError` without an error class, a `ValueError` and a
`TypeError`.
### Does this PR introduce _any_ user-facing change?
Yes. `explain` and `StreamingQuery.explain` print as PySpark 4.3.0 does,
and raise its errors. The errors of PySpark's error classes that the client
raises carry their classes and parameters.
### How was this patch tested?
- A new live test captures the plans that `explain` and
`StreamingQuery.explain` print with `redirect_stdout`. It passes with official
PySpark 4.3.0, and fails without this change.
- A new offline test checks the errors of the arguments and of
`json_tuple`, and the plan that `explain` prints of the explain string of a
mock Connect server and the mode it sends. The mock now answers `Explain`
requests. The test fails without this change.
- A new lib test checks the explain modes of `explain_string`.
- A probe of `explain`, `_explain_string`, `printSchema`, `show` and
streaming `explain` matches official PySpark 4.3.0.
- The lib tests and every integration test, e2e included, against a local
Spark 4.3.0 server (672 passed, and 129 of the transport crate), the build of
all tests with `wasm-udf`, the offline Python suites (1786 passed, 8 skipped),
and the live drop-in and Arrow suites (95 passed) and live cases (10 passed).
Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #222 from HyukjinKwon/SPARK-61019-explain.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 63 +++++++++++++++++++++--------
crates/pyspark-rs/src/errors.rs | 30 ++++++++++++--
crates/pyspark-rs/src/streaming.rs | 9 +++--
crates/spark-connect/src/dataframe.rs | 75 +++++++++++++++++++++++++----------
python/tests/_mock_connect_server.py | 6 +++
python/tests/test_dropin_e2e.py | 29 ++++++++++++++
python/tests/test_dropin_offline.py | 47 ++++++++++++++++++++++
7 files changed, 216 insertions(+), 43 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 9ed6072..eab80f0 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -763,37 +763,66 @@ impl PyDataFrame {
Ok(PyDataFrame::new(df))
}
- /// Print the plan. Mirrors `DataFrame.explain(extended=None, mode=None)`:
- /// `extended` may be a bool or a mode string; `mode` is a mode string;
setting
- /// both is an error.
+ /// Print the plan with Python's `print`, as PySpark's
`DataFrame.explain`, which prints
+ /// `_explain_string(extended, mode)`.
#[pyo3(signature = (extended=None, mode=None))]
fn explain(
&self,
py: Python<'_>,
extended: Option<&Bound<'_, PyAny>>,
- mode: Option<String>,
+ mode: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
- let resolved: String = match (extended, mode.as_deref()) {
+ let plan = self.explain_string(py, extended, mode)?;
+ py.import("builtins")?.call_method1("print", (plan,))?;
+ Ok(())
+ }
+
+ /// The plan that `explain` prints, as PySpark's
`DataFrame._explain_string`: `extended` is
+ /// a bool, for the "extended" or the "simple" mode, or a mode, and `mode`
a mode, but not
+ /// both; the mode is "simple" without either.
+ #[pyo3(name = "_explain_string", signature = (extended=None, mode=None))]
+ fn explain_string(
+ &self,
+ py: Python<'_>,
+ extended: Option<&Bound<'_, PyAny>>,
+ mode: Option<&Bound<'_, PyAny>>,
+ ) -> PyResult<String> {
+ use pyo3::types::{PyBool, PyString};
+ let explain_mode: String = match (extended, mode) {
(Some(_), Some(_)) => {
- return Err(PyErr::new::<pyo3::exceptions::PyValueError, _>(
- "extended and mode cannot be set together",
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkValueError",
+ "CANNOT_SET_TOGETHER",
+ &[("arg_list", "extended and mode")],
))
}
(None, None) => "simple".to_string(),
- (None, Some(m)) => m.to_string(),
- (Some(e), None) => {
- if let Ok(b) = e.extract::<bool>() {
- if b { "extended" } else { "simple" }.to_string()
- } else if let Ok(s) = e.extract::<String>() {
- s
+ (Some(extended), None) if extended.is_instance_of::<PyBool>() => {
+ if extended.extract::<bool>()? {
+ "extended"
} else {
- return Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(
- "extended must be a bool or a mode string",
- ));
+ "simple"
}
+ .to_string()
+ }
+ (Some(arg), None) | (None, Some(arg)) if
arg.is_instance_of::<PyString>() => {
+ arg.extract()?
+ }
+ (Some(arg), None) | (None, Some(arg)) => {
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "NOT_EXPECTED_TYPE",
+ &[
+ ("expected_type", "bool or str"),
+ ("arg_name", "extended (optional) and mode
(optional)"),
+ ("arg_type", &arg.get_type().str()?.to_string()),
+ ],
+ ))
}
};
- py.detach(|| self.dataframe.explain_mode(&resolved))
+ py.detach(|| self.dataframe.explain_string(&explain_mode))
.to_pyerr()
}
diff --git a/crates/pyspark-rs/src/errors.rs b/crates/pyspark-rs/src/errors.rs
index d82bc46..c471010 100644
--- a/crates/pyspark-rs/src/errors.rs
+++ b/crates/pyspark-rs/src/errors.rs
@@ -176,9 +176,13 @@ pub fn spark_error_to_py_exception(err: SparkError) ->
PyErr {
// Get the exception class from the module
match errors_module.getattr(class_name) {
Ok(exc_class) => {
- // Try to instantiate the exception class by calling it with
the message
- // This creates a PyObject exception instance
- match exc_class.call1((msg.clone(),)) {
+ // Instantiate the exception class: of the error class and
message parameters
+ // of an error of a PySpark error class, otherwise of the
message.
+ let created = match classed_exception(&exc_class, &err) {
+ Some(exc_instance) => Ok(exc_instance),
+ None => exc_class.call1((msg.clone(),)),
+ };
+ match created {
Ok(exc_instance) => {
// Mirror the reference client, which stamps
`_operation_id` on
// errors raised for an ExecutePlan (read via
`operation_id`).
@@ -203,6 +207,26 @@ pub fn spark_error_to_py_exception(err: SparkError) ->
PyErr {
})
}
+/// The exception of `exc_class` for an error of a PySpark error class that
the client raised,
+/// as PySpark raises it: of the error class and the message parameters, from
which the
+/// exception renders its message, so that `getCondition` and
`getMessageParameters` return
+/// them. None for an error without an error class, or of one that PySpark
does not know.
+fn classed_exception<'py>(
+ exc_class: &Bound<'py, PyAny>,
+ err: &SparkError,
+) -> Option<Bound<'py, PyAny>> {
+ let error_class = err.get_condition()?;
+ let py = exc_class.py();
+ let parameters = PyDict::new(py);
+ for (key, value) in err.get_message_parameters().unwrap_or_default() {
+ parameters.set_item(key, value).ok()?;
+ }
+ let kwargs = PyDict::new(py);
+ kwargs.set_item("errorClass", error_class).ok()?;
+ kwargs.set_item("messageParameters", parameters).ok()?;
+ exc_class.call((), Some(&kwargs)).ok()
+}
+
/// Helper to convert Result<T, SparkError> to PyResult<T>.
pub trait ResultExt<T> {
fn to_pyerr(self) -> PyResult<T>;
diff --git a/crates/pyspark-rs/src/streaming.rs
b/crates/pyspark-rs/src/streaming.rs
index ef080b1..8f47836 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -897,10 +897,13 @@ impl PyStreamingQuery {
self.inner.process_all_available().to_pyerr()
}
- // Mirrors reference `StreamingQuery.explain(extended=False)` - the arg is
optional.
+ /// Print the plan of the query with Python's `print`, as PySpark's
+ /// `StreamingQuery.explain(extended=False)`.
#[pyo3(signature = (extended=false))]
- fn explain(&self, extended: bool) -> PyResult<String> {
- self.inner.explain(extended).to_pyerr()
+ fn explain(&self, py: Python<'_>, extended: bool) -> PyResult<()> {
+ let plan = py.detach(|| self.inner.explain(extended)).to_pyerr()?;
+ py.import("builtins")?.call_method1("print", (plan,))?;
+ Ok(())
}
fn exception(&self) -> PyResult<Option<PyStreamingQueryException>> {
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 1a708b4..cc335a4 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1233,39 +1233,57 @@ impl DataFrame {
self.explain_mode("simple")
}
- /// Print the execution plan in a specific mode. Mirrors the `mode`
argument of
- /// `pyspark.sql.DataFrame.explain`: one of "simple", "extended",
"codegen",
- /// "cost", "formatted" (case-insensitive).
+ /// Print the execution plan in an explain mode,
[`DataFrame::explain_string`].
pub fn explain_mode(&self, mode: &str) -> Result<()> {
+ println!("{}", self.explain_string(mode)?);
+ Ok(())
+ }
+
+ /// The execution plan in an explain mode, mirroring
`pyspark.sql.DataFrame._explain_string`
+ /// of a mode, which PySpark Connect's client checks: one of "simple",
"extended",
+ /// "codegen", "cost" and "formatted", else a `VALUE_NOT_ALLOWED` error.
+ pub fn explain_string(&self, mode: &str) -> Result<String> {
use proto::analyze_plan_request::explain::ExplainMode;
- let explain_mode = match mode.to_lowercase().as_str() {
+ let explain_mode = match mode {
"simple" => ExplainMode::Simple,
"extended" => ExplainMode::Extended,
"codegen" => ExplainMode::Codegen,
"cost" => ExplainMode::Cost,
"formatted" => ExplainMode::Formatted,
- other => {
+ _ => {
return Err(SparkError::value(
- "UNSUPPORTED_EXPLAIN_MODE",
- &[("mode", other)],
+ "VALUE_NOT_ALLOWED",
+ &[
+ ("arg_name", "explain_mode"),
+ (
+ "allowed_values",
+ "['simple', 'extended', 'codegen', 'cost',
'formatted']",
+ ),
+ ],
))
}
};
- let relation = self.relation();
- let mut plan = proto::Plan::default();
- plan.op_type = Some(proto::plan::OpType::Root(relation));
- let mut ex = proto::analyze_plan_request::Explain::default();
- ex.plan = Some(plan);
- ex.explain_mode = explain_mode as i32;
- let mut request = proto::AnalyzePlanRequest::default();
- request.session_id = self.session.client().session_id().to_string();
- request.user_context = Some(proto::UserContext::default());
- request.analyze =
Some(proto::analyze_plan_request::Analyze::Explain(ex));
+ let plan = proto::Plan {
+ op_type: Some(proto::plan::OpType::Root(self.relation())),
+ };
+ let request = proto::AnalyzePlanRequest {
+ session_id: self.session.client().session_id().to_string(),
+ user_context: Some(proto::UserContext::default()),
+ analyze: Some(proto::analyze_plan_request::Analyze::Explain(
+ proto::analyze_plan_request::Explain {
+ plan: Some(plan),
+ explain_mode: explain_mode as i32,
+ },
+ )),
+ ..Default::default()
+ };
let response = block_on(self.session.client().analyze_plan(request))?;
- if let Some(proto::analyze_plan_response::Result::Explain(e)) =
response.result {
- println!("{}", e.explain_string);
+ match response.result {
+ Some(proto::analyze_plan_response::Result::Explain(e)) =>
Ok(e.explain_string),
+ _ => Err(SparkError::connect_msg(
+ "AnalyzePlan response did not contain an explain result",
+ )),
}
- Ok(())
}
/// Add a watermark to this DataFrame for event-time based windows.
@@ -3605,6 +3623,23 @@ mod plan_construction_tests {
assert_eq!(relation.common.and_then(|common| common.plan_id), Some(1));
}
+ #[test]
+ fn explain_string_takes_the_explain_modes_of_pyspark() {
+ // As PySpark Connect's client, which checks the mode before it asks
the server.
+ let spark = session();
+ let df = spark.range(1).unwrap();
+ for mode in ["FORMATTED", "bogus", ""] {
+ let err = df.explain_string(mode).unwrap_err();
+ assert_eq!(err.get_condition().as_deref(),
Some("VALUE_NOT_ALLOWED"));
+ let parameters = err.get_message_parameters().unwrap();
+ assert_eq!(parameters["arg_name"], "explain_mode");
+ assert_eq!(
+ parameters["allowed_values"],
+ "['simple', 'extended', 'codegen', 'cost', 'formatted']"
+ );
+ }
+ }
+
#[test]
fn with_watermark_plan() {
let spark = session();
diff --git a/python/tests/_mock_connect_server.py
b/python/tests/_mock_connect_server.py
index 452e9c3..847504b 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -98,6 +98,8 @@ def serve(connection):
calls = 0
execute_calls = 0
data_type = struct_schema()
+ # AnalyzePlan returns `explain_string` for an Explain request, else
`data_type`.
+ explain_string = "== Physical Plan ==\n*(1) Range (0, 1, step=1,
splits=1)\n"
failure = None
block_response = False
response_gate = Event()
@@ -143,6 +145,10 @@ def serve(connection):
return message_field(2, b"")
if self.failure == "invalid-type":
return message_field(2, message_field(1, b""))
+ # base.proto: AnalyzePlanRequest.explain=5;
AnalyzePlanResponse.explain=3,
+ # Explain.explain_string=1.
+ if 5 in parse_message(request):
+ return message_field(3, message_field(1, self.explain_string))
# base.proto: AnalyzePlanResponse.schema=2, Schema.schema=1.
return message_field(2, message_field(1, self.data_type))
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 99336ef..6905009 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1087,6 +1087,35 @@ def
test_sql_keyword_arguments_are_not_parameters_as_pyspark(session):
assert error.value.getCondition() == "UNBOUND_SQL_PARAMETER"
+def test_explain_prints_with_python_print_as_pyspark(session):
+ # As PySpark's: explain, and StreamingQuery.explain, print the plan with
Python's print, to
+ # sys.stdout, which redirect_stdout captures, and return None.
+ import contextlib
+ import io
+
+ df = session.range(1)
+ printed = io.StringIO()
+ with contextlib.redirect_stdout(printed):
+ assert df.explain() is None
+ assert df.explain(mode="formatted") is None
+ assert printed.getvalue().startswith("== Physical Plan ==\n")
+ assert "\n(1) Range [codegen id : 1]\n" in printed.getvalue()
+ assert df._explain_string(True).startswith("== Parsed Logical Plan ==\n")
+
+ query = (
+ session.readStream.format("rate").load()
+ .writeStream.format("memory").queryName("explain_prints").start()
+ )
+ try:
+ query.processAllAvailable()
+ printed = io.StringIO()
+ with contextlib.redirect_stdout(printed):
+ assert query.explain(True) is None
+ assert printed.getvalue().startswith("== Parsed Logical Plan ==\n")
+ finally:
+ query.stop()
+
+
def test_column_alias_names_every_column_and_keeps_its_metadata(session):
df = session.sql("SELECT map('k1', 1) AS
m").select(F.explode("m").alias("k", "v"))
assert df.columns == ["k", "v"]
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 5535830..8f00a42 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6578,6 +6578,53 @@ def _offline_session():
return SparkSession.builder.remote("sc://localhost:1").getOrCreate()
+def
test_explain_checks_its_arguments_and_prints_with_print_as_pyspark(mock_connect):
+ # As PySpark's DataFrame._explain_string, which checks its arguments and
the mode, and
+ # explain, which prints its plan with Python's print.
+ import contextlib
+ import io
+
+ from pyspark.errors import PySparkTypeError, PySparkValueError
+ from pyspark.sql import functions as F
+
+ spark, service = mock_connect
+ df = spark.range(1)
+ not_bool_or_str = {
+ "expected_type": "bool or str",
+ "arg_name": "extended (optional) and mode (optional)",
+ "arg_type": "<class 'int'>",
+ }
+ for call, error_type, condition, parameters in (
+ (lambda: df.explain(True, "simple"), PySparkValueError,
"CANNOT_SET_TOGETHER",
+ {"arg_list": "extended and mode"}),
+ (lambda: df._explain_string(1), PySparkTypeError, "NOT_EXPECTED_TYPE",
not_bool_or_str),
+ (lambda: df.explain(mode=1), PySparkTypeError, "NOT_EXPECTED_TYPE",
not_bool_or_str),
+ (lambda: df.explain(mode="FORMATTED"), PySparkValueError,
"VALUE_NOT_ALLOWED",
+ {"arg_name": "explain_mode",
+ "allowed_values": "['simple', 'extended', 'codegen', 'cost',
'formatted']"}),
+ # An error of a PySpark error class that the client raises carries its
error class
+ # and message parameters.
+ (lambda: spark.tvf.json_tuple(F.lit("{}")), PySparkValueError,
"CANNOT_BE_EMPTY",
+ {"item": "field"}),
+ ):
+ with pytest.raises(error_type) as error:
+ call()
+ assert error.value.getCondition() == condition
+ assert error.value.getMessageParameters() == parameters
+
+ plan = service.explain_string
+ assert df._explain_string(mode="cost") == plan
+ printed = io.StringIO()
+ with contextlib.redirect_stdout(printed):
+ assert df.explain(True) is None
+ assert printed.getvalue() == plan + "\n"
+ # base.proto: AnalyzePlanRequest.explain=5, Explain.explain_mode=2, of
EXPLAIN_MODE_COST=4
+ # and EXPLAIN_MODE_EXTENDED=2.
+ modes = [parse_message(parse_message(request)[5][0])[2][0]
+ for request in service.analyze_requests[-2:]]
+ assert modes == [4, 2]
+
+
def test_session_context_manager_stops_session():
spark = _offline_session()
with spark as s:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]