This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 44db668  [SPARK-61019][CONNECT][PYTHON] Print the plans of explain and 
StreamingQuery.explain with Python's print, and raise the errors of explain, as 
PySpark does
44db668 is described below

commit 44db668c2217f1308ae921798ff5cc0513b3912c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 02:20:00 2026 +0900

    [SPARK-61019][CONNECT][PYTHON] Print the plans of explain and 
StreamingQuery.explain with Python's print, and raise the errors of explain, as 
PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61019](https://issues.apache.org/jira/browse/SPARK-61019): `explain` 
and `StreamingQuery.explain` now print with Python's `print`, and `explain` 
raises the errors of PySpark.
    
    - `df.explain(extended, mode)` prints `df._explain_string(extended, mode)` 
with Python's `print`. So the plan goes to `sys.stdout`, which a notebook or 
`contextlib.redirect_stdout` captures.
    - `StreamingQuery.explain(extended)` prints its plan and returns None.
    - `_explain_string` raises `CANNOT_SET_TOGETHER` for both `extended` and 
`mode`, and `NOT_EXPECTED_TYPE` for an argument that is neither a bool nor a 
string. It raises `VALUE_NOT_ALLOWED` for a mode other than `"simple"`, 
`"extended"`, `"codegen"`, `"cost"` and `"formatted"`, as PySpark Connect's 
client checks it.
    
    An error of a PySpark error class that the client raises is now raised with 
its error class and message parameters, so that `getCondition()` and 
`getMessageParameters()` return them. Examples are the `VALUE_NOT_ALLOWED` of 
`explain`, and the `CANNOT_BE_EMPTY` of `spark.tvf.json_tuple(col)` without 
fields. An error of a class that PySpark does not know is raised with its 
message, as before.
    
    In the Rust API, `DataFrame::explain_string` returns the plan that 
`explain_mode` prints, and the mode is checked as PySpark checks it.
    
    ### Why are the changes needed?
    
    - `explain` wrote the plan to the standard output of the process from Rust, 
which neither a notebook nor `redirect_stdout` captured.
    - `StreamingQuery.explain` returned the plan as a string.
    - `_explain_string` did not exist.
    - `explain(mode="FORMATTED")` was accepted.
    - `explain(mode="bogus")`, `explain(True, "simple")` and `explain(1)` 
raised a `PySparkValueError` without an error class, a `ValueError` and a 
`TypeError`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `explain` and `StreamingQuery.explain` print as PySpark 4.3.0 does, 
and raise its errors. The errors of PySpark's error classes that the client 
raises carry their classes and parameters.
    
    ### How was this patch tested?
    
    - A new live test captures the plans that `explain` and 
`StreamingQuery.explain` print with `redirect_stdout`. It passes with official 
PySpark 4.3.0, and fails without this change.
    - A new offline test checks the errors of the arguments and of 
`json_tuple`, and the plan that `explain` prints of the explain string of a 
mock Connect server and the mode it sends. The mock now answers `Explain` 
requests. The test fails without this change.
    - A new lib test checks the explain modes of `explain_string`.
    - A probe of `explain`, `_explain_string`, `printSchema`, `show` and 
streaming `explain` matches official PySpark 4.3.0.
    - The lib tests and every integration test, e2e included, against a local 
Spark 4.3.0 server (672 passed, and 129 of the transport crate), the build of 
all tests with `wasm-udf`, the offline Python suites (1786 passed, 8 skipped), 
and the live drop-in and Arrow suites (95 passed) and live cases (10 passed). 
Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #222 from HyukjinKwon/SPARK-61019-explain.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs    | 63 +++++++++++++++++++++--------
 crates/pyspark-rs/src/errors.rs       | 30 ++++++++++++--
 crates/pyspark-rs/src/streaming.rs    |  9 +++--
 crates/spark-connect/src/dataframe.rs | 75 +++++++++++++++++++++++++----------
 python/tests/_mock_connect_server.py  |  6 +++
 python/tests/test_dropin_e2e.py       | 29 ++++++++++++++
 python/tests/test_dropin_offline.py   | 47 ++++++++++++++++++++++
 7 files changed, 216 insertions(+), 43 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 9ed6072..eab80f0 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -763,37 +763,66 @@ impl PyDataFrame {
         Ok(PyDataFrame::new(df))
     }
 
-    /// Print the plan. Mirrors `DataFrame.explain(extended=None, mode=None)`:
-    /// `extended` may be a bool or a mode string; `mode` is a mode string; 
setting
-    /// both is an error.
+    /// Print the plan with Python's `print`, as PySpark's 
`DataFrame.explain`, which prints
+    /// `_explain_string(extended, mode)`.
     #[pyo3(signature = (extended=None, mode=None))]
     fn explain(
         &self,
         py: Python<'_>,
         extended: Option<&Bound<'_, PyAny>>,
-        mode: Option<String>,
+        mode: Option<&Bound<'_, PyAny>>,
     ) -> PyResult<()> {
-        let resolved: String = match (extended, mode.as_deref()) {
+        let plan = self.explain_string(py, extended, mode)?;
+        py.import("builtins")?.call_method1("print", (plan,))?;
+        Ok(())
+    }
+
+    /// The plan that `explain` prints, as PySpark's 
`DataFrame._explain_string`: `extended` is
+    /// a bool, for the "extended" or the "simple" mode, or a mode, and `mode` 
a mode, but not
+    /// both; the mode is "simple" without either.
+    #[pyo3(name = "_explain_string", signature = (extended=None, mode=None))]
+    fn explain_string(
+        &self,
+        py: Python<'_>,
+        extended: Option<&Bound<'_, PyAny>>,
+        mode: Option<&Bound<'_, PyAny>>,
+    ) -> PyResult<String> {
+        use pyo3::types::{PyBool, PyString};
+        let explain_mode: String = match (extended, mode) {
             (Some(_), Some(_)) => {
-                return Err(PyErr::new::<pyo3::exceptions::PyValueError, _>(
-                    "extended and mode cannot be set together",
+                return Err(crate::column::raise_pyspark(
+                    py,
+                    "PySparkValueError",
+                    "CANNOT_SET_TOGETHER",
+                    &[("arg_list", "extended and mode")],
                 ))
             }
             (None, None) => "simple".to_string(),
-            (None, Some(m)) => m.to_string(),
-            (Some(e), None) => {
-                if let Ok(b) = e.extract::<bool>() {
-                    if b { "extended" } else { "simple" }.to_string()
-                } else if let Ok(s) = e.extract::<String>() {
-                    s
+            (Some(extended), None) if extended.is_instance_of::<PyBool>() => {
+                if extended.extract::<bool>()? {
+                    "extended"
                 } else {
-                    return Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(
-                        "extended must be a bool or a mode string",
-                    ));
+                    "simple"
                 }
+                .to_string()
+            }
+            (Some(arg), None) | (None, Some(arg)) if 
arg.is_instance_of::<PyString>() => {
+                arg.extract()?
+            }
+            (Some(arg), None) | (None, Some(arg)) => {
+                return Err(crate::column::raise_pyspark(
+                    py,
+                    "PySparkTypeError",
+                    "NOT_EXPECTED_TYPE",
+                    &[
+                        ("expected_type", "bool or str"),
+                        ("arg_name", "extended (optional) and mode 
(optional)"),
+                        ("arg_type", &arg.get_type().str()?.to_string()),
+                    ],
+                ))
             }
         };
-        py.detach(|| self.dataframe.explain_mode(&resolved))
+        py.detach(|| self.dataframe.explain_string(&explain_mode))
             .to_pyerr()
     }
 
diff --git a/crates/pyspark-rs/src/errors.rs b/crates/pyspark-rs/src/errors.rs
index d82bc46..c471010 100644
--- a/crates/pyspark-rs/src/errors.rs
+++ b/crates/pyspark-rs/src/errors.rs
@@ -176,9 +176,13 @@ pub fn spark_error_to_py_exception(err: SparkError) -> 
PyErr {
         // Get the exception class from the module
         match errors_module.getattr(class_name) {
             Ok(exc_class) => {
-                // Try to instantiate the exception class by calling it with 
the message
-                // This creates a PyObject exception instance
-                match exc_class.call1((msg.clone(),)) {
+                // Instantiate the exception class: of the error class and 
message parameters
+                // of an error of a PySpark error class, otherwise of the 
message.
+                let created = match classed_exception(&exc_class, &err) {
+                    Some(exc_instance) => Ok(exc_instance),
+                    None => exc_class.call1((msg.clone(),)),
+                };
+                match created {
                     Ok(exc_instance) => {
                         // Mirror the reference client, which stamps 
`_operation_id` on
                         // errors raised for an ExecutePlan (read via 
`operation_id`).
@@ -203,6 +207,26 @@ pub fn spark_error_to_py_exception(err: SparkError) -> 
PyErr {
     })
 }
 
+/// The exception of `exc_class` for an error of a PySpark error class that 
the client raised,
+/// as PySpark raises it: of the error class and the message parameters, from 
which the
+/// exception renders its message, so that `getCondition` and 
`getMessageParameters` return
+/// them. None for an error without an error class, or of one that PySpark 
does not know.
+fn classed_exception<'py>(
+    exc_class: &Bound<'py, PyAny>,
+    err: &SparkError,
+) -> Option<Bound<'py, PyAny>> {
+    let error_class = err.get_condition()?;
+    let py = exc_class.py();
+    let parameters = PyDict::new(py);
+    for (key, value) in err.get_message_parameters().unwrap_or_default() {
+        parameters.set_item(key, value).ok()?;
+    }
+    let kwargs = PyDict::new(py);
+    kwargs.set_item("errorClass", error_class).ok()?;
+    kwargs.set_item("messageParameters", parameters).ok()?;
+    exc_class.call((), Some(&kwargs)).ok()
+}
+
 /// Helper to convert Result<T, SparkError> to PyResult<T>.
 pub trait ResultExt<T> {
     fn to_pyerr(self) -> PyResult<T>;
diff --git a/crates/pyspark-rs/src/streaming.rs 
b/crates/pyspark-rs/src/streaming.rs
index ef080b1..8f47836 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -897,10 +897,13 @@ impl PyStreamingQuery {
         self.inner.process_all_available().to_pyerr()
     }
 
-    // Mirrors reference `StreamingQuery.explain(extended=False)` - the arg is 
optional.
+    /// Print the plan of the query with Python's `print`, as PySpark's
+    /// `StreamingQuery.explain(extended=False)`.
     #[pyo3(signature = (extended=false))]
-    fn explain(&self, extended: bool) -> PyResult<String> {
-        self.inner.explain(extended).to_pyerr()
+    fn explain(&self, py: Python<'_>, extended: bool) -> PyResult<()> {
+        let plan = py.detach(|| self.inner.explain(extended)).to_pyerr()?;
+        py.import("builtins")?.call_method1("print", (plan,))?;
+        Ok(())
     }
 
     fn exception(&self) -> PyResult<Option<PyStreamingQueryException>> {
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index 1a708b4..cc335a4 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1233,39 +1233,57 @@ impl DataFrame {
         self.explain_mode("simple")
     }
 
-    /// Print the execution plan in a specific mode. Mirrors the `mode` 
argument of
-    /// `pyspark.sql.DataFrame.explain`: one of "simple", "extended", 
"codegen",
-    /// "cost", "formatted" (case-insensitive).
+    /// Print the execution plan in an explain mode, 
[`DataFrame::explain_string`].
     pub fn explain_mode(&self, mode: &str) -> Result<()> {
+        println!("{}", self.explain_string(mode)?);
+        Ok(())
+    }
+
+    /// The execution plan in an explain mode, mirroring 
`pyspark.sql.DataFrame._explain_string`
+    /// of a mode, which PySpark Connect's client checks: one of "simple", 
"extended",
+    /// "codegen", "cost" and "formatted", else a `VALUE_NOT_ALLOWED` error.
+    pub fn explain_string(&self, mode: &str) -> Result<String> {
         use proto::analyze_plan_request::explain::ExplainMode;
-        let explain_mode = match mode.to_lowercase().as_str() {
+        let explain_mode = match mode {
             "simple" => ExplainMode::Simple,
             "extended" => ExplainMode::Extended,
             "codegen" => ExplainMode::Codegen,
             "cost" => ExplainMode::Cost,
             "formatted" => ExplainMode::Formatted,
-            other => {
+            _ => {
                 return Err(SparkError::value(
-                    "UNSUPPORTED_EXPLAIN_MODE",
-                    &[("mode", other)],
+                    "VALUE_NOT_ALLOWED",
+                    &[
+                        ("arg_name", "explain_mode"),
+                        (
+                            "allowed_values",
+                            "['simple', 'extended', 'codegen', 'cost', 
'formatted']",
+                        ),
+                    ],
                 ))
             }
         };
-        let relation = self.relation();
-        let mut plan = proto::Plan::default();
-        plan.op_type = Some(proto::plan::OpType::Root(relation));
-        let mut ex = proto::analyze_plan_request::Explain::default();
-        ex.plan = Some(plan);
-        ex.explain_mode = explain_mode as i32;
-        let mut request = proto::AnalyzePlanRequest::default();
-        request.session_id = self.session.client().session_id().to_string();
-        request.user_context = Some(proto::UserContext::default());
-        request.analyze = 
Some(proto::analyze_plan_request::Analyze::Explain(ex));
+        let plan = proto::Plan {
+            op_type: Some(proto::plan::OpType::Root(self.relation())),
+        };
+        let request = proto::AnalyzePlanRequest {
+            session_id: self.session.client().session_id().to_string(),
+            user_context: Some(proto::UserContext::default()),
+            analyze: Some(proto::analyze_plan_request::Analyze::Explain(
+                proto::analyze_plan_request::Explain {
+                    plan: Some(plan),
+                    explain_mode: explain_mode as i32,
+                },
+            )),
+            ..Default::default()
+        };
         let response = block_on(self.session.client().analyze_plan(request))?;
-        if let Some(proto::analyze_plan_response::Result::Explain(e)) = 
response.result {
-            println!("{}", e.explain_string);
+        match response.result {
+            Some(proto::analyze_plan_response::Result::Explain(e)) => 
Ok(e.explain_string),
+            _ => Err(SparkError::connect_msg(
+                "AnalyzePlan response did not contain an explain result",
+            )),
         }
-        Ok(())
     }
 
     /// Add a watermark to this DataFrame for event-time based windows.
@@ -3605,6 +3623,23 @@ mod plan_construction_tests {
         assert_eq!(relation.common.and_then(|common| common.plan_id), Some(1));
     }
 
+    #[test]
+    fn explain_string_takes_the_explain_modes_of_pyspark() {
+        // As PySpark Connect's client, which checks the mode before it asks 
the server.
+        let spark = session();
+        let df = spark.range(1).unwrap();
+        for mode in ["FORMATTED", "bogus", ""] {
+            let err = df.explain_string(mode).unwrap_err();
+            assert_eq!(err.get_condition().as_deref(), 
Some("VALUE_NOT_ALLOWED"));
+            let parameters = err.get_message_parameters().unwrap();
+            assert_eq!(parameters["arg_name"], "explain_mode");
+            assert_eq!(
+                parameters["allowed_values"],
+                "['simple', 'extended', 'codegen', 'cost', 'formatted']"
+            );
+        }
+    }
+
     #[test]
     fn with_watermark_plan() {
         let spark = session();
diff --git a/python/tests/_mock_connect_server.py 
b/python/tests/_mock_connect_server.py
index 452e9c3..847504b 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -98,6 +98,8 @@ def serve(connection):
         calls = 0
         execute_calls = 0
         data_type = struct_schema()
+        # AnalyzePlan returns `explain_string` for an Explain request, else 
`data_type`.
+        explain_string = "== Physical Plan ==\n*(1) Range (0, 1, step=1, 
splits=1)\n"
         failure = None
         block_response = False
         response_gate = Event()
@@ -143,6 +145,10 @@ def serve(connection):
                 return message_field(2, b"")
             if self.failure == "invalid-type":
                 return message_field(2, message_field(1, b""))
+            # base.proto: AnalyzePlanRequest.explain=5; 
AnalyzePlanResponse.explain=3,
+            # Explain.explain_string=1.
+            if 5 in parse_message(request):
+                return message_field(3, message_field(1, self.explain_string))
             # base.proto: AnalyzePlanResponse.schema=2, Schema.schema=1.
             return message_field(2, message_field(1, self.data_type))
 
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 99336ef..6905009 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1087,6 +1087,35 @@ def 
test_sql_keyword_arguments_are_not_parameters_as_pyspark(session):
     assert error.value.getCondition() == "UNBOUND_SQL_PARAMETER"
 
 
+def test_explain_prints_with_python_print_as_pyspark(session):
+    # As PySpark's: explain, and StreamingQuery.explain, print the plan with 
Python's print, to
+    # sys.stdout, which redirect_stdout captures, and return None.
+    import contextlib
+    import io
+
+    df = session.range(1)
+    printed = io.StringIO()
+    with contextlib.redirect_stdout(printed):
+        assert df.explain() is None
+        assert df.explain(mode="formatted") is None
+    assert printed.getvalue().startswith("== Physical Plan ==\n")
+    assert "\n(1) Range [codegen id : 1]\n" in printed.getvalue()
+    assert df._explain_string(True).startswith("== Parsed Logical Plan ==\n")
+
+    query = (
+        session.readStream.format("rate").load()
+        .writeStream.format("memory").queryName("explain_prints").start()
+    )
+    try:
+        query.processAllAvailable()
+        printed = io.StringIO()
+        with contextlib.redirect_stdout(printed):
+            assert query.explain(True) is None
+        assert printed.getvalue().startswith("== Parsed Logical Plan ==\n")
+    finally:
+        query.stop()
+
+
 def test_column_alias_names_every_column_and_keeps_its_metadata(session):
     df = session.sql("SELECT map('k1', 1) AS 
m").select(F.explode("m").alias("k", "v"))
     assert df.columns == ["k", "v"]
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 5535830..8f00a42 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6578,6 +6578,53 @@ def _offline_session():
     return SparkSession.builder.remote("sc://localhost:1").getOrCreate()
 
 
+def 
test_explain_checks_its_arguments_and_prints_with_print_as_pyspark(mock_connect):
+    # As PySpark's DataFrame._explain_string, which checks its arguments and 
the mode, and
+    # explain, which prints its plan with Python's print.
+    import contextlib
+    import io
+
+    from pyspark.errors import PySparkTypeError, PySparkValueError
+    from pyspark.sql import functions as F
+
+    spark, service = mock_connect
+    df = spark.range(1)
+    not_bool_or_str = {
+        "expected_type": "bool or str",
+        "arg_name": "extended (optional) and mode (optional)",
+        "arg_type": "<class 'int'>",
+    }
+    for call, error_type, condition, parameters in (
+        (lambda: df.explain(True, "simple"), PySparkValueError, 
"CANNOT_SET_TOGETHER",
+         {"arg_list": "extended and mode"}),
+        (lambda: df._explain_string(1), PySparkTypeError, "NOT_EXPECTED_TYPE", 
not_bool_or_str),
+        (lambda: df.explain(mode=1), PySparkTypeError, "NOT_EXPECTED_TYPE", 
not_bool_or_str),
+        (lambda: df.explain(mode="FORMATTED"), PySparkValueError, 
"VALUE_NOT_ALLOWED",
+         {"arg_name": "explain_mode",
+          "allowed_values": "['simple', 'extended', 'codegen', 'cost', 
'formatted']"}),
+        # An error of a PySpark error class that the client raises carries its 
error class
+        # and message parameters.
+        (lambda: spark.tvf.json_tuple(F.lit("{}")), PySparkValueError, 
"CANNOT_BE_EMPTY",
+         {"item": "field"}),
+    ):
+        with pytest.raises(error_type) as error:
+            call()
+        assert error.value.getCondition() == condition
+        assert error.value.getMessageParameters() == parameters
+
+    plan = service.explain_string
+    assert df._explain_string(mode="cost") == plan
+    printed = io.StringIO()
+    with contextlib.redirect_stdout(printed):
+        assert df.explain(True) is None
+    assert printed.getvalue() == plan + "\n"
+    # base.proto: AnalyzePlanRequest.explain=5, Explain.explain_mode=2, of 
EXPLAIN_MODE_COST=4
+    # and EXPLAIN_MODE_EXTENDED=2.
+    modes = [parse_message(parse_message(request)[5][0])[2][0]
+             for request in service.analyze_requests[-2:]]
+    assert modes == [4, 2]
+
+
 def test_session_context_manager_stops_session():
     spark = _offline_session()
     with spark as s:


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to