This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new d4650b7  [SPARK-60991][CONNECT][PYTHON] Show a DataFrame as the table 
of the server, as PySpark does
d4650b7 is described below

commit d4650b77bb2f73f818a1d1935dfed3f9e3b03be5
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 14:59:03 2026 +0900

    [SPARK-60991][CONNECT][PYTHON] Show a DataFrame as the table of the server, 
as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60991](https://issues.apache.org/jira/browse/SPARK-60991): 
`DataFrame.show(n=20, truncate=True, vertical=False)` now sends a `ShowString` 
relation with the number of rows, the truncation and the vertical flag, as 
PySpark does. It prints the table that the server formats with Python's 
`print`. `DataFrame._show_string` returns that table.
    
    - Both check their arguments as PySpark's `_show_string` does. They raise 
`NOT_EXPECTED_TYPE` for an `n` that is not an int (a bool included), a 
`vertical` that is not a bool, and a `truncate` that `int()` cannot convert. An 
explicit None is checked as any other value. Too many, unknown or repeated 
arguments raise Python's `TypeError`s.
    - The Rust `DataFrame::show` prints the same table, and the new 
`DataFrame::show_string` returns it, through a new `LogicalPlan::ShowString`.
    
    ### Why are the changes needed?
    
    `df.show()` collected the first rows and printed each with Rust's 
`println!`, ignoring `truncate` and `vertical`. For example, 
`spark.range(2).show()` printed `[0]` and `[1]` instead of the table:
    
    ```
    +---+
    | id|
    +---+
    |  0|
    |  1|
    +---+
    ```
    
    As it wrote past Python's `sys.stdout`, a notebook or 
`contextlib.redirect_stdout` did not get the output.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, `show` prints the table of PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, a probe of 18 calls of `show` and 
`_show_string` gives the same output as official PySpark 4.3.0. It covers the 
default, `n`, a `truncate` of False, an int, a string and a float, `vertical`, 
the errors of each argument (None included), and too many, unknown and repeated 
arguments.
    - A new live test checks the tables of `show` with each argument, through 
`contextlib.redirect_stdout`. It also passes with official PySpark 4.3.0, and 
fails without this change.
    - A new offline test checks the errors of the arguments, which are raised 
before any request, and the signature.
    - The lib tests and every integration test, e2e included, against the local 
server (647 passed), the build of all tests with `wasm-udf`, the offline Python 
suites (719 passed, 9 skipped), and the live drop-in and Arrow suites (73 
passed) and live cases (10 passed). Clippy reports only one diagnostic in the 
changed lines: the `result_large_err` of the `py.detach` closure, which every 
action of the Python `DataFrame` has.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #194 from HyukjinKwon/SPARK-60991-show-string.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs    | 138 ++++++++++++++++++++++++++++++++--
 crates/spark-connect/src/dataframe.rs |  30 ++++++--
 crates/spark-connect/src/plan.rs      |  24 ++++++
 python/tests/test_dropin_e2e.py       |  29 +++++++
 python/tests/test_dropin_offline.py   |  30 ++++++++
 5 files changed, 239 insertions(+), 12 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 6e5be21..944a41f 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -50,6 +50,66 @@ pub struct PyDataFrame {
 }
 
 impl PyDataFrame {
+    /// The table of `show` and `_show_string`, for the arguments of 
`function`.
+    fn show_table(
+        &self,
+        py: Python<'_>,
+        function: &str,
+        args: &Bound<'_, PyTuple>,
+        kwargs: Option<&Bound<'_, PyDict>>,
+    ) -> PyResult<String> {
+        let [n, truncate, vertical] = bind_arguments(
+            function,
+            args,
+            kwargs,
+            ["n", "truncate", "vertical"],
+            [
+                20i32.into_pyobject(py)?.into_any(),
+                pyo3::types::PyBool::new(py, true).to_owned().into_any(),
+                pyo3::types::PyBool::new(py, false).to_owned().into_any(),
+            ],
+        )?;
+        let not_expected = |expected: &str, name: &str, value: &Bound<'_, 
PyAny>| -> PyErr {
+            let type_name = value
+                .get_type()
+                .name()
+                .map(|name| name.to_string())
+                .unwrap_or_default();
+            crate::column::raise_pyspark(
+                py,
+                "PySparkTypeError",
+                "NOT_EXPECTED_TYPE",
+                &[
+                    ("expected_type", expected),
+                    ("arg_name", name),
+                    ("arg_type", &type_name),
+                ],
+            )
+        };
+        if n.is_instance_of::<pyo3::types::PyBool>() || 
!n.is_instance_of::<pyo3::types::PyInt>() {
+            return Err(not_expected("int", "n", &n));
+        }
+        if !vertical.is_instance_of::<pyo3::types::PyBool>() {
+            return Err(not_expected("bool", "vertical", &vertical));
+        }
+        let truncate =
+            if truncate.is_instance_of::<pyo3::types::PyBool>() && 
truncate.is_truthy()? {
+                20
+            } else {
+                // As `int(truncate)`, whose ValueError is PySpark's 
NOT_EXPECTED_TYPE.
+                match 
py.import("builtins")?.getattr("int")?.call1((&truncate,)) {
+                    Ok(value) => value.extract::<i32>()?,
+                    Err(err) if 
err.is_instance_of::<pyo3::exceptions::PyValueError>(py) => {
+                        return Err(not_expected("bool", "truncate", 
&truncate));
+                    }
+                    Err(err) => return Err(err),
+                }
+            };
+        let (n, vertical) = (n.extract::<i32>()?, vertical.extract::<bool>()?);
+        py.detach(|| self.dataframe.show_string(n, truncate, vertical))
+            .to_pyerr()
+    }
+
     pub fn new(dataframe: DataFrame) -> Self {
         PyDataFrame { dataframe }
     }
@@ -1221,13 +1281,32 @@ impl PyDataFrame {
         py.detach(|| self.dataframe.count()).to_pyerr()
     }
 
-    /// Show the first n rows.
-    #[pyo3(signature = (n=20, truncate=true, vertical=false))]
-    fn show(&self, py: Python<'_>, n: usize, truncate: bool, vertical: bool) 
-> PyResult<()> {
-        // truncate/vertical are accepted for PySpark signature parity; the 
Rust core
-        // currently renders with its default formatting.
-        let _ = (truncate, vertical);
-        py.detach(|| self.dataframe.show(n)).to_pyerr()
+    /// Print the first `n` rows as the server formats them. Mirrors 
`DataFrame.show`, which
+    /// prints the table of `_show_string` with Python's `print`.
+    #[pyo3(signature = (*args, **kwargs), text_signature = "($self, n=20, 
truncate=True, vertical=False)")]
+    fn show(
+        &self,
+        py: Python<'_>,
+        args: &Bound<'_, PyTuple>,
+        kwargs: Option<&Bound<'_, PyDict>>,
+    ) -> PyResult<()> {
+        let table = self.show_table(py, "DataFrame.show", args, kwargs)?;
+        py.import("builtins")?.call_method1("print", (table,))?;
+        Ok(())
+    }
+
+    /// The table that `show` prints: the first `n` rows (20 by default), 
values truncated
+    /// to 20 characters for `truncate=True`, to `int(truncate)` otherwise (0 
for none), and
+    /// vertical with a line per value. Mirrors `DataFrame._show_string`, with 
its errors,
+    /// for any value given, None included.
+    #[pyo3(name = "_show_string", signature = (*args, **kwargs), 
text_signature = "($self, n=20, truncate=True, vertical=False)")]
+    fn show_string(
+        &self,
+        py: Python<'_>,
+        args: &Bound<'_, PyTuple>,
+        kwargs: Option<&Bound<'_, PyDict>>,
+    ) -> PyResult<String> {
+        self.show_table(py, "DataFrame._show_string", args, kwargs)
     }
 
     /// Execution info from the most recent action, as a 
`pyspark.sql.metrics.ExecutionInfo`
@@ -2440,6 +2519,51 @@ pub(crate) fn py_version(py: Python<'_>) -> String {
     .unwrap_or_else(|| "3.11".to_string())
 }
 
+/// The arguments of a call, bound to `names` positionally and by keyword as 
Python binds a
+/// signature of those names with `defaults`, raising Python's `TypeError`s of 
`function`
+/// for too many, repeated or unexpected arguments. For a method that tells a 
value of None
+/// from an argument not given.
+fn bind_arguments<'py, const N: usize>(
+    function: &str,
+    args: &Bound<'py, PyTuple>,
+    kwargs: Option<&Bound<'py, PyDict>>,
+    names: [&str; N],
+    defaults: [Bound<'py, PyAny>; N],
+) -> PyResult<[Bound<'py, PyAny>; N]> {
+    if args.len() > N {
+        return Err(pyo3::exceptions::PyTypeError::new_err(format!(
+            "{function}() takes from 1 to {} positional arguments but {} were 
given",
+            N + 1,
+            args.len() + 1
+        )));
+    }
+    let mut values: [Option<Bound<'py, PyAny>>; N] = std::array::from_fn(|_| 
None);
+    for (i, value) in args.iter().enumerate() {
+        values[i] = Some(value);
+    }
+    if let Some(kwargs) = kwargs {
+        for (key, value) in kwargs.iter() {
+            let key: String = key.extract()?;
+            let Some(i) = names.iter().position(|name| *name == key) else {
+                return Err(pyo3::exceptions::PyTypeError::new_err(format!(
+                    "{function}() got an unexpected keyword argument '{key}'"
+                )));
+            };
+            if values[i].is_some() {
+                return Err(pyo3::exceptions::PyTypeError::new_err(format!(
+                    "{function}() got multiple values for argument '{key}'"
+                )));
+            }
+            values[i] = Some(value);
+        }
+    }
+    let mut defaults = defaults.into_iter();
+    Ok(values.map(|value| {
+        let default = defaults.next().expect("a default per name");
+        value.unwrap_or(default)
+    }))
+}
+
 /// Resolve a Python schema argument (a `DataType` wrapper or a DDL string) 
into a Rust DataType.
 pub(crate) fn resolve_datatype(
     schema: &Bound<'_, PyAny>,
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index 7ddf16e..1004999 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -830,12 +830,32 @@ impl DataFrame {
         }
     }
 
-    /// Show the first n rows.
-    pub fn show(&self, n: usize) -> Result<()> {
-        let limited = self.limit(n as i32).collect()?;
-        for row in limited {
-            println!("{}", row);
+    /// The table that `show` prints, as the server formats it: the first `n` 
rows, values
+    /// truncated to `truncate` characters (0 for none), vertical with a line 
per value.
+    /// Mirrors `DataFrame._show_string`, which sends a `ShowString` relation.
+    pub fn show_string(&self, n: i32, truncate: i32, vertical: bool) -> 
Result<String> {
+        let shown = DataFrame::new(
+            self.session.clone(),
+            LogicalPlan::ShowString {
+                input: Arc::clone(&self.plan),
+                num_rows: n,
+                truncate,
+                vertical,
+            },
+        );
+        match shown.collect()?.first().map(|row| row.values()) {
+            Some([Value::String(table)]) => Ok(table.clone()),
+            _ => Err(SparkError::connect_msg(
+                "ShowString returned no table of one string",
+            )),
         }
+    }
+
+    /// Print the first `n` rows as the server formats them, with values 
truncated to 20
+    /// characters. Mirrors `DataFrame.show(n)`.
+    pub fn show(&self, n: usize) -> Result<()> {
+        let n = i32::try_from(n).unwrap_or(i32::MAX);
+        println!("{}", self.show_string(n, 20, false)?);
         Ok(())
     }
 
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 878a9fd..1fe426b 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -97,6 +97,15 @@ pub enum LogicalPlan {
     },
     /// Tail: `df.tail(n)` - returns last n rows.
     Tail { input: Arc<LogicalPlan>, limit: i32 },
+    /// ShowString: the table of `df.show(...)`, which the server formats as 
one string:
+    /// `num_rows` rows, values truncated to `truncate` characters (0 for 
none), vertical
+    /// with a line per value.
+    ShowString {
+        input: Arc<LogicalPlan>,
+        num_rows: i32,
+        truncate: i32,
+        vertical: bool,
+    },
     /// Deduplicate: `df.distinct()` or `df.dropDuplicates()`.
     Deduplicate {
         input: Arc<LogicalPlan>,
@@ -706,6 +715,21 @@ impl LogicalPlan {
                 relation.rel_type = 
Some(proto::relation::RelType::Tail(Box::new(tail)));
             }
 
+            LogicalPlan::ShowString {
+                input,
+                num_rows,
+                truncate,
+                vertical,
+            } => {
+                let show = proto::ShowString {
+                    input: Some(Box::new(relation_of(input, plan_ids))),
+                    num_rows: *num_rows,
+                    truncate: *truncate,
+                    vertical: *vertical,
+                };
+                relation.rel_type = 
Some(proto::relation::RelType::ShowString(Box::new(show)));
+            }
+
             LogicalPlan::Deduplicate {
                 input,
                 all_columns_as_keys,
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 15ab7cf..0fbb97b 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -696,6 +696,35 @@ def 
test_sql_formats_the_query_with_keyword_arguments_as_pyspark(session):
         assert [tuple(r) for r in run().collect()] == rows
 
 
+def test_show_prints_the_table_of_the_server_as_pyspark(session):
+    import contextlib
+    import io
+
+    df = session.sql("SELECT id, repeat('x', 25) AS s FROM range(2)")
+
+    def shown(**kwargs):
+        out = io.StringIO()
+        with contextlib.redirect_stdout(out):  # Python's print, as PySpark's 
show.
+            df.show(**kwargs)
+        return out.getvalue()
+
+    assert shown() == (
+        "+---+--------------------+\n"
+        "| id|                   s|\n"
+        "+---+--------------------+\n"
+        "|  0|xxxxxxxxxxxxxxxxx...|\n"
+        "|  1|xxxxxxxxxxxxxxxxx...|\n"
+        "+---+--------------------+\n\n")
+    assert shown(n=1, truncate=5) == (
+        "+---+-----+\n| id|    s|\n+---+-----+\n|  0|xx...|\n+---+-----+\nonly 
showing top 1 row\n")
+    assert shown(truncate=False).splitlines()[3] == "|0  |" + "x" * 25 + "|"
+    assert shown(n=1, vertical=True) == (
+        "-RECORD 0-------------------\n"
+        " id  | 0                    \n"
+        " s   | xxxxxxxxxxxxxxxxx... \n"
+        "only showing top 1 row\n")
+
+
 def test_sql_keyword_arguments_are_not_parameters_as_pyspark(session):
     with pytest.raises(AnalysisException) as error:
         session.sql("SELECT :p AS v", p=5).collect()
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 849835e..380fe6d 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -578,6 +578,36 @@ def 
test_add_artifacts_raises_for_paths_that_pyspark_does_not_upload(tmp_path):
         "arg_names": "'pyfile', 'archive' and/or 'file'", "condition": "True 
together"}
 
 
+def test_show_checks_its_arguments_as_pyspark():
+    import inspect
+
+    from pyspark.errors import PySparkTypeError
+    from pyspark.sql import DataFrame
+
+    assert str(inspect.signature(DataFrame.show)) == "(self, /, n=20, 
truncate=True, vertical=False)"
+    df = _offline_session().range(3)
+    # Raised before any request, as by DataFrame._show_string.
+    for kwargs, name, expected, arg_type in (
+        ({"n": True}, "n", "int", "bool"),
+        ({"n": "3"}, "n", "int", "str"),
+        ({"n": None}, "n", "int", "NoneType"),
+        ({"vertical": 1}, "vertical", "bool", "int"),
+        ({"vertical": None}, "vertical", "bool", "NoneType"),
+        ({"truncate": "x"}, "truncate", "bool", "str"),
+    ):
+        with pytest.raises(PySparkTypeError) as error:
+            df.show(**kwargs)
+        assert error.value.getCondition() == "NOT_EXPECTED_TYPE"
+        assert error.value.getMessageParameters() == {
+            "expected_type": expected, "arg_name": name, "arg_type": arg_type}
+    with pytest.raises(TypeError, match=r"^int\(\) argument must be"):
+        df.show(truncate=None)
+    with pytest.raises(TypeError, match=r"^DataFrame\.show\(\) got an 
unexpected keyword argument 'foo'$"):
+        df.show(foo=1)
+    with pytest.raises(TypeError, match=r"^DataFrame\._show_string\(\) got 
multiple values for argument 'n'$"):
+        df._show_string(1, n=2)
+
+
 def test_dropin_module_imports():
     # The thin re-export shims import cleanly.
     import pyspark.sql.dataframe  # noqa: F401


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to