This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new d4650b7 [SPARK-60991][CONNECT][PYTHON] Show a DataFrame as the table
of the server, as PySpark does
d4650b7 is described below
commit d4650b77bb2f73f818a1d1935dfed3f9e3b03be5
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 14:59:03 2026 +0900
[SPARK-60991][CONNECT][PYTHON] Show a DataFrame as the table of the server,
as PySpark does
### What changes were proposed in this pull request?
[SPARK-60991](https://issues.apache.org/jira/browse/SPARK-60991):
`DataFrame.show(n=20, truncate=True, vertical=False)` now sends a `ShowString`
relation with the number of rows, the truncation and the vertical flag, as
PySpark does. It prints the table that the server formats with Python's
`print`. `DataFrame._show_string` returns that table.
- Both check their arguments as PySpark's `_show_string` does. They raise
`NOT_EXPECTED_TYPE` for an `n` that is not an int (a bool included), a
`vertical` that is not a bool, and a `truncate` that `int()` cannot convert. An
explicit None is checked as any other value. Too many, unknown or repeated
arguments raise Python's `TypeError`s.
- The Rust `DataFrame::show` prints the same table, and the new
`DataFrame::show_string` returns it, through a new `LogicalPlan::ShowString`.
### Why are the changes needed?
`df.show()` collected the first rows and printed each with Rust's
`println!`, ignoring `truncate` and `vertical`. For example,
`spark.range(2).show()` printed `[0]` and `[1]` instead of the table:
```
+---+
| id|
+---+
| 0|
| 1|
+---+
```
As it wrote past Python's `sys.stdout`, a notebook or
`contextlib.redirect_stdout` did not get the output.
### Does this PR introduce _any_ user-facing change?
Yes, `show` prints the table of PySpark 4.3.0.
### How was this patch tested?
- Against a local Spark 4.3.0 server, a probe of 18 calls of `show` and
`_show_string` gives the same output as official PySpark 4.3.0. It covers the
default, `n`, a `truncate` of False, an int, a string and a float, `vertical`,
the errors of each argument (None included), and too many, unknown and repeated
arguments.
- A new live test checks the tables of `show` with each argument, through
`contextlib.redirect_stdout`. It also passes with official PySpark 4.3.0, and
fails without this change.
- A new offline test checks the errors of the arguments, which are raised
before any request, and the signature.
- The lib tests and every integration test, e2e included, against the local
server (647 passed), the build of all tests with `wasm-udf`, the offline Python
suites (719 passed, 9 skipped), and the live drop-in and Arrow suites (73
passed) and live cases (10 passed). Clippy reports only one diagnostic in the
changed lines: the `result_large_err` of the `py.detach` closure, which every
action of the Python `DataFrame` has.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #194 from HyukjinKwon/SPARK-60991-show-string.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 138 ++++++++++++++++++++++++++++++++--
crates/spark-connect/src/dataframe.rs | 30 ++++++--
crates/spark-connect/src/plan.rs | 24 ++++++
python/tests/test_dropin_e2e.py | 29 +++++++
python/tests/test_dropin_offline.py | 30 ++++++++
5 files changed, 239 insertions(+), 12 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 6e5be21..944a41f 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -50,6 +50,66 @@ pub struct PyDataFrame {
}
impl PyDataFrame {
+ /// The table of `show` and `_show_string`, for the arguments of
`function`.
+ fn show_table(
+ &self,
+ py: Python<'_>,
+ function: &str,
+ args: &Bound<'_, PyTuple>,
+ kwargs: Option<&Bound<'_, PyDict>>,
+ ) -> PyResult<String> {
+ let [n, truncate, vertical] = bind_arguments(
+ function,
+ args,
+ kwargs,
+ ["n", "truncate", "vertical"],
+ [
+ 20i32.into_pyobject(py)?.into_any(),
+ pyo3::types::PyBool::new(py, true).to_owned().into_any(),
+ pyo3::types::PyBool::new(py, false).to_owned().into_any(),
+ ],
+ )?;
+ let not_expected = |expected: &str, name: &str, value: &Bound<'_,
PyAny>| -> PyErr {
+ let type_name = value
+ .get_type()
+ .name()
+ .map(|name| name.to_string())
+ .unwrap_or_default();
+ crate::column::raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "NOT_EXPECTED_TYPE",
+ &[
+ ("expected_type", expected),
+ ("arg_name", name),
+ ("arg_type", &type_name),
+ ],
+ )
+ };
+ if n.is_instance_of::<pyo3::types::PyBool>() ||
!n.is_instance_of::<pyo3::types::PyInt>() {
+ return Err(not_expected("int", "n", &n));
+ }
+ if !vertical.is_instance_of::<pyo3::types::PyBool>() {
+ return Err(not_expected("bool", "vertical", &vertical));
+ }
+ let truncate =
+ if truncate.is_instance_of::<pyo3::types::PyBool>() &&
truncate.is_truthy()? {
+ 20
+ } else {
+ // As `int(truncate)`, whose ValueError is PySpark's
NOT_EXPECTED_TYPE.
+ match
py.import("builtins")?.getattr("int")?.call1((&truncate,)) {
+ Ok(value) => value.extract::<i32>()?,
+ Err(err) if
err.is_instance_of::<pyo3::exceptions::PyValueError>(py) => {
+ return Err(not_expected("bool", "truncate",
&truncate));
+ }
+ Err(err) => return Err(err),
+ }
+ };
+ let (n, vertical) = (n.extract::<i32>()?, vertical.extract::<bool>()?);
+ py.detach(|| self.dataframe.show_string(n, truncate, vertical))
+ .to_pyerr()
+ }
+
pub fn new(dataframe: DataFrame) -> Self {
PyDataFrame { dataframe }
}
@@ -1221,13 +1281,32 @@ impl PyDataFrame {
py.detach(|| self.dataframe.count()).to_pyerr()
}
- /// Show the first n rows.
- #[pyo3(signature = (n=20, truncate=true, vertical=false))]
- fn show(&self, py: Python<'_>, n: usize, truncate: bool, vertical: bool)
-> PyResult<()> {
- // truncate/vertical are accepted for PySpark signature parity; the
Rust core
- // currently renders with its default formatting.
- let _ = (truncate, vertical);
- py.detach(|| self.dataframe.show(n)).to_pyerr()
+ /// Print the first `n` rows as the server formats them. Mirrors
`DataFrame.show`, which
+ /// prints the table of `_show_string` with Python's `print`.
+ #[pyo3(signature = (*args, **kwargs), text_signature = "($self, n=20,
truncate=True, vertical=False)")]
+ fn show(
+ &self,
+ py: Python<'_>,
+ args: &Bound<'_, PyTuple>,
+ kwargs: Option<&Bound<'_, PyDict>>,
+ ) -> PyResult<()> {
+ let table = self.show_table(py, "DataFrame.show", args, kwargs)?;
+ py.import("builtins")?.call_method1("print", (table,))?;
+ Ok(())
+ }
+
+ /// The table that `show` prints: the first `n` rows (20 by default),
values truncated
+ /// to 20 characters for `truncate=True`, to `int(truncate)` otherwise (0
for none), and
+ /// vertical with a line per value. Mirrors `DataFrame._show_string`, with
its errors,
+ /// for any value given, None included.
+ #[pyo3(name = "_show_string", signature = (*args, **kwargs),
text_signature = "($self, n=20, truncate=True, vertical=False)")]
+ fn show_string(
+ &self,
+ py: Python<'_>,
+ args: &Bound<'_, PyTuple>,
+ kwargs: Option<&Bound<'_, PyDict>>,
+ ) -> PyResult<String> {
+ self.show_table(py, "DataFrame._show_string", args, kwargs)
}
/// Execution info from the most recent action, as a
`pyspark.sql.metrics.ExecutionInfo`
@@ -2440,6 +2519,51 @@ pub(crate) fn py_version(py: Python<'_>) -> String {
.unwrap_or_else(|| "3.11".to_string())
}
+/// The arguments of a call, bound to `names` positionally and by keyword as
Python binds a
+/// signature of those names with `defaults`, raising Python's `TypeError`s of
`function`
+/// for too many, repeated or unexpected arguments. For a method that tells a
value of None
+/// from an argument not given.
+fn bind_arguments<'py, const N: usize>(
+ function: &str,
+ args: &Bound<'py, PyTuple>,
+ kwargs: Option<&Bound<'py, PyDict>>,
+ names: [&str; N],
+ defaults: [Bound<'py, PyAny>; N],
+) -> PyResult<[Bound<'py, PyAny>; N]> {
+ if args.len() > N {
+ return Err(pyo3::exceptions::PyTypeError::new_err(format!(
+ "{function}() takes from 1 to {} positional arguments but {} were
given",
+ N + 1,
+ args.len() + 1
+ )));
+ }
+ let mut values: [Option<Bound<'py, PyAny>>; N] = std::array::from_fn(|_|
None);
+ for (i, value) in args.iter().enumerate() {
+ values[i] = Some(value);
+ }
+ if let Some(kwargs) = kwargs {
+ for (key, value) in kwargs.iter() {
+ let key: String = key.extract()?;
+ let Some(i) = names.iter().position(|name| *name == key) else {
+ return Err(pyo3::exceptions::PyTypeError::new_err(format!(
+ "{function}() got an unexpected keyword argument '{key}'"
+ )));
+ };
+ if values[i].is_some() {
+ return Err(pyo3::exceptions::PyTypeError::new_err(format!(
+ "{function}() got multiple values for argument '{key}'"
+ )));
+ }
+ values[i] = Some(value);
+ }
+ }
+ let mut defaults = defaults.into_iter();
+ Ok(values.map(|value| {
+ let default = defaults.next().expect("a default per name");
+ value.unwrap_or(default)
+ }))
+}
+
/// Resolve a Python schema argument (a `DataType` wrapper or a DDL string)
into a Rust DataType.
pub(crate) fn resolve_datatype(
schema: &Bound<'_, PyAny>,
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 7ddf16e..1004999 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -830,12 +830,32 @@ impl DataFrame {
}
}
- /// Show the first n rows.
- pub fn show(&self, n: usize) -> Result<()> {
- let limited = self.limit(n as i32).collect()?;
- for row in limited {
- println!("{}", row);
+ /// The table that `show` prints, as the server formats it: the first `n`
rows, values
+ /// truncated to `truncate` characters (0 for none), vertical with a line
per value.
+ /// Mirrors `DataFrame._show_string`, which sends a `ShowString` relation.
+ pub fn show_string(&self, n: i32, truncate: i32, vertical: bool) ->
Result<String> {
+ let shown = DataFrame::new(
+ self.session.clone(),
+ LogicalPlan::ShowString {
+ input: Arc::clone(&self.plan),
+ num_rows: n,
+ truncate,
+ vertical,
+ },
+ );
+ match shown.collect()?.first().map(|row| row.values()) {
+ Some([Value::String(table)]) => Ok(table.clone()),
+ _ => Err(SparkError::connect_msg(
+ "ShowString returned no table of one string",
+ )),
}
+ }
+
+ /// Print the first `n` rows as the server formats them, with values
truncated to 20
+ /// characters. Mirrors `DataFrame.show(n)`.
+ pub fn show(&self, n: usize) -> Result<()> {
+ let n = i32::try_from(n).unwrap_or(i32::MAX);
+ println!("{}", self.show_string(n, 20, false)?);
Ok(())
}
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 878a9fd..1fe426b 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -97,6 +97,15 @@ pub enum LogicalPlan {
},
/// Tail: `df.tail(n)` - returns last n rows.
Tail { input: Arc<LogicalPlan>, limit: i32 },
+ /// ShowString: the table of `df.show(...)`, which the server formats as
one string:
+ /// `num_rows` rows, values truncated to `truncate` characters (0 for
none), vertical
+ /// with a line per value.
+ ShowString {
+ input: Arc<LogicalPlan>,
+ num_rows: i32,
+ truncate: i32,
+ vertical: bool,
+ },
/// Deduplicate: `df.distinct()` or `df.dropDuplicates()`.
Deduplicate {
input: Arc<LogicalPlan>,
@@ -706,6 +715,21 @@ impl LogicalPlan {
relation.rel_type =
Some(proto::relation::RelType::Tail(Box::new(tail)));
}
+ LogicalPlan::ShowString {
+ input,
+ num_rows,
+ truncate,
+ vertical,
+ } => {
+ let show = proto::ShowString {
+ input: Some(Box::new(relation_of(input, plan_ids))),
+ num_rows: *num_rows,
+ truncate: *truncate,
+ vertical: *vertical,
+ };
+ relation.rel_type =
Some(proto::relation::RelType::ShowString(Box::new(show)));
+ }
+
LogicalPlan::Deduplicate {
input,
all_columns_as_keys,
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 15ab7cf..0fbb97b 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -696,6 +696,35 @@ def
test_sql_formats_the_query_with_keyword_arguments_as_pyspark(session):
assert [tuple(r) for r in run().collect()] == rows
+def test_show_prints_the_table_of_the_server_as_pyspark(session):
+ import contextlib
+ import io
+
+ df = session.sql("SELECT id, repeat('x', 25) AS s FROM range(2)")
+
+ def shown(**kwargs):
+ out = io.StringIO()
+ with contextlib.redirect_stdout(out): # Python's print, as PySpark's
show.
+ df.show(**kwargs)
+ return out.getvalue()
+
+ assert shown() == (
+ "+---+--------------------+\n"
+ "| id| s|\n"
+ "+---+--------------------+\n"
+ "| 0|xxxxxxxxxxxxxxxxx...|\n"
+ "| 1|xxxxxxxxxxxxxxxxx...|\n"
+ "+---+--------------------+\n\n")
+ assert shown(n=1, truncate=5) == (
+ "+---+-----+\n| id| s|\n+---+-----+\n| 0|xx...|\n+---+-----+\nonly
showing top 1 row\n")
+ assert shown(truncate=False).splitlines()[3] == "|0 |" + "x" * 25 + "|"
+ assert shown(n=1, vertical=True) == (
+ "-RECORD 0-------------------\n"
+ " id | 0 \n"
+ " s | xxxxxxxxxxxxxxxxx... \n"
+ "only showing top 1 row\n")
+
+
def test_sql_keyword_arguments_are_not_parameters_as_pyspark(session):
with pytest.raises(AnalysisException) as error:
session.sql("SELECT :p AS v", p=5).collect()
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 849835e..380fe6d 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -578,6 +578,36 @@ def
test_add_artifacts_raises_for_paths_that_pyspark_does_not_upload(tmp_path):
"arg_names": "'pyfile', 'archive' and/or 'file'", "condition": "True
together"}
+def test_show_checks_its_arguments_as_pyspark():
+ import inspect
+
+ from pyspark.errors import PySparkTypeError
+ from pyspark.sql import DataFrame
+
+ assert str(inspect.signature(DataFrame.show)) == "(self, /, n=20,
truncate=True, vertical=False)"
+ df = _offline_session().range(3)
+ # Raised before any request, as by DataFrame._show_string.
+ for kwargs, name, expected, arg_type in (
+ ({"n": True}, "n", "int", "bool"),
+ ({"n": "3"}, "n", "int", "str"),
+ ({"n": None}, "n", "int", "NoneType"),
+ ({"vertical": 1}, "vertical", "bool", "int"),
+ ({"vertical": None}, "vertical", "bool", "NoneType"),
+ ({"truncate": "x"}, "truncate", "bool", "str"),
+ ):
+ with pytest.raises(PySparkTypeError) as error:
+ df.show(**kwargs)
+ assert error.value.getCondition() == "NOT_EXPECTED_TYPE"
+ assert error.value.getMessageParameters() == {
+ "expected_type": expected, "arg_name": name, "arg_type": arg_type}
+ with pytest.raises(TypeError, match=r"^int\(\) argument must be"):
+ df.show(truncate=None)
+ with pytest.raises(TypeError, match=r"^DataFrame\.show\(\) got an
unexpected keyword argument 'foo'$"):
+ df.show(foo=1)
+ with pytest.raises(TypeError, match=r"^DataFrame\._show_string\(\) got
multiple values for argument 'n'$"):
+ df._show_string(1, n=2)
+
+
def test_dropin_module_imports():
# The thin re-export shims import cleanly.
import pyspark.sql.dataframe # noqa: F401
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]