This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new c9d48d3 [SPARK-60695][CONNECT] Print DataFrame schemas without
executing queries
c9d48d3 is described below
commit c9d48d39e59cb07cfa409daf8efafa4e86f2ca8d
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 08:06:20 2026 +0900
[SPARK-60695][CONNECT] Print DataFrame schemas without executing queries
### What changes were proposed in this pull request?
Fix `DataFrame.printSchema()` calling `show(0)`, which executes a zero-row
query and emits no schema. Resolve schema metadata instead, render its tree at
the requested depth, and call Python's `print` so `sys.stdout` capture works.
Preserve reference behavior for default, nonpositive, false, fractional and
invalid depth arguments.
Match reference tree formatting for scalar type names, nested
struct/array/map children and nullability, field-name control-character
escaping, and unlimited nonpositive depths. UDTs with Python class metadata use
the lowercase class name, such as `vectorudt`; unavailable class metadata
retains the generic `udt` fallback. Field names are escaped in one pass through
a dedicated helper, tree lines are written directly to the output buffer, and
scalar leaves avoid unused prefix allocatio [...]
Reuse the existing mock Connect server with a module-scoped process and
per-test resets. This also changes the existing schema-cache fixtures because
both test groups share mock_connect: one process avoids repeated Python/gRPC
startup, while a function-scoped reset clears schema, counters, failures and
blocking state. Each test creates fresh DataFrames, whose schema caches belong
to the individual frame, preserving test isolation. Representative depth cases
check both Python entry poi [...]
### Why are the changes needed?
[SPARK-60695](https://issues.apache.org/jira/browse/SPARK-60695). Printing
a schema should emit schema text without sending ExecutePlan requests.
### Does this PR introduce _any_ user-facing change?
Yes. `printSchema()` now prints the schema through Python stdout, honors
nesting depth, and performs schema analysis instead of query execution. Schema
tree text uses reference formatting for nested collection types, scalar type
names, and Python UDT class names when available. Repeated calls reuse the
schema cache already merged into master.
### How was this patch tested?
Local LC-review validation at `f807ba5` (master base `617fa148`) on Python
3.12 with an optimized release wheel:
- **331 offline Python tests passed**, covering both schema-tree entry
points, exact stdout, representative depth arguments, RPC-error propagation,
schema-cache reuse, UDT names and zero ExecutePlan requests.
- **398 Rust library tests and 13 type-behavior tests passed**. All Rust
client test targets compiled with and without `wasm-udf`.
- Reference PySpark 4.2.0 and the rebuilt candidate matched five
representative exact tree depths, the default signature, explicit invalid
tree-depth arguments and the escaping golden test. Reference printSchema also
matched the corresponding depth outputs and its falsey-default behavior.
- Release wheel build, Rust formatting, whitespace checks, and normal
commit hooks passed.
- Advisory workspace Clippy still fails on existing code. Workspace and
focused binding checks were run on this commit; an exact-source audit placed
all emitted errors and located warnings on unchanged base lines, with none on
this PR's changes. This audit compares source text and does not claim a
separate baseline build.
No JVM/live Spark server was used. Mock gRPC servers exercise the real
client transport in a separate process.
Historical measurements from the preceding implementation, **not rerun on
the rebased review-fix head**:
- An unchanged frozen harness and validators passed **195/195 samples**,
with five fresh-process samples per case/client across five printSchema cases
and eight metadata/show/API/error guards.
- Both measurement wheels included PR120
`f87973b1e02fe500e3c4266a8ad3f353244994af`. Repeated depth-256 printSchema
measured **2.607229 ms → 0.001406 ms** (Python reference **0.005496 ms**);
timed ExecutePlan calls **20 → 0**. First-call depth-256 measured **3.174625 →
2.921541 ms**. All eight unrelated guard sample ranges overlapped; median
changes ranged from -14.6% to +8.0%, without a statistical equivalence claim.
The frozen inspection validator did not validate stdout, so those benchmark
passes supplement the independent exact-output tests. Raw samples and setup
provenance remain in the campaign artifacts. PR120 is now merged into the
master base; this PR has no unmerged dependency.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6).
This pull request and its description were written by Isaac.
Closes #131 from dbtsai/codex/spark-60695-print-schema.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 22 +++-
crates/pyspark-rs/src/types.rs | 22 +++-
crates/spark-connect/src/dataframe.rs | 2 +-
crates/spark-connect/src/types.rs | 130 ++++++++++++++-----
crates/spark-connect/tests/types_behavior.rs | 84 +++++++++++-
python/tests/_mock_connect_server.py | 28 ++++
python/tests/test_dropin_offline.py | 185 ++++++++++++++++++++++++++-
7 files changed, 429 insertions(+), 44 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 8df1b83..f491d23 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1404,13 +1404,23 @@ impl PyDataFrame {
self.dataframe.is_empty().to_pyerr()
}
- /// Print the schema of the DataFrame. `level` (max nesting depth to
print) is
- /// accepted for signature parity with reference pyspark; the connect core
does not
- /// yet thread a depth into the schema tree-string, so it is currently
ignored.
+ /// Print the schema through Python stdout, honoring the maximum nesting
depth.
#[pyo3(signature = (level=None))]
- #[allow(unused_variables)]
- fn printSchema(&self, level: Option<i32>) -> PyResult<()> {
- self.dataframe.show(0).to_pyerr()
+ fn printSchema(&self, py: Python<'_>, level: Option<&Bound<'_, PyAny>>) ->
PyResult<()> {
+ // Reference's `if level:` selects the default for false values before
+ // resolving the schema; positive fractional depths behave as
ceil(level).
+ let level = match level {
+ Some(level) if level.is_truthy()? => Some(level),
+ _ => None,
+ };
+ let schema = py.detach(|| self.dataframe.schema().to_pyerr())?;
+ let depth = level
+ .map(crate::types::normalize_tree_depth)
+ .transpose()?
+ .unwrap_or(i32::MAX);
+ let tree = schema.tree_string_with_depth(depth).to_pyerr()?;
+ py.import("builtins")?.getattr("print")?.call1((tree,))?;
+ Ok(())
}
/// Get the schema as a list of (name, type) tuples.
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 4eab163..24e147c 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -4,6 +4,17 @@ use pyo3::prelude::*;
use spark_connect::types::{DataType, StructField};
use std::collections::BTreeMap;
+/// Normalize the depth used by both Python schema-tree entry points.
+pub(crate) fn normalize_tree_depth(depth: &Bound<'_, PyAny>) -> PyResult<i32> {
+ // Reference compares > 0 at each level, so positive fractional depths
+ // behave as ceil(depth). Nonpositive depths and NaN print the full tree.
+ if !depth.gt(0)? || depth.ge(i32::MAX)? {
+ Ok(i32::MAX)
+ } else {
+ Ok(depth.extract::<f64>()?.ceil() as i32)
+ }
+}
+
/// Make a DataType picklable so it round-trips to an official-pyspark UDF
worker:
/// pickle reconstructs it via
`pyspark.sql.types._parse_datatype_json_string(json)`
/// (session-free, present in both our shim and the worker's official pyspark).
@@ -1765,12 +1776,17 @@ impl PyStructType {
/// Tree string. Mirrors `StructType.treeString(maxDepth)`.
#[allow(non_snake_case)]
- #[pyo3(name = "treeString", signature = (maxDepth = 2147483647))]
- fn __st_tree_string(&self, maxDepth: i32) -> PyResult<String> {
+ #[pyo3(
+ name = "treeString",
+ signature = (maxDepth = Python::attach(|py|
pyo3::types::PyInt::new(py, i32::MAX).into_any().unbind())),
+ text_signature = "($self, maxDepth=2147483647)"
+ )]
+ fn __st_tree_string(&self, py: Python<'_>, maxDepth: Py<PyAny>) ->
PyResult<String> {
+ let depth = normalize_tree_depth(maxDepth.bind(py))?;
DataType::Struct {
fields: self.fields.clone(),
}
- .tree_string_with_depth(maxDepth)
+ .tree_string_with_depth(depth)
.map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError,
_>(e.to_string()))
}
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 8aa9db0..5edae1a 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1491,7 +1491,7 @@ impl DataFrame {
/// Print the schema of this DataFrame.
pub fn print_schema(&self) -> Result<()> {
let schema = self.schema()?;
- println!("{}", schema);
+ print!("{}", schema.tree_string()?);
Ok(())
}
diff --git a/crates/spark-connect/src/types.rs
b/crates/spark-connect/src/types.rs
index fd2894b..c357273 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -4,7 +4,7 @@
//! and the Spark Connect protobuf representation.
use std::collections::BTreeMap;
-use std::fmt;
+use std::fmt::{self, Write};
use std::hash::{Hash, Hasher};
use spark_connect_core::error::{Result, SparkError};
@@ -943,14 +943,19 @@ impl DataType {
self.tree_string_with_depth(i32::MAX)
}
- /// Like [`DataType::tree_string`], but stops recursing into nested structs
- /// once `max_depth` nesting levels have been printed (top-level fields are
- /// depth 1). Mirrors `StructType.treeString(maxDepth)`.
+ /// Like [`DataType::tree_string`], but limits nesting through structs,
+ /// arrays, and maps to `max_depth` levels. Top-level fields are depth 1;
+ /// zero or negative values print the full tree.
+ /// Mirrors `StructType.treeString(maxDepth)`.
pub fn tree_string_with_depth(&self, max_depth: i32) -> Result<String> {
match self {
DataType::Struct { .. } => {
let mut out = String::from("root\n");
- self.append_tree(&mut out, " |", 1, max_depth);
+ self.append_tree(
+ &mut out,
+ " |",
+ if max_depth > 0 { max_depth } else { i32::MAX },
+ );
Ok(out)
}
_ => Err(SparkError::value(
@@ -960,33 +965,85 @@ impl DataType {
}
}
- fn append_tree(&self, out: &mut String, prefix: &str, depth: i32,
max_depth: i32) {
- if let DataType::Struct { fields } = self {
- for f in fields {
- // A nested struct is rendered as the bare type name
("struct") and its
- // fields are shown by recursing below, so `max_depth` can
truncate them.
- // Using `simple_string()` here would inline the children (e.g.
- // "struct<inner:int>"), leaking them past `max_depth`.
Non-struct types
- // have no depth-controlled children, so their simple string
is used.
- let is_struct = matches!(f.data_type, DataType::Struct { .. });
- let type_repr = if is_struct {
- f.data_type.type_name()
- } else {
- f.data_type.simple_string()
- };
- out.push_str(&format!(
- "{}-- {}: {} (nullable = {})\n",
- prefix, f.name, type_repr, f.nullable
- ));
- if is_struct && depth < max_depth {
- f.data_type.append_tree(
+ fn tree_type_name(&self) -> String {
+ match self {
+ DataType::Decimal { .. }
+ | DataType::Char { .. }
+ | DataType::Varchar { .. }
+ | DataType::Time { .. }
+ | DataType::YearMonthInterval { .. }
+ | DataType::DayTimeInterval { .. } => self.simple_string(),
+ DataType::Udt {
+ python_class: Some(python_class),
+ ..
+ } => python_class
+ .rsplit('.')
+ .next()
+ .filter(|class_name| !class_name.is_empty())
+ // UserDefinedType.typeName() preserves the entire class name,
+ // unlike DataType.typeName(), which strips the "Type" suffix.
+ .map(str::to_lowercase)
+ .unwrap_or_else(|| self.type_name()),
+ _ => self.type_name(),
+ }
+ }
+
+ fn append_tree(&self, out: &mut String, prefix: &str, remaining: i32) {
+ if remaining <= 0 {
+ return;
+ }
+ match self {
+ DataType::Struct { fields } => {
+ let child_prefix = format!("{prefix} |");
+ for f in fields {
+ out.push_str(prefix);
+ out.push_str("-- ");
+ push_escaped_name(out, &f.name);
+ writeln!(
out,
- &format!("{} |", prefix.trim_end_matches('|')),
- depth + 1,
- max_depth,
- );
+ ": {} (nullable = {})",
+ f.data_type.tree_type_name(),
+ f.nullable
+ )
+ .expect("writing to a String cannot fail");
+ f.data_type.append_tree(out, &child_prefix, remaining - 1);
}
}
+ DataType::Array {
+ element_type,
+ contains_null,
+ } => {
+ let child_prefix = format!("{prefix} |");
+ writeln!(
+ out,
+ "{}-- element: {} (containsNull = {})",
+ prefix,
+ element_type.tree_type_name(),
+ contains_null
+ )
+ .expect("writing to a String cannot fail");
+ element_type.append_tree(out, &child_prefix, remaining - 1);
+ }
+ DataType::Map {
+ key_type,
+ value_type,
+ value_contains_null,
+ } => {
+ let child_prefix = format!("{prefix} |");
+ writeln!(out, "{}-- key: {}", prefix,
key_type.tree_type_name())
+ .expect("writing to a String cannot fail");
+ key_type.append_tree(out, &child_prefix, remaining - 1);
+ writeln!(
+ out,
+ "{}-- value: {} (valueContainsNull = {})",
+ prefix,
+ value_type.tree_type_name(),
+ value_contains_null
+ )
+ .expect("writing to a String cannot fail");
+ value_type.append_tree(out, &child_prefix, remaining - 1);
+ }
+ _ => {}
}
}
@@ -1063,6 +1120,21 @@ impl DataType {
}
}
+fn push_escaped_name(out: &mut String, name: &str) {
+ for ch in name.chars() {
+ match ch {
+ '\n' => out.push_str("\\n"),
+ '\r' => out.push_str("\\r"),
+ '\t' => out.push_str("\\t"),
+ '\x0c' => out.push_str("\\f"),
+ '\x08' => out.push_str("\\b"),
+ '\x0b' => out.push_str("\\v"),
+ '\x07' => out.push_str("\\a"),
+ _ => out.push(ch),
+ }
+ }
+}
+
impl fmt::Display for DataType {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.simple_string())
diff --git a/crates/spark-connect/tests/types_behavior.rs
b/crates/spark-connect/tests/types_behavior.rs
index 8f32b79..56ee941 100644
--- a/crates/spark-connect/tests/types_behavior.rs
+++ b/crates/spark-connect/tests/types_behavior.rs
@@ -280,7 +280,7 @@ fn struct_to_ddl_tree_and_nullable() {
assert_eq!(st.to_ddl().unwrap(), "a int,b string NOT NULL");
let tree = st.tree_string().unwrap();
assert!(tree.starts_with("root\n"));
- assert!(tree.contains("a: int (nullable = true)"));
+ assert!(tree.contains("a: integer (nullable = true)"));
assert!(tree.contains("b: string (nullable = false)"));
// toNullable makes every field nullable.
if let DataType::Struct { fields } = st.to_nullable() {
@@ -314,13 +314,93 @@ fn tree_string_with_depth_truncates_nested_structs() {
// Full depth shows the nested field, rendering the struct as the bare
type name.
let full = nested.tree_string().unwrap();
assert!(full.contains("outer: struct (nullable = true)"));
- assert!(full.contains("inner: int (nullable = true)"));
+ assert!(full.contains("inner: integer (nullable = true)"));
// max_depth = 1 prints only the top level; the nested field is omitted.
let shallow = nested.tree_string_with_depth(1).unwrap();
assert!(shallow.contains("outer: struct (nullable = true)"));
assert!(!shallow.contains("inner"));
}
+#[test]
+fn tree_string_uses_python_udt_class_names() {
+ use spark_connect::types::StructField;
+ let vector_udt = || DataType::Udt {
+ type_str: "udt".to_string(),
+ jvm_class: None,
+ python_class: Some("pyspark.ml.linalg.VectorUDT".to_string()),
+ serialized_python_class: None,
+ sql_type: Some(Box::new(DataType::Integer)),
+ };
+ let schema = DataType::Struct {
+ fields: vec![
+ StructField {
+ name: "features".to_string(),
+ data_type: vector_udt(),
+ nullable: true,
+ metadata: Default::default(),
+ },
+ StructField {
+ name: "array".to_string(),
+ data_type: DataType::Array {
+ element_type: Box::new(vector_udt()),
+ contains_null: false,
+ },
+ nullable: false,
+ metadata: Default::default(),
+ },
+ StructField {
+ name: "map".to_string(),
+ data_type: DataType::Map {
+ key_type: Box::new(DataType::String {
+ collation: String::new(),
+ }),
+ value_type: Box::new(vector_udt()),
+ value_contains_null: true,
+ },
+ nullable: true,
+ metadata: Default::default(),
+ },
+ ],
+ };
+
+ assert_eq!(
+ schema.tree_string().unwrap(),
+ concat!(
+ "root\n",
+ " |-- features: vectorudt (nullable = true)\n",
+ " |-- array: array (nullable = false)\n",
+ " | |-- element: vectorudt (containsNull = false)\n",
+ " |-- map: map (nullable = true)\n",
+ " | |-- key: string\n",
+ " | |-- value: vectorudt (valueContainsNull = true)\n",
+ )
+ );
+}
+
+#[test]
+fn tree_string_keeps_generic_udt_name_without_python_class() {
+ use spark_connect::types::StructField;
+ let schema = DataType::Struct {
+ fields: vec![StructField {
+ name: "features".to_string(),
+ data_type: DataType::Udt {
+ type_str: "udt".to_string(),
+ jvm_class:
Some("org.apache.spark.ml.linalg.VectorUDT".to_string()),
+ python_class: None,
+ serialized_python_class: None,
+ sql_type: Some(Box::new(DataType::Integer)),
+ },
+ nullable: true,
+ metadata: Default::default(),
+ }],
+ };
+
+ assert_eq!(
+ schema.tree_string().unwrap(),
+ "root\n |-- features: udt (nullable = true)\n",
+ );
+}
+
#[test]
fn from_json_str_roundtrips() {
use spark_connect::types::StructField;
diff --git a/python/tests/_mock_connect_server.py
b/python/tests/_mock_connect_server.py
index ae573fa..b05469e 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -96,6 +96,7 @@ def serve(connection):
# they wait for a response.
class Service:
calls = 0
+ execute_calls = 0
data_type = struct_schema()
failure = None
block_response = False
@@ -104,6 +105,24 @@ def serve(connection):
# Each AnalyzePlanRequest, recorded.
analyze_requests = []
+ def reset(self):
+ # Back to the state of a new server: the tests of a module share
one.
+ self.calls = 0
+ self.execute_calls = 0
+ self.data_type = struct_schema()
+ self.failure = None
+ self.block_response = False
+ self.response_gate.clear()
+ self.analyze_requests = []
+ self.execute_requests = []
+ self.execute_payload = b""
+ self.execute_failure = None
+ self.execute_error = None
+ self.error_details = b""
+ self.fetched_error_ids = []
+ self.confs = {}
+ self.read_confs = []
+
def analyze(self, request, context):
self.calls += 1
self.analyze_requests.append(request)
@@ -143,6 +162,7 @@ def serve(connection):
read_confs = []
def execute(self, request, context):
+ self.execute_calls += 1
self.execute_requests.append(request)
if self.execute_failure:
context.abort(grpc.StatusCode.INVALID_ARGUMENT,
self.execute_failure)
@@ -226,6 +246,9 @@ def serve(connection):
elif command[0] == "release":
service.response_gate.set()
connection.send(None)
+ elif command[0] == "reset":
+ service.reset()
+ connection.send(None)
finally:
server.stop(0).wait()
@@ -249,6 +272,11 @@ class RemoteService:
assert self.connection.poll(10), "Mock server did not respond"
self.connection.recv()
+ def reset(self):
+ self.connection.send(("reset",))
+ assert self.connection.poll(10), "Mock server did not respond"
+ self.connection.recv()
+
@contextmanager
def mock_connect_session():
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 0ff1437..a4214c6 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -780,7 +780,7 @@ def test_structtype_object_model():
assert st.typeName() == "struct"
assert st.fieldNames() == ["a", "b"]
assert st.toDDL() == "a int,b string NOT NULL"
- assert "root" in st.treeString() and "a: int (nullable = true)" in
st.treeString()
+ assert "root" in st.treeString() and "a: integer (nullable = true)" in
st.treeString()
# toNullable makes every field nullable.
assert st.toNullable().simpleString() == "struct<a:int,b:string>"
# json round-trips through fromJson.
@@ -3465,13 +3465,20 @@ def
test_create_dataframe_cannot_type_an_all_null_field_as_pyspark(data):
# ---------------------------------------------------------------------------
schema cache
[email protected]
-def mock_connect():
[email protected](scope="module")
+def _shared_mock_connect():
pytest.importorskip("grpc")
with mock_connect_session() as session:
yield session
[email protected]
+def mock_connect(_shared_mock_connect):
+ spark, service = _shared_mock_connect
+ service.reset()
+ return spark, service
+
+
@pytest.mark.parametrize("first", ["schema", "columns", "dtypes"])
def test_schema_cache_one_analysis_shared_by_metadata_accessors(mock_connect,
first):
spark, service = mock_connect
@@ -4823,3 +4830,175 @@ def
test_exact_string_subclass_uses_generic_conversion():
assert _local_create_dataframe([Row(value=IndexString("x")),
Row(value=5)]).isLocal()
with pytest.raises(PySparkRuntimeError):
_local_create_dataframe([Row(value=IndexString("x")), Row(value="y")])
+
+
+# ---------------------------------------------------------------------------
printSchema
+
+
+def _print_schema_field(name, data_type, nullable=True):
+ return (
+ message_field(1, name)
+ + message_field(2, data_type)
+ + (b"\x18\x01" if nullable else b"\x18\x00")
+ )
+
+
+def _print_schema_type():
+ def struct(*fields):
+ return message_field(21, b"".join(message_field(1, field) for field in
fields))
+
+ integer = message_field(6, b"")
+ long = message_field(7, b"")
+ string = message_field(11, b"")
+ array = message_field(
+ 20, message_field(1, struct(_print_schema_field("item", long))) +
b"\x10\x00"
+ )
+ mapping = message_field(
+ 22,
+ message_field(1, string)
+ + message_field(2, struct(_print_schema_field("value", integer)))
+ + b"\x18\x01",
+ )
+ return struct(
+ _print_schema_field("outer", struct(_print_schema_field("inner",
integer, False))),
+ _print_schema_field("items", array, False),
+ _print_schema_field("lookup", mapping),
+ )
+
+
+_PRINT_SCHEMA_LEVEL1 = (
+ "root\n"
+ " |-- outer: struct (nullable = true)\n"
+ " |-- items: array (nullable = false)\n"
+ " |-- lookup: map (nullable = true)\n\n"
+)
+_PRINT_SCHEMA_LEVEL2 = (
+ "root\n"
+ " |-- outer: struct (nullable = true)\n"
+ " | |-- inner: integer (nullable = false)\n"
+ " |-- items: array (nullable = false)\n"
+ " | |-- element: struct (containsNull = false)\n"
+ " |-- lookup: map (nullable = true)\n"
+ " | |-- key: string\n"
+ " | |-- value: struct (valueContainsNull = true)\n\n"
+)
+_PRINT_SCHEMA_FULL = (
+ "root\n"
+ " |-- outer: struct (nullable = true)\n"
+ " | |-- inner: integer (nullable = false)\n"
+ " |-- items: array (nullable = false)\n"
+ " | |-- element: struct (containsNull = false)\n"
+ " | | |-- item: long (nullable = true)\n"
+ " |-- lookup: map (nullable = true)\n"
+ " | |-- key: string\n"
+ " | |-- value: struct (valueContainsNull = true)\n"
+ " | | |-- value: integer (nullable = true)\n\n"
+)
+
+
[email protected](
+ "level, expected",
+ [
+ (0, _PRINT_SCHEMA_FULL),
+ (1, _PRINT_SCHEMA_LEVEL1),
+ (1.5, _PRINT_SCHEMA_LEVEL2),
+ (2**100, _PRINT_SCHEMA_FULL),
+ (float("nan"), _PRINT_SCHEMA_FULL),
+ ],
+)
+def test_print_schema_exact_python_stdout(mock_connect, capsys, level,
expected):
+ spark, service = mock_connect
+ service.data_type = _print_schema_type()
+ df = spark.range(10)
+ assert df.printSchema(level) is None
+ captured = capsys.readouterr()
+ assert captured.out == expected
+ assert captured.err == ""
+ assert df.schema.treeString(maxDepth=level) == expected[:-1]
+ assert service.execute_calls == 0
+
+
+def test_print_schema_default_reuses_cached_schema(mock_connect, capsys):
+ spark, service = mock_connect
+ service.data_type = _print_schema_type()
+ df = spark.range(10)
+ for _ in range(2):
+ assert df.printSchema() is None
+ assert capsys.readouterr().out == _PRINT_SCHEMA_FULL
+ assert service.calls == 1
+ assert service.execute_calls == 0
+
+
[email protected]("level", [None, []])
+def test_print_schema_falsey_level_uses_default(mock_connect, capsys, level):
+ spark, service = mock_connect
+ service.data_type = _print_schema_type()
+ df = spark.range(10)
+ assert df.printSchema(level) is None
+ assert capsys.readouterr().out == _PRINT_SCHEMA_FULL
+ with pytest.raises(TypeError):
+ df.schema.treeString(level)
+ assert service.execute_calls == 0
+
+
+def test_schema_tree_invalid_depth(mock_connect, capsys):
+ spark, service = mock_connect
+ service.data_type = _print_schema_type()
+ df = spark.range(10)
+ with pytest.raises(TypeError):
+ df.printSchema("two")
+ with pytest.raises(TypeError):
+ df.schema.treeString("two")
+ assert capsys.readouterr().out == ""
+ assert service.execute_calls == 0
+
+
+def test_print_schema_propagates_schema_error(mock_connect, capsys):
+ spark, service = mock_connect
+ service.failure = "rpc"
+ with pytest.raises(Exception, match="schema unavailable"):
+ spark.range(10).printSchema()
+ assert capsys.readouterr().out == ""
+ assert service.execute_calls == 0
+
+
+def test_schema_tree_type_names_and_escaped_field_names():
+ schema = T.StructType(
+ [
+ T.StructField("é🌱\\n\n\r\t\f\b\v\a", T.LongType(), False),
+ T.StructField("decimal", T.DecimalType(10, 2)),
+ T.StructField("small", T.ByteType()),
+ T.StructField("short", T.ShortType()),
+ ]
+ )
+ expected = (
+ "root\n"
+ " |-- é🌱\\n\\n\\r\\t\\f\\b\\v\\a: long (nullable = false)\n"
+ " |-- decimal: decimal(10,2) (nullable = true)\n"
+ " |-- small: byte (nullable = true)\n"
+ " |-- short: short (nullable = true)\n"
+ )
+ assert schema.treeString() == expected
+ assert schema.treeString(0) == expected
+ assert schema.treeString(-1) == expected
+
+
[email protected](
+ "python_class, expected",
+ [
+ ("pyspark.ml.linalg.VectorUDT", "vectorudt"),
+ (None, "udt"),
+ ],
+)
+def test_print_schema_udt_class_name(mock_connect, capsys, python_class,
expected):
+ spark, service = mock_connect
+ udt = message_field(1, "udt")
+ if python_class is not None:
+ udt += message_field(3, python_class)
+ service.data_type = message_field(
+ 21, message_field(1, _print_schema_field("features", message_field(23,
udt)))
+ )
+ assert spark.range(10).printSchema() is None
+ assert capsys.readouterr().out == f"root\n |-- features: {expected}
(nullable = true)\n\n"
+ assert service.calls == 1
+ assert service.execute_calls == 0
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]