This is an automated email from the ASF dual-hosted git repository.
dbtsai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 9feb32b [SPARK-60695][CONNECT][TESTS] Assert native schema-printing
stdout
9feb32b is described below
commit 9feb32b3b0a4545f4e7217f8b4b3f23a9b7cdf52
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 22:05:12 2026 -0800
[SPARK-60695][CONNECT][TESTS] Assert native schema-printing stdout
### What changes were proposed in this pull request?
PR #169 aligned Rust `DataFrame::print_schema()` with the reference
trailing blank line, but its exact Rust stdout check was not committed. Add an
offline regression that calls the real method in a child process and asserts
exact output for nested and empty cached schemas. A 30-second child watchdog
bounds cache/RPC regressions. Document the intentional final blank line;
production behavior and public APIs are unchanged.
### Why are the changes needed?
Keep the output contract from
[SPARK-60695](https://issues.apache.org/jira/browse/SPARK-60695) under durable
regression coverage. Python stdout tests exercise a separate print path. The
new test fails when the native `println!` is temporarily changed back to
`print!`.
### How was this patch tested?
- 462 Rust release library tests passed, including the actual-stdout
regression; the negative newline mutation failed as expected and was restored.
- All client integration targets compiled with and without `wasm-udf`;
formatting and whitespace checks passed.
- Fresh native Python extension release build passed; 14 focused Python
schema tests passed.
- Official Spark 4.3 type-parity gate: 70 passed, 21 skipped on an isolated
UTC server with `spark.sql.timeType.enabled=true`. The initial TIME-disabled
failure reproduced with official PySpark before correcting the server
configuration.
- Twelve-class attribute-name comparison found zero unexpected missing
public names; the documented `Column.to_plan` transport gap and inherited
private/dunder differences remain visible. This is not a claim of complete
behavioral parity.
- Genuine local Claude Code reviewed the original landed patch twice and
the complete follow-up patch, assessed all findings/nits, and approved this
final source with no defects.
Draft pending broader repository validation: workspace Clippy exits 101
with 14 existing approximate-constant errors in files byte-identical to the
base (source audit, not a separate baseline Clippy run). The complete official
Connect suite and CI have not been certified by these scoped checks.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6), with independent local Claude Code
review.
This pull request and its description were written by Isaac.
Closes #285 from dbtsai/codex/postmerge-pr-169-schema-stdout-test.
Authored-by: DB Tsai <[email protected]>
Signed-off-by: DB Tsai <[email protected]>
Co-authored-by: Isaac <[email protected]>
---
crates/spark-connect/src/dataframe.rs | 74 +++++++++++++++++++++++++++++++++++
1 file changed, 74 insertions(+)
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 5666c67..0265375 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1780,6 +1780,8 @@ impl DataFrame {
}
/// Print the schema of this DataFrame.
+ ///
+ /// The output ends with a blank line, matching Scala and Python.
pub fn print_schema(&self) -> Result<()> {
let schema = self.schema()?;
println!("{}", schema.tree_string()?);
@@ -3369,6 +3371,78 @@ mod cache_tests {
use super::*;
use prost::Message;
+ #[test]
+ fn print_schema_includes_final_blank_line() {
+ const CHILD: &str = "SPARK_CONNECT_PRINT_SCHEMA_STDOUT_CHILD";
+ const BEGIN: &str = "__PRINT_SCHEMA_BEGIN__\n";
+ const END: &str = "__PRINT_SCHEMA_END__";
+ if std::env::var_os(CHILD).is_some() {
+ // A cache regression must fail promptly instead of waiting on an
RPC.
+ let _watchdog = std::thread::spawn(|| {
+ std::thread::sleep(std::time::Duration::from_secs(30));
+ eprintln!("print_schema stdout child timed out");
+ std::process::exit(1);
+ });
+ let spark = SparkSession::builder()
+ .remote("sc://localhost:1")
+ .get_or_create()
+ .unwrap();
+ let nested = DataType::from_json(&serde_json::json!({
+ "type": "struct",
+ "fields": [{"name": "outer", "type": {
+ "type": "struct",
+ "fields": [{"name": "inner", "type": "integer",
"nullable": false, "metadata": {}}]
+ }, "nullable": true, "metadata": {}}]
+ }))
+ .unwrap();
+ print!("{BEGIN}");
+ for schema in [nested, DataType::Struct { fields: vec![] }] {
+ let df = spark.range(0).unwrap();
+ *df.schema_cache.lock().unwrap() = Some(Arc::new(schema));
+ df.print_schema().unwrap();
+ }
+ println!("{END}");
+ return;
+ }
+
+ // A separate process exposes real stdout without changing global
capture
+ // or sharing it with the other parallel tests in this process.
+ let output =
std::process::Command::new(std::env::current_exe().unwrap())
+ .args([
+ "--exact",
+
"dataframe::cache_tests::print_schema_includes_final_blank_line",
+ "--nocapture",
+ ])
+ .env(CHILD, "1")
+ .output()
+ .unwrap();
+ assert!(
+ output.status.success(),
+ "stdout: {}\nstderr: {}",
+ String::from_utf8_lossy(&output.stdout),
+ String::from_utf8_lossy(&output.stderr),
+ );
+ let stdout = String::from_utf8(output.stdout).unwrap();
+ let printed = stdout
+ .split_once(BEGIN)
+ .expect("child must run the schema print test")
+ .1
+ .split_once(END)
+ .expect("child must finish printing schemas")
+ .0;
+ assert_eq!(
+ printed,
+ concat!(
+ "root\n",
+ " |-- outer: struct (nullable = true)\n",
+ " | |-- inner: integer (nullable = false)\n",
+ "\n",
+ "root\n",
+ "\n",
+ ),
+ );
+ }
+
#[test]
fn schema_cache_shares_exact_clones_but_not_changed_plans_or_new_sources()
{
let spark = SparkSession::builder()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]