This is an automated email from the ASF dual-hosted git repository.
dbtsai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new f9fa9ee [SPARK-60695][CONNECT][PYTHON] Keep tiny positive schema
depths bounded
f9fa9ee is described below
commit f9fa9eecb67ccf9b35a11bf47bc8efa1deb6b60e
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 21:48:01 2026 -0800
[SPARK-60695][CONNECT][PYTHON] Keep tiny positive schema depths bounded
### What changes were proposed in this pull request?
Keep a positive schema depth at least one after numeric conversion. With
`Fraction(1, 10**400)` or `Decimal("1e-1000")`, the current conversion
underflows to zero, which the tree renderer interprets as unlimited depth. A
request to print one level therefore prints the complete tree. This follow-up
to #131 clamps that positive conversion result and adds cases to the existing
exact-output tests for both `DataFrame.printSchema` and
`StructType.treeString`, including the no-ExecutePlan as [...]
### Why are the changes needed?
Follow-up to
[SPARK-60695](https://issues.apache.org/jira/browse/SPARK-60695). Positive
limits should remain bounded. Other floating-point normalization semantics are
unchanged; this does not claim exact arbitrary-number parity near every integer
boundary.
### Does this PR introduce any user-facing change?
Tiny positive numeric depth values print only top-level fields, as official
PySpark does, instead of printing all nested fields.
### How was this patch tested?
**Source review is complete. GitHub CI has passed the rebuilt-extension
offline suite, Rust build/tests, WASM, lint, and behavioral e2e. The long
Connect and Pandas parity jobs were still running at merge preparation.**
- Official PySpark 4.2.0 produced the expected exact output for tiny
Fraction/Decimal values through both schema entry points, using an in-memory
schema holder for `DataFrame.printSchema`; this did not use a live server.
- Changed Python test source parses, `git diff --check` passes, and normal
commit hooks passed.
- Genuine local Claude Code 2.1.314 completed two full-patch source review
rounds (`system.ai.claude-fable-5-1`, session
`05614004-d1c5-419b-a553-9c0f07dc888d`). Codex assessed every finding; an
unverified formatting concern was disproved with official source/runtime
evidence, and final Claude review found no actionable source finding. Accepted
exact-number precision, UDT fallback and optional nits remain documented.
- Local validation initially remained blocked because the campaign-locked
Cargo command failed with `FileNotFoundError: cargo`. GitHub CI subsequently
built the native Python extension and passed the full offline Python/Arrow
suite, including both new schema-depth parameters. Rust build/tests, WASM,
rustfmt, behavioral e2e, pipelines parity, documentation and vendored-source
checks also passed. The Clippy workflow is advisory; a green job alone does not
prove warning-free Clippy. Full [...]
A separate read-only source assessment found no regression in the narrowed
change; it does not replace mandatory Claude review or compiled validation. The
earlier newline/documentation nits from #131 were already fixed by #169.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6).
This pull request and its description were written by Isaac.
Closes #288 from dbtsai/codex/postmerge-pr-131-positive-schema-depth.
Authored-by: DB Tsai <[email protected]>
Signed-off-by: DB Tsai <[email protected]>
Co-authored-by: Isaac <[email protected]>
---
crates/pyspark-rs/src/types.rs | 4 +++-
python/tests/test_dropin_offline.py | 3 +++
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index eee200a..6b16bf1 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -11,7 +11,9 @@ pub(crate) fn normalize_tree_depth(depth: &Bound<'_, PyAny>)
-> PyResult<i32> {
if !depth.gt(0)? || depth.ge(i32::MAX)? {
Ok(i32::MAX)
} else {
- Ok(depth.extract::<f64>()?.ceil() as i32)
+ // A positive Fraction or Decimal may underflow to zero as f64. Keep
+ // it at depth one: zero would request an unlimited tree instead.
+ Ok((depth.extract::<f64>()?.ceil() as i32).max(1))
}
}
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 6686fe5..f834631 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -17,6 +17,7 @@ import pickle
import time
from collections import namedtuple
from concurrent.futures import ThreadPoolExecutor
+from fractions import Fraction
from threading import Barrier
import pytest
@@ -11324,6 +11325,8 @@ _PRINT_SCHEMA_FULL = (
(0, _PRINT_SCHEMA_FULL),
(1, _PRINT_SCHEMA_LEVEL1),
(1.5, _PRINT_SCHEMA_LEVEL2),
+ (Fraction(1, 10**400), _PRINT_SCHEMA_LEVEL1),
+ (decimal.Decimal("1e-1000"), _PRINT_SCHEMA_LEVEL1),
(2**100, _PRINT_SCHEMA_FULL),
(float("nan"), _PRINT_SCHEMA_FULL),
],
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]