This is an automated email from the ASF dual-hosted git repository.

dbtsai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new f9fa9ee  [SPARK-60695][CONNECT][PYTHON] Keep tiny positive schema 
depths bounded
f9fa9ee is described below

commit f9fa9eecb67ccf9b35a11bf47bc8efa1deb6b60e
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 21:48:01 2026 -0800

    [SPARK-60695][CONNECT][PYTHON] Keep tiny positive schema depths bounded
    
    ### What changes were proposed in this pull request?
    
    Keep a positive schema depth at least one after numeric conversion. With 
`Fraction(1, 10**400)` or `Decimal("1e-1000")`, the current conversion 
underflows to zero, which the tree renderer interprets as unlimited depth. A 
request to print one level therefore prints the complete tree. This follow-up 
to #131 clamps that positive conversion result and adds cases to the existing 
exact-output tests for both `DataFrame.printSchema` and 
`StructType.treeString`, including the no-ExecutePlan as [...]
    
    ### Why are the changes needed?
    
    Follow-up to 
[SPARK-60695](https://issues.apache.org/jira/browse/SPARK-60695). Positive 
limits should remain bounded. Other floating-point normalization semantics are 
unchanged; this does not claim exact arbitrary-number parity near every integer 
boundary.
    
    ### Does this PR introduce any user-facing change?
    
    Tiny positive numeric depth values print only top-level fields, as official 
PySpark does, instead of printing all nested fields.
    
    ### How was this patch tested?
    
    **Source review is complete. GitHub CI has passed the rebuilt-extension 
offline suite, Rust build/tests, WASM, lint, and behavioral e2e. The long 
Connect and Pandas parity jobs were still running at merge preparation.**
    
    - Official PySpark 4.2.0 produced the expected exact output for tiny 
Fraction/Decimal values through both schema entry points, using an in-memory 
schema holder for `DataFrame.printSchema`; this did not use a live server.
    - Changed Python test source parses, `git diff --check` passes, and normal 
commit hooks passed.
    - Genuine local Claude Code 2.1.314 completed two full-patch source review 
rounds (`system.ai.claude-fable-5-1`, session 
`05614004-d1c5-419b-a553-9c0f07dc888d`). Codex assessed every finding; an 
unverified formatting concern was disproved with official source/runtime 
evidence, and final Claude review found no actionable source finding. Accepted 
exact-number precision, UDT fallback and optional nits remain documented.
    - Local validation initially remained blocked because the campaign-locked 
Cargo command failed with `FileNotFoundError: cargo`. GitHub CI subsequently 
built the native Python extension and passed the full offline Python/Arrow 
suite, including both new schema-depth parameters. Rust build/tests, WASM, 
rustfmt, behavioral e2e, pipelines parity, documentation and vendored-source 
checks also passed. The Clippy workflow is advisory; a green job alone does not 
prove warning-free Clippy. Full [...]
    
    A separate read-only source assessment found no regression in the narrowed 
change; it does not replace mandatory Claude review or compiled validation. The 
earlier newline/documentation nits from #131 were already fixed by #169.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6).
    
    This pull request and its description were written by Isaac.
    
    Closes #288 from dbtsai/codex/postmerge-pr-131-positive-schema-depth.
    
    Authored-by: DB Tsai <[email protected]>
    Signed-off-by: DB Tsai <[email protected]>
    Co-authored-by: Isaac <[email protected]>
---
 crates/pyspark-rs/src/types.rs      | 4 +++-
 python/tests/test_dropin_offline.py | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index eee200a..6b16bf1 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -11,7 +11,9 @@ pub(crate) fn normalize_tree_depth(depth: &Bound<'_, PyAny>) 
-> PyResult<i32> {
     if !depth.gt(0)? || depth.ge(i32::MAX)? {
         Ok(i32::MAX)
     } else {
-        Ok(depth.extract::<f64>()?.ceil() as i32)
+        // A positive Fraction or Decimal may underflow to zero as f64. Keep
+        // it at depth one: zero would request an unlimited tree instead.
+        Ok((depth.extract::<f64>()?.ceil() as i32).max(1))
     }
 }
 
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 6686fe5..f834631 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -17,6 +17,7 @@ import pickle
 import time
 from collections import namedtuple
 from concurrent.futures import ThreadPoolExecutor
+from fractions import Fraction
 from threading import Barrier
 
 import pytest
@@ -11324,6 +11325,8 @@ _PRINT_SCHEMA_FULL = (
         (0, _PRINT_SCHEMA_FULL),
         (1, _PRINT_SCHEMA_LEVEL1),
         (1.5, _PRINT_SCHEMA_LEVEL2),
+        (Fraction(1, 10**400), _PRINT_SCHEMA_LEVEL1),
+        (decimal.Decimal("1e-1000"), _PRINT_SCHEMA_LEVEL1),
         (2**100, _PRINT_SCHEMA_FULL),
         (float("nan"), _PRINT_SCHEMA_FULL),
     ],


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to