This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 0afbb916 [SPARK-61136][CONNECT][PYTHON] Collect the decimals of a 
negative scale in the Python client as PySpark does
0afbb916 is described below

commit 0afbb916b929436db132495673a4be015ed13cc8
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 09:07:51 2026 +0900

    [SPARK-61136][CONNECT][PYTHON] Collect the decimals of a negative scale in 
the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61136](https://issues.apache.org/jira/browse/SPARK-61136): With 
`spark.sql.legacy.allowNegativeScaleOfDecimal`, a decimal can have a negative 
scale. For example, `CAST(11 AS DECIMAL(1, -1))` is `1E+1`: the unscaled value 
1 at the scale -1.
    
    PySpark collects such a decimal as PyArrow gives it: `Decimal("1E+1")`, of 
which `int()` is 10. The Rust core formats the decimal string of a collected 
value. It now writes a negative scale as the unscaled value times ten to the 
negated scale, in scientific notation, as PyArrow's `Decimal` of it. So the 
client collects `Decimal("1E+1")` too, and `Decimal("-1E+2")` for `-123` cast 
to `DECIMAL(3, -2)`.
    
    ### Why are the changes needed?
    
    For a scale that is not positive, the core formatted the unscaled value 
alone. So the client collected `Decimal("1")` for `1E+1`, a value ten times 
smaller. Apache Spark's `test_negative_decimal` failed on this client: `[0, 1] 
!= [0, 10]`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The decimals of a negative scale are collected with their values, as 
with PySpark 4.3.0. They were wrong before.
    
    ### How was this patch tested?
    
    - A new live test collects decimals cast to `DECIMAL(1, -1)` and 
`DECIMAL(3, -2)`, and checks their values and their strings. It passes with 
official PySpark 4.3.0, and fails without this change.
    - The unit test of the decimal strings of the core checks negative scales.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2196 passed, 8 skipped), and the 
live drop-in and Arrow suites (183 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #328 from HyukjinKwon/SPARK-61136-negative-scale-decimals.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect/src/dataframe.rs | 11 ++++++++++-
 python/tests/test_dropin_e2e.py       | 24 ++++++++++++++++++++++++
 2 files changed, 34 insertions(+), 1 deletion(-)

diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index daac2956..14496d6b 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -2495,7 +2495,13 @@ fn decode_arrow_record_batches(
 /// Extract a value at a specific index from an Arrow array.
 /// Format an unscaled `i128` and scale as a decimal string (e.g. 150, scale 2 
-> "1.50").
 fn i128_to_decimal_string(unscaled: i128, scale: i32) -> String {
-    if scale <= 0 {
+    if scale < 0 {
+        // A negative scale, of spark.sql.legacy.allowNegativeScaleOfDecimal: 
the unscaled value
+        // times ten to the negated scale, in scientific notation, as 
PyArrow's Decimal of it,
+        // which PySpark collects: Decimal("1E+1") of 1 at the scale -1.
+        return format!("{unscaled}E+{}", -i64::from(scale));
+    }
+    if scale == 0 {
         return unscaled.to_string();
     }
     let scale = scale as usize;
@@ -5221,6 +5227,9 @@ mod arrow_value_tests {
         assert_eq!(i128_to_decimal_string(12345, 2), "123.45");
         assert_eq!(i128_to_decimal_string(5, 4), "0.0005");
         assert_eq!(i128_to_decimal_string(-5, 4), "-0.0005");
+        assert_eq!(i128_to_decimal_string(1, -1), "1E+1");
+        assert_eq!(i128_to_decimal_string(0, -1), "0E+1");
+        assert_eq!(i128_to_decimal_string(-123, -2), "-123E+2");
     }
 }
 
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index ba4bcbef..5ea78921 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2448,6 +2448,30 @@ def 
test_types_of_schemas_have_the_attributes_of_pyspark(session):
 
 
 
+def test_decimals_of_a_negative_scale_are_collected_as_pyspark(session):
+    # As PySpark's, of Arrow: with 
spark.sql.legacy.allowNegativeScaleOfDecimal, a decimal of a
+    # negative scale is collected as the Decimal of its unscaled value times 
ten to the negated
+    # scale, in scientific notation: Decimal("1E+1") of 1 at the scale -1.
+    from decimal import Decimal
+
+    from pyspark.sql.types import DecimalType
+
+    key = "spark.sql.legacy.allowNegativeScaleOfDecimal"
+    session.conf.set(key, "true")
+    try:
+        df = session.createDataFrame([(1,), (11,)], ["value"])
+        rows = df.select(F.col("value").cast(DecimalType(1, 
-1)).alias("d")).collect()
+        assert [int(r.d) for r in rows] == [0, 10]
+        assert rows[1].d == Decimal("10") and str(rows[1].d) == "1E+1"
+        df = session.createDataFrame([(-123,), (49,), (51,)], ["value"])
+        rows = df.select(F.col("value").cast(DecimalType(3, 
-2)).alias("d")).collect()
+        assert [str(r.d) for r in rows] == ["-1E+2", "0E+2", "1E+2"]
+    finally:
+        session.conf.unset(key)
+
+
+
+
 def test_decimal_literals_are_of_the_types_of_pyspark(session):
     # The types of official PySpark 4.3.0: at least the precision 10 of 
DecimalType().
     from decimal import Decimal


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to