This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 0afbb916 [SPARK-61136][CONNECT][PYTHON] Collect the decimals of a
negative scale in the Python client as PySpark does
0afbb916 is described below
commit 0afbb916b929436db132495673a4be015ed13cc8
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 09:07:51 2026 +0900
[SPARK-61136][CONNECT][PYTHON] Collect the decimals of a negative scale in
the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61136](https://issues.apache.org/jira/browse/SPARK-61136): With
`spark.sql.legacy.allowNegativeScaleOfDecimal`, a decimal can have a negative
scale. For example, `CAST(11 AS DECIMAL(1, -1))` is `1E+1`: the unscaled value
1 at the scale -1.
PySpark collects such a decimal as PyArrow gives it: `Decimal("1E+1")`, of
which `int()` is 10. The Rust core formats the decimal string of a collected
value. It now writes a negative scale as the unscaled value times ten to the
negated scale, in scientific notation, as PyArrow's `Decimal` of it. So the
client collects `Decimal("1E+1")` too, and `Decimal("-1E+2")` for `-123` cast
to `DECIMAL(3, -2)`.
### Why are the changes needed?
For a scale that is not positive, the core formatted the unscaled value
alone. So the client collected `Decimal("1")` for `1E+1`, a value ten times
smaller. Apache Spark's `test_negative_decimal` failed on this client: `[0, 1]
!= [0, 10]`.
### Does this PR introduce _any_ user-facing change?
Yes. The decimals of a negative scale are collected with their values, as
with PySpark 4.3.0. They were wrong before.
### How was this patch tested?
- A new live test collects decimals cast to `DECIMAL(1, -1)` and
`DECIMAL(3, -2)`, and checks their values and their strings. It passes with
official PySpark 4.3.0, and fails without this change.
- The unit test of the decimal strings of the core checks negative scales.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2196 passed, 8 skipped), and the
live drop-in and Arrow suites (183 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #328 from HyukjinKwon/SPARK-61136-negative-scale-decimals.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect/src/dataframe.rs | 11 ++++++++++-
python/tests/test_dropin_e2e.py | 24 ++++++++++++++++++++++++
2 files changed, 34 insertions(+), 1 deletion(-)
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index daac2956..14496d6b 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -2495,7 +2495,13 @@ fn decode_arrow_record_batches(
/// Extract a value at a specific index from an Arrow array.
/// Format an unscaled `i128` and scale as a decimal string (e.g. 150, scale 2
-> "1.50").
fn i128_to_decimal_string(unscaled: i128, scale: i32) -> String {
- if scale <= 0 {
+ if scale < 0 {
+ // A negative scale, of spark.sql.legacy.allowNegativeScaleOfDecimal:
the unscaled value
+ // times ten to the negated scale, in scientific notation, as
PyArrow's Decimal of it,
+ // which PySpark collects: Decimal("1E+1") of 1 at the scale -1.
+ return format!("{unscaled}E+{}", -i64::from(scale));
+ }
+ if scale == 0 {
return unscaled.to_string();
}
let scale = scale as usize;
@@ -5221,6 +5227,9 @@ mod arrow_value_tests {
assert_eq!(i128_to_decimal_string(12345, 2), "123.45");
assert_eq!(i128_to_decimal_string(5, 4), "0.0005");
assert_eq!(i128_to_decimal_string(-5, 4), "-0.0005");
+ assert_eq!(i128_to_decimal_string(1, -1), "1E+1");
+ assert_eq!(i128_to_decimal_string(0, -1), "0E+1");
+ assert_eq!(i128_to_decimal_string(-123, -2), "-123E+2");
}
}
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index ba4bcbef..5ea78921 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2448,6 +2448,30 @@ def
test_types_of_schemas_have_the_attributes_of_pyspark(session):
+def test_decimals_of_a_negative_scale_are_collected_as_pyspark(session):
+ # As PySpark's, of Arrow: with
spark.sql.legacy.allowNegativeScaleOfDecimal, a decimal of a
+ # negative scale is collected as the Decimal of its unscaled value times
ten to the negated
+ # scale, in scientific notation: Decimal("1E+1") of 1 at the scale -1.
+ from decimal import Decimal
+
+ from pyspark.sql.types import DecimalType
+
+ key = "spark.sql.legacy.allowNegativeScaleOfDecimal"
+ session.conf.set(key, "true")
+ try:
+ df = session.createDataFrame([(1,), (11,)], ["value"])
+ rows = df.select(F.col("value").cast(DecimalType(1,
-1)).alias("d")).collect()
+ assert [int(r.d) for r in rows] == [0, 10]
+ assert rows[1].d == Decimal("10") and str(rows[1].d) == "1E+1"
+ df = session.createDataFrame([(-123,), (49,), (51,)], ["value"])
+ rows = df.select(F.col("value").cast(DecimalType(3,
-2)).alias("d")).collect()
+ assert [str(r.d) for r in rows] == ["-1E+2", "0E+2", "1E+2"]
+ finally:
+ session.conf.unset(key)
+
+
+
+
def test_decimal_literals_are_of_the_types_of_pyspark(session):
# The types of official PySpark 4.3.0: at least the precision 10 of
DecimalType().
from decimal import Decimal
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]