This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 62080cb  [SPARK-60696][CONNECT] Convert Decimal inputs exactly in 
either notation, and fail pandas Decimals that do not fit as PySpark does
62080cb is described below

commit 62080cbe5ac168a112dea656134c580b9ae4df03
Author: DB Tsai <[email protected]>
AuthorDate: Sat Nov 7 09:25:14 2026 +0900

    [SPARK-60696][CONNECT] Convert Decimal inputs exactly in either notation, 
and fail pandas Decimals that do not fit as PySpark does
    
    ### What changes were proposed in this pull request?
    
    Decimal inputs of `createDataFrame` convert exactly in either notation, as 
PySpark 4.3.0 does:
    
    - The Arrow encoder (`decimal_str_to_unscaled` in 
`crates/spark-connect/src/session.rs`) parses fixed-point (`-1.50`) and 
scientific (`1.5E-7`) text alike, in work bounded by the length of the text 
rather than by its exponent, and requires the value to fit `decimal(precision, 
scale)` exactly: a nonzero digit beyond the scale, or a value beyond the 
precision, is an error for either notation. Text without digits (`""`, `"."`, 
`"-"`) and decimal types that Spark does not have (a negative [...]
    - The Python binding sends `str()` of a Decimal again instead of its 
fixed-point form, so `Decimal("1E+1000000000")` is not written out in a billion 
digits first.
    - A pandas column whose Decimal does not fit its field fails as PySpark's 
Arrow conversion does, with `PySparkValueError: Exception thrown when 
converting pandas.Series (object) with name 'v' to Arrow Array (decimal128(18, 
7)).` and the sentence that 
`spark.sql.execution.pandas.convertToArrowArraySafely` adds, which is read only 
to raise. Local rows keep the half-even quantization to the scale of the 
Decimal conversion, as in PySpark.
    
    ### Why are the changes needed?
    
    [SPARK-60696](https://issues.apache.org/jira/browse/SPARK-60696): the 
encoder truncated the fixed-point text of pandas Decimals to the scale of the 
field, so data was silently lost where PySpark raises, and a value beyond the 
precision reached the server. Against a local Spark 4.3.0 server, for a pandas 
object column with an explicit `decimal(18, 7)` (or `decimal(10, 2)`) field:
    
    | Value | Current master | PySpark 4.3.0 and this change |
    | --- | --- | --- |
    | `1.5E-7` | `1E-7` | `PySparkValueError` |
    | `1E-8` | `0E-7` | `PySparkValueError` |
    | `1.23456789` | `1.2345678` | `PySparkValueError` |
    | `1.235` into `decimal(10, 2)` | `1.23` | `PySparkValueError` |
    | `100000000000` | server `NUMERIC_VALUE_OUT_OF_RANGE` | 
`PySparkValueError` |
    | `1E+100` | `PySparkRuntimeError: invalid decimal` | `PySparkValueError` |
    
    The same comparison of 30 pandas and local-row cases (rounding, overflow, 
signed zeros, NaN, huge exponents) gives identical results with this change and 
with PySpark 4.3.0.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A pandas Decimal that does not fit its field raises 
`PySparkValueError`, as in PySpark, instead of losing digits. Rust callers that 
pass `Value::Decimal` text get the same exactness check, and scientific 
notation is accepted.
    
    ### How was this patch tested?
    
    - Unit tests of the parser: both notations, trailing zeros, leading zeros, 
signs, whitespace, the largest values of each precision, exponents up to 
`i64::MIN`/`i64::MAX`, inexact and out-of-precision values, malformed text and 
unsupported types.
    - Offline tests in `python/tests/test_dropin_offline.py`, against a mock 
server that records the uploaded Arrow IPC: exact scientific and fixed-point 
inputs give the IPC cells pyarrow builds, local rows round half-even as 
PySpark, and inexact or overflowing pandas Decimals raise PySpark's error, with 
its message, before anything is sent.
    - Against the current master and a local Spark 4.3.0 server: the lib and 
every integration test of `apache-spark-connect` and 
`apache-spark-connect-core`, e2e included (818 passed), the build of all tests 
with `wasm-udf`, the offline Python suites (1,840 passed, 8 skipped), the live 
drop-in and Arrow suites (109 passed) and the live cases (10 passed). `cargo 
fmt --all --check` passes.
    
    Changes on top of the original commits: the merge with master, which 
formats Decimals in fixed point; one exactness rule for both notations, where 
the original kept truncating fixed point; sending `str()` again; and PySpark's 
error for pandas columns. An inexact Decimal nested in an array or struct of a 
pandas cell also fails now, but with the encoder's `PySparkRuntimeError` rather 
than PySpark's `PySparkValueError`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6); the rework on the current master by 
Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #132 from dbtsai/codex/spark-60696-decimal-exponents.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Co-authored-by: DB Tsai <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs    |  51 +++++++-
 crates/spark-connect/src/session.rs | 244 ++++++++++++++++++++++++++++++++----
 python/tests/test_dropin_offline.py | 232 ++++++++++++++++++++++++++++++++++
 3 files changed, 496 insertions(+), 31 deletions(-)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index a6b297d..d17c8c3 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1,7 +1,7 @@
 //! PyO3 wrapper for Spark Connect sessions.
 
 use pyo3::prelude::*;
-use pyo3::types::{PyDict, PyList, PyString};
+use pyo3::types::{PyDict, PyList};
 use spark_connect::row::{Row, StructValue, Value};
 use spark_connect::session::SparkSession;
 use spark_connect::types::{DataType, StructField};
@@ -666,6 +666,47 @@ impl PySparkSession {
                 fields
             }
         };
+        // As PySpark's Arrow conversion of a pandas column: a Decimal that 
does not fit the
+        // precision and scale of its field exactly fails, rather than losing 
digits.
+        for (i, field) in fields.iter().enumerate() {
+            let DataType::Decimal { precision, scale } = field.data_type else {
+                continue;
+            };
+            let fits = |value: &Value| match value {
+                Value::Decimal { value, .. } => {
+                    spark_connect::session::decimal_str_to_unscaled(value, 
precision, scale).is_ok()
+                }
+                _ => true,
+            };
+            if !rows.iter().filter_map(|row| row.get(i)).all(fits) {
+                let mut message = format!(
+                    "Exception thrown when converting pandas.Series ({}) with 
name '{}' to Arrow \
+                     Array (decimal128({precision}, {scale})).",
+                    dtypes.get(i).map_or("object", String::as_str),
+                    labels.get(i).unwrap_or(&field.name),
+                );
+                // Read only to raise, as the conf changes the message, not 
the failure.
+                let safecheck = py
+                    .detach(|| {
+                        self.session
+                            .conf()
+                            
.get("spark.sql.execution.pandas.convertToArrowArraySafely")
+                    })
+                    .map_or(true, |value| value.is_none_or(|value| value == 
"true"));
+                if safecheck {
+                    message.push_str(
+                        " It can be caused by overflows or other unsafe 
conversions warned by \
+                         Arrow. Arrow safe type check can be disabled by using 
SQL config \
+                         
`spark.sql.execution.pandas.convertToArrowArraySafely`.",
+                    );
+                }
+                return Err(PyErr::from_value(
+                    py.import("pyspark.errors")?
+                        .getattr("PySparkValueError")?
+                        .call1((message,))?,
+                ));
+            }
+        }
         let schema = DataType::Struct { fields };
         let df = py
             .detach(|| {
@@ -2021,10 +2062,10 @@ fn decimal_to_value(obj: &Bound<'_, PyAny>) -> 
PyResult<Option<Value>> {
     if !obj.is_instance(&decimal_cls)? {
         return Ok(None);
     }
-    // Without an exponent, as `1E+2` is written, which the Arrow encoder 
parses.
-    let dec_str: String = PyString::new(py, "{:f}")
-        .call_method1("format", (obj,))?
-        .extract()?;
+    // As written, in scientific notation for `1E+2` or `1.5E-7`: the Arrow 
encoder parses
+    // either notation, in work bounded by the length of the text, which a 
fixed-point form
+    // of `1E+1000000000` would not be.
+    let dec_str: String = obj.str()?.extract()?;
     let as_tuple = obj.call_method0("as_tuple")?;
     let mut scale: Option<i32> = None;
     if let Ok(exp_obj) = as_tuple.get_item(2) {
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 5de1433..5b6698e 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -1228,7 +1228,7 @@ fn values_to_arrow(values: &[&Value], dt: &DataType) -> 
Result<Arc<dyn arrow::ar
                 .map(|v| match *v {
                     Value::Null => Ok(None),
                     Value::Decimal { value, .. } => {
-                        decimal_str_to_unscaled(value, col_scale).map(Some)
+                        decimal_str_to_unscaled(value, *precision, 
col_scale).map(Some)
                     }
                     _ => Err(SparkError::connect_msg("Type mismatch in row 
data")),
                 })
@@ -1479,37 +1479,85 @@ fn values_to_arrow(values: &[&Value], dt: &DataType) -> 
Result<Arc<dyn arrow::ar
     }
 }
 
-/// Parse a decimal string (e.g. "-1.50") into an unscaled `i128` at the given 
scale
-/// (e.g. scale 2 → -150). Pads/truncates the fractional part to `scale` 
digits.
-fn decimal_str_to_unscaled(s: &str, scale: i32) -> Result<i128> {
-    let s = s.trim();
-    let (neg, s) = match s.strip_prefix('-') {
+/// Converts a decimal string, in fixed-point (`-1.50`) or scientific 
(`1.5E-7`) notation, to
+/// its unscaled `i128` at `scale` (`-1.50` at scale 2 is -150). The value 
must fit
+/// `decimal(precision, scale)` exactly, as Arrow requires of the Decimals of 
PySpark's pandas
+/// conversion: a nonzero digit beyond the scale, or a value beyond the 
precision, is an error
+/// rather than truncated. The work is bounded by the length of the string, 
never by its
+/// exponent, so `1E+1000000000` fails at once.
+pub fn decimal_str_to_unscaled(s: &str, precision: i32, scale: i32) -> 
Result<i128> {
+    if !(1..=38).contains(&precision) || !(0..=precision).contains(&scale) {
+        return Err(SparkError::connect_msg(format!(
+            "unsupported decimal type decimal({precision},{scale})"
+        )));
+    }
+    let invalid = || SparkError::connect_msg(format!("invalid decimal '{s}'"));
+    let inexact = || {
+        SparkError::connect_msg(format!(
+            "decimal '{s}' does not fit decimal({precision},{scale}) exactly"
+        ))
+    };
+    let text = s.trim();
+    let (negative, unsigned) = match text.strip_prefix('-') {
         Some(rest) => (true, rest),
-        None => (false, s.strip_prefix('+').unwrap_or(s)),
+        None => (false, text.strip_prefix('+').unwrap_or(text)),
     };
-    let (int_part, frac_part) = match s.split_once('.') {
-        Some((i, f)) => (i, f),
-        None => (s, ""),
+    let (coefficient, exponent) = match unsigned.split_once(['e', 'E']) {
+        Some((coefficient, exponent)) => {
+            (coefficient, exponent.parse::<i64>().map_err(|_| invalid())?)
+        }
+        None => (unsigned, 0),
     };
-    let scale = scale.max(0) as usize;
-    let mut digits = String::with_capacity(int_part.len() + scale);
-    digits.push_str(int_part);
-    // Pad or truncate the fractional digits to exactly `scale`.
-    let frac: String = frac_part
-        .chars()
-        .chain(std::iter::repeat('0'))
-        .take(scale)
+    let mut point = false;
+    let mut fractional = 0_i128;
+    let mut digits = 0_usize;
+    for byte in coefficient.bytes() {
+        match byte {
+            b'.' if !point => point = true,
+            b'0'..=b'9' => {
+                digits += 1;
+                if point {
+                    fractional += 1;
+                }
+            }
+            _ => return Err(invalid()),
+        }
+    }
+    if digits == 0 {
+        return Err(invalid());
+    }
+    // The digits from the first nonzero one: the value is them times 
10^(exponent -
+    // fractional), and the unscaled value them times 10^shift.
+    let significant: Vec<u8> = coefficient
+        .bytes()
+        .filter(|b| *b != b'.')
+        .skip_while(|b| *b == b'0')
         .collect();
-    digits.push_str(&frac);
-    let digits = digits.trim_start_matches('0');
-    let mag: i128 = if digits.is_empty() {
-        0
+    // i128 arithmetic, so that i64::MIN and i64::MAX exponents cannot 
overflow.
+    let shift = i128::from(scale) + i128::from(exponent) - fractional;
+    let dropped = if shift < 0 {
+        (-shift).min(significant.len() as i128) as usize
     } else {
-        digits
-            .parse::<i128>()
-            .map_err(|e| SparkError::connect_msg(format!("invalid decimal 
'{s}': {e}")))?
+        0
     };
-    Ok(if neg { -mag } else { mag })
+    let (kept, lost) = significant.split_at(significant.len() - dropped);
+    if lost.iter().any(|b| *b != b'0') {
+        return Err(inexact());
+    }
+    if kept.is_empty() {
+        return Ok(0);
+    }
+    if kept.len() as i128 + shift.max(0) > i128::from(precision) {
+        return Err(inexact());
+    }
+    // At most 38 digits: no overflow.
+    let mut value = kept
+        .iter()
+        .fold(0_i128, |value, digit| value * 10 + i128::from(digit - b'0'));
+    for _ in 0..shift.max(0) {
+        value *= 10;
+    }
+    Ok(if negative { -value } else { value })
 }
 
 #[cfg(test)]
@@ -1522,6 +1570,150 @@ mod tests {
     use arrow::record_batch::RecordBatch;
     use std::io::Cursor;
 
+    #[test]
+    fn decimal_strings_convert_in_fixed_point_and_scientific_notation() {
+        for (text, precision, scale, expected) in [
+            ("1E+3", 18, 7, 10_000_000_000),
+            ("1E-7", 18, 7, 1),
+            ("-1E-7", 18, 7, -1),
+            ("+1.25e+2", 38, 2, 12_500),
+            ("-1.25E-2", 38, 4, -125),
+            ("0001.2500e+2", 38, 2, 12_500),
+            ("0E+999999999", 18, 7, 0),
+            ("-0E-999999999", 18, 7, 0),
+            ("123.45", 38, 2, 12_345),
+            ("-123.450", 38, 2, -12_345),
+            (".00100E+3", 38, 2, 100),
+            ("  +1.e2  ", 38, 0, 100),
+            // Trailing zeros beyond the scale lose nothing.
+            ("1.2300E+0", 3, 2, 123),
+            ("1.2300", 3, 2, 123),
+            ("9.9E+1", 2, 0, 99),
+            ("99999999999.9999999", 18, 7, 999_999_999_999_999_999),
+            ("-99999999999.9999999", 18, 7, -999_999_999_999_999_999),
+            ("1E-38", 38, 38, 1),
+            // A long coefficient may still fit once it is scaled.
+            ("100000000000000000000000000000000000000000E-41", 38, 0, 1),
+        ] {
+            assert_eq!(
+                decimal_str_to_unscaled(text, precision, scale).unwrap(),
+                expected,
+                "{text}"
+            );
+        }
+        let max = "9".repeat(38);
+        assert_eq!(
+            decimal_str_to_unscaled(&max, 38, 0).unwrap(),
+            max.parse::<i128>().unwrap()
+        );
+        assert_eq!(
+            decimal_str_to_unscaled(&format!("-{max}"), 38, 0).unwrap(),
+            -max.parse::<i128>().unwrap()
+        );
+    }
+
+    #[test]
+    fn decimal_strings_that_do_not_fit_exactly_are_errors() {
+        for (text, precision, scale) in [
+            // A nonzero digit beyond the scale, in either notation: PySpark's 
Arrow
+            // conversion of a pandas column fails rather than dropping it.
+            ("1.5E-7", 18, 7),
+            ("-1.5E-7", 18, 7),
+            ("1E-8", 18, 7),
+            ("0.00000015", 18, 7),
+            ("1.235", 10, 2),
+            ("1E-9223372036854775808", 18, 7),
+            // Beyond the precision.
+            ("123456E-3", 9, 7),
+            ("1E+2", 2, 0),
+            ("-1E+2", 2, 0),
+            ("100", 2, 0),
+            ("100000000000", 18, 7),
+            ("1E+38", 38, 0),
+            ("1E+999999999", 38, 0),
+            ("1E+9223372036854775807", 38, 0),
+            ("170141183460469231731687303715884105727", 38, 0),
+        ] {
+            let error = decimal_str_to_unscaled(text, precision, 
scale).unwrap_err();
+            assert!(
+                error.to_string().contains("does not fit"),
+                "{text}: {error}"
+            );
+        }
+        for text in [
+            "",
+            ".",
+            "-",
+            "+",
+            "1E",
+            "E3",
+            "1E1E1",
+            "1.2.3",
+            "1E1.5",
+            "NaN",
+            "Infinity",
+            "1E 3",
+            "1_000",
+            "1E+9223372036854775808",
+        ] {
+            let error = decimal_str_to_unscaled(text, 38, 0).unwrap_err();
+            assert!(
+                error.to_string().contains("invalid decimal"),
+                "{text}: {error}"
+            );
+        }
+        // Spark has no such decimal types.
+        for (precision, scale) in [(0, 0), (39, 0), (10, -2), (10, 11), (38, 
i32::MAX)] {
+            assert!(decimal_str_to_unscaled("1", precision, scale).is_err());
+        }
+    }
+
+    #[test]
+    fn scientific_decimal_ipc_cells_match_fixed_point() {
+        use arrow::array::Decimal128Array;
+        let values = [
+            "1E+3",
+            "1E-7",
+            "-1E-7",
+            "-0E+3",
+            "99999999999.9999999",
+            "-99999999999.9999999",
+        ];
+        let rows: Vec<_> = values
+            .iter()
+            .map(|value| {
+                row(vec![Value::Decimal {
+                    value: value.to_string(),
+                    precision: None,
+                    scale: None,
+                }])
+            })
+            .collect();
+        let batch = read_batch(
+            &rows,
+            &schema(vec![DataType::Decimal {
+                precision: 18,
+                scale: 7,
+            }]),
+        );
+        let array = batch
+            .column(0)
+            .as_any()
+            .downcast_ref::<Decimal128Array>()
+            .unwrap();
+        assert_eq!(
+            array.values().as_ref(),
+            &[
+                10_000_000_000,
+                1,
+                -1,
+                0,
+                999_999_999_999_999_999,
+                -999_999_999_999_999_999
+            ]
+        );
+    }
+
     fn struct_field(name: &str, data_type: DataType) -> 
crate::types::StructField {
         crate::types::StructField {
             name: name.to_string(),
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 5da39f8..f9fc331 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -5930,6 +5930,238 @@ def 
test_exact_date_ipc_calendar_range(exact_date_capture_input, source, schema_
     assert table.column(0).cast(pa.int32()).to_pylist() == expected_days
 
 
+# Inspect the local-input IPC through a real transport boundary. The subprocess
+# avoids sharing the GIL with native RPC calls and needs no JVM or Spark 
server.
+_DECIMAL_IPC_SERVER = r"""
+import sys
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+import grpc
+import pyarrow as pa
+from google.protobuf import descriptor_pb2, descriptor_pool, message_factory
+pool = descriptor_pool.DescriptorPool()
+for file in 
descriptor_pb2.FileDescriptorSet.FromString(Path(sys.argv[1]).read_bytes()).file:
+    pool.Add(file)
+def message(name, **fields):
+    cls = 
message_factory.GetMessageClass(pool.FindMessageTypeByName('spark.connect.' + 
name))
+    return cls(**fields)
+def config(raw, context):
+    request = message('ConfigRequest').FromString(raw)
+    response = message('ConfigResponse', session_id=request.session_id,
+                       
server_side_session_id='00000000-0000-0000-0000-000000000001')
+    values = {
+        'spark.sql.session.timeZone': 'UTC',
+        'spark.sql.timestampType': 'TIMESTAMP_LTZ',
+        'spark.sql.session.localRelationCacheThreshold': str(1 << 60),
+        'spark.sql.session.localRelationSizeLimit': str(1 << 60),
+        'spark.sql.session.localRelationChunkSizeRows': '10000',
+        'spark.sql.session.localRelationChunkSizeBytes': str(16 << 20),
+        'spark.sql.session.localRelationBatchOfChunksSizeBytes': str(64 << 20),
+        # As the default of a server.
+        'spark.sql.execution.pandas.convertToArrowArraySafely': 'true',
+    }
+    operation = request.operation
+    if operation.HasField('get'):
+        keys = operation.get.keys
+    elif operation.HasField('get_with_default'):
+        keys = [pair.key for pair in operation.get_with_default.pairs]
+    else:
+        context.abort(grpc.StatusCode.INVALID_ARGUMENT, 'unexpected config 
operation')
+    for key in keys:
+        response.pairs.add(key=key, value=values.get(key, 'false'))
+    return response.SerializeToString()
+def execute(raw, context):
+    request = message('ExecutePlanRequest').FromString(raw)
+    relation = request.plan.root
+    # Python's pandas path wraps the local relation to assign column names.
+    if relation.HasField('to_df'):
+        relation = relation.to_df.input
+    assert relation.HasField('local_relation'), relation
+    payload = relation.local_relation.data
+    Path(sys.argv[2]).write_bytes(payload)
+    table = pa.ipc.open_stream(payload).read_all()
+    header = dict(session_id=request.session_id, 
operation_id=request.operation_id,
+                  
server_side_session_id='00000000-0000-0000-0000-000000000001')
+    response = message('ExecutePlanResponse', **header, response_id='batch',
+                       arrow_batch=dict(data=payload, 
row_count=table.num_rows))
+    for field in table.schema:
+        output = response.schema.struct.fields.add(name=field.name, 
nullable=True)
+        output.data_type.decimal.precision = field.type.precision
+        output.data_type.decimal.scale = field.type.scale
+    yield response.SerializeToString()
+    yield message('ExecutePlanResponse', **header, response_id='complete',
+                  result_complete={}).SerializeToString()
+def release_session(raw, context):
+    request = message('ReleaseSessionRequest').FromString(raw)
+    return message('ReleaseSessionResponse', session_id=request.session_id,
+                   
server_side_session_id='00000000-0000-0000-0000-000000000001').SerializeToString()
+def release_execute(raw, context):
+    request = message('ReleaseExecuteRequest').FromString(raw)
+    return message('ReleaseExecuteResponse', session_id=request.session_id,
+                   operation_id=request.operation_id,
+                   
server_side_session_id='00000000-0000-0000-0000-000000000001').SerializeToString()
+with ThreadPoolExecutor(max_workers=4) as executor:
+    server = grpc.server(executor)
+    server.add_generic_rpc_handlers((grpc.method_handlers_generic_handler(
+        'spark.connect.SparkConnectService', {
+            'Config': grpc.unary_unary_rpc_method_handler(config),
+            'ExecutePlan': grpc.unary_stream_rpc_method_handler(execute),
+            'ReleaseSession': 
grpc.unary_unary_rpc_method_handler(release_session),
+            'ReleaseExecute': 
grpc.unary_unary_rpc_method_handler(release_execute),
+        }),))
+    port = server.add_insecure_port('127.0.0.1:0')
+    assert port
+    server.start()
+    print(port, flush=True)
+    sys.stdin.readline()
+    server.stop(0).wait()
+"""
+
+
[email protected](scope="module")
+def decimal_ipc_protocol(tmp_path_factory):
+    import os
+    from pathlib import Path
+    import shutil
+    import subprocess
+
+    pytest.importorskip("grpc")
+    pytest.importorskip("pyarrow")
+    protoc = shutil.which(os.environ.get("PROTOC", "protoc"))
+    if protoc is None:
+        pytest.skip("Decimal IPC transport tests require protoc")
+    proto_root = Path(__file__).resolve().parents[2] / 
"crates/spark-connect-proto/proto"
+    descriptor = tmp_path_factory.mktemp("decimal-proto") / "connect.pb"
+    command = [protoc, f"-I{proto_root}"]
+    if include := os.environ.get("PROTOC_INCLUDE"):
+        command.append(f"-I{include}")
+    subprocess.run(command + ["--include_imports", 
f"--descriptor_set_out={descriptor}",
+                             str(proto_root / "spark/connect/base.proto")], 
check=True)
+    return descriptor
+
+
[email protected]
+def decimal_ipc_session(decimal_ipc_protocol, tmp_path):
+    import subprocess
+    import sys
+    from pyspark.sql import SparkSession
+
+    payload = tmp_path / "input.arrow"
+    process = subprocess.Popen(
+        [sys.executable, "-u", "-c", _DECIMAL_IPC_SERVER, 
str(decimal_ipc_protocol), str(payload)],
+        stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True,
+    )
+    spark = None
+    try:
+        port = int(process.stdout.readline())
+        spark = SparkSession.builder.remote(f"sc://127.0.0.1:{port}").create()
+        yield spark, payload
+    finally:
+        try:
+            if spark is not None:
+                spark.stop()
+        finally:
+            process.communicate("stop\n", timeout=20)
+            assert process.returncode == 0
+
+
[email protected]("kind", ["tuples", "pandas"])
[email protected]("precision, scale, texts", [
+    (18, 7, ["1E+3", "1E-7", "-1E-7", "+1.25E+2", "-1.25E-2", "0E+3", "-0E-7", 
None]),
+    (18, 7, ["99999999999.9999999", "-99999999999.9999999", "1.234500E-3"]),
+    (38, 0, ["1E+37", "-1E+37", 
"9.9999999999999999999999999999999999999E+37"]),
+    (38, 38, ["1E-38", "-1E-38", "0E-38"]),
+    (10, 2, ["123.45", "-123.45", "1.2300", "0.00", "-0.00"]),
+])
+def test_decimal_scientific_local_input_ipc(decimal_ipc_session, kind, 
precision, scale, texts):
+    from decimal import Decimal
+    import pyarrow as pa
+
+    spark, payload = decimal_ipc_session
+    values = [None if text is None else Decimal(text) for text in texts]
+    data = [(value,) for value in values]
+    if kind == "pandas":
+        pd = pytest.importorskip("pandas")
+        data = pd.DataFrame({"v": values})
+    schema = T.StructType([T.StructField("v", T.DecimalType(precision, 
scale))])
+    spark.createDataFrame(data, schema).collect()
+    # Read the uploaded IPC independently rather than trusting collect's 
decoder.
+    table = pa.ipc.open_stream(payload.read_bytes()).read_all()
+    expected = pa.table({"v": pa.array(values, type=pa.decimal128(precision, 
scale))})
+    assert table.schema == expected.schema
+    assert table.equals(expected)
+    # Check the physical little-endian Decimal128 cells as well as their 
values.
+    actual = table.column(0).combine_chunks()
+    wanted = expected.column(0).combine_chunks()
+    for index, value in enumerate(values):
+        if value is not None:
+            offset = index * 16
+            assert actual.buffers()[1][offset:offset + 16] == 
wanted.buffers()[1][offset:offset + 16]
+
+
[email protected]("text", ["1E+1000000000", "-1E+1000000000"])
+def test_decimal_scientific_local_input_overflow(decimal_ipc_session, text):
+    from decimal import Decimal
+
+    spark, payload = decimal_ipc_session
+    schema = T.StructType([T.StructField("v", T.DecimalType(38, 0))])
+    with pytest.raises((ArithmeticError, RuntimeError)):
+        spark.createDataFrame([(Decimal(text),)], schema).collect()
+    assert not payload.exists()
+
+
[email protected]("kind", ["tuples", "pandas"])
[email protected]("text, precision, scale, local_expected", [
+    ("1.5E-7", 18, 7, "0.0000002"),
+    ("-1.5E-7", 18, 7, "-0.0000002"),
+    ("1E-8", 18, 7, "0.0000000"),
+    ("0.00000015", 18, 7, "0.0000002"),
+    ("1.235", 10, 2, "1.24"),
+    ("1E+2", 2, 0, None),
+    ("-1E+2", 2, 0, None),
+    ("100000000000", 18, 7, None),
+    ("1E+1000000000", 38, 0, None),
+])
+def test_decimal_rescaling_follows_pyspark(
+    decimal_ipc_session, kind, text, precision, scale, local_expected
+):
+    # As PySpark 4.3.0: local rows are quantized half-even to the scale, and 
fail beyond
+    # the precision, in the Decimal conversion; a pandas column fails its 
Arrow conversion
+    # for any value that does not fit exactly, in either notation, rather than 
losing digits.
+    from decimal import Decimal
+    import pyarrow as pa
+    from pyspark.errors import PySparkValueError
+
+    spark, payload = decimal_ipc_session
+    data = [(Decimal(text),)]
+    if kind == "pandas":
+        pd = pytest.importorskip("pandas")
+        data = pd.DataFrame({"v": [Decimal(text)]}, dtype=object)
+    schema = T.StructType([T.StructField("v", T.DecimalType(precision, 
scale))])
+    if kind == "tuples" and local_expected is not None:
+        spark.createDataFrame(data, schema).collect()
+        actual = pa.ipc.open_stream(payload.read_bytes()).read_all()
+        expected = pa.table({"v": pa.array([Decimal(local_expected)],
+                                           type=pa.decimal128(precision, 
scale))})
+        actual.validate(full=True)
+        assert actual.schema == expected.schema
+        assert actual.equals(expected)
+        return
+    if kind == "tuples":
+        with pytest.raises(ArithmeticError):
+            spark.createDataFrame(data, schema)
+    else:
+        with pytest.raises(PySparkValueError) as error:
+            spark.createDataFrame(data, schema)
+        assert str(error.value) == (
+            "Exception thrown when converting pandas.Series (object) with name 
'v' to Arrow "
+            f"Array (decimal128({precision}, {scale})). It can be caused by 
overflows or other "
+            "unsafe conversions warned by Arrow. Arrow safe type check can be 
disabled by using "
+            "SQL config 
`spark.sql.execution.pandas.convertToArrowArraySafely`."
+        )
+    assert not payload.exists()
+
+
 # --------------------------------------------------------------------------- 
schema cache
 
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to