This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 5861b84  [SPARK-60704][CONNECT] Look up the Decimal type once, and 
build the quantum and context of each decimal type once per conversion
5861b84 is described below

commit 5861b849d037ff4bc4615176c21db7c097a815b4
Author: DB Tsai <[email protected]>
AuthorDate: Sat Nov 7 12:26:33 2026 +0900

    [SPARK-60704][CONNECT] Look up the Decimal type once, and build the quantum 
and context of each decimal type once per conversion
    
    ### What changes were proposed in this pull request?
    
    Stacked on #133 (SPARK-60697) and #132 (SPARK-60696); until they merge, 
this diff includes them.
    
    The Decimal conversions of `createDataFrame` resolve what they need once 
rather than for every value:
    
    - `decimal_to_value` (pandas cells and other generic values) and the pandas 
Decimal type inference take the `decimal.Decimal` class from the cache that 
`infer` already keeps, rather than importing `decimal` and reading its 
attribute per value.
    - The local-row conversion builds the quantum (`Decimal("1E-<scale>")`) and 
the context (`Context(prec=<precision>, rounding=ROUND_HALF_EVEN)`) of each 
decimal type once per conversion, as PySpark builds them once per converter, 
rather than for every value. They are kept only while a conversion runs 
(`DecimalQuanta`), so a change of `decimal.DefaultContext` applies from the 
next `createDataFrame`, as in PySpark. The quantized value is sent as written, 
which the encoder parses in eithe [...]
    
    This replaces the original change, which ran a second pass over every 
pandas cell behind checks of the `decimal` and `_decimal` modules and of the 
type's flags: the cached class gives the same saving to every path without them.
    
    ### Why are the changes needed?
    
    [SPARK-60704](https://issues.apache.org/jira/browse/SPARK-60704): the 
per-value lookups and constructions dominated the conversion of Decimal inputs. 
Measured on macOS against a local Spark 4.3.0 server, as the client CPU time of 
`createDataFrame` of 200,000 rows of two Decimal columns (median of 5):
    
    | Input | #133 | This change | PySpark 4.3.0 |
    | --- | --- | --- | --- |
    | Local rows, `decimal(12,4), decimal(10,2)` | 917 ms | 254 ms | 332 ms |
    | Local rows, column names (inferred) | 1,624 ms | 894 ms | 2,590 ms |
    | pandas, `decimal(12,4), decimal(10,2)` | 627 ms | 441 ms | 194 ms |
    | pandas, column names (inferred) | 634 ms | 465 ms | 632 ms |
    
    All give the same sums of both columns.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, other than speed.
    
    ### How was this patch tested?
    
    - A new offline test in `python/tests/test_dropin_offline.py`, against 
#132's mock server that records the uploaded Arrow IPC: each decimal type of 
one conversion is quantized half-even to its own scale, and a trap set in 
`decimal.DefaultContext` applies to the next `createDataFrame` and not after it 
is reset, as with PySpark 4.3.0.
    - The Decimal tests of the offline suite (78 passed, 1 skipped); the full 
suites run before the merge.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6); the rework on the current master by 
Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #136 from dbtsai/codex/spark-60704-decimal-lookup.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Co-authored-by: DB Tsai <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/infer.rs      | 94 ++++++++++++++++++++++++++++---------
 crates/pyspark-rs/src/session.rs    | 17 ++++---
 python/tests/test_dropin_offline.py | 30 ++++++++++++
 3 files changed, 110 insertions(+), 31 deletions(-)

diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 5ae84ca..f62bd61 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -24,7 +24,7 @@ use crate::errors::ResultExt;
 use crate::row::PyRow;
 use crate::values::{PyGeography, PyGeometry, PyVariantVal};
 
-static DECIMAL: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+pub(crate) static DECIMAL: PyOnceLock<Py<PyType>> = PyOnceLock::new();
 static DATE: PyOnceLock<Py<PyType>> = PyOnceLock::new();
 static DATETIME: PyOnceLock<Py<PyType>> = PyOnceLock::new();
 static TIME: PyOnceLock<Py<PyType>> = PyOnceLock::new();
@@ -32,6 +32,73 @@ static TIMEDELTA: PyOnceLock<Py<PyType>> = PyOnceLock::new();
 static ARRAY: PyOnceLock<Py<PyType>> = PyOnceLock::new();
 static SORTED: PyOnceLock<Py<PyAny>> = PyOnceLock::new();
 
+thread_local! {
+    /// The quantum and the context of each decimal type, while a 
[`DecimalQuanta`] lives: PySpark
+    /// builds them once for the converter of a field, rather than once per 
value.
+    static DECIMAL_QUANTA: std::cell::RefCell<Option<HashMap<(i32, i32), 
(Py<PyAny>, Py<PyAny>)>>> =
+        const { std::cell::RefCell::new(None) };
+}
+
+/// While alive, the conversions of this thread build the quantum and the 
context of each decimal
+/// type once. Each conversion of rows has its own, as each `createDataFrame` 
of PySpark has its
+/// converters, so that its contexts take the `decimal.DefaultContext` of 
their time.
+pub(crate) struct DecimalQuanta;
+
+impl DecimalQuanta {
+    pub(crate) fn new() -> Self {
+        DECIMAL_QUANTA.with(|quanta| *quanta.borrow_mut() = 
Some(HashMap::new()));
+        DecimalQuanta
+    }
+}
+
+impl Drop for DecimalQuanta {
+    fn drop(&mut self) {
+        DECIMAL_QUANTA.with(|quanta| quanta.borrow_mut().take());
+    }
+}
+
+/// `Decimal("1E-<scale>")` and `Context(prec=<precision>, 
rounding=ROUND_HALF_EVEN)`, with which
+/// PySpark quantizes a value of `decimal(precision, scale)`.
+fn decimal_quantum<'py>(
+    py: Python<'py>,
+    precision: i32,
+    scale: i32,
+) -> PyResult<(Bound<'py, PyAny>, Bound<'py, PyAny>)> {
+    let cached = DECIMAL_QUANTA.with(|quanta| {
+        quanta.borrow().as_ref().and_then(|quanta| {
+            quanta
+                .get(&(precision, scale))
+                .map(|(exp, context)| (exp.bind(py).clone(), 
context.bind(py).clone()))
+        })
+    });
+    if let Some(cached) = cached {
+        return Ok(cached);
+    }
+    let module = py.import(intern!(py, "decimal"))?;
+    let exp = DECIMAL
+        .import(py, "decimal", "Decimal")?
+        .call1((format!("1E-{scale}"),))?;
+    let context = module.getattr(intern!(py, "Context"))?.call(
+        (),
+        Some(
+            &[
+                ("prec", precision.into_pyobject(py)?.into_any()),
+                ("rounding", module.getattr(intern!(py, "ROUND_HALF_EVEN"))?),
+            ]
+            .into_py_dict(py)?,
+        ),
+    )?;
+    DECIMAL_QUANTA.with(|quanta| {
+        if let Some(quanta) = quanta.borrow_mut().as_mut() {
+            quanta.insert(
+                (precision, scale),
+                (exp.clone().unbind(), context.clone().unbind()),
+            );
+        }
+    });
+    Ok((exp, context))
+}
+
 /// The session configurations that inference reads, as PySpark reads them for
 /// `createDataFrame`.
 struct Confs {
@@ -954,28 +1021,11 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: 
&DataType, nullable: bool) ->
                 return Ok(Value::Null);
             }
             // `value.quantize(Decimal("1E-<scale>"), 
context=Context(prec=<precision>,
-            // rounding=ROUND_HALF_EVEN))`, written without an exponent.
-            let module = py.import(intern!(py, "decimal"))?;
-            let exp = decimal.call1((format!("1E-{scale}"),))?;
-            let context = module.getattr(intern!(py, "Context"))?.call(
-                (),
-                Some(
-                    &[
-                        ("prec", precision.into_pyobject(py)?.into_any()),
-                        ("rounding", module.getattr(intern!(py, 
"ROUND_HALF_EVEN"))?),
-                    ]
-                    .into_py_dict(py)?,
-                ),
-            )?;
-            let quantized = obj.call_method(
-                intern!(py, "quantize"),
-                (exp,),
-                Some(&[("context", context)].into_py_dict(py)?),
-            )?;
-            let text =
-                PyString::new(py, "{:f}").call_method1(intern!(py, "format"), 
(quantized,))?;
+            // rounding=ROUND_HALF_EVEN))`, as written: the encoder parses 
either notation.
+            let (exp, context) = decimal_quantum(py, *precision, *scale)?;
+            let quantized = obj.call_method1(intern!(py, "quantize"), (exp, 
py.None(), context))?;
             Ok(Value::Decimal {
-                value: text.extract()?,
+                value: quantized.str()?.extract()?,
                 precision: Some(*precision),
                 scale: Some(*scale),
             })
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 4ce35d1..2b05a00 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -509,6 +509,7 @@ impl PySparkSession {
         let mut values: Vec<Vec<Value>> = (0..parts.len())
             .map(|_| Vec::with_capacity(fields.len()))
             .collect();
+        let _quanta = infer::DecimalQuanta::new();
         for (i, field) in fields.iter().enumerate() {
             for (row, out) in parts.iter().zip(values.iter_mut()) {
                 out.push(infer::convert(&row[i], &field.data_type, 
field.nullable)?);
@@ -1752,7 +1753,7 @@ fn pandas_decimal_type(data: &Bound<'_, PyList>, rows: 
&[Row], i: usize) -> PyRe
     // The digits and exponent of an exact Decimal come from the text that 
`py_to_value` took
     // of it, rather than from `as_tuple()`, which builds a DecimalTuple of 
every digit; a
     // subclass, which can override either, is asked for `as_tuple()`, as 
Arrow does.
-    let decimal_cls = data.py().import("decimal")?.getattr("Decimal")?;
+    let decimal_cls = crate::infer::DECIMAL.import(data.py(), "decimal", 
"Decimal")?;
     let (mut integer_digits, mut scale) = (0i64, 0i64);
     for (row, converted) in data.iter().zip(rows) {
         let value = row.get_item(i)?;
@@ -1760,7 +1761,7 @@ fn pandas_decimal_type(data: &Bound<'_, PyList>, rows: 
&[Row], i: usize) -> PyRe
             continue;
         }
         let from_text = match converted.get(i) {
-            Some(Value::Decimal { value: text, .. }) if 
value.get_type().is(&decimal_cls) => {
+            Some(Value::Decimal { value: text, .. }) if 
value.get_type().is(decimal_cls) => {
                 decimal_text_digits(text)
             }
             _ => None,
@@ -2131,19 +2132,17 @@ pub(crate) fn datetime_to_micros(obj: &Bound<'_, 
PyAny>) -> PyResult<i64> {
 /// Decimal defines `__float__`/`__int__`.
 fn decimal_to_value(obj: &Bound<'_, PyAny>) -> PyResult<Option<Value>> {
     let py = obj.py();
-    let dec_mod = match py.import("decimal") {
-        Ok(m) => m,
-        Err(_) => return Ok(None),
-    };
-    let decimal_cls = dec_mod.getattr("Decimal")?;
-    if !obj.is_instance(&decimal_cls)? {
+    // Imported once: looking the class up per value was a module import and 
an attribute
+    // read of every converted value.
+    let decimal_cls = crate::infer::DECIMAL.import(py, "decimal", "Decimal")?;
+    if !obj.is_instance(decimal_cls.as_any())? {
         return Ok(None);
     }
     // As written, in scientific notation for `1E+2` or `1.5E-7`: the Arrow 
encoder parses
     // either notation, in work bounded by the length of the text, which a 
fixed-point form
     // of `1E+1000000000` would not be.
     let dec_str: String = obj.str()?.extract()?;
-    let scale = if obj.get_type().is(&decimal_cls) {
+    let scale = if obj.get_type().is(decimal_cls) {
         // Exact Decimal strings preserve the coefficient's trailing zeros and 
exponent.
         // Subclasses may override either __str__ or as_tuple, so keep their 
old path.
         decimal_string_scale(&dec_str)
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 8d68cc0..c3b18d0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6429,6 +6429,36 @@ def 
test_decimal_scale_of_non_finite_values(decimal_ipc_session, text):
     assert not payload.exists()
 
 
+def test_decimal_quanta_of_each_type_and_conversion(decimal_ipc_session):
+    # Each decimal type of a conversion is quantized with its own quantum and 
context, which
+    # are built once per conversion, as PySpark builds them once per 
converter: a change of
+    # decimal.DefaultContext applies from the next createDataFrame.
+    import decimal
+    from decimal import Decimal
+    import pyarrow as pa
+
+    spark, payload = decimal_ipc_session
+    schema = T.StructType([T.StructField("a", T.DecimalType(10, 2)),
+                           T.StructField("b", T.DecimalType(5, 1)),
+                           T.StructField("c", T.DecimalType(10, 2))])
+    rows = [(Decimal("1.235"), Decimal("2.25"), Decimal("-0.005")),
+            (Decimal("1.245"), Decimal("2.35"), None)]
+    spark.createDataFrame(rows, schema).collect()
+    table = pa.ipc.open_stream(payload.read_bytes()).read_all()
+    assert table.to_pydict() == {
+        "a": [Decimal("1.24"), Decimal("1.24")],
+        "b": [Decimal("2.2"), Decimal("2.4")],
+        "c": [Decimal("-0.00"), None],
+    }
+    decimal.DefaultContext.traps[decimal.Inexact] = True
+    try:
+        with pytest.raises(decimal.Inexact):
+            spark.createDataFrame(rows, schema)
+    finally:
+        decimal.DefaultContext.traps[decimal.Inexact] = False
+    spark.createDataFrame(rows[1:], schema)
+
+
 # --------------------------------------------------------------------------- 
schema cache
 
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to