This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 30910182 [SPARK-60704][CONNECT][FOLLOWUP] Preserve Decimal hooks and 
nested inferred scales
30910182 is described below

commit 30910182d7ae1955895450e1c8def5cfad417343
Author: DB Tsai <[email protected]>
AuthorDate: Sun Nov 8 20:31:25 2026 +0900

    [SPARK-60704][CONNECT][FOLLOWUP] Preserve Decimal hooks and nested inferred 
scales
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60704](https://issues.apache.org/jira/browse/SPARK-60704): fix 
Decimal hook dispatch and conversion-cache lifetime in the conversion already 
on master. Nested pandas inference also relates to 
[SPARK-60696](https://issues.apache.org/jira/browse/SPARK-60696).
    
    - Resolve the canonical C Decimal class independently of a rebound public 
alias, caching both accelerator presence and absence. Exact C values retain the 
text shortcuts; subclasses and supported pure-Python values retain their hooks 
and keyword `quantize(exp, context=...)` call.
    - Restore the outer quantum/context cache after reentrant conversion, 
including errors. Release cache borrows before invoking Python or dropping 
displaced Python objects; both quantize call forms share one context per 
Decimal type.
    - On native pandas row fallback, widen inferred Decimal scales across 
observed nested array, struct and map values. Preserve container shape, 
trailing zeros and precision 38. Ordinary pandas inputs retain master's 
columnar conversion.
    
    ### Why are the changes needed?
    
    Subclass hooks can otherwise be bypassed or receive incompatible arguments, 
and reentry can discard the outer conversion's cache. First-value-only 
inference can reject a later `Decimal("1.234")` after `Decimal("1.2")`, 
although both fit at the later scale.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. These are narrow correctness fixes with explicit compatibility limits:
    
    - **Class selection and precision loss:** the effective class and 
accelerator availability are cached at first resolution; later alias or 
module-availability changes are not rediscovered. With C Decimal available, 
rebinding `decimal.Decimal` to `_pydecimal.Decimal` does not make pure-Python 
instances native Decimals. Local-row conversion/inference can fail; inferred or 
names-only native pandas fallback can instead coerce them through `__float__` 
to doubles and lose precision. Quantums [...]
    - **Context hooks:** one context per Decimal type is created lazily at its 
first non-null, non-NaN value within a conversion. Reference PySpark constructs 
contexts per field converter before processing values. Hooks that alter context 
factories, `DefaultContext` or a supplied context can observe different timing 
and sharing. Changes between conversions apply to the next conversion.
    - **Nested scale, precision and errors:** preserving the greatest observed 
scale can make previously accepted values exceed precision 38. Native fallback 
can raise `PySparkRuntimeError` during DataFrame construction before sending a 
relation; reference PySpark Connect 4.3.0 can construct that input and later 
raise `ArithmeticException` during an action. A newly increased scale above 38 
raises `ValueError` locally. Existing explicit-nested conversion errors and 
Decimal-before-other-inv [...]
    
    This does not establish full Python behavioral parity or introduce a new 
scientific-notation parser.
    
    ### How was this patch tested?
    
    Candidate `bbf493219fdda8bc668471163b936f34b5e46429` against baseline 
`e8b1d4284e11d119c4fda9563a913ece22f43e7c`:
    
    - Focused native tests: **132 passed, one skipped**. Full native suite: 
**2,260 passed, eight skipped, one failed**. The sole failure matches the 
baseline's unsupported-NumPy-scalar case; the raw full-suite result remains 
failed.
    - Independent full-patch source review found no actionable findings. Rust 
core/API/deadline checks and default/Wasm compilation passed in the 
exact-source focused receipt and are reused by the full-suite run. Paired 
binding Clippy evidence is carried through audited unchanged lint inputs: raw 
exit **101**, **zero new binding diagnostics**. This is not clean final-head 
lint.
    - Fresh official transport gate: **91 selected files, 2,601 passed, 655 
runtime skips, zero unexpected failures/errors** (87 local files: 2,586/641; 
four cluster files: 15/14). Eleven modules contain only skips, totaling 267 
cases. A separately drift-run, checked-in excluded catalog case still fails 
because the distribution lacks `InMemoryCatalog`; it is outside those selected 
totals. Skips and exclusions are not passing coverage. Reference Python 
planning/conversion/decoding over Rus [...]
    - The initial final-revision run completed all six measurement workers and 
contracts, then its large-upload live check exhausted the **1 GiB JVM heap**; 
cleanup also raised `PermissionError`. The incomplete raw summary and 
failed-execution receipt are preserved; that run is not accepted as passed. A 
complete repeat uses an independently verified **3 GiB JVM heap**, with 
unchanged sources, worker, driver and plan.
    - The complete **3 GiB rerun passed**: all six timing workers, 
output-contract checks and **eight native live tests per candidate** (including 
the >128 MiB upload), with no live skips or failures. The 12-case Decimal 
contract probe passed on the candidate and official PySpark; the baseline had 
three contract failures. All final source, import, binary and reference-input 
guards passed; the owned JVM was reaped.
    - Published-head CI remains pending publication and exact-head verification.
    
    Measurements use macOS, Spark Connect 4.3.0, pandas 2.3.3, PyArrow 26.0.0 
and one shared owned local server. Decimal workloads have 200,000 rows and two 
fields; string workloads have three fields. Each worker is a separate process. 
The following values are client `createDataFrame` CPU / wall milliseconds. 
Complete output and schema validation ran outside the timer.
    
    **Warm medians of five calls:**
    
    | Workload | Frozen master CPU / wall | Candidate CPU / wall | Pure Python 
CPU / wall |
    | --- | ---: | ---: | ---: |
    | local-explicit | 100.292 / 100.304 | 106.842 / 106.852 | 103.479 / 
103.957 |
    | local-names | 332.024 / 332.036 | 329.253 / 329.278 | 913.490 / 913.352 |
    | pandas-explicit | 60.314 / 61.665 | 59.594 / 60.640 | 60.837 / 61.207 |
    | pandas-names | 228.319 / 231.083 | 228.047 / 229.806 | 228.727 / 228.699 |
    
    **First measured call per workload:** imports, session setup and input 
preparation precede these samples; later workloads follow earlier ones in their 
worker.
    
    | Workload | Frozen master CPU / wall | Candidate CPU / wall | Pure Python 
CPU / wall |
    | --- | ---: | ---: | ---: |
    | local-explicit | 105.641 / 105.667 | 106.004 / 106.014 | 118.921 / 
165.571 |
    | local-names | 331.917 / 331.925 | 330.871 / 330.879 | 925.916 / 956.206 |
    | pandas-explicit | 76.151 / 139.416 | 73.075 / 115.458 | 61.859 / 62.985 |
    | pandas-names | 224.299 / 263.237 | 217.553 / 281.154 | 227.704 / 267.435 |
    
    Observed warm local-explicit CPU cost increased **6.5% versus master**; the 
other three CPU medians changed by -1.2% to -0.1%. Whole-worker cumulative peak 
RSS was 273.1 MiB for master, 282.8 MiB for the candidate and 257.6 MiB for 
Python.
    
    This is one fixed-order paired run without confidence intervals. RSS 
includes input preparation, expected tables, collection, validation and 
allocator retention. Guard-only cost, fallback/wide/nested workload 
performance, RPC/upload counts, conversion-only memory, process cold starts and 
whole-query performance are outside these measurements.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    OpenAI Codex.
    
    This pull request and its description were written by OpenAI Codex.
    
    Closes #365 from dbtsai/codex/followup-decimal-conversion-current.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/infer.rs      | 222 ++++++++++++----
 crates/pyspark-rs/src/session.rs    | 232 ++++++++++++++--
 crates/spark-connect/src/row.rs     |   5 +-
 dev/parity/known-differences.md     |  55 ++++
 python/tests/test_dropin_offline.py | 512 +++++++++++++++++++++++++++++++++++-
 5 files changed, 953 insertions(+), 73 deletions(-)

diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 0234547d..0b682543 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -26,7 +26,8 @@ use crate::errors::ResultExt;
 use crate::row::PyRow;
 use crate::values::{PyGeography, PyGeometry, PyVariantVal};
 
-pub(crate) static DECIMAL: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static DECIMAL: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+static C_DECIMAL: PyOnceLock<Option<Py<PyType>>> = PyOnceLock::new();
 static DATE: PyOnceLock<Py<PyType>> = PyOnceLock::new();
 static DATETIME: PyOnceLock<Py<PyType>> = PyOnceLock::new();
 static TIME: PyOnceLock<Py<PyType>> = PyOnceLock::new();
@@ -34,71 +35,193 @@ static TIMEDELTA: PyOnceLock<Py<PyType>> = 
PyOnceLock::new();
 static ARRAY: PyOnceLock<Py<PyType>> = PyOnceLock::new();
 static SORTED: PyOnceLock<Py<PyAny>> = PyOnceLock::new();
 
+/// Cache the accelerator class, including its absence, independently of the 
public alias.
+pub(crate) fn c_decimal_type(py: Python<'_>) -> PyResult<Option<&Bound<'_, 
PyType>>> {
+    let decimal = C_DECIMAL.get_or_try_init(py, || match py.import("_decimal") 
{
+        Ok(module) => Ok(Some(
+            module.getattr("Decimal")?.cast_into::<PyType>()?.unbind(),
+        )),
+        Err(error) if 
error.is_instance_of::<pyo3::exceptions::PyImportError>(py) => Ok(None),
+        Err(error) => Err(error),
+    })?;
+    Ok(decimal.as_ref().map(|decimal| decimal.bind(py)))
+}
+
+/// The effective Decimal class from [`decimal_classes`]: C `_decimal.Decimal` 
when
+/// available, otherwise the cached public alias.
+fn decimal_type(py: Python<'_>) -> PyResult<&Bound<'_, PyType>> {
+    Ok(decimal_classes(py)?.1)
+}
+
+/// Resolve the accelerator once and return it with the effective conversion 
class.
+pub(crate) fn decimal_classes(
+    py: Python<'_>,
+) -> PyResult<(Option<&Bound<'_, PyType>>, &Bound<'_, PyType>)> {
+    let c_decimal = c_decimal_type(py)?;
+    let decimal = match c_decimal {
+        Some(decimal) => decimal,
+        None => DECIMAL.import(py, "decimal", "Decimal")?,
+    };
+    Ok((c_decimal, decimal))
+}
+
+/// Whether an object has the exact accelerator Decimal type. The shortcuts 
assume
+/// this type is immutable, as on CPython; subclasses and pure-Python values 
keep hooks.
+pub(crate) fn is_exact_c_decimal(
+    obj: &Bound<'_, PyAny>,
+    decimal: Option<&Bound<'_, PyType>>,
+) -> bool {
+    decimal.is_some_and(|decimal| obj.get_type().is(decimal))
+}
+
+/// Cached per Decimal type: a quantum, its context and lazily built context 
kwargs.
+struct QuantizeArgs {
+    exp: Py<PyAny>,
+    context: Py<PyAny>,
+    kwargs: Option<Py<PyDict>>,
+}
+
+/// Owned arguments for one call, released from the cache borrow before Python 
runs.
+enum QuantizeCallArgs<'py> {
+    Positional {
+        exp: Bound<'py, PyAny>,
+        context: Bound<'py, PyAny>,
+    },
+    Keyword {
+        exp: Bound<'py, PyAny>,
+        kwargs: Bound<'py, PyDict>,
+    },
+}
+
+/// A cache entry's arguments: a ready call, or a keyword dict to build 
outside the borrow.
+enum QuantizeLookup<'py> {
+    Call(QuantizeCallArgs<'py>),
+    Keywords {
+        exp: Bound<'py, PyAny>,
+        context: Bound<'py, PyAny>,
+    },
+}
+
+impl QuantizeArgs {
+    /// Clone only the pair needed by the selected call form, without running 
Python.
+    fn call_args<'py>(&self, py: Python<'py>, exact_c: bool) -> 
QuantizeLookup<'py> {
+        if exact_c {
+            QuantizeLookup::Call(QuantizeCallArgs::Positional {
+                exp: self.exp.bind(py).clone(),
+                context: self.context.bind(py).clone(),
+            })
+        } else if let Some(kwargs) = &self.kwargs {
+            QuantizeLookup::Call(QuantizeCallArgs::Keyword {
+                exp: self.exp.bind(py).clone(),
+                kwargs: kwargs.bind(py).clone(),
+            })
+        } else {
+            QuantizeLookup::Keywords {
+                exp: self.exp.bind(py).clone(),
+                context: self.context.bind(py).clone(),
+            }
+        }
+    }
+}
+
+type Quanta = HashMap<(i32, i32), QuantizeArgs>;
+
 thread_local! {
-    /// The quantum and the context of each decimal type, while a 
[`DecimalQuanta`] lives: PySpark
-    /// builds them once for the converter of a field, rather than once per 
value.
-    static DECIMAL_QUANTA: std::cell::RefCell<Option<HashMap<(i32, i32), 
(Py<PyAny>, Py<PyAny>)>>> =
+    /// Quantum, context and cached context kwargs per Decimal type, while a 
[`DecimalQuanta`]
+    /// lives. Both call forms share one context, built once per type rather 
than per value.
+    static DECIMAL_QUANTA: std::cell::RefCell<Option<Quanta>> =
         const { std::cell::RefCell::new(None) };
 }
 
-/// While alive, the conversions of this thread build the quantum and the 
context of each decimal
-/// type once. Each conversion of rows has its own, as each `createDataFrame` 
of PySpark has its
-/// converters, so that its contexts take the `decimal.DefaultContext` of 
their time.
-pub(crate) struct DecimalQuanta;
+/// While alive, this thread's conversions build each Decimal type's quantum 
and context
+/// once, and its context keyword dict only if needed. Each conversion of rows 
has its own,
+/// as each `createDataFrame` of PySpark has its converters. A type's context 
is built lazily
+/// at its first non-null, non-NaN value, using `decimal.DefaultContext` at 
that time.
+#[must_use = "bind the guard for the duration of the conversion"]
+pub(crate) struct DecimalQuanta(Option<Quanta>);
 
 impl DecimalQuanta {
     pub(crate) fn new() -> Self {
-        DECIMAL_QUANTA.with(|quanta| *quanta.borrow_mut() = 
Some(HashMap::new()));
-        DecimalQuanta
+        let previous = DECIMAL_QUANTA.with(|quanta| 
quanta.replace(Some(HashMap::new())));
+        DecimalQuanta(previous)
     }
 }
 
 impl Drop for DecimalQuanta {
     fn drop(&mut self) {
-        DECIMAL_QUANTA.with(|quanta| quanta.borrow_mut().take());
+        DECIMAL_QUANTA.with(|quanta| quanta.replace(self.0.take()));
     }
 }
 
-/// `Decimal("1E-<scale>")` and `Context(prec=<precision>, 
rounding=ROUND_HALF_EVEN)`, with which
-/// PySpark quantizes a value of `decimal(precision, scale)`.
+/// Cache the quantum, context and context keyword dict for 
`decimal(precision, scale)`.
+/// Return only the arguments needed by the selected call form, sharing the 
same context.
+/// `decimal` must be the effective class from [`decimal_classes`]: the C 
class when
+/// available, otherwise the cached public alias.
 fn decimal_quantum<'py>(
     py: Python<'py>,
+    decimal: &Bound<'py, PyType>,
     precision: i32,
     scale: i32,
-) -> PyResult<(Bound<'py, PyAny>, Bound<'py, PyAny>)> {
+    exact_c: bool,
+) -> PyResult<QuantizeCallArgs<'py>> {
     let cached = DECIMAL_QUANTA.with(|quanta| {
         quanta.borrow().as_ref().and_then(|quanta| {
             quanta
                 .get(&(precision, scale))
-                .map(|(exp, context)| (exp.bind(py).clone(), 
context.bind(py).clone()))
+                .map(|args| args.call_args(py, exact_c))
         })
     });
-    if let Some(cached) = cached {
-        return Ok(cached);
-    }
-    let module = py.import(intern!(py, "decimal"))?;
-    let exp = DECIMAL
-        .import(py, "decimal", "Decimal")?
-        .call1((format!("1E-{scale}"),))?;
-    let context = module.getattr(intern!(py, "Context"))?.call(
-        (),
-        Some(
-            &[
-                ("prec", precision.into_pyobject(py)?.into_any()),
-                ("rounding", module.getattr(intern!(py, "ROUND_HALF_EVEN"))?),
-            ]
-            .into_py_dict(py)?,
-        ),
-    )?;
-    DECIMAL_QUANTA.with(|quanta| {
-        if let Some(quanta) = quanta.borrow_mut().as_mut() {
-            quanta.insert(
-                (precision, scale),
-                (exp.clone().unbind(), context.clone().unbind()),
-            );
+    let cached = match cached {
+        Some(cached) => cached,
+        None => {
+            let module = py.import(intern!(py, "decimal"))?;
+            let exp = decimal.call1((format!("1E-{scale}"),))?;
+            // Read Context at construction time, so each conversion observes 
the current factory.
+            let context = module.getattr(intern!(py, "Context"))?.call(
+                (),
+                Some(
+                    &[
+                        ("prec", precision.into_pyobject(py)?.into_any()),
+                        ("rounding", module.getattr(intern!(py, 
"ROUND_HALF_EVEN"))?),
+                    ]
+                    .into_py_dict(py)?,
+                ),
+            )?;
+            let args = QuantizeArgs {
+                exp: exp.unbind(),
+                context: context.unbind(),
+                kwargs: None,
+            };
+            let cached = args.call_args(py, exact_c);
+            let discarded = DECIMAL_QUANTA.with(|quanta| match 
quanta.borrow_mut().as_mut() {
+                Some(quanta) => quanta.insert((precision, scale), args),
+                None => Some(args),
+            });
+            // Release displaced or uncached Python objects only after the 
borrow ends.
+            drop(discarded);
+            cached
         }
-    });
-    Ok((exp, context))
+    };
+    match cached {
+        QuantizeLookup::Call(args) => Ok(args),
+        QuantizeLookup::Keywords { exp, context } => {
+            // Dict allocation may run GC callbacks: release the cache borrow 
first, and
+            // share the context already used by positional calls of this 
Decimal type.
+            let kwargs = [("context", context)].into_py_dict(py)?;
+            DECIMAL_QUANTA.with(|quanta| {
+                if let Some(args) = quanta
+                    .borrow_mut()
+                    .as_mut()
+                    .and_then(|quanta| quanta.get_mut(&(precision, scale)))
+                {
+                    if args.kwargs.is_none() {
+                        args.kwargs = Some(kwargs.clone().unbind());
+                    }
+                }
+            });
+            Ok(QuantizeCallArgs::Keyword { exp, kwargs })
+        }
+    }
 }
 
 /// The session configurations that inference reads, as PySpark reads them for
@@ -308,7 +431,7 @@ pub(crate) fn infer_type(obj: &Bound<'_, PyAny>, o: 
Options<'_>) -> PyResult<Dat
         return Ok(DataType::Binary);
     }
     let ty = obj.get_type();
-    if ty.is(DECIMAL.import(py, "decimal", "Decimal")?) {
+    if ty.is(decimal_type(py)?) {
         // As PySpark: the precision and scale of a value may differ from row 
to row.
         return Ok(DataType::Decimal {
             precision: 38,
@@ -1128,7 +1251,7 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: 
&DataType, nullable: bool) ->
             Ok(Value::Timestamp(wall_clock_micros(obj)?))
         }
         DataType::Decimal { precision, scale } => {
-            let decimal = DECIMAL.import(py, "decimal", "Decimal")?;
+            let (c_decimal, decimal) = decimal_classes(py)?;
             if !obj.is_instance(decimal.as_any())? {
                 return Err(assertion());
             }
@@ -1138,10 +1261,17 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: 
&DataType, nullable: bool) ->
                 }
                 return Ok(Value::Null);
             }
-            // `value.quantize(Decimal("1E-<scale>"), 
context=Context(prec=<precision>,
-            // rounding=ROUND_HALF_EVEN))`, as written: the encoder parses 
either notation.
-            let (exp, context) = decimal_quantum(py, *precision, *scale)?;
-            let quantized = obj.call_method1(intern!(py, "quantize"), (exp, 
py.None(), context))?;
+            // Exact C Decimals accept positional context; other values keep 
PySpark's
+            // keyword call so subclass overrides and the pure-Python fallback 
retain it.
+            let exact_c = is_exact_c_decimal(obj, c_decimal);
+            let quantized = match decimal_quantum(py, decimal, *precision, 
*scale, exact_c)? {
+                QuantizeCallArgs::Positional { exp, context } => {
+                    obj.call_method1(intern!(py, "quantize"), (exp, py.None(), 
context))?
+                }
+                QuantizeCallArgs::Keyword { exp, kwargs } => {
+                    obj.call_method(intern!(py, "quantize"), (exp,), 
Some(&kwargs))?
+                }
+            };
             Ok(Value::Decimal {
                 value: quantized.str()?.extract()?,
                 precision: Some(*precision),
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 94aa92dd..773514e7 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -704,10 +704,12 @@ impl PySparkSession {
         let mut values: Vec<Vec<Value>> = (0..parts.len())
             .map(|_| Vec::with_capacity(fields.len()))
             .collect();
-        let _quanta = infer::DecimalQuanta::new();
-        for (i, field) in fields.iter().enumerate() {
-            for (row, out) in parts.iter().zip(values.iter_mut()) {
-                out.push(infer::convert(&row[i], &field.data_type, 
field.nullable)?);
+        {
+            let _quanta = infer::DecimalQuanta::new();
+            for (i, field) in fields.iter().enumerate() {
+                for (row, out) in parts.iter().zip(values.iter_mut()) {
+                    out.push(infer::convert(&row[i], &field.data_type, 
field.nullable)?);
+                }
             }
         }
         // Then, as PySpark, the schema converts to Arrow, which has no char, 
varchar, year-month or
@@ -1136,7 +1138,14 @@ impl PySparkSession {
                     let cells = || rows.iter().filter_map(|row| row.get(i));
                     let data_type = if !cells().any(|v| matches!(v, 
Value::Decimal { .. })) {
                         match first {
-                            Some(value) => value_to_datatype(value),
+                            Some(value) => {
+                                let mut data_type = value_to_datatype(value);
+                                if has_decimal_type(&data_type) {
+                                    let column: Vec<&Value> = 
cells().collect();
+                                    widen_pandas_decimal_scales(&mut 
data_type, &column)?;
+                                }
+                                data_type
+                            }
                             None => {
                                 
pandas_dtype_type(dtypes.get(i).map_or("object", String::as_str))
                             }
@@ -2546,18 +2555,20 @@ fn pandas_dtype_type(dtype: &str) -> DataType {
 /// The decimal type of column `i` of pandas rows, as Arrow infers it from all 
its Decimals:
 /// the most integer digits and the largest scale of the values.
 fn pandas_decimal_type(data: &Bound<'_, PyList>, rows: &[Row], i: usize) -> 
PyResult<DataType> {
-    // The digits and exponent of an exact Decimal come from the text that 
`py_to_value` took
-    // of it, rather than from `as_tuple()`, which builds a DecimalTuple of 
every digit; a
-    // subclass, which can override either, is asked for `as_tuple()`, as 
Arrow does.
-    let decimal_cls = crate::infer::DECIMAL.import(data.py(), "decimal", 
"Decimal")?;
+    // The digits and exponent of an exact C Decimal come from the text that 
`py_to_value`
+    // took of it, rather than from `as_tuple()`, which builds a DecimalTuple 
of every digit.
+    // Subclasses and pure-Python Decimals can override either, so keep their 
`as_tuple()`.
     let (mut integer_digits, mut scale) = (0i64, 0i64);
+    let c_decimal = crate::infer::c_decimal_type(data.py())?;
     for (row, converted) in data.iter().zip(rows) {
         let value = row.get_item(i)?;
         if value.is_none() {
             continue;
         }
         let from_text = match converted.get(i) {
-            Some(Value::Decimal { value: text, .. }) if 
value.get_type().is(decimal_cls) => {
+            Some(Value::Decimal { value: text, .. })
+                if crate::infer::is_exact_c_decimal(&value, c_decimal) =>
+            {
                 decimal_text_digits(text)
             }
             _ => None,
@@ -2597,6 +2608,140 @@ fn pandas_decimal_type(data: &Bound<'_, PyList>, rows: 
&[Row], i: usize) -> PyRe
     })
 }
 
+/// Whether a type contains Decimal, to prune unrelated nested inference 
subtrees.
+fn has_decimal_type(data_type: &DataType) -> bool {
+    match data_type {
+        DataType::Decimal { .. } => true,
+        DataType::Array { element_type, .. } => has_decimal_type(element_type),
+        DataType::Struct { fields } => fields
+            .iter()
+            .any(|field| has_decimal_type(&field.data_type)),
+        DataType::Map {
+            key_type,
+            value_type,
+            ..
+        } => has_decimal_type(key_type) || has_decimal_type(value_type),
+        _ => false,
+    }
+}
+
+/// Preserve the existing nested shape and precision, but use every observed 
Decimal's
+/// scale. Choosing only the first scale would make the exact encoder reject a 
later value
+/// such as 1.25 after 1.5. This pass uses already converted values and 
invokes no Python hooks.
+/// Callers prune types without Decimal before allocating child references.
+/// Raises `ValueError` when widening increases a scale beyond its precision; 
an
+/// unchanged, already-invalid first scale is left to the encoder.
+fn widen_pandas_decimal_scales(data_type: &mut DataType, values: &[&Value]) -> 
PyResult<()> {
+    match data_type {
+        DataType::Decimal { precision, scale } => {
+            let initial_scale = *scale;
+            for value in values {
+                if let Value::Decimal {
+                    scale: Some(observed),
+                    ..
+                } = value
+                {
+                    *scale = (*scale).max(*observed);
+                }
+            }
+            // Reject invalid types introduced by widening before the encoder 
can cast the
+            // scale to i8. Keep the inherited handling of an already-invalid 
first value.
+            if *scale > *precision && *scale > initial_scale {
+                return Err(pyo3::exceptions::PyValueError::new_err(format!(
+                    "Decimal scale is out of the range of DecimalType, up to 
{precision}"
+                )));
+            }
+        }
+        DataType::Array { element_type, .. } => {
+            let children = values
+                .iter()
+                .filter_map(|value| match value {
+                    Value::List(items) => Some(items.iter()),
+                    _ => None,
+                })
+                .flatten()
+                .collect::<Vec<_>>();
+            widen_pandas_decimal_scales(element_type, &children)?;
+        }
+        DataType::Struct { fields } => {
+            let in_order = values
+                .iter()
+                .map(|value| match value {
+                    Value::Struct(row) => {
+                        row.len() == fields.len()
+                            && row
+                                .fields()
+                                .iter()
+                                .zip(fields.iter())
+                                .all(|(name, field)| *name == field.name)
+                    }
+                    _ => false,
+                })
+                .collect::<Vec<_>>();
+            for (index, field) in fields.iter_mut().enumerate() {
+                if !has_decimal_type(&field.data_type) {
+                    continue;
+                }
+                let children = values
+                    .iter()
+                    .zip(&in_order)
+                    .filter_map(|(value, &in_order)| match value {
+                        Value::Struct(row) => {
+                            if in_order {
+                                row.get(index)
+                            } else {
+                                // Match the encoder: by field name, else by 
position,
+                                // so widening inspects the same value it will 
encode.
+                                row.get_by_name(&field.name).or_else(|| 
row.get(index))
+                            }
+                        }
+                        // An earlier Row can establish the Struct type while 
a later
+                        // dict in the same pandas column is converted to 
Value::Map.
+                        Value::Map(entries) => entries.iter().find_map(|(key, 
value)| match key {
+                            Value::String(name) if *name == field.name => 
Some(value),
+                            _ => None,
+                        }),
+                        _ => None,
+                    })
+                    .collect::<Vec<_>>();
+                widen_pandas_decimal_scales(&mut field.data_type, &children)?;
+            }
+        }
+        DataType::Map {
+            key_type,
+            value_type,
+            ..
+        } => {
+            if has_decimal_type(key_type) {
+                let keys = values
+                    .iter()
+                    .filter_map(|value| match value {
+                        Value::Map(entries) => Some(entries.iter()),
+                        _ => None,
+                    })
+                    .flatten()
+                    .map(|(key, _)| key)
+                    .collect::<Vec<_>>();
+                widen_pandas_decimal_scales(key_type, &keys)?;
+            }
+            if has_decimal_type(value_type) {
+                let children = values
+                    .iter()
+                    .filter_map(|value| match value {
+                        Value::Map(entries) => Some(entries.iter()),
+                        _ => None,
+                    })
+                    .flatten()
+                    .map(|(_, value)| value)
+                    .collect::<Vec<_>>();
+                widen_pandas_decimal_scales(value_type, &children)?;
+            }
+        }
+        _ => {}
+    }
+    Ok(())
+}
+
 /// Infer the Spark `DataType` for a `createDataFrame` column from its (first) 
value.
 /// Nested collections are inferred recursively; nulls and empty collections 
use
 /// String as a fallback.
@@ -2928,14 +3073,16 @@ pub(crate) fn datetime_to_micros(obj: &Bound<'_, 
PyAny>) -> PyResult<i64> {
     Ok((timestamp_f64 * 1_000_000.0).round() as i64)
 }
 
-/// If `obj` is a `decimal.Decimal`, return it as a `Value::Decimal` (string 
value + scale
-/// from its exponent); otherwise `None`. Checked before int/float in 
`py_to_value` because
+/// If `obj` is an instance of the cached Decimal class (C `_decimal.Decimal`, 
else the
+/// public fallback), return it as a `Value::Decimal` (string value + scale 
from its
+/// exponent); otherwise `None`. See the class-rebinding entry in
+/// `dev/parity/known-differences.md`. Checked before int/float in 
`py_to_value` because
 /// Decimal defines `__float__`/`__int__`.
 fn decimal_to_value(obj: &Bound<'_, PyAny>) -> PyResult<Option<Value>> {
     let py = obj.py();
-    // Imported once: looking the class up per value was a module import and 
an attribute
-    // read of every converted value.
-    let decimal_cls = crate::infer::DECIMAL.import(py, "decimal", "Decimal")?;
+    // Cache the accelerator class or its absence. Interpreters without it use 
the
+    // public Python class, whose exact instances retain the as_tuple hook 
path.
+    let (c_decimal, decimal_cls) = crate::infer::decimal_classes(py)?;
     if !obj.is_instance(decimal_cls.as_any())? {
         return Ok(None);
     }
@@ -2943,9 +3090,10 @@ fn decimal_to_value(obj: &Bound<'_, PyAny>) -> 
PyResult<Option<Value>> {
     // either notation, in work bounded by the length of the text, which a 
fixed-point form
     // of `1E+1000000000` would not be.
     let dec_str: String = obj.str()?.extract()?;
-    let scale = if obj.get_type().is(decimal_cls) {
-        // Exact Decimal strings preserve the coefficient's trailing zeros and 
exponent.
-        // Subclasses may override either __str__ or as_tuple, so keep their 
old path.
+    let scale = if crate::infer::is_exact_c_decimal(obj, c_decimal) {
+        // Exact C Decimal strings preserve the coefficient's trailing zeros 
and exponent.
+        // Subclasses and pure-Python Decimals can override __str__ or 
as_tuple, so keep
+        // their tuple hooks.
         decimal_string_scale(&dec_str)
     } else {
         let as_tuple = obj.call_method0("as_tuple")?;
@@ -3054,3 +3202,51 @@ fn value_out_of_range(int: &Bound<'_, PyAny>) -> PyErr {
     let shown = int.str().map(|text| text.to_string()).unwrap_or_default();
     pyo3::exceptions::PyValueError::new_err(format!("Value out of range: 
{shown}"))
 }
+
+#[cfg(test)]
+mod decimal_text_tests {
+    use super::{decimal_string_scale, decimal_text_digits};
+
+    #[test]
+    fn coefficient_digits_and_exponent_match_decimal_tuple() {
+        for (text, digits, exponent) in [
+            ("0", 1, 0),
+            ("0.000", 1, -3),
+            ("-0E-7", 1, -7),
+            ("1E+3", 1, 3),
+            ("1.0E+4", 2, 3),
+            ("123.4500", 7, -4),
+            ("0.001234500", 7, -9),
+            ("1.5E-7", 2, -8),
+        ] {
+            assert_eq!(
+                decimal_text_digits(text),
+                Some((digits, exponent)),
+                "{text}"
+            );
+        }
+    }
+
+    #[test]
+    fn scale_preserves_trailing_zeros_and_nonpositive_exponents() {
+        for (text, scale) in [
+            ("0", Some(0)),
+            ("0.000", Some(3)),
+            ("-0E-7", Some(7)),
+            ("123.4500", Some(4)),
+            ("1.5E-7", Some(8)),
+            ("1E+3", None),
+            ("1.0E+4", None),
+        ] {
+            assert_eq!(decimal_string_scale(text), scale, "{text}");
+        }
+    }
+
+    #[test]
+    fn nonfinite_values_have_no_numeric_exponent() {
+        for text in ["NaN", "sNaN", "Infinity", "-Infinity"] {
+            assert_eq!(decimal_text_digits(text), None, "{text}");
+            assert_eq!(decimal_string_scale(text), None, "{text}");
+        }
+    }
+}
diff --git a/crates/spark-connect/src/row.rs b/crates/spark-connect/src/row.rs
index b7e95597..06edcadd 100644
--- a/crates/spark-connect/src/row.rs
+++ b/crates/spark-connect/src/row.rs
@@ -42,7 +42,10 @@ pub enum Value {
     /// representable by Python's datetime.timedelta without nanosecond 
conversion.
     /// Display uses raw microseconds, like Date and Timestamp use their raw 
units.
     DayTimeInterval(i64),
-    /// Decimal (string value with optional precision and scale)
+    /// Decimal text in fixed-point or scientific notation, with optional 
precision and scale.
+    /// Text passed to `create_dataframe` must fit the declared precision and 
scale exactly.
+    /// Callers that rescale, such as Python local-row conversion, do so 
before building
+    /// this value.
     Decimal {
         value: String,
         precision: Option<i32>,
diff --git a/dev/parity/known-differences.md b/dev/parity/known-differences.md
index bfe9fed6..c6ae7a3d 100644
--- a/dev/parity/known-differences.md
+++ b/dev/parity/known-differences.md
@@ -12,6 +12,61 @@ difference is intentional or tracked for a fix.
   raises PyArrow's `ArrowInvalid`. This client raises `AXIS_LENGTH_MISMATCH` 
for both
   rather than drop data.
 - **NumPy ndarray input** (tracked by SPARK-60772). Not accepted yet.
+- **Nested pandas Decimal inference and failures** (intentional; related 
change:
+  SPARK-60696).
+
+  - **Inferred precision and scale.** On the native row fallback, inferred 
nested
+    Decimal fields keep precision 38 and the greatest observed scale; PySpark 
infers the
+    smallest precision that covers the values. Increasing scale can make a 
large value
+    exceed precision 38 even when the first value's scale could encode every 
value by
+    discarding only trailing zeros. For example, with `[Decimal("9" * 37 + 
".0")]`
+    followed by `[Decimal("1.0000000")]`, the first value would need 44 digits 
at scale
+    7, beyond precision 38; the previous native fallback accepted both at 
scale 1.
+    PySpark Connect 4.3.0 constructs a DataFrame for this example, but 
requesting its
+    schema or collecting results raises `ArithmeticException`
+    (`DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION`, precision 44, maximum 38). 
This client's
+    native fallback instead raises `PySparkRuntimeError` during construction, 
before
+    sending a relation. The rejection stage and error class differ.
+
+  - **Scale beyond 38.** A later value that increases the scale beyond 38 
raises
+    `ValueError` before upload. This can reject later zero or trailing-zero 
values that
+    the previous native fallback accepted at the first value's scale. Existing 
container
+    and non-Decimal inference rules are unchanged.
+
+  - **Explicit schema.** With an explicit schema, a nested Decimal that does 
not fit
+    raises `PySparkRuntimeError` only on the native row fallback, for frames 
rejected by
+    `accepts_pandas_table`. Ordinary frames use PySpark's Arrow conversion and 
raise its
+    errors. Both paths reject an explicit-schema value that does not fit 
before a
+    relation is sent.
+- **Multiple invalid pandas columns with an explicit Decimal field** 
(intentional,
+  SPARK-60696). On native row fallback, Decimal exactness is checked before 
the encoder
+  converts other fields. If an earlier non-Decimal column and a later Decimal 
column are
+  both invalid, this client reports the Decimal column first; PySpark reports 
the first
+  invalid column in schema order. Both reject the input before a relation is 
sent.
+- **Decimal context hooks during local-row conversion** (intentional, 
SPARK-60704).
+  Quantum and context objects are shared by Decimal type within one conversion 
and built
+  when its first non-null, non-NaN value is converted. PySpark constructs a 
context for
+  each field's converter before converting values. A custom hook that changes
+  `decimal.DefaultContext` or mutates its supplied context during a conversion 
can
+  observe different timing or sharing. Changes between `createDataFrame` calls 
apply to
+  the next conversion.
+- **Decimal class rebinding** (intentional, SPARK-60704). Native Decimal 
conversion and
+  inference for local rows and pandas input recognize the cached C 
`_decimal.Decimal`
+  class, falling back to the cached public `decimal.Decimal` when the 
accelerator is
+  unavailable; accelerator absence is cached too. If `_decimal` is importable 
at first
+  resolution, its class takes precedence even when the public 
`decimal.Decimal` is the
+  pure-Python class. Values of that pure-Python class are then not recognized 
as
+  Decimals: local-row Decimal conversion raises `AssertionError` and inference 
raises
+  `CANNOT_INFER_TYPE_FOR_FIELD`, while the native row fallback of pandas input 
(frames
+  that the Arrow path does not take) without a schema or with names only reads 
them as
+  doubles through `__float__`, potentially losing precision. Pandas frames on 
the Arrow
+  path convert them as PySpark does. Quantization also uses a quantum of the 
selected C
+  class, even when the public alias is rebound. The context factory and 
rounding mode
+  still come from the public `decimal` module at first use; mixing its 
pure-Python
+  context with explicit C Decimal values is unsupported. The text shortcut 
assumes
+  `_decimal.Decimal` is immutable, as on CPython; mutable implementations of 
that class
+  on other interpreters are not covered. This can differ from PySpark when an
+  application rebinds the public class alias during the process.
 
 ## `DataFrame`
 
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index de3e89e5..34f7a1f1 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -8322,7 +8322,10 @@ def decimal_ipc_protocol(tmp_path_factory):
     pytest.importorskip("pyarrow")
     protoc = shutil.which(os.environ.get("PROTOC", "protoc"))
     if protoc is None:
-        pytest.skip("Decimal IPC transport tests require protoc")
+        message = "Decimal IPC transport tests require protoc"
+        if os.environ.get("CI", "").strip().lower() in {"1", "true", "yes"}:
+            pytest.fail(message)
+        pytest.skip(message)
     proto_root = Path(__file__).resolve().parents[2] / 
"crates/spark-connect-proto/proto"
     descriptor = tmp_path_factory.mktemp("decimal-proto") / "connect.pb"
     command = [protoc, f"-I{proto_root}"]
@@ -8358,6 +8361,213 @@ def decimal_ipc_session(decimal_ipc_protocol, tmp_path):
             assert process.returncode == 0
 
 
+def 
test_decimal_missing_c_accelerator_is_cached_and_keeps_hooks(decimal_ipc_protocol,
 tmp_path):
+    # The extension's type cache belongs to the process, so reset decimal's 
modules and
+    # hide the accelerator before importing pyspark. Count only conversion 
lookups.
+    pytest.importorskip("pandas")
+    import subprocess
+    import sys
+
+    script = r'''
+import sys
+# Exercise startup hooks that preload decimal before configuring the fallback.
+import decimal
+sys.modules.pop("decimal", None)
+sys.modules.pop("_pydecimal", None)
+sys.modules["_decimal"] = None
+import decimal
+import _pydecimal
+assert decimal.Decimal is _pydecimal.Decimal
+import builtins
+import contextlib
+from pathlib import Path
+import subprocess
+import pandas as pd
+import pyarrow as pa
+import pytest
+from pyspark.sql import SparkSession
+from pyspark.sql import types as T
+
+class RowPathFrame(pd.DataFrame):
+    pass
+
[email protected]
+def session():
+    payload = Path(sys.argv[3]) / "input.arrow"
+    process = subprocess.Popen(
+        [sys.executable, "-u", "-c", sys.argv[1], sys.argv[2], str(payload)],
+        stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True,
+    )
+    spark = None
+    try:
+        port = int(process.stdout.readline())
+        spark = SparkSession.builder.remote(f"sc://127.0.0.1:{port}").create()
+        yield spark, payload
+    finally:
+        try:
+            if spark is not None:
+                spark.stop()
+        finally:
+            try:
+                process.communicate("stop\n", timeout=20)
+            except subprocess.TimeoutExpired:
+                process.kill()
+                process.communicate()
+                raise
+            assert process.returncode == 0
+
+original_import = builtins.__import__
+imports = []
+conversion_import_counts = []
+
+def tracked_import(name, *args, **kwargs):
+    if name == "_decimal":
+        imports.append(name)
+    return original_import(name, *args, **kwargs)
+
+def build(spark, data, schema=None):
+    before = len(imports)
+    builtins.__import__ = tracked_import
+    try:
+        return spark.createDataFrame(data, schema)
+    finally:
+        builtins.__import__ = original_import
+        conversion_import_counts.append(len(imports) - before)
+
+def assert_decimal_cells(payload, precision, scale, unscaled):
+    # Compare raw signed Decimal128 cells without constructing expected Arrow 
Decimals
+    # or relying on the client's result decoder to validate its own uploaded 
payload.
+    table = pa.ipc.open_stream(payload.read_bytes()).read_all()
+    table.validate(full=True)
+    assert table.schema == pa.schema([pa.field("v", pa.decimal128(precision, 
scale))])
+    column = table.column("v").combine_chunks()
+    assert column.is_valid().to_pylist() == [value is not None for value in 
unscaled]
+    cells = column.buffers()[1].to_pybytes()
+    assert len(column) == len(unscaled)
+    for index, value in enumerate(unscaled):
+        if value is not None:
+            assert cells[index * 16:(index + 1) * 16] == value.to_bytes(
+                16, "little", signed=True
+            )
+
+with session() as (spark, payload):
+    schema = T.StructType([T.StructField("v", T.DecimalType(10, 2))])
+    rows = [(decimal.Decimal("1.235"),), (None,), (decimal.Decimal("-2.345"),)]
+    build(spark, rows, schema).collect()
+    assert_decimal_cells(payload, 10, 2, [124, None, -234])
+
+    calls = []
+    class Traced(decimal.Decimal):
+        def __str__(self):
+            calls.append("str")
+            return super().__str__()
+
+        def as_tuple(self):
+            calls.append("tuple")
+            return super().as_tuple()
+
+    frame = RowPathFrame({"v": [Traced("1.20"), None, Traced("-30.5"),
+                              decimal.Decimal("1E+3")]}, dtype=object)
+    # Customized frames retain the native conversion path and its Decimal 
hooks.
+    build(spark, frame).collect()
+    assert calls == ["str", "tuple", "str", "tuple", "tuple", "tuple"], calls
+    assert_decimal_cells(payload, 6, 2, [120, None, -3050, 100000])
+
+    class Broken(decimal.Decimal):
+        def as_tuple(self):
+            raise LookupError("pure Decimal tuple hook")
+
+    failing = RowPathFrame({"v": [Broken("1.25")]}, dtype=object)
+    payload.unlink()
+    with pytest.raises(LookupError, match="^pure Decimal tuple hook$"):
+        build(spark, failing)
+    assert not payload.exists()
+    assert imports == ["_decimal"], imports
+    assert conversion_import_counts == [1, 0, 0], conversion_import_counts
+'''
+    try:
+        result = subprocess.run(
+            [sys.executable, "-c", script, _DECIMAL_IPC_SERVER,
+             str(decimal_ipc_protocol), str(tmp_path)],
+            capture_output=True, text=True, timeout=180,
+        )
+    except subprocess.TimeoutExpired as error:
+        output = []
+        for name, value in (("stdout", error.stdout), ("stderr", 
error.stderr)):
+            if isinstance(value, bytes):
+                value = value.decode(errors="replace")
+            output.append(f"{name}:\n{value or ''}")
+        pytest.fail(
+            "Decimal fallback child exceeded 180 seconds.\n" + 
"\n".join(output),
+            pytrace=False,
+        )
+    assert result.returncode == 0, result.stdout + result.stderr
+
+
+def test_decimal_public_pure_python_alias_native_pandas_uses_float():
+    # Type caches are process-local: set the public alias before the extension 
is loaded,
+    # while leaving the C accelerator available. Inspect the inline IPC 
without an RPC.
+    # This pins "Decimal class rebinding" in dev/parity/known-differences.md; 
update the
+    # expected lossy values if pure-Python alias values become recognized as 
Decimals.
+    import subprocess
+    import sys
+    from pathlib import Path
+
+    pytest.importorskip("_decimal")
+    pytest.importorskip("pandas")
+    pytest.importorskip("pyarrow")
+    script = r'''
+import sys
+import decimal
+import _decimal
+import _pydecimal
+assert "pyspark._pyspark" not in sys.modules
+assert _decimal.Decimal is not _pydecimal.Decimal
+decimal.Decimal = _pydecimal.Decimal
+import pandas as pd
+import pyarrow as pa
+assert "pyspark._pyspark" not in sys.modules
+sys.path.insert(0, sys.argv[1])
+from _mock_connect_server import parse_message
+from pyspark.sql import SparkSession
+assert decimal.Decimal is _pydecimal.Decimal
+
+class RowPathFrame(pd.DataFrame):
+    pass
+
+values = [_pydecimal.Decimal("1.0000000000000000000001"), None,
+          _pydecimal.Decimal("9007199254740993")]
+spark = SparkSession.builder.remote("sc://localhost:1").create()
+for schema in (None, ["v"]):
+    frame = RowPathFrame({"v": values}, dtype=object)
+    relation = parse_message(spark.createDataFrame(frame, 
schema)._relation_bytes())
+    while 18 in relation:
+        relation = parse_message(parse_message(relation[18][0])[1][0])
+    local = parse_message(relation[11][0])
+    actual = pa.ipc.open_stream(local[1][0]).read_all()
+    expected = pa.table({"v": pa.array([1.0, None, 9007199254740992.0], 
type=pa.float64())})
+    actual.validate(full=True)
+    assert actual.schema == expected.schema
+    assert actual.equals(expected)
+'''
+    try:
+        result = subprocess.run(
+            [sys.executable, "-c", script, 
str(Path(__file__).resolve().parent)],
+            capture_output=True, text=True, timeout=60,
+        )
+    except subprocess.TimeoutExpired as error:
+        output = []
+        for name, value in (("stdout", error.stdout), ("stderr", 
error.stderr)):
+            if isinstance(value, bytes):
+                value = value.decode(errors="replace")
+            output.append(f"{name}:\n{value or ''}")
+        pytest.fail(
+            "Decimal public-alias child exceeded 60 seconds.\n" + 
"\n".join(output),
+            pytrace=False,
+        )
+    assert result.returncode == 0, result.stdout + result.stderr
+
+
 @pytest.mark.parametrize("kind", ["tuples", "pandas"])
 @pytest.mark.parametrize("precision, scale, texts", [
     (18, 7, ["1E+3", "1E-7", "-1E-7", "+1.25E+2", "-1.25E-2", "0E+3", "-0E-7", 
None]),
@@ -8394,11 +8604,13 @@ def 
test_decimal_scientific_local_input_ipc(decimal_ipc_session, kind, precision
 
 @pytest.mark.parametrize("text", ["1E+1000000000", "-1E+1000000000"])
 def test_decimal_scientific_local_input_overflow(decimal_ipc_session, text):
-    from decimal import Decimal
+    from decimal import Decimal, InvalidOperation
 
     spark, payload = decimal_ipc_session
     schema = T.StructType([T.StructField("v", T.DecimalType(38, 0))])
-    with pytest.raises((ArithmeticError, RuntimeError)):
+    # Trap-mutating tests restore DefaultContext; its normal InvalidOperation 
trap
+    # makes quantize reject this overflow before any IPC relation is uploaded.
+    with pytest.raises(InvalidOperation):
         spark.createDataFrame([(Decimal(text),)], schema).collect()
     assert not payload.exists()
 
@@ -8463,6 +8675,7 @@ def test_decimal_rescaling_follows_pyspark(
     ["0.001234500", "-0.000002300", "0.000000", None],
     ["123", "-4", "0", None],
     ["1E+3", "1E-7", "-0E-7", "1.2300E-7", None],
+    ["1.0E+4", "-2.00E+3", "-0E-7", None],
 ])
 def test_decimal_scale_input_ipc(decimal_ipc_session, kind, schema_mode, 
texts):
     # The scale of a Decimal comes from its text, not from as_tuple(): the IPC 
holds the same
@@ -8535,6 +8748,67 @@ def 
test_decimal_scale_of_subclasses(decimal_ipc_session):
             spark.createDataFrame(pd.DataFrame({"v": [failing("1.5")]}, 
dtype=object), ["v"])
 
 
[email protected]("schema_mode", ["explicit", "names", "inferred"])
[email protected]("hook_behavior", ["error", "scale"])
+def test_decimal_scale_rebound_subclass_keeps_tuple_hook(
+    decimal_ipc_session, monkeypatch, schema_mode, hook_behavior
+):
+    # A subclass rebound as decimal.Decimal is still a subclass, not the 
immutable C type.
+    # Both conversion and inferred pandas typing must retain its as_tuple hook.
+    import decimal
+    from unittest.mock import Mock
+
+    import pyarrow as pa
+    from pyspark.sql import _pandas_input
+
+    pd = pytest.importorskip("pandas")
+    original = decimal.Decimal
+    calls = []
+    parts = original("123.45000").as_tuple()
+
+    class Rebound(original):
+        def __str__(self):
+            calls.append("str")
+            return "1.20"
+
+        def as_tuple(self):
+            calls.append("tuple")
+            if hook_behavior == "error":
+                raise LookupError("rebound Decimal tuple hook")
+            return parts
+
+    # Canonical-looking metadata must not make a Python heap type eligible 
either.
+    Rebound.__name__ = "Decimal"
+    Rebound.__module__ = "decimal"
+    frame = _RowPathFrame(pd)({"v": [Rebound("1.20"), None]}, dtype=object)
+    schema = T.StructType([T.StructField("v", T.DecimalType(10, 2))])
+    if schema_mode != "explicit":
+        schema = ["v"] if schema_mode == "names" else None
+    spark, payload = decimal_ipc_session
+    explicit_arrow = Mock(side_effect=AssertionError("native Decimal test 
reached explicit Arrow"))
+    inferred_arrow = Mock(side_effect=AssertionError("native Decimal test 
reached inferred Arrow"))
+    with monkeypatch.context() as alias:
+        alias.setattr(decimal, "Decimal", Rebound)
+        alias.setattr(_pandas_input, "prepare_pandas_table", explicit_arrow)
+        alias.setattr(_pandas_input, "prepare_inferred_pandas_table", 
inferred_arrow)
+        if hook_behavior == "error":
+            with pytest.raises(LookupError, match="^rebound Decimal tuple 
hook$"):
+                spark.createDataFrame(frame, schema)
+        else:
+            spark.createDataFrame(frame, schema).collect()
+    explicit_arrow.assert_not_called()
+    inferred_arrow.assert_not_called()
+    if hook_behavior == "error":
+        assert calls == ["str", "tuple"]
+        assert not payload.exists()
+    else:
+        assert calls == ["str", "tuple"] + ([] if schema_mode == "explicit" 
else ["tuple"])
+        table = pa.ipc.open_stream(payload.read_bytes()).read_all()
+        expected_type = pa.decimal128(10, 2) if schema_mode == "explicit" else 
pa.decimal128(8, 5)
+        assert table.schema.field("v").type == expected_type
+        assert table.column(0).to_pylist() == [original("1.20"), None]
+
+
 @pytest.mark.parametrize("schema", [["v"], T.StructType([T.StructField("v", 
T.DecimalType(10, 4))])])
 def test_decimal_scale_avoids_coefficient_tuple(decimal_ipc_session, schema):
     # Exact Decimals are typed and converted from their text: no DecimalTuple 
is built.
@@ -8556,6 +8830,32 @@ def 
test_decimal_scale_avoids_coefficient_tuple(decimal_ipc_session, schema):
     assert not any(entry.code is tuple_constructor for entry in 
profile.getstats())
 
 
+def 
test_decimal_explicit_pandas_schema_uses_arrow_conversion(decimal_ipc_session, 
monkeypatch):
+    from decimal import Decimal
+    import pyarrow as pa
+    from pyspark.sql import _pandas_input
+
+    pd = pytest.importorskip("pandas")
+    spark, payload = decimal_ipc_session
+    prepare = _pandas_input.prepare_pandas_table
+    calls = []
+
+    def prepare_table(*args):
+        calls.append(True)
+        return prepare(*args)
+
+    monkeypatch.setattr(_pandas_input, "prepare_pandas_table", prepare_table)
+    values = [Decimal("1.20"), None, Decimal("-30.5")]
+    schema = T.StructType([T.StructField("v", T.DecimalType(10, 2))])
+    spark.createDataFrame(pd.DataFrame({"v": values}, dtype=object), 
schema).collect()
+    assert calls == [True]
+    table = pa.ipc.open_stream(payload.read_bytes()).read_all()
+    expected = pa.table({"v": pa.array(values, type=pa.decimal128(10, 2))})
+    table.validate(full=True)
+    assert table.schema == expected.schema
+    assert table.equals(expected)
+
+
 @pytest.mark.parametrize("text", ["Infinity", "-Infinity"])
 def test_decimal_scale_of_non_finite_values(decimal_ipc_session, text):
     # An infinity has no precision to type a column by.
@@ -8570,7 +8870,7 @@ def 
test_decimal_scale_of_non_finite_values(decimal_ipc_session, text):
     assert not payload.exists()
 
 
-def test_decimal_quanta_of_each_type_and_conversion(decimal_ipc_session):
+def test_decimal_quanta_of_each_type_and_conversion(decimal_ipc_session, 
monkeypatch):
     # Each decimal type of a conversion is quantized with its own quantum and 
context, which
     # are built once per conversion, as PySpark builds them once per 
converter: a change of
     # decimal.DefaultContext applies from the next createDataFrame.
@@ -8579,25 +8879,221 @@ def 
test_decimal_quanta_of_each_type_and_conversion(decimal_ipc_session):
     import pyarrow as pa
 
     spark, payload = decimal_ipc_session
-    schema = T.StructType([T.StructField("a", T.DecimalType(10, 2)),
-                           T.StructField("b", T.DecimalType(5, 1)),
-                           T.StructField("c", T.DecimalType(10, 2))])
+    schema = T.StructType([
+        T.StructField("a", T.DecimalType(10, 2)),
+        T.StructField("b", T.DecimalType(5, 1)),
+        T.StructField("c", T.DecimalType(10, 2)),
+    ])
+    context_type = decimal.Context
+    constructions = []
+
+    def context(*args, **kwargs):
+        constructions.append(kwargs["prec"])
+        return context_type(*args, **kwargs)
+
+    monkeypatch.setattr(decimal, "Context", context)
     rows = [(Decimal("1.235"), Decimal("2.25"), Decimal("-0.005")),
             (Decimal("1.245"), Decimal("2.35"), None)]
     spark.createDataFrame(rows, schema).collect()
+    assert constructions == [10, 5]
     table = pa.ipc.open_stream(payload.read_bytes()).read_all()
     assert table.to_pydict() == {
         "a": [Decimal("1.24"), Decimal("1.24")],
         "b": [Decimal("2.2"), Decimal("2.4")],
         "c": [Decimal("-0.00"), None],
     }
+    previous_trap = decimal.DefaultContext.traps[decimal.Inexact]
     decimal.DefaultContext.traps[decimal.Inexact] = True
     try:
         with pytest.raises(decimal.Inexact):
             spark.createDataFrame(rows, schema)
     finally:
-        decimal.DefaultContext.traps[decimal.Inexact] = False
+        decimal.DefaultContext.traps[decimal.Inexact] = previous_trap
+    assert constructions == [10, 5, 10]
     spark.createDataFrame(rows[1:], schema)
+    assert constructions == [10, 5, 10, 10, 5]
+
+
[email protected]("prefix_exact", [False, True])
+def test_decimal_quanta_preserve_keyword_quantize_overrides(
+    decimal_ipc_session, monkeypatch, prefix_exact
+):
+    import decimal
+    import pyarrow as pa
+
+    spark, payload = decimal_ipc_session
+    schema = T.StructType([T.StructField("v", T.DecimalType(10, 2))])
+    calls = []
+    contexts = []
+    context_type = decimal.Context
+    constructions = []
+
+    def context(*args, **kwargs):
+        created = context_type(*args, **kwargs)
+        constructions.append(created)
+        return created
+
+    monkeypatch.setattr(decimal, "Context", context)
+
+    class KeywordContext(decimal.Decimal):
+        # PySpark passes context by keyword; this valid override has no 
rounding argument.
+        def quantize(self, exp, context=None):
+            calls.append((str(exp), context.prec, context.rounding))
+            contexts.append(context)
+            return super().quantize(exp, context=context)
+
+    rows = [(KeywordContext("1.235"),), (KeywordContext("1.245"),)]
+    if prefix_exact:
+        # Build the positional cache first: its later lazy keyword dict must 
share the
+        # existing context rather than creating one at the first subclass 
value.
+        rows.insert(0, (decimal.Decimal("1.235"),))
+    spark.createDataFrame(rows, schema).collect()
+    assert calls == [("0.01", 10, decimal.ROUND_HALF_EVEN)] * 2
+    assert len(constructions) == 1
+    assert all(value is constructions[0] for value in contexts)
+    assert 
pa.ipc.open_stream(payload.read_bytes()).read_all().column(0).to_pylist() == [
+        decimal.Decimal("1.24"),
+    ] * len(rows)
+
+
+def 
test_decimal_quanta_restore_outer_conversion_after_reentry(decimal_ipc_session):
+    import decimal
+    import pyarrow as pa
+
+    spark, payload = decimal_ipc_session
+    schema = T.StructType([T.StructField("v", T.DecimalType(10, 2))])
+    previous_trap = decimal.DefaultContext.traps[decimal.Inexact]
+    calls = []
+
+    class Reentrant(decimal.Decimal):
+        def quantize(self, exp, rounding=None, context=None):
+            calls.append(context.traps[decimal.Inexact])
+            if len(calls) == 1:
+                spark.createDataFrame([(decimal.Decimal("1.23"),)], 
schema).collect()
+                decimal.DefaultContext.traps[decimal.Inexact] = True
+            return super().quantize(exp, rounding, context)
+
+    try:
+        decimal.DefaultContext.traps[decimal.Inexact] = False
+        rows = [(Reentrant("1.235"),), (Reentrant("1.245"),)]
+        spark.createDataFrame(rows, schema).collect()
+    finally:
+        decimal.DefaultContext.traps[decimal.Inexact] = previous_trap
+    assert calls == [False, False]
+    assert 
pa.ipc.open_stream(payload.read_bytes()).read_all().column(0).to_pylist() == [
+        decimal.Decimal("1.24"), decimal.Decimal("1.24"),
+    ]
+
+
[email protected]("schema_kind", ["inferred", "names"])
[email protected]("layout", [
+    "array", "nested_array", "struct", "mixed_struct", "array_struct", 
"map_values", "map_keys",
+])
+def test_pandas_nested_decimals_use_every_observed_scale(mock_connect, layout, 
schema_kind):
+    from decimal import Decimal
+    import pyarrow as pa
+
+    decimals = [Decimal("1.5"), Decimal("1.25"), Decimal("0.003"), 
Decimal("1E+3")]
+    # Nested inferred precision remains 38 (SPARK-60696; see 
dev/parity/known-differences.md).
+    # Widen only the scale to preserve all values without changing the 
inherited container
+    # or non-Decimal type inference.
+    decimal_type = pa.decimal128(38, 3)
+    if layout == "array":
+        values = [[decimals[0], decimals[1]], None, [], decimals[2:]]
+        expected_values = values
+        expected_type = pa.list_(decimal_type)
+    elif layout == "nested_array":
+        values = [[[decimals[0]], []], None, [[decimals[1]], decimals[2:]]]
+        expected_values = values
+        expected_type = pa.list_(pa.list_(decimal_type))
+    elif layout in ("struct", "mixed_struct", "array_struct"):
+        values = [Row(d=value, keep=index) for index, value in 
enumerate(decimals)]
+        expected_values = [{"d": value, "keep": index} for index, value in 
enumerate(decimals)]
+        expected_type = pa.struct([pa.field("d", decimal_type), 
pa.field("keep", pa.int64())])
+        if layout == "mixed_struct":
+            values = [values[0], *expected_values[1:]]
+        if layout == "array_struct":
+            values = [[values[0]], None, values[1:]]
+            expected_values = [[expected_values[0]], None, expected_values[1:]]
+            expected_type = pa.list_(expected_type)
+    elif layout == "map_values":
+        values = [
+            {"a": decimals[0]}, None, {"a": decimals[1], "b": decimals[2], 
"c": decimals[3]},
+        ]
+        expected_values = values
+        expected_type = pa.map_(pa.string(), decimal_type)
+    else:
+        values = [
+            {decimals[0]: "a"}, None, {decimals[1]: "b", decimals[2]: "c", 
decimals[3]: "d"},
+        ]
+        expected_values = values
+        expected_type = pa.map_(decimal_type, pa.string())
+    names = ["renamed"] if schema_kind == "names" else None
+    actual = _native_number_input_table(
+        mock_connect, [(value,) for value in values], names, source="pandas"
+    )
+    name = "renamed" if names else "value"
+    expected = pa.table({name: pa.array(expected_values, type=expected_type)})
+    actual.validate(full=True)
+    assert actual.schema == expected.schema
+    assert actual.equals(expected)
+
+
[email protected]("scale", [38, 39, 50, 128, 5_000_000_000])
+def test_pandas_nested_decimal_scale_bounds(mock_connect, scale):
+    from decimal import Decimal
+    import pyarrow as pa
+
+    spark, service = mock_connect
+    values = [[Decimal("0.1")], [Decimal(f"1E-{scale}")]]
+    if scale > 38:
+        pd = pytest.importorskip("pandas")
+        frame = _RowPathFrame(pd)({"value": values}, dtype=object)
+        with pytest.raises(
+            ValueError,
+            match=r"^Decimal scale is out of the range of DecimalType, up to 
38$",
+        ):
+            spark.createDataFrame(frame)
+        assert not service.execute_requests
+    else:
+        actual = _native_number_input_table(
+            mock_connect, [(value,) for value in values], source="pandas"
+        )
+        expected = pa.table({"value": pa.array(values, 
type=pa.list_(pa.decimal128(38, 38)))})
+        actual.validate(full=True)
+        assert actual.schema == expected.schema
+        assert actual.equals(expected)
+
+
+def test_pandas_nested_decimal_trailing_zeros_preserve_values(mock_connect):
+    from decimal import Decimal
+    import pyarrow as pa
+
+    # The second value's trailing zeros increase the scale even though it 
represents 1.
+    values = [[Decimal("9" * 30 + ".0")], [Decimal("1.0000000")]]
+    actual = _native_number_input_table(
+        mock_connect, [(value,) for value in values], source="pandas"
+    )
+    expected = pa.table({"value": pa.array(values, 
type=pa.list_(pa.decimal128(38, 7)))})
+    actual.validate(full=True)
+    assert actual.schema == expected.schema
+    assert actual.equals(expected)
+
+
+def 
test_pandas_nested_decimal_scale_widening_can_overflow_precision(decimal_ipc_session):
+    from decimal import Decimal
+    from pyspark.errors import PySparkRuntimeError
+
+    pd = pytest.importorskip("pandas")
+    spark, payload = decimal_ipc_session
+    # With only the first scale (1), both values fit: the second discards six 
zero digits.
+    # Preserving every observed scale (7) requires 44 digits for the first 
value, beyond
+    # the native fallback's inherited precision of 38. See 
known-differences.md.
+    values = [[Decimal("9" * 37 + ".0")], [Decimal("1.0000000")]]
+    frame = _RowPathFrame(pd)({"v": values}, dtype=object)
+    with pytest.raises(PySparkRuntimeError, match=r"does not fit 
decimal\(38,7\) exactly"):
+        spark.createDataFrame(frame)
+    assert not payload.exists()
 
 
 # --------------------------------------------------------------------------- 
schema cache


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to