This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 5861b84 [SPARK-60704][CONNECT] Look up the Decimal type once, and
build the quantum and context of each decimal type once per conversion
5861b84 is described below
commit 5861b849d037ff4bc4615176c21db7c097a815b4
Author: DB Tsai <[email protected]>
AuthorDate: Sat Nov 7 12:26:33 2026 +0900
[SPARK-60704][CONNECT] Look up the Decimal type once, and build the quantum
and context of each decimal type once per conversion
### What changes were proposed in this pull request?
Stacked on #133 (SPARK-60697) and #132 (SPARK-60696); until they merge,
this diff includes them.
The Decimal conversions of `createDataFrame` resolve what they need once
rather than for every value:
- `decimal_to_value` (pandas cells and other generic values) and the pandas
Decimal type inference take the `decimal.Decimal` class from the cache that
`infer` already keeps, rather than importing `decimal` and reading its
attribute per value.
- The local-row conversion builds the quantum (`Decimal("1E-<scale>")`) and
the context (`Context(prec=<precision>, rounding=ROUND_HALF_EVEN)`) of each
decimal type once per conversion, as PySpark builds them once per converter,
rather than for every value. They are kept only while a conversion runs
(`DecimalQuanta`), so a change of `decimal.DefaultContext` applies from the
next `createDataFrame`, as in PySpark. The quantized value is sent as written,
which the encoder parses in eithe [...]
This replaces the original change, which ran a second pass over every
pandas cell behind checks of the `decimal` and `_decimal` modules and of the
type's flags: the cached class gives the same saving to every path without them.
### Why are the changes needed?
[SPARK-60704](https://issues.apache.org/jira/browse/SPARK-60704): the
per-value lookups and constructions dominated the conversion of Decimal inputs.
Measured on macOS against a local Spark 4.3.0 server, as the client CPU time of
`createDataFrame` of 200,000 rows of two Decimal columns (median of 5):
| Input | #133 | This change | PySpark 4.3.0 |
| --- | --- | --- | --- |
| Local rows, `decimal(12,4), decimal(10,2)` | 917 ms | 254 ms | 332 ms |
| Local rows, column names (inferred) | 1,624 ms | 894 ms | 2,590 ms |
| pandas, `decimal(12,4), decimal(10,2)` | 627 ms | 441 ms | 194 ms |
| pandas, column names (inferred) | 634 ms | 465 ms | 632 ms |
All give the same sums of both columns.
### Does this PR introduce _any_ user-facing change?
No, other than speed.
### How was this patch tested?
- A new offline test in `python/tests/test_dropin_offline.py`, against
#132's mock server that records the uploaded Arrow IPC: each decimal type of
one conversion is quantized half-even to its own scale, and a trap set in
`decimal.DefaultContext` applies to the next `createDataFrame` and not after it
is reset, as with PySpark 4.3.0.
- The Decimal tests of the offline suite (78 passed, 1 skipped); the full
suites run before the merge.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); the rework on the current master by
Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #136 from dbtsai/codex/spark-60704-decimal-lookup.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Co-authored-by: DB Tsai <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/infer.rs | 94 ++++++++++++++++++++++++++++---------
crates/pyspark-rs/src/session.rs | 17 ++++---
python/tests/test_dropin_offline.py | 30 ++++++++++++
3 files changed, 110 insertions(+), 31 deletions(-)
diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 5ae84ca..f62bd61 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -24,7 +24,7 @@ use crate::errors::ResultExt;
use crate::row::PyRow;
use crate::values::{PyGeography, PyGeometry, PyVariantVal};
-static DECIMAL: PyOnceLock<Py<PyType>> = PyOnceLock::new();
+pub(crate) static DECIMAL: PyOnceLock<Py<PyType>> = PyOnceLock::new();
static DATE: PyOnceLock<Py<PyType>> = PyOnceLock::new();
static DATETIME: PyOnceLock<Py<PyType>> = PyOnceLock::new();
static TIME: PyOnceLock<Py<PyType>> = PyOnceLock::new();
@@ -32,6 +32,73 @@ static TIMEDELTA: PyOnceLock<Py<PyType>> = PyOnceLock::new();
static ARRAY: PyOnceLock<Py<PyType>> = PyOnceLock::new();
static SORTED: PyOnceLock<Py<PyAny>> = PyOnceLock::new();
+thread_local! {
+ /// The quantum and the context of each decimal type, while a
[`DecimalQuanta`] lives: PySpark
+ /// builds them once for the converter of a field, rather than once per
value.
+ static DECIMAL_QUANTA: std::cell::RefCell<Option<HashMap<(i32, i32),
(Py<PyAny>, Py<PyAny>)>>> =
+ const { std::cell::RefCell::new(None) };
+}
+
+/// While alive, the conversions of this thread build the quantum and the
context of each decimal
+/// type once. Each conversion of rows has its own, as each `createDataFrame`
of PySpark has its
+/// converters, so that its contexts take the `decimal.DefaultContext` of
their time.
+pub(crate) struct DecimalQuanta;
+
+impl DecimalQuanta {
+ pub(crate) fn new() -> Self {
+ DECIMAL_QUANTA.with(|quanta| *quanta.borrow_mut() =
Some(HashMap::new()));
+ DecimalQuanta
+ }
+}
+
+impl Drop for DecimalQuanta {
+ fn drop(&mut self) {
+ DECIMAL_QUANTA.with(|quanta| quanta.borrow_mut().take());
+ }
+}
+
+/// `Decimal("1E-<scale>")` and `Context(prec=<precision>,
rounding=ROUND_HALF_EVEN)`, with which
+/// PySpark quantizes a value of `decimal(precision, scale)`.
+fn decimal_quantum<'py>(
+ py: Python<'py>,
+ precision: i32,
+ scale: i32,
+) -> PyResult<(Bound<'py, PyAny>, Bound<'py, PyAny>)> {
+ let cached = DECIMAL_QUANTA.with(|quanta| {
+ quanta.borrow().as_ref().and_then(|quanta| {
+ quanta
+ .get(&(precision, scale))
+ .map(|(exp, context)| (exp.bind(py).clone(),
context.bind(py).clone()))
+ })
+ });
+ if let Some(cached) = cached {
+ return Ok(cached);
+ }
+ let module = py.import(intern!(py, "decimal"))?;
+ let exp = DECIMAL
+ .import(py, "decimal", "Decimal")?
+ .call1((format!("1E-{scale}"),))?;
+ let context = module.getattr(intern!(py, "Context"))?.call(
+ (),
+ Some(
+ &[
+ ("prec", precision.into_pyobject(py)?.into_any()),
+ ("rounding", module.getattr(intern!(py, "ROUND_HALF_EVEN"))?),
+ ]
+ .into_py_dict(py)?,
+ ),
+ )?;
+ DECIMAL_QUANTA.with(|quanta| {
+ if let Some(quanta) = quanta.borrow_mut().as_mut() {
+ quanta.insert(
+ (precision, scale),
+ (exp.clone().unbind(), context.clone().unbind()),
+ );
+ }
+ });
+ Ok((exp, context))
+}
+
/// The session configurations that inference reads, as PySpark reads them for
/// `createDataFrame`.
struct Confs {
@@ -954,28 +1021,11 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt:
&DataType, nullable: bool) ->
return Ok(Value::Null);
}
// `value.quantize(Decimal("1E-<scale>"),
context=Context(prec=<precision>,
- // rounding=ROUND_HALF_EVEN))`, written without an exponent.
- let module = py.import(intern!(py, "decimal"))?;
- let exp = decimal.call1((format!("1E-{scale}"),))?;
- let context = module.getattr(intern!(py, "Context"))?.call(
- (),
- Some(
- &[
- ("prec", precision.into_pyobject(py)?.into_any()),
- ("rounding", module.getattr(intern!(py,
"ROUND_HALF_EVEN"))?),
- ]
- .into_py_dict(py)?,
- ),
- )?;
- let quantized = obj.call_method(
- intern!(py, "quantize"),
- (exp,),
- Some(&[("context", context)].into_py_dict(py)?),
- )?;
- let text =
- PyString::new(py, "{:f}").call_method1(intern!(py, "format"),
(quantized,))?;
+ // rounding=ROUND_HALF_EVEN))`, as written: the encoder parses
either notation.
+ let (exp, context) = decimal_quantum(py, *precision, *scale)?;
+ let quantized = obj.call_method1(intern!(py, "quantize"), (exp,
py.None(), context))?;
Ok(Value::Decimal {
- value: text.extract()?,
+ value: quantized.str()?.extract()?,
precision: Some(*precision),
scale: Some(*scale),
})
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 4ce35d1..2b05a00 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -509,6 +509,7 @@ impl PySparkSession {
let mut values: Vec<Vec<Value>> = (0..parts.len())
.map(|_| Vec::with_capacity(fields.len()))
.collect();
+ let _quanta = infer::DecimalQuanta::new();
for (i, field) in fields.iter().enumerate() {
for (row, out) in parts.iter().zip(values.iter_mut()) {
out.push(infer::convert(&row[i], &field.data_type,
field.nullable)?);
@@ -1752,7 +1753,7 @@ fn pandas_decimal_type(data: &Bound<'_, PyList>, rows:
&[Row], i: usize) -> PyRe
// The digits and exponent of an exact Decimal come from the text that
`py_to_value` took
// of it, rather than from `as_tuple()`, which builds a DecimalTuple of
every digit; a
// subclass, which can override either, is asked for `as_tuple()`, as
Arrow does.
- let decimal_cls = data.py().import("decimal")?.getattr("Decimal")?;
+ let decimal_cls = crate::infer::DECIMAL.import(data.py(), "decimal",
"Decimal")?;
let (mut integer_digits, mut scale) = (0i64, 0i64);
for (row, converted) in data.iter().zip(rows) {
let value = row.get_item(i)?;
@@ -1760,7 +1761,7 @@ fn pandas_decimal_type(data: &Bound<'_, PyList>, rows:
&[Row], i: usize) -> PyRe
continue;
}
let from_text = match converted.get(i) {
- Some(Value::Decimal { value: text, .. }) if
value.get_type().is(&decimal_cls) => {
+ Some(Value::Decimal { value: text, .. }) if
value.get_type().is(decimal_cls) => {
decimal_text_digits(text)
}
_ => None,
@@ -2131,19 +2132,17 @@ pub(crate) fn datetime_to_micros(obj: &Bound<'_,
PyAny>) -> PyResult<i64> {
/// Decimal defines `__float__`/`__int__`.
fn decimal_to_value(obj: &Bound<'_, PyAny>) -> PyResult<Option<Value>> {
let py = obj.py();
- let dec_mod = match py.import("decimal") {
- Ok(m) => m,
- Err(_) => return Ok(None),
- };
- let decimal_cls = dec_mod.getattr("Decimal")?;
- if !obj.is_instance(&decimal_cls)? {
+ // Imported once: looking the class up per value was a module import and
an attribute
+ // read of every converted value.
+ let decimal_cls = crate::infer::DECIMAL.import(py, "decimal", "Decimal")?;
+ if !obj.is_instance(decimal_cls.as_any())? {
return Ok(None);
}
// As written, in scientific notation for `1E+2` or `1.5E-7`: the Arrow
encoder parses
// either notation, in work bounded by the length of the text, which a
fixed-point form
// of `1E+1000000000` would not be.
let dec_str: String = obj.str()?.extract()?;
- let scale = if obj.get_type().is(&decimal_cls) {
+ let scale = if obj.get_type().is(decimal_cls) {
// Exact Decimal strings preserve the coefficient's trailing zeros and
exponent.
// Subclasses may override either __str__ or as_tuple, so keep their
old path.
decimal_string_scale(&dec_str)
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 8d68cc0..c3b18d0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6429,6 +6429,36 @@ def
test_decimal_scale_of_non_finite_values(decimal_ipc_session, text):
assert not payload.exists()
+def test_decimal_quanta_of_each_type_and_conversion(decimal_ipc_session):
+ # Each decimal type of a conversion is quantized with its own quantum and
context, which
+ # are built once per conversion, as PySpark builds them once per
converter: a change of
+ # decimal.DefaultContext applies from the next createDataFrame.
+ import decimal
+ from decimal import Decimal
+ import pyarrow as pa
+
+ spark, payload = decimal_ipc_session
+ schema = T.StructType([T.StructField("a", T.DecimalType(10, 2)),
+ T.StructField("b", T.DecimalType(5, 1)),
+ T.StructField("c", T.DecimalType(10, 2))])
+ rows = [(Decimal("1.235"), Decimal("2.25"), Decimal("-0.005")),
+ (Decimal("1.245"), Decimal("2.35"), None)]
+ spark.createDataFrame(rows, schema).collect()
+ table = pa.ipc.open_stream(payload.read_bytes()).read_all()
+ assert table.to_pydict() == {
+ "a": [Decimal("1.24"), Decimal("1.24")],
+ "b": [Decimal("2.2"), Decimal("2.4")],
+ "c": [Decimal("-0.00"), None],
+ }
+ decimal.DefaultContext.traps[decimal.Inexact] = True
+ try:
+ with pytest.raises(decimal.Inexact):
+ spark.createDataFrame(rows, schema)
+ finally:
+ decimal.DefaultContext.traps[decimal.Inexact] = False
+ spark.createDataFrame(rows[1:], schema)
+
+
# ---------------------------------------------------------------------------
schema cache
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]