This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 62080cb [SPARK-60696][CONNECT] Convert Decimal inputs exactly in
either notation, and fail pandas Decimals that do not fit as PySpark does
62080cb is described below
commit 62080cbe5ac168a112dea656134c580b9ae4df03
Author: DB Tsai <[email protected]>
AuthorDate: Sat Nov 7 09:25:14 2026 +0900
[SPARK-60696][CONNECT] Convert Decimal inputs exactly in either notation,
and fail pandas Decimals that do not fit as PySpark does
### What changes were proposed in this pull request?
Decimal inputs of `createDataFrame` convert exactly in either notation, as
PySpark 4.3.0 does:
- The Arrow encoder (`decimal_str_to_unscaled` in
`crates/spark-connect/src/session.rs`) parses fixed-point (`-1.50`) and
scientific (`1.5E-7`) text alike, in work bounded by the length of the text
rather than by its exponent, and requires the value to fit `decimal(precision,
scale)` exactly: a nonzero digit beyond the scale, or a value beyond the
precision, is an error for either notation. Text without digits (`""`, `"."`,
`"-"`) and decimal types that Spark does not have (a negative [...]
- The Python binding sends `str()` of a Decimal again instead of its
fixed-point form, so `Decimal("1E+1000000000")` is not written out in a billion
digits first.
- A pandas column whose Decimal does not fit its field fails as PySpark's
Arrow conversion does, with `PySparkValueError: Exception thrown when
converting pandas.Series (object) with name 'v' to Arrow Array (decimal128(18,
7)).` and the sentence that
`spark.sql.execution.pandas.convertToArrowArraySafely` adds, which is read only
to raise. Local rows keep the half-even quantization to the scale of the
Decimal conversion, as in PySpark.
### Why are the changes needed?
[SPARK-60696](https://issues.apache.org/jira/browse/SPARK-60696): the
encoder truncated the fixed-point text of pandas Decimals to the scale of the
field, so data was silently lost where PySpark raises, and a value beyond the
precision reached the server. Against a local Spark 4.3.0 server, for a pandas
object column with an explicit `decimal(18, 7)` (or `decimal(10, 2)`) field:
| Value | Current master | PySpark 4.3.0 and this change |
| --- | --- | --- |
| `1.5E-7` | `1E-7` | `PySparkValueError` |
| `1E-8` | `0E-7` | `PySparkValueError` |
| `1.23456789` | `1.2345678` | `PySparkValueError` |
| `1.235` into `decimal(10, 2)` | `1.23` | `PySparkValueError` |
| `100000000000` | server `NUMERIC_VALUE_OUT_OF_RANGE` |
`PySparkValueError` |
| `1E+100` | `PySparkRuntimeError: invalid decimal` | `PySparkValueError` |
The same comparison of 30 pandas and local-row cases (rounding, overflow,
signed zeros, NaN, huge exponents) gives identical results with this change and
with PySpark 4.3.0.
### Does this PR introduce _any_ user-facing change?
Yes. A pandas Decimal that does not fit its field raises
`PySparkValueError`, as in PySpark, instead of losing digits. Rust callers that
pass `Value::Decimal` text get the same exactness check, and scientific
notation is accepted.
### How was this patch tested?
- Unit tests of the parser: both notations, trailing zeros, leading zeros,
signs, whitespace, the largest values of each precision, exponents up to
`i64::MIN`/`i64::MAX`, inexact and out-of-precision values, malformed text and
unsupported types.
- Offline tests in `python/tests/test_dropin_offline.py`, against a mock
server that records the uploaded Arrow IPC: exact scientific and fixed-point
inputs give the IPC cells pyarrow builds, local rows round half-even as
PySpark, and inexact or overflowing pandas Decimals raise PySpark's error, with
its message, before anything is sent.
- Against the current master and a local Spark 4.3.0 server: the lib and
every integration test of `apache-spark-connect` and
`apache-spark-connect-core`, e2e included (818 passed), the build of all tests
with `wasm-udf`, the offline Python suites (1,840 passed, 8 skipped), the live
drop-in and Arrow suites (109 passed) and the live cases (10 passed). `cargo
fmt --all --check` passes.
Changes on top of the original commits: the merge with master, which
formats Decimals in fixed point; one exactness rule for both notations, where
the original kept truncating fixed point; sending `str()` again; and PySpark's
error for pandas columns. An inexact Decimal nested in an array or struct of a
pandas cell also fails now, but with the encoder's `PySparkRuntimeError` rather
than PySpark's `PySparkValueError`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); the rework on the current master by
Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #132 from dbtsai/codex/spark-60696-decimal-exponents.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Co-authored-by: DB Tsai <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/session.rs | 51 +++++++-
crates/spark-connect/src/session.rs | 244 ++++++++++++++++++++++++++++++++----
python/tests/test_dropin_offline.py | 232 ++++++++++++++++++++++++++++++++++
3 files changed, 496 insertions(+), 31 deletions(-)
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index a6b297d..d17c8c3 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1,7 +1,7 @@
//! PyO3 wrapper for Spark Connect sessions.
use pyo3::prelude::*;
-use pyo3::types::{PyDict, PyList, PyString};
+use pyo3::types::{PyDict, PyList};
use spark_connect::row::{Row, StructValue, Value};
use spark_connect::session::SparkSession;
use spark_connect::types::{DataType, StructField};
@@ -666,6 +666,47 @@ impl PySparkSession {
fields
}
};
+ // As PySpark's Arrow conversion of a pandas column: a Decimal that
does not fit the
+ // precision and scale of its field exactly fails, rather than losing
digits.
+ for (i, field) in fields.iter().enumerate() {
+ let DataType::Decimal { precision, scale } = field.data_type else {
+ continue;
+ };
+ let fits = |value: &Value| match value {
+ Value::Decimal { value, .. } => {
+ spark_connect::session::decimal_str_to_unscaled(value,
precision, scale).is_ok()
+ }
+ _ => true,
+ };
+ if !rows.iter().filter_map(|row| row.get(i)).all(fits) {
+ let mut message = format!(
+ "Exception thrown when converting pandas.Series ({}) with
name '{}' to Arrow \
+ Array (decimal128({precision}, {scale})).",
+ dtypes.get(i).map_or("object", String::as_str),
+ labels.get(i).unwrap_or(&field.name),
+ );
+ // Read only to raise, as the conf changes the message, not
the failure.
+ let safecheck = py
+ .detach(|| {
+ self.session
+ .conf()
+
.get("spark.sql.execution.pandas.convertToArrowArraySafely")
+ })
+ .map_or(true, |value| value.is_none_or(|value| value ==
"true"));
+ if safecheck {
+ message.push_str(
+ " It can be caused by overflows or other unsafe
conversions warned by \
+ Arrow. Arrow safe type check can be disabled by using
SQL config \
+
`spark.sql.execution.pandas.convertToArrowArraySafely`.",
+ );
+ }
+ return Err(PyErr::from_value(
+ py.import("pyspark.errors")?
+ .getattr("PySparkValueError")?
+ .call1((message,))?,
+ ));
+ }
+ }
let schema = DataType::Struct { fields };
let df = py
.detach(|| {
@@ -2021,10 +2062,10 @@ fn decimal_to_value(obj: &Bound<'_, PyAny>) ->
PyResult<Option<Value>> {
if !obj.is_instance(&decimal_cls)? {
return Ok(None);
}
- // Without an exponent, as `1E+2` is written, which the Arrow encoder
parses.
- let dec_str: String = PyString::new(py, "{:f}")
- .call_method1("format", (obj,))?
- .extract()?;
+ // As written, in scientific notation for `1E+2` or `1.5E-7`: the Arrow
encoder parses
+ // either notation, in work bounded by the length of the text, which a
fixed-point form
+ // of `1E+1000000000` would not be.
+ let dec_str: String = obj.str()?.extract()?;
let as_tuple = obj.call_method0("as_tuple")?;
let mut scale: Option<i32> = None;
if let Ok(exp_obj) = as_tuple.get_item(2) {
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 5de1433..5b6698e 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -1228,7 +1228,7 @@ fn values_to_arrow(values: &[&Value], dt: &DataType) ->
Result<Arc<dyn arrow::ar
.map(|v| match *v {
Value::Null => Ok(None),
Value::Decimal { value, .. } => {
- decimal_str_to_unscaled(value, col_scale).map(Some)
+ decimal_str_to_unscaled(value, *precision,
col_scale).map(Some)
}
_ => Err(SparkError::connect_msg("Type mismatch in row
data")),
})
@@ -1479,37 +1479,85 @@ fn values_to_arrow(values: &[&Value], dt: &DataType) ->
Result<Arc<dyn arrow::ar
}
}
-/// Parse a decimal string (e.g. "-1.50") into an unscaled `i128` at the given
scale
-/// (e.g. scale 2 → -150). Pads/truncates the fractional part to `scale`
digits.
-fn decimal_str_to_unscaled(s: &str, scale: i32) -> Result<i128> {
- let s = s.trim();
- let (neg, s) = match s.strip_prefix('-') {
+/// Converts a decimal string, in fixed-point (`-1.50`) or scientific
(`1.5E-7`) notation, to
+/// its unscaled `i128` at `scale` (`-1.50` at scale 2 is -150). The value
must fit
+/// `decimal(precision, scale)` exactly, as Arrow requires of the Decimals of
PySpark's pandas
+/// conversion: a nonzero digit beyond the scale, or a value beyond the
precision, is an error
+/// rather than truncated. The work is bounded by the length of the string,
never by its
+/// exponent, so `1E+1000000000` fails at once.
+pub fn decimal_str_to_unscaled(s: &str, precision: i32, scale: i32) ->
Result<i128> {
+ if !(1..=38).contains(&precision) || !(0..=precision).contains(&scale) {
+ return Err(SparkError::connect_msg(format!(
+ "unsupported decimal type decimal({precision},{scale})"
+ )));
+ }
+ let invalid = || SparkError::connect_msg(format!("invalid decimal '{s}'"));
+ let inexact = || {
+ SparkError::connect_msg(format!(
+ "decimal '{s}' does not fit decimal({precision},{scale}) exactly"
+ ))
+ };
+ let text = s.trim();
+ let (negative, unsigned) = match text.strip_prefix('-') {
Some(rest) => (true, rest),
- None => (false, s.strip_prefix('+').unwrap_or(s)),
+ None => (false, text.strip_prefix('+').unwrap_or(text)),
};
- let (int_part, frac_part) = match s.split_once('.') {
- Some((i, f)) => (i, f),
- None => (s, ""),
+ let (coefficient, exponent) = match unsigned.split_once(['e', 'E']) {
+ Some((coefficient, exponent)) => {
+ (coefficient, exponent.parse::<i64>().map_err(|_| invalid())?)
+ }
+ None => (unsigned, 0),
};
- let scale = scale.max(0) as usize;
- let mut digits = String::with_capacity(int_part.len() + scale);
- digits.push_str(int_part);
- // Pad or truncate the fractional digits to exactly `scale`.
- let frac: String = frac_part
- .chars()
- .chain(std::iter::repeat('0'))
- .take(scale)
+ let mut point = false;
+ let mut fractional = 0_i128;
+ let mut digits = 0_usize;
+ for byte in coefficient.bytes() {
+ match byte {
+ b'.' if !point => point = true,
+ b'0'..=b'9' => {
+ digits += 1;
+ if point {
+ fractional += 1;
+ }
+ }
+ _ => return Err(invalid()),
+ }
+ }
+ if digits == 0 {
+ return Err(invalid());
+ }
+ // The digits from the first nonzero one: the value is them times
10^(exponent -
+ // fractional), and the unscaled value them times 10^shift.
+ let significant: Vec<u8> = coefficient
+ .bytes()
+ .filter(|b| *b != b'.')
+ .skip_while(|b| *b == b'0')
.collect();
- digits.push_str(&frac);
- let digits = digits.trim_start_matches('0');
- let mag: i128 = if digits.is_empty() {
- 0
+ // i128 arithmetic, so that i64::MIN and i64::MAX exponents cannot
overflow.
+ let shift = i128::from(scale) + i128::from(exponent) - fractional;
+ let dropped = if shift < 0 {
+ (-shift).min(significant.len() as i128) as usize
} else {
- digits
- .parse::<i128>()
- .map_err(|e| SparkError::connect_msg(format!("invalid decimal
'{s}': {e}")))?
+ 0
};
- Ok(if neg { -mag } else { mag })
+ let (kept, lost) = significant.split_at(significant.len() - dropped);
+ if lost.iter().any(|b| *b != b'0') {
+ return Err(inexact());
+ }
+ if kept.is_empty() {
+ return Ok(0);
+ }
+ if kept.len() as i128 + shift.max(0) > i128::from(precision) {
+ return Err(inexact());
+ }
+ // At most 38 digits: no overflow.
+ let mut value = kept
+ .iter()
+ .fold(0_i128, |value, digit| value * 10 + i128::from(digit - b'0'));
+ for _ in 0..shift.max(0) {
+ value *= 10;
+ }
+ Ok(if negative { -value } else { value })
}
#[cfg(test)]
@@ -1522,6 +1570,150 @@ mod tests {
use arrow::record_batch::RecordBatch;
use std::io::Cursor;
+ #[test]
+ fn decimal_strings_convert_in_fixed_point_and_scientific_notation() {
+ for (text, precision, scale, expected) in [
+ ("1E+3", 18, 7, 10_000_000_000),
+ ("1E-7", 18, 7, 1),
+ ("-1E-7", 18, 7, -1),
+ ("+1.25e+2", 38, 2, 12_500),
+ ("-1.25E-2", 38, 4, -125),
+ ("0001.2500e+2", 38, 2, 12_500),
+ ("0E+999999999", 18, 7, 0),
+ ("-0E-999999999", 18, 7, 0),
+ ("123.45", 38, 2, 12_345),
+ ("-123.450", 38, 2, -12_345),
+ (".00100E+3", 38, 2, 100),
+ (" +1.e2 ", 38, 0, 100),
+ // Trailing zeros beyond the scale lose nothing.
+ ("1.2300E+0", 3, 2, 123),
+ ("1.2300", 3, 2, 123),
+ ("9.9E+1", 2, 0, 99),
+ ("99999999999.9999999", 18, 7, 999_999_999_999_999_999),
+ ("-99999999999.9999999", 18, 7, -999_999_999_999_999_999),
+ ("1E-38", 38, 38, 1),
+ // A long coefficient may still fit once it is scaled.
+ ("100000000000000000000000000000000000000000E-41", 38, 0, 1),
+ ] {
+ assert_eq!(
+ decimal_str_to_unscaled(text, precision, scale).unwrap(),
+ expected,
+ "{text}"
+ );
+ }
+ let max = "9".repeat(38);
+ assert_eq!(
+ decimal_str_to_unscaled(&max, 38, 0).unwrap(),
+ max.parse::<i128>().unwrap()
+ );
+ assert_eq!(
+ decimal_str_to_unscaled(&format!("-{max}"), 38, 0).unwrap(),
+ -max.parse::<i128>().unwrap()
+ );
+ }
+
+ #[test]
+ fn decimal_strings_that_do_not_fit_exactly_are_errors() {
+ for (text, precision, scale) in [
+ // A nonzero digit beyond the scale, in either notation: PySpark's
Arrow
+ // conversion of a pandas column fails rather than dropping it.
+ ("1.5E-7", 18, 7),
+ ("-1.5E-7", 18, 7),
+ ("1E-8", 18, 7),
+ ("0.00000015", 18, 7),
+ ("1.235", 10, 2),
+ ("1E-9223372036854775808", 18, 7),
+ // Beyond the precision.
+ ("123456E-3", 9, 7),
+ ("1E+2", 2, 0),
+ ("-1E+2", 2, 0),
+ ("100", 2, 0),
+ ("100000000000", 18, 7),
+ ("1E+38", 38, 0),
+ ("1E+999999999", 38, 0),
+ ("1E+9223372036854775807", 38, 0),
+ ("170141183460469231731687303715884105727", 38, 0),
+ ] {
+ let error = decimal_str_to_unscaled(text, precision,
scale).unwrap_err();
+ assert!(
+ error.to_string().contains("does not fit"),
+ "{text}: {error}"
+ );
+ }
+ for text in [
+ "",
+ ".",
+ "-",
+ "+",
+ "1E",
+ "E3",
+ "1E1E1",
+ "1.2.3",
+ "1E1.5",
+ "NaN",
+ "Infinity",
+ "1E 3",
+ "1_000",
+ "1E+9223372036854775808",
+ ] {
+ let error = decimal_str_to_unscaled(text, 38, 0).unwrap_err();
+ assert!(
+ error.to_string().contains("invalid decimal"),
+ "{text}: {error}"
+ );
+ }
+ // Spark has no such decimal types.
+ for (precision, scale) in [(0, 0), (39, 0), (10, -2), (10, 11), (38,
i32::MAX)] {
+ assert!(decimal_str_to_unscaled("1", precision, scale).is_err());
+ }
+ }
+
+ #[test]
+ fn scientific_decimal_ipc_cells_match_fixed_point() {
+ use arrow::array::Decimal128Array;
+ let values = [
+ "1E+3",
+ "1E-7",
+ "-1E-7",
+ "-0E+3",
+ "99999999999.9999999",
+ "-99999999999.9999999",
+ ];
+ let rows: Vec<_> = values
+ .iter()
+ .map(|value| {
+ row(vec![Value::Decimal {
+ value: value.to_string(),
+ precision: None,
+ scale: None,
+ }])
+ })
+ .collect();
+ let batch = read_batch(
+ &rows,
+ &schema(vec![DataType::Decimal {
+ precision: 18,
+ scale: 7,
+ }]),
+ );
+ let array = batch
+ .column(0)
+ .as_any()
+ .downcast_ref::<Decimal128Array>()
+ .unwrap();
+ assert_eq!(
+ array.values().as_ref(),
+ &[
+ 10_000_000_000,
+ 1,
+ -1,
+ 0,
+ 999_999_999_999_999_999,
+ -999_999_999_999_999_999
+ ]
+ );
+ }
+
fn struct_field(name: &str, data_type: DataType) ->
crate::types::StructField {
crate::types::StructField {
name: name.to_string(),
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 5da39f8..f9fc331 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -5930,6 +5930,238 @@ def
test_exact_date_ipc_calendar_range(exact_date_capture_input, source, schema_
assert table.column(0).cast(pa.int32()).to_pylist() == expected_days
+# Inspect the local-input IPC through a real transport boundary. The subprocess
+# avoids sharing the GIL with native RPC calls and needs no JVM or Spark
server.
+_DECIMAL_IPC_SERVER = r"""
+import sys
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+import grpc
+import pyarrow as pa
+from google.protobuf import descriptor_pb2, descriptor_pool, message_factory
+pool = descriptor_pool.DescriptorPool()
+for file in
descriptor_pb2.FileDescriptorSet.FromString(Path(sys.argv[1]).read_bytes()).file:
+ pool.Add(file)
+def message(name, **fields):
+ cls =
message_factory.GetMessageClass(pool.FindMessageTypeByName('spark.connect.' +
name))
+ return cls(**fields)
+def config(raw, context):
+ request = message('ConfigRequest').FromString(raw)
+ response = message('ConfigResponse', session_id=request.session_id,
+
server_side_session_id='00000000-0000-0000-0000-000000000001')
+ values = {
+ 'spark.sql.session.timeZone': 'UTC',
+ 'spark.sql.timestampType': 'TIMESTAMP_LTZ',
+ 'spark.sql.session.localRelationCacheThreshold': str(1 << 60),
+ 'spark.sql.session.localRelationSizeLimit': str(1 << 60),
+ 'spark.sql.session.localRelationChunkSizeRows': '10000',
+ 'spark.sql.session.localRelationChunkSizeBytes': str(16 << 20),
+ 'spark.sql.session.localRelationBatchOfChunksSizeBytes': str(64 << 20),
+ # As the default of a server.
+ 'spark.sql.execution.pandas.convertToArrowArraySafely': 'true',
+ }
+ operation = request.operation
+ if operation.HasField('get'):
+ keys = operation.get.keys
+ elif operation.HasField('get_with_default'):
+ keys = [pair.key for pair in operation.get_with_default.pairs]
+ else:
+ context.abort(grpc.StatusCode.INVALID_ARGUMENT, 'unexpected config
operation')
+ for key in keys:
+ response.pairs.add(key=key, value=values.get(key, 'false'))
+ return response.SerializeToString()
+def execute(raw, context):
+ request = message('ExecutePlanRequest').FromString(raw)
+ relation = request.plan.root
+ # Python's pandas path wraps the local relation to assign column names.
+ if relation.HasField('to_df'):
+ relation = relation.to_df.input
+ assert relation.HasField('local_relation'), relation
+ payload = relation.local_relation.data
+ Path(sys.argv[2]).write_bytes(payload)
+ table = pa.ipc.open_stream(payload).read_all()
+ header = dict(session_id=request.session_id,
operation_id=request.operation_id,
+
server_side_session_id='00000000-0000-0000-0000-000000000001')
+ response = message('ExecutePlanResponse', **header, response_id='batch',
+ arrow_batch=dict(data=payload,
row_count=table.num_rows))
+ for field in table.schema:
+ output = response.schema.struct.fields.add(name=field.name,
nullable=True)
+ output.data_type.decimal.precision = field.type.precision
+ output.data_type.decimal.scale = field.type.scale
+ yield response.SerializeToString()
+ yield message('ExecutePlanResponse', **header, response_id='complete',
+ result_complete={}).SerializeToString()
+def release_session(raw, context):
+ request = message('ReleaseSessionRequest').FromString(raw)
+ return message('ReleaseSessionResponse', session_id=request.session_id,
+
server_side_session_id='00000000-0000-0000-0000-000000000001').SerializeToString()
+def release_execute(raw, context):
+ request = message('ReleaseExecuteRequest').FromString(raw)
+ return message('ReleaseExecuteResponse', session_id=request.session_id,
+ operation_id=request.operation_id,
+
server_side_session_id='00000000-0000-0000-0000-000000000001').SerializeToString()
+with ThreadPoolExecutor(max_workers=4) as executor:
+ server = grpc.server(executor)
+ server.add_generic_rpc_handlers((grpc.method_handlers_generic_handler(
+ 'spark.connect.SparkConnectService', {
+ 'Config': grpc.unary_unary_rpc_method_handler(config),
+ 'ExecutePlan': grpc.unary_stream_rpc_method_handler(execute),
+ 'ReleaseSession':
grpc.unary_unary_rpc_method_handler(release_session),
+ 'ReleaseExecute':
grpc.unary_unary_rpc_method_handler(release_execute),
+ }),))
+ port = server.add_insecure_port('127.0.0.1:0')
+ assert port
+ server.start()
+ print(port, flush=True)
+ sys.stdin.readline()
+ server.stop(0).wait()
+"""
+
+
[email protected](scope="module")
+def decimal_ipc_protocol(tmp_path_factory):
+ import os
+ from pathlib import Path
+ import shutil
+ import subprocess
+
+ pytest.importorskip("grpc")
+ pytest.importorskip("pyarrow")
+ protoc = shutil.which(os.environ.get("PROTOC", "protoc"))
+ if protoc is None:
+ pytest.skip("Decimal IPC transport tests require protoc")
+ proto_root = Path(__file__).resolve().parents[2] /
"crates/spark-connect-proto/proto"
+ descriptor = tmp_path_factory.mktemp("decimal-proto") / "connect.pb"
+ command = [protoc, f"-I{proto_root}"]
+ if include := os.environ.get("PROTOC_INCLUDE"):
+ command.append(f"-I{include}")
+ subprocess.run(command + ["--include_imports",
f"--descriptor_set_out={descriptor}",
+ str(proto_root / "spark/connect/base.proto")],
check=True)
+ return descriptor
+
+
[email protected]
+def decimal_ipc_session(decimal_ipc_protocol, tmp_path):
+ import subprocess
+ import sys
+ from pyspark.sql import SparkSession
+
+ payload = tmp_path / "input.arrow"
+ process = subprocess.Popen(
+ [sys.executable, "-u", "-c", _DECIMAL_IPC_SERVER,
str(decimal_ipc_protocol), str(payload)],
+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True,
+ )
+ spark = None
+ try:
+ port = int(process.stdout.readline())
+ spark = SparkSession.builder.remote(f"sc://127.0.0.1:{port}").create()
+ yield spark, payload
+ finally:
+ try:
+ if spark is not None:
+ spark.stop()
+ finally:
+ process.communicate("stop\n", timeout=20)
+ assert process.returncode == 0
+
+
[email protected]("kind", ["tuples", "pandas"])
[email protected]("precision, scale, texts", [
+ (18, 7, ["1E+3", "1E-7", "-1E-7", "+1.25E+2", "-1.25E-2", "0E+3", "-0E-7",
None]),
+ (18, 7, ["99999999999.9999999", "-99999999999.9999999", "1.234500E-3"]),
+ (38, 0, ["1E+37", "-1E+37",
"9.9999999999999999999999999999999999999E+37"]),
+ (38, 38, ["1E-38", "-1E-38", "0E-38"]),
+ (10, 2, ["123.45", "-123.45", "1.2300", "0.00", "-0.00"]),
+])
+def test_decimal_scientific_local_input_ipc(decimal_ipc_session, kind,
precision, scale, texts):
+ from decimal import Decimal
+ import pyarrow as pa
+
+ spark, payload = decimal_ipc_session
+ values = [None if text is None else Decimal(text) for text in texts]
+ data = [(value,) for value in values]
+ if kind == "pandas":
+ pd = pytest.importorskip("pandas")
+ data = pd.DataFrame({"v": values})
+ schema = T.StructType([T.StructField("v", T.DecimalType(precision,
scale))])
+ spark.createDataFrame(data, schema).collect()
+ # Read the uploaded IPC independently rather than trusting collect's
decoder.
+ table = pa.ipc.open_stream(payload.read_bytes()).read_all()
+ expected = pa.table({"v": pa.array(values, type=pa.decimal128(precision,
scale))})
+ assert table.schema == expected.schema
+ assert table.equals(expected)
+ # Check the physical little-endian Decimal128 cells as well as their
values.
+ actual = table.column(0).combine_chunks()
+ wanted = expected.column(0).combine_chunks()
+ for index, value in enumerate(values):
+ if value is not None:
+ offset = index * 16
+ assert actual.buffers()[1][offset:offset + 16] ==
wanted.buffers()[1][offset:offset + 16]
+
+
[email protected]("text", ["1E+1000000000", "-1E+1000000000"])
+def test_decimal_scientific_local_input_overflow(decimal_ipc_session, text):
+ from decimal import Decimal
+
+ spark, payload = decimal_ipc_session
+ schema = T.StructType([T.StructField("v", T.DecimalType(38, 0))])
+ with pytest.raises((ArithmeticError, RuntimeError)):
+ spark.createDataFrame([(Decimal(text),)], schema).collect()
+ assert not payload.exists()
+
+
[email protected]("kind", ["tuples", "pandas"])
[email protected]("text, precision, scale, local_expected", [
+ ("1.5E-7", 18, 7, "0.0000002"),
+ ("-1.5E-7", 18, 7, "-0.0000002"),
+ ("1E-8", 18, 7, "0.0000000"),
+ ("0.00000015", 18, 7, "0.0000002"),
+ ("1.235", 10, 2, "1.24"),
+ ("1E+2", 2, 0, None),
+ ("-1E+2", 2, 0, None),
+ ("100000000000", 18, 7, None),
+ ("1E+1000000000", 38, 0, None),
+])
+def test_decimal_rescaling_follows_pyspark(
+ decimal_ipc_session, kind, text, precision, scale, local_expected
+):
+ # As PySpark 4.3.0: local rows are quantized half-even to the scale, and
fail beyond
+ # the precision, in the Decimal conversion; a pandas column fails its
Arrow conversion
+ # for any value that does not fit exactly, in either notation, rather than
losing digits.
+ from decimal import Decimal
+ import pyarrow as pa
+ from pyspark.errors import PySparkValueError
+
+ spark, payload = decimal_ipc_session
+ data = [(Decimal(text),)]
+ if kind == "pandas":
+ pd = pytest.importorskip("pandas")
+ data = pd.DataFrame({"v": [Decimal(text)]}, dtype=object)
+ schema = T.StructType([T.StructField("v", T.DecimalType(precision,
scale))])
+ if kind == "tuples" and local_expected is not None:
+ spark.createDataFrame(data, schema).collect()
+ actual = pa.ipc.open_stream(payload.read_bytes()).read_all()
+ expected = pa.table({"v": pa.array([Decimal(local_expected)],
+ type=pa.decimal128(precision,
scale))})
+ actual.validate(full=True)
+ assert actual.schema == expected.schema
+ assert actual.equals(expected)
+ return
+ if kind == "tuples":
+ with pytest.raises(ArithmeticError):
+ spark.createDataFrame(data, schema)
+ else:
+ with pytest.raises(PySparkValueError) as error:
+ spark.createDataFrame(data, schema)
+ assert str(error.value) == (
+ "Exception thrown when converting pandas.Series (object) with name
'v' to Arrow "
+ f"Array (decimal128({precision}, {scale})). It can be caused by
overflows or other "
+ "unsafe conversions warned by Arrow. Arrow safe type check can be
disabled by using "
+ "SQL config
`spark.sql.execution.pandas.convertToArrowArraySafely`."
+ )
+ assert not payload.exists()
+
+
# ---------------------------------------------------------------------------
schema cache
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]