This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 63207987 [SPARK-61154][CONNECT][PYTHON] Raise VALUE_NOT_BETWEEN for a
literal of an int beyond the range of a long, in the Python client as PySpark
does
63207987 is described below
commit 632079876cca0acce04b848d07439c7e95020834
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 13:01:05 2026 +0900
[SPARK-61154][CONNECT][PYTHON] Raise VALUE_NOT_BETWEEN for a literal of an
int beyond the range of a long, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61154](https://issues.apache.org/jira/browse/SPARK-61154): Raise
VALUE_NOT_BETWEEN for a literal of an int beyond the range of a long, in the
Python client as PySpark does.
- A Python int (not a bool) that does not fit in a long is an error
wherever it becomes a literal, instead of a double. This covers `lit`, the
Column operators and comparisons, `isin`, `between`, `when`/`otherwise`,
`array`, list literals and `pivot` values. The error is `PySparkValueError`
`VALUE_NOT_BETWEEN` with `arg_name` "value", `min` "-9223372036854775808" and
`max` "32767", as `LiteralExpression._infer_type` raises it (its maximum is
`JVM_SHORT_MAX`, which PySpark writes this way).
- `fillna` and `na.fill` with such an int raise `ValueError: Value out of
range: <n>`, as PySpark's proto raises for an out-of-range long.
- Floats, bools, NumPy scalars and ints within the range of a long are
unchanged. `createDataFrame` and `spark.sql(..., kwargs)` keep their results,
which already matched PySpark.
Official PySpark 4.3.0 for the same expressions (checked against the same
server): `df.where(F.col("a") == 2**63)`, `F.lit(2**70)`, `F.col("a") > 2**70`,
`isin`, `between`, `when`, `array`, `pivot` raise `PySparkValueError
VALUE_NOT_BETWEEN`; `fillna(2**70)` raises `ValueError Value out of range:
1180591620717411303424`.
### Why are the changes needed?
A Python int beyond the int64 range was silently converted to a double. The
comparison `df.where(F.col("a") == 2**63)` returned the rows with a =
9223372036854775807 and a = 9223372036854775806, because the double rounds, and
`F.col("a") > 2**70` returned no rows. PySpark raises for these values.
### Does this PR introduce _any_ user-facing change?
Yes. An int beyond the range of a long raises `PySparkValueError`
`VALUE_NOT_BETWEEN` where it used to become a double, and `fillna` raises
`ValueError` for it.
### How was this patch tested?
- `test_int_literals_beyond_a_long_raise_value_not_between_as_pyspark`
(offline, no server): `lit`, `==`, `>`, `isin`, `between`, `when`, `array` and
list literals raise `VALUE_NOT_BETWEEN` with the message parameters of PySpark;
`fillna(2**70)` raises the `ValueError`; the bounds of a long and a float
beyond them are accepted. The same cases give the same results with official
PySpark 4.3.0.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped), and
the live tests of literals, isin, between, fillna, when and pivot (51 passed)
against a local Spark 4.3.0 server; rustfmt passes, and clippy reports no
diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #343 from HyukjinKwon/SPARK-61154-int-literal-beyond-long.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/functions.rs | 19 ++++++++++++++++++
crates/pyspark-rs/src/session.rs | 4 ++++
crates/pyspark-rs/src/stat_arguments.rs | 24 ++++++++++++++++++-----
python/tests/test_dropin_offline.py | 34 +++++++++++++++++++++++++++++++++
4 files changed, 76 insertions(+), 5 deletions(-)
diff --git a/crates/pyspark-rs/src/functions.rs
b/crates/pyspark-rs/src/functions.rs
index 16c527fc..74da2f4f 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -205,6 +205,21 @@ pub(crate) fn enum_to_value<'py>(obj: &Bound<'py, PyAny>)
-> PyResult<Bound<'py,
Ok(value)
}
+/// The error of a Python int beyond the range of a long, as PySpark's
`LiteralExpression`
+/// raises it: `VALUE_NOT_BETWEEN`, whose maximum is `JVM_SHORT_MAX`, as
PySpark writes it.
+pub(crate) fn value_not_between(py: Python<'_>) -> PyErr {
+ crate::column::raise_pyspark(
+ py,
+ "PySparkValueError",
+ "VALUE_NOT_BETWEEN",
+ &[
+ ("arg_name", "value"),
+ ("min", "-9223372036854775808"),
+ ("max", "32767"),
+ ],
+ )
+}
+
/// Convert a Python object to a Rust Column (handles Column or scalars that
become literals).
pub fn to_column(obj: &Bound<'_, PyAny>) -> PyResult<Column> {
// Try to get as PyColumn first
@@ -271,6 +286,10 @@ pub fn to_column(obj: &Bound<'_, PyAny>) ->
PyResult<Column> {
},
)));
}
+ // An int beyond a long is not a double, as PySpark's literal of it: it is
an error.
+ if obj.is_instance_of::<pyo3::types::PyInt>() {
+ return Err(value_not_between(obj.py()));
+ }
if let Ok(f) = obj.extract::<f64>() {
return Ok(Column::new(Expression::literal(LiteralExpression::double(
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index ba5aa38d..a5ab6d16 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -2754,6 +2754,10 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) ->
PyResult<Value> {
if let Ok(i) = obj.extract::<i64>() {
return Ok(Value::Long(i));
}
+ // An int beyond a long is not a double, as PySpark's literal of it: it is
an error.
+ if obj.is_instance_of::<pyo3::types::PyInt>() {
+ return Err(crate::functions::value_not_between(obj.py()));
+ }
// Check for float
if let Ok(f) = obj.extract::<f64>() {
diff --git a/crates/pyspark-rs/src/stat_arguments.rs
b/crates/pyspark-rs/src/stat_arguments.rs
index 099069f7..e3cd4210 100644
--- a/crates/pyspark-rs/src/stat_arguments.rs
+++ b/crates/pyspark-rs/src/stat_arguments.rs
@@ -157,14 +157,28 @@ pub(crate) fn fillna(
if let Some(dict) = dict {
let mut pairs = Vec::with_capacity(dict.len());
for (column, fill) in dict.iter() {
- pairs.push((column.extract()?,
crate::session::py_to_value(&fill)?));
+ pairs.push((column.extract()?, fill_value(&fill)?));
}
return Ok(PyDataFrame::new(df.fillna_map(pairs)));
}
- Ok(PyDataFrame::new(df.fillna_value(
- crate::session::py_to_value(value)?,
- column_refs(&columns),
- )))
+ Ok(PyDataFrame::new(
+ df.fillna_value(fill_value(value)?, column_refs(&columns)),
+ ))
+}
+
+/// The value of a fill. As PySpark sends it as a proto literal, an int beyond
a long is out of
+/// range for the proto, which raises `ValueError`, not the error of a literal.
+fn fill_value(value: &Bound<'_, PyAny>) -> PyResult<spark_connect::row::Value>
{
+ if value.is_instance_of::<PyInt>()
+ && !value.is_instance_of::<pyo3::types::PyBool>()
+ && value.extract::<i64>().is_err()
+ {
+ return Err(pyo3::exceptions::PyValueError::new_err(format!(
+ "Value out of range: {}",
+ value.str()?
+ )));
+ }
+ crate::session::py_to_value(value)
}
/// The columns of `df.describe(*cols)`, as PySpark's: column names, or a list
of them.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 8ff09063..c7a299c2 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9100,6 +9100,40 @@ def completion_analyzed():
child.close()
+def test_int_literals_beyond_a_long_raise_value_not_between_as_pyspark():
+ # As PySpark's LiteralExpression._infer_type: an int beyond the range of a
long is an error,
+ # not a double, which would make `col == 2**63` match the rows of the
longs next to it. The
+ # maximum is JVM_SHORT_MAX, as PySpark writes it. fillna sends the value
as a proto literal,
+ # which raises ValueError for it.
+ from pyspark.errors import PySparkValueError
+ from pyspark.sql import functions as F
+
+ spark = _offline_session()
+ df = spark.range(1)
+ for case in (
+ lambda: F.lit(2**70),
+ lambda: F.lit(-(2**63) - 1),
+ lambda: F.col("id") == 2**63,
+ lambda: F.col("id") > 2**70,
+ lambda: F.col("id").isin([1, 2**70]),
+ lambda: F.col("id").between(1, 2**70),
+ lambda: F.when(F.col("id") > 0, 2**70),
+ lambda: F.array(F.lit(1), F.lit(2**70)),
+ lambda: F.lit([2**70]),
+ ):
+ with pytest.raises(PySparkValueError) as e:
+ case()
+ assert e.value.getCondition() == "VALUE_NOT_BETWEEN"
+ assert e.value.getMessageParameters() == {
+ "arg_name": "value", "min": "-9223372036854775808", "max": "32767"}
+ with pytest.raises(ValueError, match="Value out of range:
1180591620717411303424"):
+ df.fillna(2**70)
+ # The bounds of a long, and a float beyond them, are fine.
+ assert F.lit(2**63 - 1) is not None and F.lit(-(2**63)) is not None
+ assert F.lit(2.0**70) is not None
+ spark.stop()
+
+
@pytest.mark.parametrize("names", [
["z", "a"],
["with space", "a.b", "with`tick", "", "123", "class", "雪", "_private"],
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]