This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 63207987 [SPARK-61154][CONNECT][PYTHON] Raise VALUE_NOT_BETWEEN for a 
literal of an int beyond the range of a long, in the Python client as PySpark 
does
63207987 is described below

commit 632079876cca0acce04b848d07439c7e95020834
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 13:01:05 2026 +0900

    [SPARK-61154][CONNECT][PYTHON] Raise VALUE_NOT_BETWEEN for a literal of an 
int beyond the range of a long, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61154](https://issues.apache.org/jira/browse/SPARK-61154): Raise 
VALUE_NOT_BETWEEN for a literal of an int beyond the range of a long, in the 
Python client as PySpark does.
    
    - A Python int (not a bool) that does not fit in a long is an error 
wherever it becomes a literal, instead of a double. This covers `lit`, the 
Column operators and comparisons, `isin`, `between`, `when`/`otherwise`, 
`array`, list literals and `pivot` values. The error is `PySparkValueError` 
`VALUE_NOT_BETWEEN` with `arg_name` "value", `min` "-9223372036854775808" and 
`max` "32767", as `LiteralExpression._infer_type` raises it (its maximum is 
`JVM_SHORT_MAX`, which PySpark writes this way).
    - `fillna` and `na.fill` with such an int raise `ValueError: Value out of 
range: <n>`, as PySpark's proto raises for an out-of-range long.
    - Floats, bools, NumPy scalars and ints within the range of a long are 
unchanged. `createDataFrame` and `spark.sql(..., kwargs)` keep their results, 
which already matched PySpark.
    
    Official PySpark 4.3.0 for the same expressions (checked against the same 
server): `df.where(F.col("a") == 2**63)`, `F.lit(2**70)`, `F.col("a") > 2**70`, 
`isin`, `between`, `when`, `array`, `pivot` raise `PySparkValueError 
VALUE_NOT_BETWEEN`; `fillna(2**70)` raises `ValueError Value out of range: 
1180591620717411303424`.
    
    ### Why are the changes needed?
    
    A Python int beyond the int64 range was silently converted to a double. The 
comparison `df.where(F.col("a") == 2**63)` returned the rows with a = 
9223372036854775807 and a = 9223372036854775806, because the double rounds, and 
`F.col("a") > 2**70` returned no rows. PySpark raises for these values.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. An int beyond the range of a long raises `PySparkValueError` 
`VALUE_NOT_BETWEEN` where it used to become a double, and `fillna` raises 
`ValueError` for it.
    
    ### How was this patch tested?
    
    - `test_int_literals_beyond_a_long_raise_value_not_between_as_pyspark` 
(offline, no server): `lit`, `==`, `>`, `isin`, `between`, `when`, `array` and 
list literals raise `VALUE_NOT_BETWEEN` with the message parameters of PySpark; 
`fillna(2**70)` raises the `ValueError`; the bounds of a long and a float 
beyond them are accepted. The same cases give the same results with official 
PySpark 4.3.0.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped), and 
the live tests of literals, isin, between, fillna, when and pivot (51 passed) 
against a local Spark 4.3.0 server; rustfmt passes, and clippy reports no 
diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #343 from HyukjinKwon/SPARK-61154-int-literal-beyond-long.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/functions.rs      | 19 ++++++++++++++++++
 crates/pyspark-rs/src/session.rs        |  4 ++++
 crates/pyspark-rs/src/stat_arguments.rs | 24 ++++++++++++++++++-----
 python/tests/test_dropin_offline.py     | 34 +++++++++++++++++++++++++++++++++
 4 files changed, 76 insertions(+), 5 deletions(-)

diff --git a/crates/pyspark-rs/src/functions.rs 
b/crates/pyspark-rs/src/functions.rs
index 16c527fc..74da2f4f 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -205,6 +205,21 @@ pub(crate) fn enum_to_value<'py>(obj: &Bound<'py, PyAny>) 
-> PyResult<Bound<'py,
     Ok(value)
 }
 
+/// The error of a Python int beyond the range of a long, as PySpark's 
`LiteralExpression`
+/// raises it: `VALUE_NOT_BETWEEN`, whose maximum is `JVM_SHORT_MAX`, as 
PySpark writes it.
+pub(crate) fn value_not_between(py: Python<'_>) -> PyErr {
+    crate::column::raise_pyspark(
+        py,
+        "PySparkValueError",
+        "VALUE_NOT_BETWEEN",
+        &[
+            ("arg_name", "value"),
+            ("min", "-9223372036854775808"),
+            ("max", "32767"),
+        ],
+    )
+}
+
 /// Convert a Python object to a Rust Column (handles Column or scalars that 
become literals).
 pub fn to_column(obj: &Bound<'_, PyAny>) -> PyResult<Column> {
     // Try to get as PyColumn first
@@ -271,6 +286,10 @@ pub fn to_column(obj: &Bound<'_, PyAny>) -> 
PyResult<Column> {
             },
         )));
     }
+    // An int beyond a long is not a double, as PySpark's literal of it: it is 
an error.
+    if obj.is_instance_of::<pyo3::types::PyInt>() {
+        return Err(value_not_between(obj.py()));
+    }
 
     if let Ok(f) = obj.extract::<f64>() {
         return Ok(Column::new(Expression::literal(LiteralExpression::double(
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index ba5aa38d..a5ab6d16 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -2754,6 +2754,10 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) -> 
PyResult<Value> {
     if let Ok(i) = obj.extract::<i64>() {
         return Ok(Value::Long(i));
     }
+    // An int beyond a long is not a double, as PySpark's literal of it: it is 
an error.
+    if obj.is_instance_of::<pyo3::types::PyInt>() {
+        return Err(crate::functions::value_not_between(obj.py()));
+    }
 
     // Check for float
     if let Ok(f) = obj.extract::<f64>() {
diff --git a/crates/pyspark-rs/src/stat_arguments.rs 
b/crates/pyspark-rs/src/stat_arguments.rs
index 099069f7..e3cd4210 100644
--- a/crates/pyspark-rs/src/stat_arguments.rs
+++ b/crates/pyspark-rs/src/stat_arguments.rs
@@ -157,14 +157,28 @@ pub(crate) fn fillna(
     if let Some(dict) = dict {
         let mut pairs = Vec::with_capacity(dict.len());
         for (column, fill) in dict.iter() {
-            pairs.push((column.extract()?, 
crate::session::py_to_value(&fill)?));
+            pairs.push((column.extract()?, fill_value(&fill)?));
         }
         return Ok(PyDataFrame::new(df.fillna_map(pairs)));
     }
-    Ok(PyDataFrame::new(df.fillna_value(
-        crate::session::py_to_value(value)?,
-        column_refs(&columns),
-    )))
+    Ok(PyDataFrame::new(
+        df.fillna_value(fill_value(value)?, column_refs(&columns)),
+    ))
+}
+
+/// The value of a fill. As PySpark sends it as a proto literal, an int beyond 
a long is out of
+/// range for the proto, which raises `ValueError`, not the error of a literal.
+fn fill_value(value: &Bound<'_, PyAny>) -> PyResult<spark_connect::row::Value> 
{
+    if value.is_instance_of::<PyInt>()
+        && !value.is_instance_of::<pyo3::types::PyBool>()
+        && value.extract::<i64>().is_err()
+    {
+        return Err(pyo3::exceptions::PyValueError::new_err(format!(
+            "Value out of range: {}",
+            value.str()?
+        )));
+    }
+    crate::session::py_to_value(value)
 }
 
 /// The columns of `df.describe(*cols)`, as PySpark's: column names, or a list 
of them.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 8ff09063..c7a299c2 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9100,6 +9100,40 @@ def completion_analyzed():
         child.close()
 
 
+def test_int_literals_beyond_a_long_raise_value_not_between_as_pyspark():
+    # As PySpark's LiteralExpression._infer_type: an int beyond the range of a 
long is an error,
+    # not a double, which would make `col == 2**63` match the rows of the 
longs next to it. The
+    # maximum is JVM_SHORT_MAX, as PySpark writes it. fillna sends the value 
as a proto literal,
+    # which raises ValueError for it.
+    from pyspark.errors import PySparkValueError
+    from pyspark.sql import functions as F
+
+    spark = _offline_session()
+    df = spark.range(1)
+    for case in (
+        lambda: F.lit(2**70),
+        lambda: F.lit(-(2**63) - 1),
+        lambda: F.col("id") == 2**63,
+        lambda: F.col("id") > 2**70,
+        lambda: F.col("id").isin([1, 2**70]),
+        lambda: F.col("id").between(1, 2**70),
+        lambda: F.when(F.col("id") > 0, 2**70),
+        lambda: F.array(F.lit(1), F.lit(2**70)),
+        lambda: F.lit([2**70]),
+    ):
+        with pytest.raises(PySparkValueError) as e:
+            case()
+        assert e.value.getCondition() == "VALUE_NOT_BETWEEN"
+        assert e.value.getMessageParameters() == {
+            "arg_name": "value", "min": "-9223372036854775808", "max": "32767"}
+    with pytest.raises(ValueError, match="Value out of range: 
1180591620717411303424"):
+        df.fillna(2**70)
+    # The bounds of a long, and a float beyond them, are fine.
+    assert F.lit(2**63 - 1) is not None and F.lit(-(2**63)) is not None
+    assert F.lit(2.0**70) is not None
+    spark.stop()
+
+
 @pytest.mark.parametrize("names", [
     ["z", "a"],
     ["with space", "a.b", "with`tick", "", "123", "class", "雪", "_private"],


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to