This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new aec003c  [SPARK-60982][CONNECT][PYTHON] Build an array of a list or a 
NumPy array in lit() as PySpark does
aec003c is described below

commit aec003c57fccb1b3481eeef0f6cf5e9e33913821
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 13:31:01 2026 +0900

    [SPARK-60982][CONNECT][PYTHON] Build an array of a list or a NumPy array in 
lit() as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60982](https://issues.apache.org/jira/browse/SPARK-60982): lists and 
NumPy arrays become arrays as in PySpark 4.3.0.
    
    - **`lit` of a list** is the array of the literals of its elements, as 
PySpark's `lit` builds it:
      - `F.lit([1, 2])` is `array(1, 2)`, and nested lists nest;
      - a list with a Column raises `PySparkValueError` `COLUMN_IN_LIST`.
    - **`lit` of a one-dimensional NumPy array** is that array cast to the 
array type of its dtype, by PySpark's `_from_numpy_type`, which 
`pyspark.sql.types` now has:
      - `F.lit(np.array([1, 2]))` is `CAST(array(1, 2) AS ARRAY<BIGINT>)`;
      - a dtype without a Spark type raises `UNSUPPORTED_NUMPY_ARRAY_SCALAR`.
    - **Function arguments:** a list in a function's argument is such an array 
too, as PySpark's functions take a value with `lit`:
      - `F.array_contains("a", [1])` is `array_contains(a, array(1))`;
      - `F.when(c, [1, 2])` has the value `array(1, 2)`.
    - **A single list of columns:** `array`, `create_map`, `map_concat` and 
`struct` take a single list, set or tuple as their columns, as in PySpark: 
`F.array(["a", "b"])` is `array(a, b)`.
    - **Other values:** a value of another type raises `PySparkTypeError` 
`UNSUPPORTED_DATA_TYPE`, as PySpark's literal conversion does, instead of 
`TypeError`. This covers a tuple, a dict, and any other NumPy array.
    
    ### Why are the changes needed?
    
    - `F.lit([1, 2])` and `F.array(["a", "b"])` raised `TypeError: Expected 
Column or scalar ...; got list`.
    - A NumPy array of one element was converted to that element:
      - `F.lit(np.array([1.5]))` was the scalar `1.5`;
      - `F.lit(np.array([1], dtype=np.uint8))` was `1`;
      - `F.col("a") == np.array([1])` compared with `1`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, these values become arrays, or raise, as in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - Probes of 44 cases give the same results and errors as official PySpark 
4.3.0, apart from reprs that 
[SPARK-60948](https://issues.apache.org/jira/browse/SPARK-60948) covers, such 
as the case of a cast's type. The cases cover `lit` of lists, NumPy arrays and 
scalars and other values, lists in function arguments, and single lists of 
columns.
      - Two kinds of value stay different. `datetime.time` and 
`datetime.timedelta`, which PySpark's `lit` takes, now raise 
`UNSUPPORTED_DATA_TYPE`, as before but with PySpark's error class. TIME is 
tracked by [SPARK-60947](https://issues.apache.org/jira/browse/SPARK-60947).
      - An operator with a list, such as `F.col("a") == [1, 2]`, compares with 
the array, as classic PySpark does, where PySpark 4.3.0's Connect client fails 
with `AttributeError`.
    - Against a local Spark 4.3.0 server, a select of these columns gives the 
same types and values as official PySpark 4.3.0.
    - New offline tests check the reprs and errors, and a new live test checks 
the types and values. They also pass with official PySpark 4.3.0, and fail 
without this change.
    - The lib tests and every integration test, e2e included, against the local 
server (645 passed), the build of all tests with `wasm-udf`, the offline Python 
suites (658 passed, 8 live cases skipped), and the live drop-in and Arrow 
suites (64 passed) and live cases (10 passed). Clippy reports no diagnostics in 
the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #188 from HyukjinKwon/SPARK-60982-lit-list.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/functions.rs  | 44 +++++++++++++++++++++++++--
 python/pyspark/sql/functions.py     | 60 ++++++++++++++++++++++++++++++++++++-
 python/pyspark/sql/types.py         | 24 +++++++++++++++
 python/tests/test_dropin_e2e.py     | 17 +++++++++++
 python/tests/test_dropin_offline.py | 55 ++++++++++++++++++++++++++++++++++
 5 files changed, 196 insertions(+), 4 deletions(-)

diff --git a/crates/pyspark-rs/src/functions.rs 
b/crates/pyspark-rs/src/functions.rs
index 3b628ad..0e95069 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -157,6 +157,27 @@ pub fn to_column(obj: &Bound<'_, PyAny>) -> 
PyResult<Column> {
         ))));
     }
 
+    // A list, or a one-dimensional NumPy array, is an array, as PySpark's 
functions take a
+    // value with `lit`: `pyspark.sql.functions.lit` builds it. Another NumPy 
array is not a
+    // literal value; NumPy would convert one of a single element to a scalar 
below.
+    let ndarray = numpy_ndarray_ndim(obj)?;
+    if obj.is_instance_of::<pyo3::types::PyList>() || ndarray == Some(1) {
+        let column = obj
+            .py()
+            .import("pyspark.sql.functions")?
+            .getattr("lit")?
+            .call1((obj,))?;
+        return Ok(column.extract::<PyRef<'_, PyColumn>>()?.column.clone());
+    }
+    if ndarray.is_some() {
+        return Err(crate::column::raise_pyspark(
+            obj.py(),
+            "PySparkTypeError",
+            "UNSUPPORTED_DATA_TYPE",
+            &[("data_type", "ndarray")],
+        ));
+    }
+
     // Non-primitive scalars must be checked BEFORE int/float: Decimal defines
     // __float__/__int__, and a pandas Timestamp (a datetime subclass) defines 
__int__
     // (nanoseconds), so an early int/float extract would silently mis-encode 
them.
@@ -194,14 +215,31 @@ pub fn to_column(obj: &Bound<'_, PyAny>) -> 
PyResult<Column> {
         ))));
     }
 
+    // As PySpark's literal conversion for a value of another type.
     let type_name = obj
         .get_type()
         .name()
         .map(|n| n.to_string())
         .unwrap_or_else(|_| "<unknown>".to_string());
-    Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(format!(
-        "Expected Column or scalar (int, float, str, bool, bytes, datetime, 
date, Decimal, None); got {type_name}"
-    )))
+    Err(crate::column::raise_pyspark(
+        obj.py(),
+        "PySparkTypeError",
+        "UNSUPPORTED_DATA_TYPE",
+        &[("data_type", &type_name)],
+    ))
+}
+
+/// The number of dimensions of `obj` if it is a `numpy.ndarray`. NumPy is 
looked up in
+/// `sys.modules`, as an array exists only when it is imported, so this does 
not import it.
+fn numpy_ndarray_ndim(obj: &Bound<'_, PyAny>) -> PyResult<Option<usize>> {
+    let py = obj.py();
+    let modules = py.import("sys")?.getattr("modules")?;
+    match modules.cast::<pyo3::types::PyDict>()?.get_item("numpy")? {
+        Some(numpy) if !numpy.is_none() && 
obj.is_instance(&numpy.getattr("ndarray")?)? => {
+            Ok(Some(obj.getattr("ndim")?.extract()?))
+        }
+        _ => Ok(None),
+    }
 }
 
 /// Build a literal for the non-primitive scalar Python types (`datetime`, 
`date`,
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index e81be66..64b7ea8 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -331,7 +331,37 @@ def col(col):
 
 
 def lit(col):
-    """Creates a Column of literal value."""
+    """Creates a Column of literal value.
+
+    Mirrors pyspark.sql.connect.functions.lit: a list is the array of the 
literals of its
+    elements, and a one-dimensional NumPy array that array cast to the array 
type of its
+    dtype.
+    """
+    if isinstance(col, _pyspark.Column):
+        return col
+    if isinstance(col, list):
+        if any(isinstance(c, _pyspark.Column) for c in col):
+            from pyspark.errors import PySparkValueError
+
+            raise PySparkValueError(
+                errorClass="COLUMN_IN_LIST", messageParameters={"func_name": 
"lit"}
+            )
+        return array(*[lit(c) for c in col])
+    import sys
+
+    # An ndarray exists only when NumPy is imported, so this does not import 
it.
+    np = sys.modules.get("numpy")
+    if np is not None and isinstance(col, np.ndarray) and col.ndim == 1:
+        from pyspark.errors import PySparkTypeError
+        from pyspark.sql.types import ArrayType, _from_numpy_type
+
+        dt = _from_numpy_type(col.dtype)
+        if dt is None:
+            raise PySparkTypeError(
+                errorClass="UNSUPPORTED_NUMPY_ARRAY_SCALAR",
+                messageParameters={"dtype": col.dtype.name},
+            )
+        return array(*[lit(c) for c in col]).cast(ArrayType(dt))
     return _pyfunc_lit(col)
 
 
@@ -542,6 +572,34 @@ def counter_diff(value, startTime=None):
     return _dispatch("counter_diff", [value] if startTime is None else [value, 
startTime])
 
 
+def _columns_of(cols):
+    # As PySpark's array, create_map, map_concat and struct: a single list, 
set or tuple
+    # is the columns.
+    if len(cols) == 1 and isinstance(cols[0], (list, set, tuple)):
+        return list(cols[0])
+    return list(cols)
+
+
+def array(*cols):
+    """array(*cols): an array of the columns, or of a single list, set or 
tuple of them."""
+    return _dispatch("array", _columns_of(cols))
+
+
+def create_map(*cols):
+    """create_map(*cols): a map of the key and value columns, or of a single 
list of them."""
+    return _dispatch("create_map", _columns_of(cols))
+
+
+def map_concat(*cols):
+    """map_concat(*cols): the union of the maps, or of a single list, set or 
tuple of them."""
+    return _dispatch("map_concat", _columns_of(cols))
+
+
+def struct(*cols):
+    """struct(*cols): a struct of the columns, or of a single list, set or 
tuple of them."""
+    return _dispatch("struct", _columns_of(cols))
+
+
 def lag(col, offset=1, default=None):
     """lag(col, offset=1, default=None): value offset rows before the current 
row."""
     return _wrap(_call_function("lag", _to_col(col), _pyfunc_lit(offset), 
_pyfunc_lit(default)))
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 83a584d..174e04e 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -363,6 +363,30 @@ def _has_type(dt, dts):
         return False
 
 
+def _from_numpy_type(nt):
+    """Convert NumPy type to Spark data type."""
+    import numpy as np
+
+    if nt == np.dtype("bool"):
+        return BooleanType()
+    elif nt == np.dtype("int8"):
+        return ByteType()
+    elif nt == np.dtype("int16"):
+        return ShortType()
+    elif nt == np.dtype("int32"):
+        return IntegerType()
+    elif nt == np.dtype("int64"):
+        return LongType()
+    elif nt == np.dtype("float32"):
+        return FloatType()
+    elif nt == np.dtype("float64"):
+        return DoubleType()
+    elif nt.type == np.dtype("str"):
+        return StringType()
+
+    return None
+
+
 def _create_row(fields, values):
     """A Row of ``values`` named by ``fields``, as in PySpark, whose pickled 
rows call it."""
     row = Row(*values)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 46c3a3a..b153d0e 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -573,6 +573,23 @@ def test_sql_runs_statements_eagerly(session):
     assert [tuple(r) for r in query.collect()] == [(1,)]
 
 
+def test_lists_and_numpy_arrays_are_arrays_as_pyspark(session):
+    np = pytest.importorskip("numpy")
+
+    df = session.range(1).select(
+        F.lit([1, 2]).alias("l"), F.lit(np.array([1, 2])).alias("n"),
+        F.lit(np.array([1.5], dtype=np.float32)).alias("f"), 
F.lit(np.array(["x"])).alias("s"),
+        F.array_contains(F.lit([[1], [2]]), [1]).alias("c"), F.array(["id", 
"id"]).alias("a"),
+        F.struct(["id"]).alias("t"), F.when(F.lit(True), [3]).alias("w"),
+    )
+    assert df.dtypes == [
+        ("l", "array<int>"), ("n", "array<bigint>"), ("f", "array<float>"), 
("s", "array<string>"),
+        ("c", "boolean"), ("a", "array<bigint>"), ("t", "struct<id:bigint>"), 
("w", "array<int>"),
+    ]
+    assert [tuple(r) for r in df.collect()] == [
+        ([1, 2], [1, 2], [1.5], ["x"], True, [0, 0], Row(id=0), [3])]
+
+
 def test_sql_fails_at_a_statement_that_does_not_parse(session):
     with pytest.raises(AnalysisException, match="PARSE_SYNTAX_ERROR"):
         session.sql("SELEC 1")
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index e6a6b3f..05ed52f 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4458,6 +4458,61 @@ def test_function_calls_render_as_pyspark(column, 
expected):
     assert repr(column(F)) == f"Column<'{expected}'>"
 
 
[email protected]("build,expected", [
+    # As PySpark's lit: a list is the array of the literals of its elements.
+    (lambda F: F.lit([1, 2]), "array(1, 2)"),
+    (lambda F: F.lit([[1], [2, 3]]), "array(array(1), array(2, 3))"),
+    (lambda F: F.lit([]), "array()"),
+    # Functions take a value with lit.
+    (lambda F: F.array_contains("a", [1]), "array_contains(a, array(1))"),
+    # array, create_map, map_concat and struct take a single list, set or 
tuple of columns.
+    (lambda F: F.array(["a", "b"]), "array(a, b)"),
+    (lambda F: F.array(("a", "b")), "array(a, b)"),
+    (lambda F: F.struct(["a", "b"]), "struct(a, b)"),
+    (lambda F: F.create_map([F.lit("k"), F.col("v")]), "map(k, v)"),
+    (lambda F: F.map_concat(["m1", "m2"]), "map_concat(m1, m2)"),
+])
+def test_lists_are_arrays_and_columns_as_pyspark(build, expected):
+    from pyspark.sql import functions as F
+
+    assert repr(build(F)) == f"Column<'{expected}'>"
+
+
+def test_lit_of_a_numpy_array_is_an_array_of_its_type_as_pyspark():
+    np = pytest.importorskip("numpy")
+    from pyspark.sql import functions as F
+
+    # The case of the type in the repr differs (SPARK-60948).
+    for value, expected in (
+        (np.array([1, 2]), "cast(array(1, 2) as array<bigint>)"),
+        (np.array([1.5], dtype=np.float32), "cast(array(1.5) as 
array<float>)"),
+        (np.array(["x"]), "cast(array(x) as array<string>)"),
+    ):
+        assert repr(F.lit(value)).lower() == f"column<'{expected}'>"
+
+
+def test_lit_errors_as_pyspark():
+    np = pytest.importorskip("numpy")
+    from pyspark.errors import PySparkTypeError, PySparkValueError
+    from pyspark.sql import functions as F
+
+    with pytest.raises(PySparkValueError) as error:
+        F.lit([F.col("a")])
+    assert error.value.getCondition() == "COLUMN_IN_LIST"
+    assert error.value.getMessageParameters() == {"func_name": "lit"}
+    for value, dtype in ((np.array([1], dtype=np.uint8), "uint8"), 
(np.array([1, "a"], dtype=object), "object")):
+        with pytest.raises(PySparkTypeError) as error:
+            F.lit(value)
+        assert error.value.getCondition() == "UNSUPPORTED_NUMPY_ARRAY_SCALAR"
+        assert error.value.getMessageParameters() == {"dtype": dtype}
+    # Rather than a scalar of an array of one element.
+    for value, name in (((1, 2), "tuple"), ({"a": 1}, "dict"), (object(), 
"object"), (np.array([[1]]), "ndarray")):
+        with pytest.raises(PySparkTypeError) as error:
+            F.lit(value)
+        assert error.value.getCondition() == "UNSUPPORTED_DATA_TYPE"
+        assert error.value.getMessageParameters() == {"data_type": name}
+
+
 def test_variant_delete_requires_paths():
     from pyspark.errors import PySparkValueError
     from pyspark.sql import functions as F


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to