This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new aec003c [SPARK-60982][CONNECT][PYTHON] Build an array of a list or a
NumPy array in lit() as PySpark does
aec003c is described below
commit aec003c57fccb1b3481eeef0f6cf5e9e33913821
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 13:31:01 2026 +0900
[SPARK-60982][CONNECT][PYTHON] Build an array of a list or a NumPy array in
lit() as PySpark does
### What changes were proposed in this pull request?
[SPARK-60982](https://issues.apache.org/jira/browse/SPARK-60982): lists and
NumPy arrays become arrays as in PySpark 4.3.0.
- **`lit` of a list** is the array of the literals of its elements, as
PySpark's `lit` builds it:
- `F.lit([1, 2])` is `array(1, 2)`, and nested lists nest;
- a list with a Column raises `PySparkValueError` `COLUMN_IN_LIST`.
- **`lit` of a one-dimensional NumPy array** is that array cast to the
array type of its dtype, by PySpark's `_from_numpy_type`, which
`pyspark.sql.types` now has:
- `F.lit(np.array([1, 2]))` is `CAST(array(1, 2) AS ARRAY<BIGINT>)`;
- a dtype without a Spark type raises `UNSUPPORTED_NUMPY_ARRAY_SCALAR`.
- **Function arguments:** a list in a function's argument is such an array
too, as PySpark's functions take a value with `lit`:
- `F.array_contains("a", [1])` is `array_contains(a, array(1))`;
- `F.when(c, [1, 2])` has the value `array(1, 2)`.
- **A single list of columns:** `array`, `create_map`, `map_concat` and
`struct` take a single list, set or tuple as their columns, as in PySpark:
`F.array(["a", "b"])` is `array(a, b)`.
- **Other values:** a value of another type raises `PySparkTypeError`
`UNSUPPORTED_DATA_TYPE`, as PySpark's literal conversion does, instead of
`TypeError`. This covers a tuple, a dict, and any other NumPy array.
### Why are the changes needed?
- `F.lit([1, 2])` and `F.array(["a", "b"])` raised `TypeError: Expected
Column or scalar ...; got list`.
- A NumPy array of one element was converted to that element:
- `F.lit(np.array([1.5]))` was the scalar `1.5`;
- `F.lit(np.array([1], dtype=np.uint8))` was `1`;
- `F.col("a") == np.array([1])` compared with `1`.
### Does this PR introduce _any_ user-facing change?
Yes, these values become arrays, or raise, as in PySpark 4.3.0.
### How was this patch tested?
- Probes of 44 cases give the same results and errors as official PySpark
4.3.0, apart from reprs that
[SPARK-60948](https://issues.apache.org/jira/browse/SPARK-60948) covers, such
as the case of a cast's type. The cases cover `lit` of lists, NumPy arrays and
scalars and other values, lists in function arguments, and single lists of
columns.
- Two kinds of value stay different. `datetime.time` and
`datetime.timedelta`, which PySpark's `lit` takes, now raise
`UNSUPPORTED_DATA_TYPE`, as before but with PySpark's error class. TIME is
tracked by [SPARK-60947](https://issues.apache.org/jira/browse/SPARK-60947).
- An operator with a list, such as `F.col("a") == [1, 2]`, compares with
the array, as classic PySpark does, where PySpark 4.3.0's Connect client fails
with `AttributeError`.
- Against a local Spark 4.3.0 server, a select of these columns gives the
same types and values as official PySpark 4.3.0.
- New offline tests check the reprs and errors, and a new live test checks
the types and values. They also pass with official PySpark 4.3.0, and fail
without this change.
- The lib tests and every integration test, e2e included, against the local
server (645 passed), the build of all tests with `wasm-udf`, the offline Python
suites (658 passed, 8 live cases skipped), and the live drop-in and Arrow
suites (64 passed) and live cases (10 passed). Clippy reports no diagnostics in
the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #188 from HyukjinKwon/SPARK-60982-lit-list.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/functions.rs | 44 +++++++++++++++++++++++++--
python/pyspark/sql/functions.py | 60 ++++++++++++++++++++++++++++++++++++-
python/pyspark/sql/types.py | 24 +++++++++++++++
python/tests/test_dropin_e2e.py | 17 +++++++++++
python/tests/test_dropin_offline.py | 55 ++++++++++++++++++++++++++++++++++
5 files changed, 196 insertions(+), 4 deletions(-)
diff --git a/crates/pyspark-rs/src/functions.rs
b/crates/pyspark-rs/src/functions.rs
index 3b628ad..0e95069 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -157,6 +157,27 @@ pub fn to_column(obj: &Bound<'_, PyAny>) ->
PyResult<Column> {
))));
}
+ // A list, or a one-dimensional NumPy array, is an array, as PySpark's
functions take a
+ // value with `lit`: `pyspark.sql.functions.lit` builds it. Another NumPy
array is not a
+ // literal value; NumPy would convert one of a single element to a scalar
below.
+ let ndarray = numpy_ndarray_ndim(obj)?;
+ if obj.is_instance_of::<pyo3::types::PyList>() || ndarray == Some(1) {
+ let column = obj
+ .py()
+ .import("pyspark.sql.functions")?
+ .getattr("lit")?
+ .call1((obj,))?;
+ return Ok(column.extract::<PyRef<'_, PyColumn>>()?.column.clone());
+ }
+ if ndarray.is_some() {
+ return Err(crate::column::raise_pyspark(
+ obj.py(),
+ "PySparkTypeError",
+ "UNSUPPORTED_DATA_TYPE",
+ &[("data_type", "ndarray")],
+ ));
+ }
+
// Non-primitive scalars must be checked BEFORE int/float: Decimal defines
// __float__/__int__, and a pandas Timestamp (a datetime subclass) defines
__int__
// (nanoseconds), so an early int/float extract would silently mis-encode
them.
@@ -194,14 +215,31 @@ pub fn to_column(obj: &Bound<'_, PyAny>) ->
PyResult<Column> {
))));
}
+ // As PySpark's literal conversion for a value of another type.
let type_name = obj
.get_type()
.name()
.map(|n| n.to_string())
.unwrap_or_else(|_| "<unknown>".to_string());
- Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(format!(
- "Expected Column or scalar (int, float, str, bool, bytes, datetime,
date, Decimal, None); got {type_name}"
- )))
+ Err(crate::column::raise_pyspark(
+ obj.py(),
+ "PySparkTypeError",
+ "UNSUPPORTED_DATA_TYPE",
+ &[("data_type", &type_name)],
+ ))
+}
+
+/// The number of dimensions of `obj` if it is a `numpy.ndarray`. NumPy is
looked up in
+/// `sys.modules`, as an array exists only when it is imported, so this does
not import it.
+fn numpy_ndarray_ndim(obj: &Bound<'_, PyAny>) -> PyResult<Option<usize>> {
+ let py = obj.py();
+ let modules = py.import("sys")?.getattr("modules")?;
+ match modules.cast::<pyo3::types::PyDict>()?.get_item("numpy")? {
+ Some(numpy) if !numpy.is_none() &&
obj.is_instance(&numpy.getattr("ndarray")?)? => {
+ Ok(Some(obj.getattr("ndim")?.extract()?))
+ }
+ _ => Ok(None),
+ }
}
/// Build a literal for the non-primitive scalar Python types (`datetime`,
`date`,
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index e81be66..64b7ea8 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -331,7 +331,37 @@ def col(col):
def lit(col):
- """Creates a Column of literal value."""
+ """Creates a Column of literal value.
+
+ Mirrors pyspark.sql.connect.functions.lit: a list is the array of the
literals of its
+ elements, and a one-dimensional NumPy array that array cast to the array
type of its
+ dtype.
+ """
+ if isinstance(col, _pyspark.Column):
+ return col
+ if isinstance(col, list):
+ if any(isinstance(c, _pyspark.Column) for c in col):
+ from pyspark.errors import PySparkValueError
+
+ raise PySparkValueError(
+ errorClass="COLUMN_IN_LIST", messageParameters={"func_name":
"lit"}
+ )
+ return array(*[lit(c) for c in col])
+ import sys
+
+ # An ndarray exists only when NumPy is imported, so this does not import
it.
+ np = sys.modules.get("numpy")
+ if np is not None and isinstance(col, np.ndarray) and col.ndim == 1:
+ from pyspark.errors import PySparkTypeError
+ from pyspark.sql.types import ArrayType, _from_numpy_type
+
+ dt = _from_numpy_type(col.dtype)
+ if dt is None:
+ raise PySparkTypeError(
+ errorClass="UNSUPPORTED_NUMPY_ARRAY_SCALAR",
+ messageParameters={"dtype": col.dtype.name},
+ )
+ return array(*[lit(c) for c in col]).cast(ArrayType(dt))
return _pyfunc_lit(col)
@@ -542,6 +572,34 @@ def counter_diff(value, startTime=None):
return _dispatch("counter_diff", [value] if startTime is None else [value,
startTime])
+def _columns_of(cols):
+ # As PySpark's array, create_map, map_concat and struct: a single list,
set or tuple
+ # is the columns.
+ if len(cols) == 1 and isinstance(cols[0], (list, set, tuple)):
+ return list(cols[0])
+ return list(cols)
+
+
+def array(*cols):
+ """array(*cols): an array of the columns, or of a single list, set or
tuple of them."""
+ return _dispatch("array", _columns_of(cols))
+
+
+def create_map(*cols):
+ """create_map(*cols): a map of the key and value columns, or of a single
list of them."""
+ return _dispatch("create_map", _columns_of(cols))
+
+
+def map_concat(*cols):
+ """map_concat(*cols): the union of the maps, or of a single list, set or
tuple of them."""
+ return _dispatch("map_concat", _columns_of(cols))
+
+
+def struct(*cols):
+ """struct(*cols): a struct of the columns, or of a single list, set or
tuple of them."""
+ return _dispatch("struct", _columns_of(cols))
+
+
def lag(col, offset=1, default=None):
"""lag(col, offset=1, default=None): value offset rows before the current
row."""
return _wrap(_call_function("lag", _to_col(col), _pyfunc_lit(offset),
_pyfunc_lit(default)))
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 83a584d..174e04e 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -363,6 +363,30 @@ def _has_type(dt, dts):
return False
+def _from_numpy_type(nt):
+ """Convert NumPy type to Spark data type."""
+ import numpy as np
+
+ if nt == np.dtype("bool"):
+ return BooleanType()
+ elif nt == np.dtype("int8"):
+ return ByteType()
+ elif nt == np.dtype("int16"):
+ return ShortType()
+ elif nt == np.dtype("int32"):
+ return IntegerType()
+ elif nt == np.dtype("int64"):
+ return LongType()
+ elif nt == np.dtype("float32"):
+ return FloatType()
+ elif nt == np.dtype("float64"):
+ return DoubleType()
+ elif nt.type == np.dtype("str"):
+ return StringType()
+
+ return None
+
+
def _create_row(fields, values):
"""A Row of ``values`` named by ``fields``, as in PySpark, whose pickled
rows call it."""
row = Row(*values)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 46c3a3a..b153d0e 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -573,6 +573,23 @@ def test_sql_runs_statements_eagerly(session):
assert [tuple(r) for r in query.collect()] == [(1,)]
+def test_lists_and_numpy_arrays_are_arrays_as_pyspark(session):
+ np = pytest.importorskip("numpy")
+
+ df = session.range(1).select(
+ F.lit([1, 2]).alias("l"), F.lit(np.array([1, 2])).alias("n"),
+ F.lit(np.array([1.5], dtype=np.float32)).alias("f"),
F.lit(np.array(["x"])).alias("s"),
+ F.array_contains(F.lit([[1], [2]]), [1]).alias("c"), F.array(["id",
"id"]).alias("a"),
+ F.struct(["id"]).alias("t"), F.when(F.lit(True), [3]).alias("w"),
+ )
+ assert df.dtypes == [
+ ("l", "array<int>"), ("n", "array<bigint>"), ("f", "array<float>"),
("s", "array<string>"),
+ ("c", "boolean"), ("a", "array<bigint>"), ("t", "struct<id:bigint>"),
("w", "array<int>"),
+ ]
+ assert [tuple(r) for r in df.collect()] == [
+ ([1, 2], [1, 2], [1.5], ["x"], True, [0, 0], Row(id=0), [3])]
+
+
def test_sql_fails_at_a_statement_that_does_not_parse(session):
with pytest.raises(AnalysisException, match="PARSE_SYNTAX_ERROR"):
session.sql("SELEC 1")
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index e6a6b3f..05ed52f 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4458,6 +4458,61 @@ def test_function_calls_render_as_pyspark(column,
expected):
assert repr(column(F)) == f"Column<'{expected}'>"
[email protected]("build,expected", [
+ # As PySpark's lit: a list is the array of the literals of its elements.
+ (lambda F: F.lit([1, 2]), "array(1, 2)"),
+ (lambda F: F.lit([[1], [2, 3]]), "array(array(1), array(2, 3))"),
+ (lambda F: F.lit([]), "array()"),
+ # Functions take a value with lit.
+ (lambda F: F.array_contains("a", [1]), "array_contains(a, array(1))"),
+ # array, create_map, map_concat and struct take a single list, set or
tuple of columns.
+ (lambda F: F.array(["a", "b"]), "array(a, b)"),
+ (lambda F: F.array(("a", "b")), "array(a, b)"),
+ (lambda F: F.struct(["a", "b"]), "struct(a, b)"),
+ (lambda F: F.create_map([F.lit("k"), F.col("v")]), "map(k, v)"),
+ (lambda F: F.map_concat(["m1", "m2"]), "map_concat(m1, m2)"),
+])
+def test_lists_are_arrays_and_columns_as_pyspark(build, expected):
+ from pyspark.sql import functions as F
+
+ assert repr(build(F)) == f"Column<'{expected}'>"
+
+
+def test_lit_of_a_numpy_array_is_an_array_of_its_type_as_pyspark():
+ np = pytest.importorskip("numpy")
+ from pyspark.sql import functions as F
+
+ # The case of the type in the repr differs (SPARK-60948).
+ for value, expected in (
+ (np.array([1, 2]), "cast(array(1, 2) as array<bigint>)"),
+ (np.array([1.5], dtype=np.float32), "cast(array(1.5) as
array<float>)"),
+ (np.array(["x"]), "cast(array(x) as array<string>)"),
+ ):
+ assert repr(F.lit(value)).lower() == f"column<'{expected}'>"
+
+
+def test_lit_errors_as_pyspark():
+ np = pytest.importorskip("numpy")
+ from pyspark.errors import PySparkTypeError, PySparkValueError
+ from pyspark.sql import functions as F
+
+ with pytest.raises(PySparkValueError) as error:
+ F.lit([F.col("a")])
+ assert error.value.getCondition() == "COLUMN_IN_LIST"
+ assert error.value.getMessageParameters() == {"func_name": "lit"}
+ for value, dtype in ((np.array([1], dtype=np.uint8), "uint8"),
(np.array([1, "a"], dtype=object), "object")):
+ with pytest.raises(PySparkTypeError) as error:
+ F.lit(value)
+ assert error.value.getCondition() == "UNSUPPORTED_NUMPY_ARRAY_SCALAR"
+ assert error.value.getMessageParameters() == {"dtype": dtype}
+ # Rather than a scalar of an array of one element.
+ for value, name in (((1, 2), "tuple"), ({"a": 1}, "dict"), (object(),
"object"), (np.array([[1]]), "ndarray")):
+ with pytest.raises(PySparkTypeError) as error:
+ F.lit(value)
+ assert error.value.getCondition() == "UNSUPPORTED_DATA_TYPE"
+ assert error.value.getMessageParameters() == {"data_type": name}
+
+
def test_variant_delete_requires_paths():
from pyspark.errors import PySparkValueError
from pyspark.sql import functions as F
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]