This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 4ad36945 [SPARK-61102][CONNECT][PYTHON] Match PySpark NumPy
scalar-class checks
4ad36945 is described below
commit 4ad36945b752f28a2e3213344781b3aa0cac8357
Author: DB Tsai <[email protected]>
AuthorDate: Mon Nov 9 06:46:23 2026 +0900
[SPARK-61102][CONNECT][PYTHON] Match PySpark NumPy scalar-class checks
### What changes were proposed in this pull request?
Apply PySpark 4.3.0's numeric scalar-class checks after NumPy dtype
inference. Distinct aliases such as `np.longlong` or an eight-byte
`np.longdouble` now raise PySpark's bare `AssertionError` when their class is
rejected. Supported classes retain their literal types, and wider unsupported
dtypes retain their structured errors. Update scalar and array regressions for
platform-dependent aliases.
### Why are the changes needed?
[SPARK-61102](https://issues.apache.org/jira/browse/SPARK-61102).
Dtype-kind and width checks alone accept classes that official PySpark rejects,
including longdouble on macOS arm64.
### Does this PR introduce _any_ user-facing change?
Yes. Scalar and array literals apply PySpark's class validation as well as
dtype inference.
### How was this patch tested?
- Formatting, Rust core/API units, integration-test compilation with and
without WASM, and native-extension Python suites: 2,170 passed, eight
live-server skips.
- Twenty no-server scalar/array expectation checks against official PySpark
4.3.0. Two native live cases passed with verified imports, covering three
supported NumPy schema types and existing listener behavior; this is not
exhaustive NumPy value coverage.
- API inspection found no missing inspected members across 12 class
surfaces. This checks member presence; signatures and behavior differ in
places, including the separate Row tuple gap.
- Full raw transport on the default four-thread fixture: 87 normal files,
2,586 passed, 641 skipped, one deselected, no failures/errors/retries. Eleven
files were entirely skipped. The existing metadata-column manifest entry failed
its separate advisory check because the test catalog plugin was unavailable
(one failure, 57 deselections).
- Separate local-cluster resource/artifact phase: 15 passing invocations
across four modules (ten substantive cases and five helpers), 14 JVM-guarded
skips, no failures/errors/retries.
The original full run's seeded-random failure remains recorded: the
unchanged test failed identically in official Python and both saved Rust
transports on two threads, then passed in all three on its default four-thread
fixture. Earlier CPU-queue expirations executed no checks.
The full/cluster suites use official Python converters over this build's
Rust raw transport; they do not establish native high-level API, native
ReleaseSession or native artifact-upload parity. They lack per-child import
attestation. Server logs retain deliberate negative-test diagnostics and
shutdown races; direct successful shutdown was recorded without final
descendant-process telemetry.
Clippy remains advisory exit 101. Linux and Windows were not run locally.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex
This pull request and its description were written by OpenAI Codex.
Closes #381 from dbtsai/codex/python-rust-gap-numpy-scalar.
Authored-by: DB Tsai <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/functions.rs | 38 ++++++++++++++--
python/tests/test_dropin_offline.py | 86 ++++++++++++++++++++++++++++++++++++-
2 files changed, 120 insertions(+), 4 deletions(-)
diff --git a/crates/pyspark-rs/src/functions.rs
b/crates/pyspark-rs/src/functions.rs
index 74da2f4f..14ac379f 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -333,9 +333,11 @@ fn numpy_ndarray_ndim(obj: &Bound<'_, PyAny>) ->
PyResult<Option<usize>> {
/// The literal of a NumPy scalar, typed as PySpark's
`LiteralExpression._infer_type` types it: a
/// bool, a signed int of 8 to 64 bits (tinyint to bigint), or a float32 or
float64. A str or bytes
/// scalar is `None`, for the string and binary conversions to take. Any other
NumPy scalar (an
-/// unsigned int, float16, longdouble, complex, datetime64 or timedelta64) is
-/// UNSUPPORTED_DATA_TYPE, named by its Python type. NumPy is looked up in
`sys.modules`, as a
-/// NumPy scalar exists only when NumPy is imported.
+/// unsigned int, float16, complex, datetime64 or timedelta64) is
UNSUPPORTED_DATA_TYPE,
+/// named by its Python type. A supported dtype with a scalar class outside
PySpark's accepted
+/// classes raises a bare `AssertionError`, such as a distinct `longlong`, or
an eight-byte
+/// `longdouble` on macOS arm64 and Windows. Wider `longdouble` dtypes are
unsupported.
+/// NumPy is looked up in `sys.modules`, as a NumPy scalar exists only when
NumPy is imported.
fn numpy_scalar_literal(obj: &Bound<'_, PyAny>) ->
PyResult<Option<LiteralExpression>> {
let py = obj.py();
let modules = py.import("sys")?.getattr("modules")?;
@@ -352,6 +354,36 @@ fn numpy_scalar_literal(obj: &Bound<'_, PyAny>) ->
PyResult<Option<LiteralExpres
let dtype = obj.getattr("dtype")?;
let kind: String = dtype.getattr("kind")?.extract()?;
let itemsize: usize = dtype.getattr("itemsize")?.extract()?;
+ // Mirror LiteralExpression's accepted scalar classes after dtype
inference. Aliases can
+ // share a supported dtype without being instances of its canonical NumPy
scalar class.
+ // Check the matching width first; the remaining classes preserve the
reference's full set.
+ let scalar_classes: &[&str] = match (kind.as_str(), itemsize) {
+ ("i", 1) => &["int8"],
+ ("i", 2) => &["int16", "int8"],
+ ("i", 4) => &["int32", "int16", "int8"],
+ ("i", 8) => &["int64", "int32", "int16", "int8"],
+ ("f", 4) => &["float32"],
+ ("f", 8) => &["float64", "float32"],
+ _ => &[],
+ };
+ if !scalar_classes.is_empty() {
+ let mut accepted = match kind.as_str() {
+ "i" => obj.is_instance_of::<pyo3::types::PyInt>(),
+ "f" => obj.is_instance_of::<pyo3::types::PyFloat>(),
+ _ => false,
+ };
+ if !accepted {
+ for class in scalar_classes {
+ if obj.is_instance(&numpy.getattr(*class)?)? {
+ accepted = true;
+ break;
+ }
+ }
+ }
+ if !accepted {
+ return Err(pyo3::exceptions::PyAssertionError::new_err(()));
+ }
+ }
let literal = match (kind.as_str(), itemsize) {
("U", _) | ("S", _) => return Ok(None),
("i", 1) => LiteralExpression::Byte(obj.extract::<i64>()? as i32),
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 36900270..fec6f738 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -10644,7 +10644,6 @@ def
test_numpy_scalars_of_other_types_are_unsupported_as_pyspark(mock_connect):
(np.uint8(5), "uint8"),
(np.uint64(5), "uint64"),
(np.float16(1.5), "float16"),
- (np.longdouble(1.5), "longdouble"),
(np.complex128(1 + 2j), "complex128"),
(np.datetime64("2020-01-01"), "datetime64"),
(np.timedelta64(1, "D"), "timedelta64"),
@@ -10659,6 +10658,90 @@ def
test_numpy_scalars_of_other_types_are_unsupported_as_pyspark(mock_connect):
assert error.value.getMessageParameters() == {"dtype": "uint8"}
+def test_numpy_longdouble_scalar_matches_dtype_and_class_checks_as_pyspark():
+ np = pytest.importorskip("numpy")
+ from pyspark.errors import PySparkTypeError
+ from pyspark.sql import functions as F
+
+ value = np.longdouble(1.5)
+ if isinstance(value, (float, np.float32, np.float64)):
+ pytest.skip("longdouble aliases a supported float scalar class here")
+ if value.dtype == np.dtype("float64"):
+ # A dtype mapping to DoubleType does not suffice: PySpark also asserts
that its
+ # scalar is a float, np.float32 or np.float64. Distinct eight-byte
longdouble
+ # classes (such as on macOS arm64 and Windows) fail without a message.
+ with pytest.raises(AssertionError) as error:
+ F.lit(value)
+ assert error.value.args == ()
+ else:
+ # Linux CI normally has a wider longdouble, which exercises the
unsupported
+ # dtype path; the distinct integer aliases below exercise the class
check there.
+ with pytest.raises(PySparkTypeError) as error:
+ F.lit(value)
+ assert error.value.getCondition() == "UNSUPPORTED_DATA_TYPE"
+ assert error.value.getMessageParameters() == {"data_type":
type(value).__name__}
+
+
+def test_numpy_longdouble_array_matches_dtype_and_element_checks_as_pyspark():
+ np = pytest.importorskip("numpy")
+ from pyspark.errors import PySparkTypeError
+ from pyspark.sql import functions as F
+
+ value = np.array([1.5], dtype=np.longdouble)
+ if isinstance(value[0], (float, np.float32, np.float64)):
+ pytest.skip("longdouble aliases a supported float scalar class here")
+ if value.dtype == np.dtype("float64"):
+ # A mapped array dtype still builds its literals through each scalar's
checks.
+ with pytest.raises(AssertionError) as error:
+ F.lit(value)
+ assert error.value.args == ()
+ else:
+ with pytest.raises(PySparkTypeError) as error:
+ F.lit(value)
+ assert error.value.getCondition() == "UNSUPPORTED_NUMPY_ARRAY_SCALAR"
+ assert error.value.getMessageParameters() == {"dtype":
value.dtype.name}
+
+
[email protected]("name", [
+ "byte", "short", "intc", "int_", "intp", "long", "longlong", "single",
"double",
+])
[email protected]("as_array", [False, True], ids=["scalar", "array"])
+def test_numpy_numeric_aliases_match_reference_scalar_classes(name, as_array):
+ np = pytest.importorskip("numpy")
+ from pyspark.sql import functions as F
+
+ # Read real scalar exports without NumPy's deprecated builtin-alias lookup
warnings.
+ scalar_type = vars(np).get(name)
+ if scalar_type is None or not issubclass(scalar_type, np.generic):
+ pytest.skip(f"numpy.{name} is not a NumPy scalar class in this NumPy")
+ value = scalar_type(1.5 if np.issubdtype(scalar_type, np.floating) else 5)
+ kind, size = value.dtype.kind, value.dtype.itemsize
+ if kind == "i":
+ # These are the classes asserted by LiteralExpression for the inferred
width.
+ accepted_classes = (int,) + tuple(
+ getattr(np, f"int{width}") for width in (8, 16, 32, 64) if width
<= size * 8
+ )
+ spark_type = {1: "tinyint", 2: "smallint", 4: "int", 8: "bigint"}[size]
+ rendered = "5"
+ else:
+ accepted_classes = (float, np.float32) + ((np.float64,) if size == 8
else ())
+ spark_type = {4: "float", 8: "double"}[size]
+ rendered = "1.5"
+ literal = np.array([value], dtype=scalar_type) if as_array else value
+ if not isinstance(value, accepted_classes):
+ # Distinct longlong scalars have int64 dtype but fail the class
assertion on
+ # Linux/macOS; platforms where it aliases int64 exercise the success
case.
+ with pytest.raises(AssertionError) as error:
+ F.lit(literal)
+ assert error.value.args == ()
+ elif as_array:
+ assert repr(F.lit(literal)).lower() == (
+ f"column<'cast(array({rendered}) as array<{spark_type}>)'>"
+ )
+ else:
+ assert repr(F.lit(literal)) == f"Column<'{rendered}'>"
+
+
# ---------------------------------------------------------------------------
4.3.0 gaps
@@ -11036,6 +11119,7 @@ def
test_lit_of_a_numpy_array_is_an_array_of_its_type_as_pyspark():
for value, expected in (
(np.array([1, 2]), "cast(array(1, 2) as array<bigint>)"),
(np.array([1.5], dtype=np.float32), "cast(array(1.5) as
array<float>)"),
+ (np.array([1.5], dtype=np.float64), "cast(array(1.5) as
array<double>)"),
(np.array(["x"]), "cast(array(x) as array<string>)"),
):
assert repr(F.lit(value)).lower() == f"column<'{expected}'>"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]