This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5f7570c8db3b [SPARK-57679][PYTHON] Fix numpy type checking
5f7570c8db3b is described below
commit 5f7570c8db3bec32f5e808b3c6d98036bcfcbef1
Author: Tian Gao <[email protected]>
AuthorDate: Tue Jun 30 14:14:29 2026 -0700
[SPARK-57679][PYTHON] Fix numpy type checking
### What changes were proposed in this pull request?
Add a branch to detect `NDArray` for `numpy >= 2.5.0`.
### Why are the changes needed?
Our detection for `NDArray` is a bit fragile. `numpy` changed the type
alias in `2.5.0` so we need another way to detect it.
CI is failing -
https://github.com/apache/spark/actions/runs/28064134730/job/83087583742
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Local test passed
### Was this patch authored or co-authored using generative AI tooling?
Yes, Claude Code (Opus 4.8 high)
Closes #56757 from gaogaotiantian/fix-numpy-type-checking.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/pandas/typedef/typehints.py | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/pandas/typedef/typehints.py
b/python/pyspark/pandas/typedef/typehints.py
index 69c20288dc12..8d99584f81a9 100644
--- a/python/pyspark/pandas/typedef/typehints.py
+++ b/python/pyspark/pandas/typedef/typehints.py
@@ -155,8 +155,18 @@ def as_spark_type(
and hasattr(tpe, "__args__")
and len(tpe.__args__) > 1
):
- # numpy.typing.NDArray
+ # numpy.typing.NDArray for numpy < 2.5
return types.ArrayType(as_spark_type(tpe.__args__[1].__args__[0],
raise_error=raise_error))
+ elif (
+ hasattr(tpe, "__origin__")
+ and hasattr(tpe.__origin__, "__value__")
+ and getattr(tpe.__origin__.__value__, "__origin__", None) is np.ndarray
+ and hasattr(tpe, "__args__")
+ and len(tpe.__args__) > 0
+ ):
+ # numpy.typing.NDArray for numpy >= 2.5: a PEP 695 type alias whose
__value__
+ # resolves to np.ndarray[shape, dtype[scalar]], with the scalar at
__args__[0]
+ return types.ArrayType(as_spark_type(tpe.__args__[0],
raise_error=raise_error))
if isinstance(tpe, np.dtype) and tpe == np.dtype("object"):
pass
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
