This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 9d4cbcc20900 [SPARK-57679][PYTHON] Fix numpy type checking
9d4cbcc20900 is described below
commit 9d4cbcc2090034c228e21e98a82b2d1ec85631aa
Author: Tian Gao <[email protected]>
AuthorDate: Tue Jun 30 14:14:29 2026 -0700
[SPARK-57679][PYTHON] Fix numpy type checking
Add a branch to detect `NDArray` for `numpy >= 2.5.0`.
Our detection for `NDArray` is a bit fragile. `numpy` changed the type
alias in `2.5.0` so we need another way to detect it.
CI is failing -
https://github.com/apache/spark/actions/runs/28064134730/job/83087583742
No.
Local test passed
Yes, Claude Code (Opus 4.8 high)
Closes #56757 from gaogaotiantian/fix-numpy-type-checking.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit 5f7570c8db3bec32f5e808b3c6d98036bcfcbef1)
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/pandas/typedef/typehints.py | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/pandas/typedef/typehints.py
b/python/pyspark/pandas/typedef/typehints.py
index a4ed9f996fe4..3759ab815afa 100644
--- a/python/pyspark/pandas/typedef/typehints.py
+++ b/python/pyspark/pandas/typedef/typehints.py
@@ -158,10 +158,20 @@ def as_spark_type(
and hasattr(tpe, "__args__")
and len(tpe.__args__) > 1
):
- # numpy.typing.NDArray
+ # numpy.typing.NDArray for numpy < 2.5
return types.ArrayType(
as_spark_type(tpe.__args__[1].__args__[0],
raise_error=raise_error)
)
+ elif (
+ hasattr(tpe, "__origin__")
+ and hasattr(tpe.__origin__, "__value__")
+ and getattr(tpe.__origin__.__value__, "__origin__", None) is
np.ndarray
+ and hasattr(tpe, "__args__")
+ and len(tpe.__args__) > 0
+ ):
+ # numpy.typing.NDArray for numpy >= 2.5: a PEP 695 type alias
whose __value__
+ # resolves to np.ndarray[shape, dtype[scalar]], with the scalar at
__args__[0]
+ return types.ArrayType(as_spark_type(tpe.__args__[0],
raise_error=raise_error))
if isinstance(tpe, np.dtype) and tpe == np.dtype("object"):
pass
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
