HyukjinKwon opened a new pull request, #56789: URL: https://github.com/apache/spark/pull/56789
### What changes were proposed in this pull request? `as_spark_type` in `python/pyspark/pandas/typedef/typehints.py` only recognized the legacy form of `numpy.typing.NDArray`, where `NDArray[T]` is a `GenericAlias` with `__origin__ is np.ndarray` and `__args__ == (shape, np.dtype[T])`. Starting with **NumPy 2.5**, `NDArray` is declared as a PEP 695 type alias (`type NDArray[ScalarT: np.generic] = np.ndarray[Any, np.dtype[ScalarT]]`). At runtime `NDArray[T]` is then a `GenericAlias` whose `__origin__` is the `NDArray` **`TypeAliasType`** (not `np.ndarray`) and whose `__args__` is the single scalar type `(T,)`. The old check no longer matched, so the annotation fell through to `issubclass(tpe.__origin__, list)`, which raised `TypeError: issubclass() arg 1 must be a class`. This PR: - adds `_extract_ndarray_scalar_type`, which detects the new PEP 695 `NDArray` form (origin is a `TypeAliasType` whose `__value__` expands to `np.ndarray`) and returns its scalar element type; - routes that scalar through `as_spark_type` to build the `ArrayType`, mirroring the legacy branch; - guards the existing `list` branch with `isinstance(tpe.__origin__, type)` so a non-class `__origin__` can never crash `issubclass` again. The legacy NumPy (<2.5) path is unchanged. ### Why are the changes needed? On runners with NumPy 2.5 (e.g. `Build / MacOS-26`, Python 3.12), pyspark-pandas tests that use `NDArray[...]` type hints — `test_apply_batch_with_type`, `test_apply_with_type`, `test_transform_batch_with_type`, and their `*-connect` parities — fail with: ``` TypeError: Cannot interpret 'NDArray[int]' as a data type TypeError: issubclass() arg 1 must be a class ``` Evidence: apache/spark run `28064134730` (`Build / ... MacOS-26`, Python 3.12), module `pyspark-pandas` / `pyspark-pandas-connect`. ### Does this PR introduce any user-facing change? No. It restores correct `NDArray` type-hint handling under NumPy 2.5. ### How was this patch tested? The existing `pyspark.pandas.tests.test_typedef.TypeHintTests.test_as_spark_type_pandas_on_spark_dtype` already exercises `as_spark_type(ntp.NDArray[...])` / `pandas_on_spark_type(ntp.NDArray[...])` across many element types; the fix makes it pass under NumPy 2.5 while keeping it green under NumPy 2.4. Validated locally on Python 3.12 against: - the real NumPy 2.4 `NDArray` (legacy form) — no regression; - a faithful simulation of NumPy 2.5's PEP 695 `NDArray` (origin is a `TypeAliasType`) — previously reproduced the exact `issubclass() arg 1 must be a class` failure; now `as_spark_type(NDArray[int]) == ArrayType(LongType())`. NumPy 2.5 is not yet on PyPI for the OSS CI image, so this cannot be reproduced on the SBT fork lane; it reproduces on the NumPy-2.5 `MacOS-26` scheduled lane. ### Was this patch authored or co-authored using generative AI tooling? Yes, Generated-by: Claude Code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
