HyukjinKwon opened a new pull request, #56788: URL: https://github.com/apache/spark/pull/56788
### What changes were proposed in this pull request? `as_spark_type` detected `numpy.typing.NDArray[...]` annotations via `tpe.__origin__ is np.ndarray` and, when that did not match, fell through to `issubclass(tpe.__origin__, list)`. numpy 2.5 changed the internal structure of `NDArray` aliases so that `tpe.__origin__` is no longer the bare `numpy.ndarray` class but a subscripted generic alias (which is not a class). As a result the NDArray branch is skipped and `issubclass(tpe.__origin__, list)` raises `TypeError: issubclass() arg 1 must be a class` (surfaced to the user as `TypeError: Cannot interpret 'NDArray[int]' as a data type`). This PR: - adds a `_numpy_ndarray_element_type` helper that unwraps the `__origin__` chain to detect `numpy.ndarray` robustly across numpy versions, then scans the type arguments for the `np.dtype[...]` scalar and returns it; - guards the list branch with `isclass(...)` so the `issubclass` call cannot raise on a non-class origin. ### Why are the changes needed? The scheduled `Build / Python-only (Python 3.12, MacOS26)` job installs numpy 2.5.0 and fails `pyspark.pandas` and `pyspark.pandas.connect` `test_apply_batch_with_type` with the `TypeError` above (e.g. apache/spark Actions run 28064134730). The breakage blocks that CI job and breaks numpy `NDArray` return-type inference for pandas-on-Spark `apply_batch`/`transform_batch` under numpy >= 2.5. ### Does this PR introduce _any_ user-facing change? No. It restores correct Spark type inference for numpy `NDArray` return annotations under numpy >= 2.5; behavior on older numpy is unchanged. ### How was this patch tested? Existing parametrized coverage in `pyspark.pandas.tests.test_typedef` exercises `as_spark_type(NDArray[...])` and `pandas_on_spark_type(NDArray[...])` over numpy/Python scalar types; that is the regression test for this path. The detection/extraction helper was additionally verified against numpy 2.3.5, numpy 2.4.6, and a simulated numpy-2.5 alias structure (where `__origin__` is a non-class generic alias): `NDArray[int]`/`NDArray[np.float64]` resolve to the correct scalar; `list[int]`, the bare `np.ndarray`, and plain `int` resolve to `None` (falling through to the existing branches). Validated on macOS-26 with numpy 2.5.0 via GitHub Actions on the fork (the `Build / Python-only (Python 3.12, MacOS26)` workflow). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
