ngoldbaum opened a new pull request, #51157: URL: https://github.com/apache/arrow/pull/51157
### Rationale for this change NumPy 2.0 added `StringDType`, a variable-width UTF-8 string dtype. `pa.array` currently rejects it with `Unsupported numpy type 2056`. Now that pyarrow requires NumPy 2.0 the conversion can use the public `NpyString_*` C API. This supersedes #50951. The first commit is @alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here. ### What changes are included in this PR? - `NumPyDtypeToArrow` maps `StringDType` to `string`, so it is also the inferred type for `pa.array`, `pa.infer_type`, `pa.from_numpy_dtype` and lists of such arrays. `large_string` and `string_view` can be requested with `type`. Any other type raises a `TypeError` naming the requested type. - The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it. - A null entry in a StringDType array becomes an Arrow null unless the dtype's `na_object` is a `str`. NumPy reports such entries as that string from `__getitem__`, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as `np.nan` and `pd.NA`, `None`, and arbitrary objects all become arrow nulls. - A short section in `docs/source/python/numpy.rst` describing the support and the semantics for converting missing data. If left unfixed, https://github.com/apache/arrow/issues/51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry. ### Are these changes tested? Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, `mask`, `None`, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering. ### Are there any user-facing changes? Yes. `pa.array` and the inference functions accept `StringDType` arrays. AI disclosure: I used an AI model to work on the follow-up commits on top of @alippai's first commit and for code review. * GitHub Issue: #42018 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
