Hi Joren, Thanks for the detailed comments and for the explanation you gave Marten.
On Thu, Aug 27, 2026 at 5:56 PM Joren Hammudoglu via NumPy-Discussion < [email protected]> wrote: > > Hi Marten, > > All valid questions; there are indeed a lot of moving parts in Python typing, and NumPy's stubs don't make it any easier, to say the least :P > >> You mention: >> >> > # missing scalar type >> >> > I think it's important that we also introduce a companion scalar type >> > for this, instead of using the `builtins.bytes` for this. >> >> To me this seems somewhat orthogonal to the NEP. If it is important for >> `ByteStringDType`, it is even more important for `StringDType`, which >> will be substantially more used, and presumably the solution would be >> the same for both (some franken-subclass of `bytes` or `str` and >> `np.generic`, just like `np.float64` (which subclasses both `float` and >> `np.generic`; its mro is `np.floating`, `np.inexact`, `np.number`, >> `np.generic`, `float`, `object`). > > > The missing scalar type for StringDType is indeed a painful problem right now for static typing. > I tried my best to make the most out of it, but this required adding a bunch of special-casing > (in the form of overloads) to function stubs that support StringDType, as well as several > # type: ignore comments to silence type-checker (rightly) errors where str is now used in places > where only numpy.generic subtypes are allowed. This was quite a lot of work, and made the stubs > even more complicated than they already were. And today, there are still functions (e.g. np.min, iirc) > that don't support StringDType because they don't have special-cased workaround overloads yet. > > So BytesDType will come without a scalar type, we'll also have to apply similar workarounds for it, > and the stubs will become an even bigger mess. > > Last meeting Nathan told me that adding these scalar types is a very tricky thing though, so I realize > that it's a big ask. But if BytesDType gets a scalar type, Naming nit: BytesDType is legacy np.bytes_ dtype, so I'm calling it ByteStringDType. > then I believe it's indeed not that much more > difficult to also add the missing StringDType scalar type, which would make this a 2-for-1 deal :) > Specifically it's tricky for any subtype of `str`, because `str` doesn't store data internally as UTF-8. It's also quite tricky to write a correct `str` subtype in C. See this discourse thread, where Petr Viktorin describes the difficulty at a high level: https://discuss.python.org/t/82543/12 I don't think anything ever came out of that thread and there still isn't an API in CPython to get a new instance of a PyUnicode subtype or a stable way to initialize our own without making a lot of assumptions about CPython internals. That said, Python's bytes instances do store the raw bytes directly, so it's much more straightforward for us to subtype PyBytes_Type: https://github.com/python/cpython/blob/486b000c6c19c555f03b481f735f4dec498f0f67/Include/cpython/bytesobject.h#L5-L15 I have a branch that updates my NEP-58 prototype to add a new np.vbytes scalar type that subtypes both np.generic and bytes. So I think that satisfies your concern. I'll be updating the NEP text shortly to discuss the new scalar and how it impacts typing. However, for downstream users who care about na_object, my plan is to preserve StringDType's behavior: missing entries can be an arbitrary object and I won't plan on making the scalar handle the na_object case. See below for more about na_object and why I think it's worth keeping as a feature despite the type checking issues. >> > Currently `StringDType` is simply type-unsafe, and it's impossible to >> > express a `StringDType` array using the widely used >> > `numpy.typing.NDArray`. For example, this leads to `f(x: >> > npt.NDArray[np.generic])` rejecting every `StringDType` array, even >> > though `npt.NDArray[np.generic]` is supposed to represent the "top >> > type" of `ndarray`. >> >> As a non-typer, this feels weird. It suggests `NDArray` is treated a >> bit like a list, but then what is the problem with `NDArray[str]`? Or >> why would one not write `NDArray[Any]`? >> >> Maybe more broadly, why treat it like a list? Not all dtypes will have >> associated scalars; e.g., the SFloatDescr that is one of numpy's test >> cases, does not have a scalar type at all (I'm building on that example >> for astropy, so it has real-world uses). >> >> Indeed, the concept of scalars associated with dtypes is inconsistent >> with the Array API. At some level, it would be rather lovely if we >> could get rid of them altogether... >> >> I guess to ask it differently, why can one not write it with the dtype, >> i.e., `NDArray[StringDType]` so that the most general form would be >> `NDArray[np.dtype]`? >> >> Now probably this has all been discussed to death already, but I guess >> the general question is whether rather than try to adjust fairly logical >> choices to typing, it is possible to expand what typing can do so that >> it can express those choices... > > > Today, numpy.typing.NDArray is defined as: > > type NDArray[ScalarT: np.generic] = np.ndarray[_AnyShape, np.dtype[ScalarT]] > > > The important bit here is that the ScalarT type parameter is restricted to (subtypes of) np.generic. > This means that NDArray[str] or NDArray[bytes] are not allowed (by static type-checkers). > Forcing it by ignoring the resulting typing errors is possible, but results in undefined behavior. So we, > and downstream users, are now not able to use NDArray to express StringDType or BytesDType arrays, > even though that's precisely what NDArray is intended for. So the only option is to spell it out in full as > np.ndarray[_AnyShape, np.dtypes.BytesDType] (where _AnyShape = tuple[Any, ...]), which is > clearly verbose, and I don't expect many downstream users to even consider it an option. Because, after all, > for any other dtype, NDArray is all you need. > > Removing the bound on ScalarT might seem like an obvious solution here, but doing so would be a breaking > change for many downstream libraries, and a backwards-incompatible one at that. Because, for example, where > it currently is always possible to assign an element of an NDArray to some x: np.generic, that will then result in > a type error, as that would indeed be type-unsafe. > > You suggested an alternative to this, where we'd allow writing NDArray[StringDType]. But that would require > some form of pattern matching in the declaration of type NDArray[ScalarT: np.generic | np.dtype] to > deal with the two possible types that ScalarT could then resolve to. Perhaps this will be possible to express in > the future, but for now it's not an option I'm afraid. > >> > # na_object >> >> > As you probably already know, this feature of `StringDType` is >> > problematic for static typing, because there is no good way to express >> > this functionality in the stubs. And although I understand that it >> > would be strange if the direct dual to `StringDType` wouldn't have the >> > same `na_object` functionality, I'd rather we not repeat the mistakes >> > of the past, taking the resulting inconsistency for granted. >> >> I'm a bit confused about this one. How is this different from, e.g., >> the concept of byte order, which is not captured by the typing either? >> Also, pandas has particular integer values to indicate missing. Isn't >> that similar? Can that be captured by typing? > > > NumPy's stubs and the numpy.typing types have been built around the idea that the elements of arrays can be > fully statically described in terms of their scalar type. But the na_object, which can be any object, is defined on > the dtype itself. So by only considering the scalar type, as we always have, we lose that na_object information, > i.e. which type it can be. A dedicated scalar type could solve this, because we could make it a generic type with > the type of na_object as type parameter. Is it really so bad? Aren't we currently only losing that information for e.g. `np.nan` or other float sentintels? None, pd.NA, etc all work because they're singletons and string missing data don't have any type issues. > But what a dedicated scalar type wouldn't help with, is the problem that na_object can be any object, even > scalar-like objects such as float that's already associated with float64 scalar types everywhere in the stubs. > So this introduces ambiguity in the inference of functions accepting array-likes like np.array. This is why the > na_object of StringDType isn't well supported in the stubs at the moment. > > In the last community call we talked about a potential solution for this ambiguity: restricting the na_object to a > limited set of allowed types, e.g. None, and/or a dedicated sentinel or some enum. Nathan suggested including > float, specifically float("nan"). But the issue with that is that, unlike e.g. int and str, float("nan") cannot > be expressed as a typing.Literal. This might be something that could be changed in the typing spec, but I'll > have to investigate a bit more to see how feasible that would be. > I'd rather not restrict it. Object arrays in real codebases represent nulls as None (pyarrow, polars), NaN (pandas' default string dtype, xarray) and pd.NA (pandas' nullable and arrow-backed columns). Pandas will hand you a single object array with NaN in one column and None in another. pd.NA is a pandas object numpy can't enumerate, so any closed set excludes pandas' arrays. I also found some discussion <https://peps.python.org/pep-0586/#illegal-parameters-for-literal-at-type-check-time> in PEP 586 where float literaly were explicitly excluded from support for Literal: > Floats: e.g. Literal[3.14]. Representing Literals of infinity or NaN in a clean way is tricky; real-world APIs are unlikely to vary their behavior based on a float parameter. I think we may have found a reason to support `Literal[np.nan]` at least in real-world uses of object arrays and StringDType. Of course fixing that requires updating the typing spec. But I'd much rather improve the typing spec to better model the behavior of real Python programs than restrict NumPy's runtime behavior because of current limitations in the typing spec. > > > I hope that answers your questions :) > > Cheers, > Joren > > On Wed, 26 Aug 2026 at 00:07, Marten van Kerkwijk via NumPy-Discussion < [email protected]> wrote: >> >> Hi Joren, >> >> As a person who doesn't use typing, some perhaps naive questions about >> your comments, but ones that express my worry that the current state of >> typing, which would seem a fast-moving and still relatively in-flux >> feature, starts to influence the convenience of code. >> >> You mention: >> >> > # missing scalar type >> >> > I think it's important that we also introduce a companion scalar type >> > for this, instead of using the `builtins.bytes` for this. >> >> To me this seems somewhat orthogonal to the NEP. If it is important for >> `ByteStringDType`, it is even more important for `StringDType`, which >> will be substantially more used, and presumably the solution would be >> the same for both (some franken-subclass of `bytes` or `str` and >> `np.generic`, just like `np.float64` (which subclasses both `float` and >> `np.generic`; its mro is `np.floating`, `np.inexact`, `np.number`, >> `np.generic`, `float`, `object`). >> >> > Currently `StringDType` is simply type-unsafe, and it's impossible to >> > express a `StringDType` array using the widely used >> > `numpy.typing.NDArray`. For example, this leads to `f(x: >> > npt.NDArray[np.generic])` rejecting every `StringDType` array, even >> > though `npt.NDArray[np.generic]` is supposed to represent the "top >> > type" of `ndarray`. >> >> As a non-typer, this feels weird. It suggests `NDArray` is treated a >> bit like a list, but then what is the problem with `NDArray[str]`? Or >> why would one not write `NDArray[Any]`? >> >> Maybe more broadly, why treat it like a list? Not all dtypes will have >> associated scalars; e.g., the SFloatDescr that is one of numpy's test >> cases, does not have a scalar type at all (I'm building on that example >> for astropy, so it has real-world uses). >> >> Indeed, the concept of scalars associated with dtypes is inconsistent >> with the Array API. At some level, it would be rather lovely if we >> could get rid of them altogether... >> >> I guess to ask it differently, why can one not write it with the dtype, >> i.e., `NDArray[StringDType]` so that the most general form would be >> `NDArray[np.dtype]`? >> >> Now probably this has all been discussed to death already, but I guess >> the general question is whether rather than try to adjust fairly logical >> choices to typing, it is possible to expand what typing can do so that >> it can express those choices... >> >> > # na_object >> >> > As you probably already know, this feature of `StringDType` is >> > problematic for static typing, because there is no good way to express >> > this functionality in the stubs. And although I understand that it >> > would be strange if the direct dual to `StringDType` wouldn't have the >> > same `na_object` functionality, I'd rather we not repeat the mistakes >> > of the past, taking the resulting inconsistency for granted. >> >> I'm a bit confused about this one. How is this different from, e.g., >> the concept of byte order, which is not captured by the typing either? >> Also, pandas has particular integer values to indicate missing. Isn't >> that similar? Can that be captured by typing? >> >> Again, apologies for what are probably naive questions... >> >> All the best, >> >> Marten >> >> _______________________________________________ >> NumPy-Discussion mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> https://mail.python.org/mailman3//lists/numpy-discussion.python.org >> Member address: [email protected] > > _______________________________________________ > NumPy-Discussion mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3//lists/numpy-discussion.python.org > Member address: [email protected]
_______________________________________________ NumPy-Discussion mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3//lists/numpy-discussion.python.org Member address: [email protected]
