On Thu, Apr 23, 2026 at 3:40 PM Sebastian Berg <[email protected]> wrote:
> Hi all, > > I would like a project like `ml_dtypes` to be able to transition to the > new DType API, I believe that is vital to make easy. > If anyone has concerns around this, please chime in. IMO, we need to > just do this to move forward. > This seems like a good idea. The details are a bit complex it looks like, but if it works for the known users of the legacy API, then that should give enough confidence. It probably does need some integration test like building `ml_dtypes` against main, and then testing it against whatever the minimum version is that `ml_dtypes` supports. Cheers, Ralf > > I.e. provide an API that allows using the new DType API all the way > back with NumPy 2.0 (by compiling with NumPy 2.5 or vendoring some > code). [1] > The trick being that we can use the new DType API, so long the DType > still is "legacy" and thus uses all the old code paths (unless the new > API clearly replaced them). > > Without such "backport" there is a problem that any regression/bug > means that we can't use the new API until the buggy NumPy version is > dropped and since nothing is bug-free that can be years of waiting... > > The path is the following PR https://github.com/numpy/numpy/pull/31302 > * Create a new DType registration slot, where the old defining struct > is passed in. (We'll use less and less of it over time.) > * We have a `static inline` function in NumPy 2.5 (which downstream can > vendor to compile with older NumPy versions!). > This static inline function does a bit of a crazy dance, but the end > result is a working "old but new" DType (`ml_dtypes` tests pass). > > This gives a path, and a path mean we can actually start deprecating > things soon. Even if some things are baked in slightly more after this > (I think this is rather little in practice), having a path now is still > better than not. > Now is also a great time, because it is a good time to drop NumPy <2. > > Testing is a bit hard. The truly exhaustive tests is probably just > indirectly `ml_dtypes` (which google tests exhaustively). (The > `ml_dtypes` pass completely with this approach.) > > Cheers, > > Sebastian > > > > [1] There is exactly one corner-case I saw, and that is that we print: > `np.array(..., dtype=repr(dtype))` for these DTypes, while legacy > ones printed `dtype=dtype.name`, which is a bit shorter. > (We can change that in NumPy 2.5+, but not on old versions.) > _______________________________________________ > NumPy-Discussion mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3//lists/numpy-discussion.python.org > Member address: [email protected] >
_______________________________________________ NumPy-Discussion mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3//lists/numpy-discussion.python.org Member address: [email protected]
