At least on arm64, this appears to be a crash in a @numba.jit function.
Given numba's known issues on non-x86 (e.g. #1033907), this suggests
that actually fixing it _may_ not be reasonable.
In pandas I don't recommend/build/test-depend on numba on the problem
architectures, but python-sparse has a hard Depends on numba so that
doesn't appear to be an option here.
A potential option is to disable the tests on these architectures and
display a warning on import (for an example, see
https://salsa.debian.org/science-team/statsmodels/-/commit/a64430166b1e48135c58ec41330e3ce9b2206b46
).