On 23. 08. 22 16:19, Sebastian Berg wrote:
On Tue, 2022-08-23 at 14:00 +0200, Petr Viktorin wrote:
On 23. 08. 22 11:46, Sebastian Berg wrote:
[snip]
One thing that I am not clear about are e.g. creation functions.
They
are public C-API so they have no way of getting a "self" or
type/module
passed in. How will such a function get the module state?
Now, we could likely replace those functions in the long run (or
even
just remove many). But it seems to me that we may need a
`PyType_GetModuleByDef()` that is passed _only_ the `module_def`?
Then you're looking at per-interpreter state, or thread-locals.
That's
problematic, e.g. you now need to handle clean-up at interpreter
shutdown, and the that isn't well supported. (Or leak -- AFAIK that's
what NumPy currently does when Python's single interpreter is
finalized?)
I do urge you to assume that there can be multiple isolated NumPy
modules created from a single def, even in a single interpreter. It's
an
additional constraint, but since it's conceptually simple I do think
it
makes up for itself in regularity/maintainability/reviewability.
And if the CPython API is lacking, it would be best to solve that in
CPython.
The issue is that we have public C-API that will be lacking the
necessary information. Maybe pretty deep API (I am not certain).
Let's find out!
Now that I think about it, even things like the type is unclear to me.
`&PyArray_Type` would not be per interpreter (unless we figure out
immortality). But it exists as public API just like `Py_None`, etc.?
Exposing PyArray_Type that way means that it must be a static type.
Those are immortal. (That said, static types are not compatible with
Stable ABI -- which is related but not strictly necessary for
subinterpreter support -- so if there's a chance to make it
`my_numpy_api->PyArray_Type`, it would be better.)
Our public C-API is currently exported as a single static struct into
the library loading NumPy. If types depend on the interpreter, it
would seem we need to redo the whole mechanism?
Right, sounds like it needs to be a dynamically allocated struct.
In the interim, one instance of the struct is static: that's the one
used for anything that doesn't support multiple interpreters yet, and
also as the module state in one “main” module object. (That would be the
first module to be loaded, and until everything switches over, it'd get
an unpaired incref to become “immortal” and leak at exit.)
Further, many of the functions would need to be adapted. We might be
able to hack that the API looks the same [1]. However, it cannot be
ABI compatible, so we would need a whole new API table/export mechnism
and some sort of shim to allow compiling against older NumPy versions
but using it with all versions (otherwise we need 2+ years of
patience).
Having one static “main” module state in the interim would also help here.
Of course there might be a point in saying that most C-API use is
initially not subinterpreter ready, but it does seem like a pretty huge
limitation...
A huge limitation, but it might be a good way to break up the work to
make it more manageable :)
Cheers,
Sebastian
[1] I.e. smuggle in module state without the library importing the
NumPy C-API having to change its code.
_______________________________________________
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
