On 23. 08. 22 11:46, Sebastian Berg wrote:
On Tue, 2022-08-23 at 03:16 +0300, Matti Picus wrote:
On 22/8/22 18:59, Eric Snow wrote:
Hi all,
<snip>
devs than just me. Do you have any preference for or against any
particular venue?
Thanks!
-eric
_______________________________________________
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com
Thanks for starting the conversation. I would personally prefer the
discussion about NumPy be here, general discussions could be
elsewhere.
Please correct me if I am wrong: I understand that multiple
interpreters
would require us to (at least):
These days, I was somewhat hoping that the HPy effort might give us
subinterpreters without having two seperate efforts going on at the
same time. Since much of the refactors are probably identical between
the two and it seemed some significant effort might go into that.
But of course starting with subinterpreter support without HPy probably
also helps the HPy effort.
Both should help each other.
- refactor all the static module global state in NumPy and make it
re-entrant or immortal including converting stack-allocated
PyTypeObjects to heap types.
What is the status of immortality? None of these seem forbidding on
first sight, so long that we can get the state everywhere.
Having immortal object seems convenient, but probably not particularly
necessary.
Most of our state is currently in static variables in functions
(usually filled in dynamically at first call). That is very convenient
since it doesn't require a global list anywhere.
I suppose moving it to module-state may well require a global list (or
is there a nice other pattern?). But while tedious, it doesn't seem
problematic.
A struct for the module state is the state of the art, yes.
Switching to heap types should not be a big deal I suspect.
- find a mechanism to access the per-interpreter module state
One thing that I am not clear about are e.g. creation functions. They
are public C-API so they have no way of getting a "self" or type/module
passed in. How will such a function get the module state?
Now, we could likely replace those functions in the long run (or even
just remove many). But it seems to me that we may need a
`PyType_GetModuleByDef()` that is passed _only_ the `module_def`?
Then you're looking at per-interpreter state, or thread-locals. That's
problematic, e.g. you now need to handle clean-up at interpreter
shutdown, and the that isn't well supported. (Or leak -- AFAIK that's
what NumPy currently does when Python's single interpreter is finalized?)
I do urge you to assume that there can be multiple isolated NumPy
modules created from a single def, even in a single interpreter. It's an
additional constraint, but since it's conceptually simple I do think it
makes up for itself in regularity/maintainability/reviewability.
And if the CPython API is lacking, it would be best to solve that in
CPython.
- carefully consider places in the code that we steal references
either
intentionally or because that is the CPython C-API we are using
This is an issue for HPy that needs to be cleared up, although I am
wondering how important it is for subinterpreters as such?
Not important. Borrowed references work mainly to enable optimized
collections that don't store full PyObjects -- currently that's HPy
territory.
If you find the C API forcing you to steal references, I do want to
eventually fix that in CPython to make switching to HPy easy (and
eventually to enable the optimizations in CPython). A lot of “better”
alternative APIs was actually added in recent versions, and I'd welcome
requests for what to prioritize for Python 3.12+.
- measure the performance implications of the necessary changes
- plan forward/backward compatibility
One other thing I am not quite sure about right now is GIL grabbing.
`PyGILState_Ensure()` will continue to work reliably?
This used to be one of my main worries. It is also something we can
fix-up (pass through additional information), but where a fallback
seems needed.
Per-interpreter GIL is an *additional* step. I believe it will need its
own opt-in mechanism. But subinterpreter support is a prerequisite for it.
So yes, PyGILState_Ensure will still acquire a global lock for you.
Cheers,
Sebastian
This seems like a significant undertaking, and is why we have
rejected
casual calls for supporting multiple interpreters in the past [2],
[3],
[4]. Supporting multiple interpreters is currently not on the NumPy
roadmap [0]. Priorities can be changed, through dialog with the NumPy
community, and others can propose changes to NumPy via NEPs, PRs, and
issues, but we are unlikely to engage directly in the work if it is
not
an agreed upon goal. There are other initiatives around NumPy that
may
dovetail with multiple interpreters. For instance the HPy group hit
many
of the issues above when creating a port of NumPy [5]. It would be
good
to get like-minded people talking about this and to pool resources,
maybe someone on this list has a strong opinion and would be willing
to
put in some work on the subject.
One thing CPython could do is to provide clear documentation how to
port
a small c-extension module [1]
Matti
[0] https://numpy.org/neps/roadmap.html
[1] https://github.com/python/cpython/issues/79601
[2] https://github.com/numpy/numpy/issues/665
[3] https://github.com/numpy/numpy/issues/14384
[4] https://github.com/numpy/numpy/issues/16963
[5]
https://github.com/hpyproject/numpy-hpy/tree/graal-team/hpy#readme
_______________________________________________
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: sebast...@sipsolutions.net
_______________________________________________
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: encu...@gmail.com
_______________________________________________
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
