On Tue, 2022-08-23 at 14:00 +0200, Petr Viktorin wrote: > On 23. 08. 22 11:46, Sebastian Berg wrote: > > On Tue, 2022-08-23 at 03:16 +0300, Matti Picus wrote: > > > > > > On 22/8/22 18:59, Eric Snow wrote: > > > > Hi all, > > > > > > > > <snip> > > > > > > devs than just me. Do you have any preference for or against > > > > any > > > > particular venue? > > > > > > > > Thanks! > > > > > > > > -eric > > > > _______________________________________________ > > > > NumPy-Discussion mailing list -- numpy-discussion@python.org > > > > To unsubscribe send an email to > > > > numpy-discussion-le...@python.org > > > > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > > > > Member address: matti.pi...@gmail.com > > > > > > Thanks for starting the conversation. I would personally prefer > > > the > > > discussion about NumPy be here, general discussions could be > > > elsewhere. > > > > > > > > > Please correct me if I am wrong: I understand that multiple > > > interpreters > > > would require us to (at least): > > > > > > These days, I was somewhat hoping that the HPy effort might give us > > subinterpreters without having two seperate efforts going on at the > > same time. Since much of the refactors are probably identical > > between > > the two and it seemed some significant effort might go into that. > > > > But of course starting with subinterpreter support without HPy > > probably > > also helps the HPy effort. > > Both should help each other. > > > > - refactor all the static module global state in NumPy and make > > > it > > > re-entrant or immortal including converting stack-allocated > > > PyTypeObjects to heap types. > > > > What is the status of immortality? None of these seem forbidding > > on > > first sight, so long that we can get the state everywhere. > > Having immortal object seems convenient, but probably not > > particularly > > necessary. > > > > Most of our state is currently in static variables in functions > > (usually filled in dynamically at first call). That is very > > convenient > > since it doesn't require a global list anywhere. > > > > I suppose moving it to module-state may well require a global list > > (or > > is there a nice other pattern?). But while tedious, it doesn't > > seem > > problematic. > > A struct for the module state is the state of the art, yes. > > > Switching to heap types should not be a big deal I suspect. > > > > > > > > - find a mechanism to access the per-interpreter module state > > > > > > > One thing that I am not clear about are e.g. creation functions. > > They > > are public C-API so they have no way of getting a "self" or > > type/module > > passed in. How will such a function get the module state? > > > > Now, we could likely replace those functions in the long run (or > > even > > just remove many). But it seems to me that we may need a > > `PyType_GetModuleByDef()` that is passed _only_ the `module_def`? > > Then you're looking at per-interpreter state, or thread-locals. > That's > problematic, e.g. you now need to handle clean-up at interpreter > shutdown, and the that isn't well supported. (Or leak -- AFAIK that's > what NumPy currently does when Python's single interpreter is > finalized?) > I do urge you to assume that there can be multiple isolated NumPy > modules created from a single def, even in a single interpreter. It's > an > additional constraint, but since it's conceptually simple I do think > it > makes up for itself in regularity/maintainability/reviewability. > > And if the CPython API is lacking, it would be best to solve that in > CPython. >
The issue is that we have public C-API that will be lacking the necessary information. Maybe pretty deep API (I am not certain). Now that I think about it, even things like the type is unclear to me. `&PyArray_Type` would not be per interpreter (unless we figure out immortality). But it exists as public API just like `Py_None`, etc.? Our public C-API is currently exported as a single static struct into the library loading NumPy. If types depend on the interpreter, it would seem we need to redo the whole mechanism? Further, many of the functions would need to be adapted. We might be able to hack that the API looks the same [1]. However, it cannot be ABI compatible, so we would need a whole new API table/export mechnism and some sort of shim to allow compiling against older NumPy versions but using it with all versions (otherwise we need 2+ years of patience). Of course there might be a point in saying that most C-API use is initially not subinterpreter ready, but it does seem like a pretty huge limitation... Cheers, Sebastian [1] I.e. smuggle in module state without the library importing the NumPy C-API having to change its code. > > > - carefully consider places in the code that we steal references > > > either > > > intentionally or because that is the CPython C-API we are using > > > > > > > This is an issue for HPy that needs to be cleared up, although I am > > wondering how important it is for subinterpreters as such? > > Not important. Borrowed references work mainly to enable optimized > collections that don't store full PyObjects -- currently that's HPy > territory. > If you find the C API forcing you to steal references, I do want to > eventually fix that in CPython to make switching to HPy easy (and > eventually to enable the optimizations in CPython). A lot of “better” > alternative APIs was actually added in recent versions, and I'd > welcome > requests for what to prioritize for Python 3.12+. > > > > - measure the performance implications of the necessary changes > > > > > > - plan forward/backward compatibility > > > > > > > > > One other thing I am not quite sure about right now is GIL > > grabbing. > > `PyGILState_Ensure()` will continue to work reliably? > > This used to be one of my main worries. It is also something we > > can > > fix-up (pass through additional information), but where a fallback > > seems needed. > > Per-interpreter GIL is an *additional* step. I believe it will need > its > own opt-in mechanism. But subinterpreter support is a prerequisite > for it. > So yes, PyGILState_Ensure will still acquire a global lock for you. > > > > Cheers, > > > > Sebastian > > > > > > > > > > > > This seems like a significant undertaking, and is why we have > > > rejected > > > casual calls for supporting multiple interpreters in the past > > > [2], > > > [3], > > > [4]. Supporting multiple interpreters is currently not on the > > > NumPy > > > roadmap [0]. Priorities can be changed, through dialog with the > > > NumPy > > > community, and others can propose changes to NumPy via NEPs, PRs, > > > and > > > issues, but we are unlikely to engage directly in the work if it > > > is > > > not > > > an agreed upon goal. There are other initiatives around NumPy > > > that > > > may > > > dovetail with multiple interpreters. For instance the HPy group > > > hit > > > many > > > of the issues above when creating a port of NumPy [5]. It would > > > be > > > good > > > to get like-minded people talking about this and to pool > > > resources, > > > maybe someone on this list has a strong opinion and would be > > > willing > > > to > > > put in some work on the subject. > > > > > > > > > One thing CPython could do is to provide clear documentation how > > > to > > > port > > > a small c-extension module [1] > > > > > > > > > Matti > > > > > > > > > [0] https://numpy.org/neps/roadmap.html > > > > > > [1] https://github.com/python/cpython/issues/79601 > > > > > > [2] https://github.com/numpy/numpy/issues/665 > > > > > > [3] https://github.com/numpy/numpy/issues/14384 > > > > > > [4] https://github.com/numpy/numpy/issues/16963 > > > > > > [5] > > > https://github.com/hpyproject/numpy-hpy/tree/graal-team/hpy#readme > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list -- numpy-discussion@python.org > > > To unsubscribe send an email to numpy-discussion-le...@python.org > > > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > > > Member address: sebast...@sipsolutions.net > > > > > > _______________________________________________ > > NumPy-Discussion mailing list -- numpy-discussion@python.org > > To unsubscribe send an email to numpy-discussion-le...@python.org > > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > > Member address: encu...@gmail.com > _______________________________________________ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: sebast...@sipsolutions.net _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
