[issue45307] Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 2ca4ab8031107f9e474e2ce26561ab9ad51faf9c by Pablo Galindo (Serhiy 
Storchaka) in branch '3.10':
[3.10] bpo-45307: Restore private C API function 
_PyImport_FindExtensionObject() (GH-28594)
https://github.com/python/cpython/commit/2ca4ab8031107f9e474e2ce26561ab9ad51faf9c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45307] Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
priority: release blocker -> 
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45307] Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset ec4e2ec241acb3bf4e04a351c06c0b05a1e4b7d2 by Serhiy Storchaka in 
branch '3.10':
[3.10] bpo-45307: Restore private C API function 
_PyImport_FindExtensionObject() (GH-28594)
https://github.com/python/cpython/commit/ec4e2ec241acb3bf4e04a351c06c0b05a1e4b7d2


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45307] Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> I don't understand which long term solution do you propose.


A kind reminder that for 3.10 we cannot add new APIs so this proposal still 
*makes sense* for the short term (3.10).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45307] Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders

[STINNER Victor]

STINNER Victor  added the comment:

> Until 3.10 makes equivalent functionality available or another workaround is 
> supported,

I don't understand which long term solution do you propose.

> a properly designed public API is probably a better solution

Can you propose a public API?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45307] Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
nosy: +pablogsal
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45307] Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Gregory, could you please build Python 3.10 with PR 28594 applied and test 
whether py2exe and PyOxidizer work well with it? The restored function no 
longer used in the CPython code, so it is not tested now.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45307] Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +26974
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28594

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45307] Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders

[Gregory Szorc]

New submission from Gregory Szorc :

https://bugs.python.org/issue41994 / commit 
4db8988420e0a122d617df741381b0c385af032c removed 
_PyImport_FindExtensionObject() from the C API and it is no longer available in 
CPython 3.10 after alpha 5.

At least py2exe and PyOxidizer rely on this API for implementing a custom 
module loader for extension modules. Essentially, both want to implement a 
custom Loader.create_module() so they can use a custom mechanism for injecting 
a shared library into the process. While the details shouldn't be important 
beyond "they can't use imp.create_dynamic()," both use a similar technique that 
hooks LoadLibrary() on Windows to enable them to load a DLL from memory (as 
opposed to a file).

While I don't have the extension module loading mechanism fully paged in to my 
head at the moment, I believe the reason that _PyImport_FindExtensionObject() 
(now effectively import_find_extension()) is important for py2exe and 
PyOxidizer is because they need to support at most once initialization, 
including honoring the multi-phase initialization semantics. Since the state of 
extension modules is stored in `static PyObject *extensions` and the thread 
state (which are opaque to the C API), the loss of 
_PyImport_FindExtensionObject() means there is no way to check for and use an 
existing extension module module object from the bowels of the importer 
machinery. And I think this means it isn't possible to implement well-behaved 
alternate extension module loaders any more.

I'm aware the deleted API was "private" and probably shouldn't have been used 
in the first place. And what py2exe and PyOxidizer are doing here is rather 
unorthodox.

In my mind the simplest path forward is restoring 
_PyImport_FindExtensionObject(). But a properly designed public API is probably 
a better solution.

Until 3.10 makes equivalent functionality available or another workaround is 
supported, PyOxidizer won't be able to support loading extension modules from 
memory on Windows on Python 3.10. This is unfortunate. But probably not a deal 
breaker and I can probably go several months shipping PyOxidizer with this 
regression without too many complaints.

--
components: C API
messages: 402754
nosy: indygreg, petr.viktorin, serhiy.storchaka, vstinner
priority: normal
severity: normal
status: open
title: Removal of _PyImport_FindExtensionObject() in 3.10 limits custom 
extension module loaders
type: enhancement
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com