[issue34309] Trouble when reloading extension modules.

[Dong-hee Na]

Change by Dong-hee Na :


--
nosy: +corona10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34309] Trouble when reloading extension modules.

[Shakeeb Alireza]

Shakeeb Alireza  added the comment:

In my project (https://github.com/shakfu/py-js), which provides an embedded 
python3 interpreter to Max/MSP in the form of an 'external' plugin, I have 
faced similar issues of being unable to reload extension modules, namely numpy, 
without reliably crashing the host application, in this case Max. 

Being able to reload extension modules cleanly is absolutely critical 
especially in case when python is embedded. Since Numpy is one of the key 
reasons why people would want to use Python, such a constraint, in this 
embedded context, becomes a sufficient reason not to use Python at all.

For example, I have recently taken note of similar frustration with this exact 
same issue from the VCV project 
(https://community.vcvrack.com/t/blowing-the-dust-off-python-in-prototype/12909).
 I quote: "I should add that CPython and more notably numpy do not support nor 
advise a complete restart of the interpreter in embedded scenarios without 
restarting the host process which kind of defeats our purpose in Prototype.
At that point I think I can safely take a step back and turn to the dev 
community looking for suggestions. Should we throw away numpy, or even python, 
altogether?"

--
nosy: +shakfu
versions: +Python 3.9 -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34309] Trouble when reloading extension modules.

[Brett Cannon]

Change by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34309] Trouble when reloading extension modules.

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34309] Trouble when reloading extension modules.

[chris]

chris  added the comment:

For short-term / mid-term we have now decided to start python as seperate 
process and interact with some kind of IPC. That leads to a limited interaction 
model between python and the embedded app but it has the advantage that 
unloading is possible (by simply restarting python).

Hopefully, at some day python will have better support for unloading / 
reloading extension modules, but as some pointed out this will take time also 
until extension modules adopt new API discussed in the PEPs.

Thanks for discussion and information!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34309] Trouble when reloading extension modules.

[Nick Coghlan]

Nick Coghlan  added the comment:

As others have noted, dynamically reloading CPython extension modules is akin 
to dynamically reloading any other C/C++ shared library, so it has enough 
opportunities for things to go wrong that we consider allowing the shared state 
to persist across initialize/finalize cycles the less problematic of two 
problematic options (at least for now).

Reliable hot reloading support is typically a property of pure Python modules 
(and even at that higher level, inter-module dependencies can still cause 
problems at runtime).

(FWIW, this problem is currently also the main reason we don't offer an in-REPL 
package installation command - while PEP 489 offers significant opportunities 
for improvement, it's likely to be years before we see widespread adoption of 
that, so we prefer to advise folks to run an installer outside the REPL, then 
restart and replay their interactive session)

If subinterpreters are an option though, then yeah, that has far more potential 
to be viable. It wouldn't be trivial, as we'd need to add dlmopen support 
(thanks Stack Overflow [1]) to give the subinterpreter a truly independent copy 
of the shared library (and also work out whatever the equivalent to dlmopen 
might be on other platforms), but going down that path could also potentially 
provide a way around the known problems with global state leaking between 
subinterpreters via extension modules.

[1] 
https://stackoverflow.com/questions/48864659/loading-shared-library-twice/4598#4598

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34309] Trouble when reloading extension modules.

[Stefan Behnel]

Stefan Behnel  added the comment:

a) Probably not something to fix in released versions any more, so increasing 
version from 3.5 to 3.8.

b) Regarding shared library unloading and the problems mentioned, I'm also not 
sure if there is a way to safely unload transitively imported libraries, e.g. 
if the extension module is a wrapper for an external C library (which then 
might come with its own dependencies again, which might still be in use by 
other extension modules, etc.).

--
versions: +Python 3.8 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34309] Trouble when reloading extension modules.

[Eric Snow]

Eric Snow  added the comment:

I've changed the issue title to reflect where things stand.

Hmm, doing so reminded me of an important consideration here.  A module object 
is effectively a fairly light wrapper around a dict.  When you call 
importlib.reload() the loader from the module's spec is used to re-execute the 
module's existing dict. [1][2]  A new module is not created and the existing 
module namespace is not reset.  So during reload the module is responsible for 
deleting anything in its namespace that wouldn't get replaced when re-executed 
(including attributes that were added to the namespace externally).  For most 
modules this isn't an issue.  However, it's something to consider when 
reloading a module.  See the docs for more explanation and caveats. [3]

[1] https://github.com/python/cpython/blob/master/Lib/importlib/__init__.py#L169
[2] 
https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L610
[3] https://docs.python.org/3/library/importlib.html#importlib.reload

--
title: Embedding Python; Py_Initialize / Py_Finalize cycles -> Trouble when 
reloading extension modules.

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com