Hi all, This is specifically about embedding Python on Windows, and I'm hoping some of the Windows Python devs might have some ideas or be interested in this. I have implemented a partial solution (linked below) and I'm interested to hear what other people think of this.
Currently when embedding Python both python3x.dll and python3.dll get loaded, with python3.dll redirecting stable API calls to python3x.dll. If a process embeds two different versions of Python 3, e.g., python39.dll and python310.dll they both load their respective python3.dlls. At this point I imagine you are thinking "don't do that", but bear with me... The problem comes when the Python loaded second imports an extension module linked with python3.dll. The python3.dll that it gets linked to will be the first one that was loaded, not the one that relates to the second Python that is actually doing the import. This results in a call into the wrong Python dll and 'bad things' happen. None of this is unexpected and I'm sure that the sensible thing to do is to simply not do this... but I've been working on a way to make this work anyway. In my case I have a plugin to another application that embeds Python into that application. It's perfectly possible (and reasonable) for other plugins to also want to embed Python. This can be dealt with by having both plugins use the same Python environment easily enough in most cases. With Python 2 it used to be possible to have two different Python interpreters embedded at the same time and not have them interfere with each other (although it is possible there would still be issues with DLL versions used by extension modules). With Python 3 if we want to have two different versions of Python 3 embedded at the same time then it will fail because of the reason outlined above. My idea is to redirect all loaded python3.dll dlls to the one we want to be used before loading any extension modules (i.e., just before any call to LoadLibraryEx) and then restore them afterwards. This can be done by manipulating the loader modules list in Windows. I have implemented this as a proof of concept in my own plugin and confirmed that this works and does allow two different versions of Python 3 to be embedded at the same time. This works with an unmodified version of Python by applying the redirect in an import hook. It uses several undocumented Windows structures and APIs in order to safely manipulate the loader table. Here is my proof of concept code that performs the redirect https://gist.github.com/tonyroberts/74888762f0063238d4f7fd7c7d36f0f0 While this works for different versions of Python 3, there is still a problem when trying to embed two different instances of the same version of Python 3. The problem is basically the same but with the added complication that the pyd files are named the same, and the ones loaded first get found by the second Python runtime and you end up again calling across versions. I managed to solve this using a similar method to the code above, but rather than redirecting just python3.dll I look for any other loaded python3x.dll and then remove *all* modules loaded under that Python distribution from the loader table. This ensures that both Python runtimes are effectively isolated from each other as neither see any of the same modules. This gets more complicated once you start thinking about user site-packages folders and venvs, but for simple distributions where everything is under the same root folder this technique works. Anyway, just keen to hear what people think or whether this has been tackled before in another way. Best regards, Tony _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IRO5XEMQPY7KEJJH5LBSMOCCL2ZKTT77/ Code of Conduct: http://python.org/psf/codeofconduct/
