[issue32516] Add a shared library mechanism for win32
xoviat <xov...@gmail.com> added the comment: Just to be clear, I'm not considering doing this with respect to the C/C++ runtimes, but all of the other shared libraries. And in case you weren't aware, this is exactly what auditwheel does (except that I believe it uses patchelf, whereas I will be using Nathaniel's tool). -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32516> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32516] Add a shared library mechanism for win32
xoviat <xov...@gmail.com> added the comment: For the record, moving the DLL path manipulation code into the interpreter would address the concern that importing a module would not manipulate the search path because the behavior would move into Python itself. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32516> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32516] Add a shared library mechanism for win32
xoviat <xov...@gmail.com> added the comment: This is what ray said: "Please do not do this. Importing a specific module should not modify the way that process loads subsequent DLLs." (https://github.com/numpy/numpy/pull/10229#issuecomment-354846459) What I'm proposing to do is write a tool, widely used like auditwheel, that will copy shared libraries into the .libs folder and then patch __init__.py in all packages in order to modify the DLL search path to add these folders. If everyone's okay with that, we can close this issue. But if everyone's not okay with that, then we need to document that it's Python's responsibility to do this. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32516> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32516] Add a shared library mechanism for win32
xoviat <xov...@gmail.com> added the comment: As Nathaniel noted, the "solution" of placing DLLs in the same directory as extension modules is not a solution. I also think that some people here misunderstand my proposal: I'm not necessarily proposing that these directories are added using an import hook: they could be added on startup through a scan of site-packages. "However, even if this were implemented we'd still need to write and maintain the __init__.py injection code for the next ~5 years, so probably the thing to do is to implement the __init__.py approach first and wait to see how it shakes out before considering interpreter changes." Yes, this approach is already implemented in NumPy and SciPy. I'm also implementing it for other packages as well. However, the principal reason that I'm opening this issue is that Ray complained that packages shouldn't be altering the DLL search path: the only other solution that I can see is to make this documented behavior, only on Windows, and only because the Windows developers (and I'm in no way anti-Windows, but I'm just frustrated with this particular issue) decided to place an arbitrary limit on probingPath. As far as the complaints about rpath: this is a mechanism used by every single manylinux and actually most OSX wheels, and it works perfectly. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32516> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32516] Add a shared library mechanism for win32
xoviat <xov...@gmail.com> added the comment: My current plan is to patch the __init__ package files to add the '.libs' folder to the search path. However, I think it would be better for Python to do this so that there is a consistent mechanism for loading shared libraries. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32516> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32516] Add a shared library mechanism for win32
xoviat <xov...@gmail.com> added the comment: So the idea here is actually to write a replacement for auditwheel that works on windows. In order to do that, I need to recursively analyze DLL dependencies, randomize the DLL filenames, rewrite the pe header table, and then copy them into the .libs folder. At that point, I need some mechanism to either preload all of the DLLs or add them to the DLL search path so that extensions can find them (example: the extension will need to load umfpack-gghsa-cp36.dll). If assemblies can accomplish such a task, then I can use them. However the restriction on the number of parent directories is a real problem. I've benchmarked preloading the DLLs and it's not really ideal. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32516> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32516] Add a shared library mechanism for win32
xoviat <xov...@gmail.com> added the comment: Sorry, that should have read: 2. A folder two levels under site-packages that is named '.libs' Please consult the auditwheel source to determine the specific pattern if there is doubt. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32516> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32516] Add a shared library mechanism for win32
New submission from xoviat <xov...@gmail.com>: On linux and macOS, a mechanism exists to relocate shared libraries inside of a wheel. Auditwheel creates a .libs folder and places the shared libraries inside of it. The problem is that on Windows, the rpath mechanism doesn't exist. We've attempted to ameliorate the situation with NumPy by modifying the DLL search path ourselves. I think this should be done in Python itself. Here is what I propose: for each folder in site packages that matches the foldername created by auditwheel, specifically: 1. A folder directly inside site-packages that ends with '.libs' 2. A folder two levels under site-packages that is named 'libs' Python should add these folders to the DLL search path at some point before the matching extensions are imported, so that DLLs located in these paths can be imported by a call to LoadLibrary. The reason that this should be done in Python is that packages shouldn't be modifying the DLL search path, but that's currently what's required. The alternative, current, recommendation is to place shared libraries in the same folder as the extension, but this approach fails when the shared library needs to be shared between more than one extension in different subpackages, but in the same distribution package. -- components: Extension Modules messages: 309642 nosy: xoviat priority: normal severity: normal status: open title: Add a shared library mechanism for win32 type: enhancement versions: Python 3.8 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32516> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31941] ImportError: DLL Load Failure: The specified module cannot be found
New submission from xoviat <xov...@gmail.com>: Yes, I know that this isn't Python's fault. I know how to resolve the problem. But what's frustrating is that this error message is totally unhelpful, because it doesn't contain the DLL that Python is looking for. Yes, I know that the error message is just directly from windows. But there has to be some way to make this error message more helpful. There has to be some way to tell the user what the name of the DLL is. Because the current state of this error message is sad. -- messages: 305531 nosy: xoviat priority: normal severity: normal status: open title: ImportError: DLL Load Failure: The specified module cannot be found type: enhancement ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31941> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20486] msilib: can't close opened database
xoviat <xov...@gmail.com> added the comment: Unfortunately, this issue has taken on a much lower importance for me, and as such, I won't be able to address it. Sorry about that. 2017-10-26 0:01 GMT-05:00 Berker Peksag <rep...@bugs.python.org>: > > Berker Peksag <berker.pek...@gmail.com> added the comment: > > xoviat, would you like to send your patch as a pull request on GitHub? It > would be nice to add a simple test that the new Close() works correctly. I > can do that if you don't have time, thank you! > > Steve, can we apply this to bugfix branches? > > -- > versions: -Python 3.5 > > ___ > Python tracker <rep...@bugs.python.org> > <https://bugs.python.org/issue20486> > ___ > -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue20486> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31595] Preferring MSVC 2017 in Python 3.6.x new minor releases
xoviat added the comment: @zooba Since you mention it, I agree with that proposal. But currently we have core developers contributing to distutils and @jaraco contributing to setuptools. @jaraco is quite competent, but I doubt that he would be able to maintain an independent fork of distutils by himself. In short, I think your proposal is a good one, but how can we allocate manpower? -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31595> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31595] Preferring MSVC 2017 in Python 3.5.x and 3.6.x new minor releases
New submission from xoviat: MSVC 2017 retains ABI compatibility with MSVC 2015, yet supports (as always with a newer compiler version) more features and compiles faster. Because it retains ABI compatibility with existing Python versions, I think it makes sense for distutils to prefer it (this could be fixed in setuptools, but distutils generally takes care of compiler detection). Thoughts? -- components: Distutils messages: 303057 nosy: dstufft, merwok, xoviat priority: normal severity: normal status: open title: Preferring MSVC 2017 in Python 3.5.x and 3.6.x new minor releases type: enhancement ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31595> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
