[Python-Dev] Re: os.add_dll_directory and DLL search order
On 6/22/20, Steve Dower wrote:
>
> What is likely happening here is that _sqlite3.pyd is being imported
> before _mapscript, and so there is already a SQLITE3 module in memory.
> Like Python, Windows will not attempt to import a second module with the
> same name, but will return the original one.
Qualified DLL loads won't interfere with each other, but dependent
DLLs are loaded by base name only. In these cases a SxS assembly
allows loading multiple DLLs that have the same base name. If the
assembly is referenced by a DLL, embed the manifest in the DLL as
resource 2. For example:
>>> import ctypes
>>> test1 = ctypes.CDLL('./test1')
>>> test2 = ctypes.CDLL('./test2')
>>> test1.call_spam.restype = None
>>> test2.call_spam.restype = None
>>> test1.call_spam()
spam v1.0
>>> test2.call_spam()
spam v2.0
>>> import win32process, win32api
>>> names = [win32api.GetModuleFileName(x)
... for x in win32process.EnumProcessModules(-1)]
>>> spams = [x for x in names if 'spam' in x]
>>> print(*spams, sep='\n')
C:\Temp\test\c\spam.dll
C:\Temp\test\c\spam_assembly\spam.dll
Source
spam1.c (spam.dll):
#include
void __declspec(dllexport) spam()
{
printf("spam v1.0\n");
}
test1.c (test1.dll):
#pragma comment(lib, "spam")
void __declspec(dllimport) spam();
void __declspec(dllexport) call_spam()
{
spam();
}
---
spam_assembly/spam_assembly.manifest:
spam2.c (spam_assembly/spam.dll):
#include
void __declspec(dllexport) spam()
{
printf("spam v2.0\n");
}
test2.c (test2.dll -- link with /manifest:embed,id=2):
#pragma comment(lib, "spam")
#pragma comment(linker, "/manifestdependency:\"\
type='win32' \
name='spam_assembly' \
version='2.0.0.0' \
processorArchitecture='amd64' \"")
void __declspec(dllimport) spam();
void __declspec(dllexport) call_spam()
{
spam();
}
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/5PDVL7KOBCCIVRSYQH4WXHBCZ23KYKG3/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: os.add_dll_directory and DLL search order
It's still a problem, even if it's a problem in the opposite direction than you first thought (Python has a newer sqlite, rather than older). Updating your API fixes the problem now, but you still need to decide how you check for and handle newer, potentially incompatible library versions. On Mon, Jun 22, 2020 at 1:13 PM Seth G wrote: > Thanks Ned. > I did double check the docs for sqlite3 after posting and wondering why > the versions were so different. > I guess the clue should have been the sqlite-3 ! > Reading the history of the module I presume sqlite3 has its own module > version number as it was integrated from a separate project. > > Seth > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NYFXECAXMGZM73DSB5AF74O5FV4TFXOL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: os.add_dll_directory and DLL search order
Thanks Ned. I did double check the docs for sqlite3 after posting and wondering why the versions were so different. I guess the clue should have been the sqlite-3 ! Reading the history of the module I presume sqlite3 has its own module version number as it was integrated from a separate project. Seth -- web:http://geographika.co.uk twitter: @geographika On Mon, Jun 22, 2020, at 6:41 PM, Ned Deily wrote: > On Jun 22, 2020, at 05:39, Seth G wrote: > > However, there is one feature of using the Windows PATH that I can't seem > > to replicate with add_dll_directory. > > The MapServer DLL builds are linked to sqlite3 3.24.0, whereas Python 3.8.2 > > is linked to 2.6.0. Building with matching versions is not something I can > > easily change. > > For what it's worth, you're looking at the wrong value (it's easy to > do!). '2.6.0' is the version number of the sqlite3 module itself, not > of the sqlite3 library. > > > >>> sqlite3.version > '2.6.0' > >>> sqlite3.sqlite_version > '3.31.1' > > -- > Ned Deily > n...@python.org -- [] > > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GMW6VL6MY5CUNAMPGQHSI6JB3PL2KLK7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: os.add_dll_directory and DLL search order
Thanks Steve for the clarifications. As a workaround placing the older sqlite3.dll in the same folder as the _mapscript.pyd file while leaving all the other DLLs in a different folder referenced by add_dll_directory allows for a successful import. For a less hacky fix - after checking the version numbers for sqlite3 again they are not too different so it may be easiest to update the upstream builds to a matching version. Apologies for the noise - I thought this was related to the new add_dll_directory function, but I can break old versions using PATH and mismatched sqlite3 .dlls too! Seth -- web:http://geographika.co.uk twitter: @geographika On Mon, Jun 22, 2020, at 5:46 PM, Steve Dower wrote: > On 22Jun2020 1039, Seth G wrote: > > However, there is one feature of using the Windows PATH that I can't seem > > to replicate with add_dll_directory. > > The MapServer DLL builds are linked to sqlite3 3.24.0, whereas Python 3.8.2 > > is linked to 2.6.0. Building with matching versions is not something I can > > easily change. > > > > When using Windows PATH the folder with the newer version could be added to > > the front of the list to take priority and import worked correctly. This > > does not seem to be possible with add_dll_directory - the Python > > sqlite3.dll in C:\Python38\DLLs always takes priority leading to: > > > > ImportError: DLL load failed while importing _mapscript: The specified > > procedure could not be found. > > > > I presume I can't remove the C:\Python38\DLLs path, is there another > > solution to this issue? > > DLLs should not be in the search path at all - it's searched by sys.path > when importing .pyd files, which are loaded by absolute path and their > dependencies found adjacent. > > What is likely happening here is that _sqlite3.pyd is being imported > before _mapscript, and so there is already a SQLITE3 module in memory. > Like Python, Windows will not attempt to import a second module with the > same name, but will return the original one. > > So your best option here is probably to rebuild sqlite3.dll with as much > of the version number (or some unique string) in the name as you need. > Or you can statically link it into your extension module, assuming you > aren't relying on shared state with other modules in your package. The > latter is probably easier. > > Cheers, > Steve > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SO37EWIVDOKEIQO6MPSR2EVL5EDPLV4J/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: os.add_dll_directory and DLL search order
On Jun 22, 2020, at 05:39, Seth G wrote: > However, there is one feature of using the Windows PATH that I can't seem to > replicate with add_dll_directory. > The MapServer DLL builds are linked to sqlite3 3.24.0, whereas Python 3.8.2 > is linked to 2.6.0. Building with matching versions is not something I can > easily change. For what it's worth, you're looking at the wrong value (it's easy to do!). '2.6.0' is the version number of the sqlite3 module itself, not of the sqlite3 library. >>> sqlite3.version '2.6.0' >>> sqlite3.sqlite_version '3.31.1' -- Ned Deily n...@python.org -- [] ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WILPNTOKCGWLNEPWCQXO47A444XL6USJ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: os.add_dll_directory and DLL search order
On 22Jun2020 1646, Steve Dower wrote: DLLs should not be in the search path at all - it's searched by sys.path when importing .pyd files, which are loaded by absolute path and their dependencies found adjacent. To clarify this - by "DLLs" I meant the DLLs directory, not DLLs in general (hence the singular "it's"). Cheers, Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YIIMGEADBMT2CTOMIOPNNWZZ7UBE5JWC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: os.add_dll_directory and DLL search order
On 22Jun2020 1039, Seth G wrote: However, there is one feature of using the Windows PATH that I can't seem to replicate with add_dll_directory. The MapServer DLL builds are linked to sqlite3 3.24.0, whereas Python 3.8.2 is linked to 2.6.0. Building with matching versions is not something I can easily change. When using Windows PATH the folder with the newer version could be added to the front of the list to take priority and import worked correctly. This does not seem to be possible with add_dll_directory - the Python sqlite3.dll in C:\Python38\DLLs always takes priority leading to: ImportError: DLL load failed while importing _mapscript: The specified procedure could not be found. I presume I can't remove the C:\Python38\DLLs path, is there another solution to this issue? DLLs should not be in the search path at all - it's searched by sys.path when importing .pyd files, which are loaded by absolute path and their dependencies found adjacent. What is likely happening here is that _sqlite3.pyd is being imported before _mapscript, and so there is already a SQLITE3 module in memory. Like Python, Windows will not attempt to import a second module with the same name, but will return the original one. So your best option here is probably to rebuild sqlite3.dll with as much of the version number (or some unique string) in the name as you need. Or you can statically link it into your extension module, assuming you aren't relying on shared state with other modules in your package. The latter is probably easier. Cheers, Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PMBOWXNIXBGPPOOTDWU3LGR6QSP73AXW/ Code of Conduct: http://python.org/psf/codeofconduct/
