[issue42731] Enhancement request for proxying PyString
New submission from Karl Nelson : When developing with JPype, the largest hole currently is that Java returns a string type which cannot be represented as a str. Java strings are string like and immutable and can be pulled to Python when needed, but it is best if they remain in Java until Python requests it as pulling all string values through the API and pushing them back can result in serious overhead. Thus they need to be represented as a Proxy to a string, which can be accessed as a string at anytime. Throughout the Python API str is treated as a concrete type (though it is somewhat polymorphic due to different storage for code points sizes.) There is also handling for an "unready" string which needs additional treatment before it can be accessed. Unfortunately this does not appear to be suitable for creating a proxy object which can be pulled from another source to create a string on demand. Having a "__str__()" method is insufficient as that merely makes an object able to become a string rather than considered to be a string by the rest of the API. Would it be possible to generalize the concept of an unready string so that when Ready is called it fetches the actually string contents, creates a piece of memory to store the string contents (outside of the object itself), and sets the access flags for so that the code points can be interpreted? Is this already possible in the API? Are there any other plans to make the str type able to operate as a proxy? -- components: Extension Modules messages: 383701 nosy: Thrameos priority: normal severity: normal status: open title: Enhancement request for proxying PyString versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue42731> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Okay, well at least now googling Python + "A dynamic link library (DLL) initialization routine failed." give something which could point a user may be able to identify the issue. It wasn't obvious to me that imports did not hold the GIL, but it is clear in retrospect that it should have been. Thanks very much and sorry for the wild goose chase. -- resolution: -> third party stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I found it. The change that hit JPype is https://bugs.python.org/issue33895 Thanks, Eryk Sun for figuring this out. I would never have gotten myself. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Drat I missed that one when I was scanning for statics because I was focusing C++ resource rather than looking for Python resources. I also wouldn't have expected this to happen on only one platform, but that was my bad. Is it possible to improve the error reporting for this? This was a horrible error to trace down at least for a non-windows programmer. At least a mention in the doc as to the symptoms so someone else can google it. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment:
Gave another series of shots at the problem. So the reason that attaching the
debugger fixes the issue is that it calls LoadLibraryExW for a bunch libraries
when it attaches.
I was also able to unconditionally trigger the bug even when loading without
the pyc. It appears that calling "import _ssh" will cause whatever conflict
is happening to trigger immediately.
Other than than progress was minimal. Attaching the debugger causes so many
changes to the control flow that I can't get the bad path to trigger and I
can't get the debugger to attach once you start down the path. I tried
creating a segfault but again I can't get the debugger to attach because
Detours is interfering with the attachment process.
I also tried disabling some of the flags that JPype was adding to the python
build process to see if any are related. "/DEBUG" on the linker does not do
anything to the problem. The other options were '/Zi', '/EHsc', '/std:c++14'.
Nothing here is out of the ordinary.
I do not see anything in the link line that shows evidence of static linkage or
likely to cause a conflict. The exact same line is being used with Python 3.8.
link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2
/MANIFESTUAC:NO
/LIBPATH:C:\Users\nelson85\AppData\Local\Programs\Python\Python39\libs
/LIBPATH:C:\Users\nelson85\AppData\Local\Programs\Python\Python39\PCbuild\amd64
/LIBPATH:C:\Program Files (x86)\Microsoft Visual
Studio\2019\Community\VC\Tools\MSVC\14.28.29333\ATLMFC\lib\x64
/LIBPATH:C:\Program Files (x86)\Microsoft Visual
Studio\2019\Community\VC\Tools\MSVC\14.28.29333\lib\x64 /LIBPATH:C:\Program
Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files
(x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64 /LIBPATH:C:\Program Files
(x86)\Windows Kits\10\lib\10.0.18362.0\um\x64 /LIBPATH:C:\Program Files
(x86)\Microsoft Visual
Studio\2019\Community\VC\Tools\MSVC\14.28.29333\ATLMFC\lib\x64
/LIBPATH:C:\Program Files (x86)\Microsoft Visual
Studio\2019\Community\VC\Tools\MSVC\14.28.29333\lib\x64 /LIBPATH:C:\Program
Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files
(x86)\Windows Kits\10\
lib\10.0.18362.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows
Kits\10\lib\10.0.18362.0\um\x64 /EXPORT:PyInit__jpype
$(OBJECTS) /OUT:build\lib.win-amd64-3.9\_jpype.cp39-win_amd64.pyd
/IMPLIB:build\temp.win-amd64-3.9\Release\native\common\_jpype.cp39-win_amd64.lib
So I remain at a loss how to proceed forward. Nothing has changed in JPype or
its symbols since Python 3.8 so this is clearly taking place in Python or at
the OS level. Is it possible that Python shipped with bad dlls for the 3.9.0
build? If so then looking at the source code may be the wrong approach.
After all depending on the order of libraries I trigger or don't trigger the
bug.
I should note that the difference between loading successfully py and pyc seems
to have been triggered in PC/getpathc.c could it be the change to remove
loading api-ms-win-core-path?
```
static void
join(wchar_t *buffer, const wchar_t *stuff)
{
-if (_PathCchCombineEx_Initialized == 0) {
-HMODULE pathapi = LoadLibraryExW(L"api-ms-win-core-path-l1-1-0.dll",
NULL,
- LOAD_LIBRARY_SEARCH_SYSTEM32);
-if (pathapi) {
-_PathCchCombineEx = (PPathCchCombineEx)GetProcAddress(pathapi,
"PathCchCombineEx");
-}
-else {
-_PathCchCombineEx = NULL;
-}
-_PathCchCombineEx_Initialized = 1;
-}
-
-if (_PathCchCombineEx) {
-if (FAILED(_PathCchCombineEx(buffer, MAXPATHLEN+1, buffer, stuff, 0)))
{
-Py_FatalError("buffer overflow in getpathp.c's join()");
-}
-} else {
-if (!PathCombineW(buffer, buffer, stuff)) {
-Py_FatalError("buffer overflow in getpathp.c's join()");
-}
+if (FAILED(PathCchCombineEx(buffer, MAXPATHLEN+1, buffer, stuff, 0))) {
+Py_FatalError("buffer overflow in getpathp.c's join()");
}
}
```
Also given JPype is really a generic C++ CPython module (with no special
options other than /Zi /EHsc why hasn't anyone else triggered this?
--
___
Python tracker
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I will look through the list of samples in Detours to see if there is something that can give us better stacktrace information. If we really need a stack trace I can make Detours bust by creating an intentional segfault and see if we can attach the debugger after the source of the problem has been triggered. Unfortunately as my windows programming knowledge is all from 1994 I may be missing something obvious. In the meantime I am upgrading the sample with some more hooks for any other potential LoadLibrary variants and FlsGetValue. Though given the GetProcAddress calls I think that it must be something in the chain leading up so the variants are likely not envolved. The other thing I can do is look at instrumenting SetErrorMode so that I can give myself a marker of when we are entering and leaving the dyload code. ``` /* Don't display a message box when Python can't load a DLL */ old_mode = SetErrorMode(SEM_FAILCRITICALERRORS); /* bpo-36085: We use LoadLibraryEx with restricted search paths to avoid DLL preloading attacks and enable use of the AddDllDirectory function. We add SEARCH_DLL_LOAD_DIR to ensure DLLs adjacent to the PYD are preferred. */ Py_BEGIN_ALLOW_THREADS hDLL = LoadLibraryExW(wpathname, NULL, LOAD_LIBRARY_SEARCH_DEFAULT_DIRS | LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR); Py_END_ALLOW_THREADS #if !USE_UNICODE_WCHAR_CACHE PyMem_Free(wpathname); #endif /* USE_UNICODE_WCHAR_CACHE */ /* restore old error mode settings */ SetErrorMode(old_mode); ``` I will post back once I get some results. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I looked more at the logs to see if I can find out what to instrument next. The log files unfortunately don't diff well because every line contains a timestamp so I can't a proper alignment as well as all the real addresses. So I wrote a short program to convert all the numbers into an "X". After doing so I find there is a single difference in the process between loading from a py file and a pyc file. # import _jpype ## Only if loaded from a Py file trclnk64: 001 GetProcAddress(7ffc4c71,FlsGetValue) X X X trclnk64: 001 GetProcAddress(,) -> X ## All X X X trclnk64: 001 LoadLibraryExW(C:\Users\nelson85\AppData\Local\Programs\Python\Python39\python3,0,1000) X X X trclnk64: 001 LoadLibraryExW(,,) -> X X X X trclnk64: ### X C:\Users\nelson85\AppData\Local\Programs\Python\Python39\python3.DLL 000140bc So here is the call that fails X X X trclnk64: 001 LoadLibraryExW(c:\users\nelson85\documents\devel\open\jpype\_jpype.cp39-win_amd64.pyd,0,1100) X X X trclnk64: 001 LoadLibraryExW(api-ms-win-core-synch-l1-2-0,0,800) X X X trclnk64: 001 LoadLibraryExW(,,) -> X X X X trclnk64: ### X C:\WINDOWS\System32\KERNELBASE.dll 002b306b X X X trclnk64: ntdll.dll [7ffc4c974138 X So the problem is something that happened before the LoadLibraryExW call when python3.dll was accessed. The correct path called FlsGetValue while the incorrect one skipped it which lead to a failure. Thus my conclusion is that the vehicle left the edge of the cliff just prior to LoadLibraryExW(_jpype) and hit the ocean when C++ runtime library was requested. Does that point to anything we can use to solve this? -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Eryk, Unfortunately, this particular bug is resistant to attaching the debugger. Whenever the debugger is attached the bug disappears. Fortunately the suggestion to use Detours by WildCard65 appears to offer a way to diagnose the issue. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: The last libraries loaded prior to the failure were... ``` 20201218192451066 20440 50.60: trclnk64: api-ms-win-eventing-provider-l1-1-0.dll [7ffc4c974108 7ffc4c8b7808] 20201218192451066 20440 50.60: trclnk64: EventUnregister0 7ffc4eab37a0 20201218192451066 20440 50.60: trclnk64: EventRegister0 7ffc4ea6a640 20201218192451066 20440 50.60: trclnk64: EventActivityIdControl0 7ffc4eac6190 20201218192451066 20440 50.60: trclnk64: EventWriteTransfer0 7ffc4eab2cf0 20201218192451066 20440 50.60: trclnk64: EventSetInformation0 7ffc4ea6a3e0 20201218192451066 20440 50.60: trclnk64: api-ms-win-core-apiquery-l1-1-0.dll [7ffc4c9740e8 7ffc4c8b77e8] 20201218192451066 20440 50.60: trclnk64: ApiSetQueryApiSetPresence0 7ffc4ead5030 20201218192451066 20440 50.60: trclnk64: api-ms-win-core-apiquery-l1-1-1.dll [7ffc4c9740f8 7ffc4c8b77f8] 20201218192451066 20440 50.60: trclnk64: ApiSetQueryApiSetPresenceEx0 7ffc4eb35730 ``` And as far as I can tell that was successful. Perhaps there is another library at this point, but my reading of the log is that LoadLibraryExW was called run though all required elements to load, got the end section before it started to load dependencies of _jpype, started fetching things that have to do with locale and then hit a failure point inside of LoadLibrary. If I look at the successful trace then I see the next action is to start linking in C++ symbols. But perhaps I am misreading... Looking farther back in the log, the last LoadLibrary to get called before the failure is. ``` 20201218192451054 20440 50.60: trclnk64: 001 LoadLibraryExW(api-ms-win-core-fibers-l1-1-1,0,800) 20201218192451054 20440 50.60: trclnk64: 001 LoadLibraryExW(,,) -> 7ffc4c71 20201218192451054 20440 50.60: trclnk64: ### 4c71: C:\WINDOWS\System32\KERNELBASE.dll 002b306b 20201218192451054 20440 50.60: trclnk64: ntdll.dll [7ffc4c974138 7ffc4c8b7838] ``` Fibers? I have no clue at this point. Is this a Python bug or a windows bug? -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I used Detours with trclnk64 to get the following log: ``` 20201218193836960 4332 50.60: trclnk64: 001 GetProcAddress(,) -> 7ffc4ccebef0 20201218193836960 4332 50.60: trclnk64: 001 GetProcAddress(7ffc4ccd,LCMapStringEx) 20201218193836960 4332 50.60: trclnk64: 001 GetProcAddress(,) -> 7ffc4cce6640 <= This is the point where it differs from a clean run. 20201218193838163 4332 50.60: trclnk64: 001 GetProcAddress(7ffc4c71,FlsFree) 20201218193838163 4332 50.60: trclnk64: 001 GetProcAddress(,) -> 7ffc4c78b000 20201218193838163 4332 50.60: trclnk64: 001 FAIL LoadLibraryExW(c:\users\nelson85\documents\devel\open\jpype\_jpype.cp39-win_amd64.pyd,0,1100) 20201218193838164 4332 50.60: trclnk64: 001 LoadLibraryExW(,,) -> 0 ``` It really is nothing to go on... Everything succeeds, then we get an extra successful request for FlsFree then it immediately returns with a fail. In the successful run (without the cache) we get... ``` 20201218192451068 20440 50.60: trclnk64: 001 GetProcAddress(7ffc4ccd,LCMapStringEx) 20201218192451068 20440 50.60: trclnk64: 001 GetProcAddress(,) -> 7ffc4cce6640 20201218192451068 20440 50.60: trclnk64: 001 LoadLibraryExW(,,) -> 7ffc349f 20201218192451068 20440 50.60: trclnk64: ### 349f: c:\users\nelson85\documents\devel\open\jpype\_jpype.cp39-win_amd64.pyd 20201218192451068 20440 50.60: trclnk64: MSVCP140.dll [7ffc34a7af48 7ffc34a5a0c8] 20201218192451068 20440 50.60: trclnk64: ?_Lock@?$basic_streambuf@DU?$char_traits@D@std@@@std@@UEAAXXZ0 7ffc36f22350 20201218192451068 20440 50.60: trclnk64: ?_Unlock@?$basic_streambuf@DU?$char_traits@D@std@@@std@@UEAAXXZ0 7ffc36f22350 ``` So the failure appears to be internal to the call before we load MSVCP140.dll. Unfortunately, I am way outside my element here as a Linux/Java/CPython programmer. My next guess would be to instrument the set system error and if we are within LoadLibrary then execute "int *i=0; *i=0;" and hope the debugger would get me the point of failure so that I can examine the stack trace to find out where we are. But if you have another direction I would be happy to hear it. Is there some function I should add to trclnk64 to that may have gone wrong? It must be something that fetched the address of LCMapStringEx, but I have no clue what that may be. This really does not look like a JPype bug. Yes, two minutes in the future from where the car ran over the pedestrian, he was going to run into the bank and demand money (monkey patch and other bad things). But at the point in time the car hit, the pedestrian was just minding his own business. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Can you be so kind as pointing me to a LoadLibraryExW detour example for Python? I have shimmed a DLL before to capture transaction logs in the past, but not with a Python or a system library. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Just for reference here are all the dependencies that _jpype has ``` MSVCP140.dll python39.dll KERNEL32.dll VCRUNTIME140_1.dll VCRUNTIME140.dll api-ms-win-crt-runtime-l1-1-0.dll api-ms-win-crt-string-l1-1-0.dll api-ms-win-crt-heap-l1-1-0.dll ``` So it isn't like there are a whole lot of things that I am doing that are likely to trigger a dependency problem. As far as I can see the only items that are not already on python39.dll dependency list are VCRUNTIME140_1.dll and MSVCP140.dll. But if that were the case then surely other projects would also be hitting this bug. So this error message makes no sense at all. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I attempted another deep dive on this one. 1) Removed manual library add platform_specific['libraries'] = ['Advapi32'] No change. 2) Compared symbol wise imports Two imports changed PyIndex_Check PyObject_CheckBuffer plus one additional "fflush" dependency in api-ms-win-crt-stdio-l1-1-0.dll 3) Compared symbol wise exports No changes detected. 4) Disassembled both binaries Other than a addressing that is about a 0x1000 change in the relocation tables. Guessing that this is just a change in the number of pages. There were functional changes likely related to Python macros differences. 5) Removed all references to printf and flush to remove api-ms-win-crt-stdio-l1-1-0.dll dependency. No change. 6) Scanned the changelog and Code changes for the past year in Python/dynload_win.c. Nothing appears likely to trigger this change. 7) Deleted the _jpype module entirely to verify that the error message is related to the copy that I am working on and not some other copy on system. Error message changes to "ModuleNotFoundError: No module named '_jpype'" 8) Scanned the source code for static variables. None detected thus far. But it is a many hours long process. Structurally these should not exist based on the module design. Statics found thus far. 2 jobject and jmethodID pointer types One C++ class called PLATFORM_ADAPTER which is the required routine determining whether to load jvm.dll or jvm.so using LoadLibrary or dlopen or shload. The global pointer to the context C++ class; a bunch of statics related to logging when enabled. Python C style structure definitions. Everything was a primitive or pointer with the exception of one class. All pointers I found were initialized to constant values. I rewrote the PLATFORM_ADAPTER to use a pointer rather than be initialized at load time. No joy. (after scanning through 17k lines of code take break and ponder life) It is certainly possible I missed something here but it looks pretty unlikely this is the source of the error. 9) Verified that the behavior does not occur on Python 3.8.5 with the current source code. Pass. This only happens on Python 3.9.0 10) Scanned the linker line for differences. One major difference. In Python 3.8 the linker line contains quotes around all the arguments while in Python 3.9 the linker line is missing the quotes on all /LIBPATH statements. However this must be a logging difference as the statement without the quotes would not work. 11) Manually ran linker command. Nothing changed, same behavior. So link command is a red herring. 12) Profit??? Okay I have once again exhausted everything that I can think of on the JPype side. So that leaves something in the build process that does not appear in the logs, a symbol conflict that so how only triggers on one path, or Python3.9.0 has a corrupt build in the wild. Steve is there anything else that I can try here? -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Well that gives me a place to search from. Unfortunately statics are not likely the source of the issue. I scrubbed all C++ structures from the project into a big structure which has only a single global pointer which I initialize in the init method. It is possible that one crept back in at some point, but it is a possibility. The use of actual C++ in this project is nearly zero. Other than exceptions, strings and the vector on the buried in some class, the rest has been purged already. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment:
Without the pyc everything goes fine...
```
...
_bootlocale' # <_frozen_importlib_external.SourceFileLoader object at
0x0222E9FD5A30>
import 'site' # <_frozen_importlib_external.SourceFileLoader object at
0x0222E9F88F40>
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# code object from
c:\users\nelson85\documents\devel\open\jpype\jpype\__init__.py
# created
'c:\\users\\nelson85\\documents\\devel\\open\\jpype\\jpype\\__pycache__\\__init__.cpython-39.pyc'
# extension module '_jpype' loaded from
'c:\\users\\nelson85\\documents\\devel\\open\\jpype\\_jpype.cp39-win_amd64.pyd'
# extension module '_jpype' executed from
'c:\\users\\nelson85\\documents\\devel\\open\\jpype\\_jpype.cp39-win_amd64.pyd'
import '_jpype' # <_frozen_importlib_external.ExtensionFileLoader object at
0x0222EA4C6250>
#
c:\users\nelson85\documents\devel\open\jpype\jpype\__pycache__\_jinit.cpython-39.pyc
matches c:\users\nelson85\documents\devel\open\jpype\jpype\_jinit.py
# code o
...
```
With the pyc file, failure...
```
...
_bootlocale' # <_frozen_importlib_external.SourceFileLoader object at
0x015278445A30>
import 'site' # <_frozen_importlib_external.SourceFileLoader object at
0x0152783F8F40>
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
#
c:\users\nelson85\documents\devel\open\jpype\jpype\__pycache__\__init__.cpython-39.pyc
matches c:\users\nelson85\documents\devel\open\jpype\jpype\__init__.py
# code object from
'c:\\users\\nelson85\\documents\\devel\\open\\jpype\\jpype\\__pycache__\\__init__.cpython-39.pyc'
Traceback (most recent call last):
File "C:\Users\nelson85\Documents\test.py", line 1, in
import jpype
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 790, in exec_module
File "", line 228, in _call_with_frames_removed
File "c:\users\nelson85\documents\devel\open\jpype\jpype\__init__.py", line
18, in
import _jpype
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 666, in _load_unlocked
File "", line 565, in module_from_spec
File "", line 1108, in create_module
File "", line 228, in _call_with_frames_removed
ImportError: DLL load failed while importing _jpype: A dynamic link library
(DLL) initialization routine failed.
# clear builtins._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# destroy jpype
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.__interactivehook__
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
...
```
I see the codecs all get loaded earlier in the trace, so that can't be the
issue. There were no differences (other than memory addresses) up to this
point. So it must be going wrong at bootstrap_external:1108.
So guessing from the message we failed in.
```
def exec_module(self, module):
"""Initialize an extension module"""
_bootstrap._call_with_frames_removed(_imp.exec_dynamic, module)
_bootstrap._verbose_message('extension module {!r} executed from {!r}',
self.name, self.path)
```
After that point it gets really fuzzy. I would guess we are going to
import.c#_imp_create_dynamic_impl which takes us to
_PyImport_LoadDynamicModuleWithSpec. But I will need some time to read code to
see if this guess is correct.
--
___
Python tracker
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment:
Thanks, I will see if I can get additional diagnostics today. Otherwise I will
have to start recompiling Python with diagnostic hooks. I have never had to
rebuild python on windows so it may take a while to figure out how to make
progress.
The monkey patch is less scary than it looks. Since I only need these slots
of specific basic types that I designate (not user selected), the extra types
all extend without altering the underlaying type, and the class layout is
controlled by a meta class so that only Java can create these type objects, it
is reasonably robust. However, yes changes in Python can break it. It is
basically an attempt to create a mixin for concrete classes without the benefit
of a dictoffset type slot. It took about a month do develop something that
safely added the slot without overrunning, leaking or creating other memory
issues (and played nice when Python added its own dict slot).
Unfortunately, as I do many 100s on JSlot tests during method resolution using
a dict based approach was unacceptably slow. I would love if there was a
formal method to create multiple inheritance mixins for Python basic types, but
that is another issue.
```
/**
* Allocate a new Python object with a slot for Java.
*
* We need extra space to store our values, but because there
* is no way to do so without disturbing the object layout.
* Fortunately, Python already handles this for dict and weakref.
* Python aligns the ends of the structure and increases the
* base type size to add additional slots to a standard object.
*
* We will use the same trick to add an additional slot for Java
* after the end of the object outside of where Python is looking.
* As the memory is aligned this is safe to do. We will use
* the alloc and finalize slot to recognize which objects have this
* extra slot appended.
*/
PyObject* PyJPValue_alloc(PyTypeObject* type, Py_ssize_t nitems )
{
// Modification from Python to add size elements
const size_t size = _PyObject_VAR_SIZE(type, nitems + 1) + sizeof
(JPValue);
PyObject *obj = (PyType_IS_GC(type)) ? _PyObject_GC_Malloc(size)
: (PyObject *) PyObject_MALLOC(size);
if (obj == NULL)
return PyErr_NoMemory(); // GCOVR_EXCL_LINE
memset(obj, 0, size);
Py_ssize_t refcnt = ((PyObject*) type)->ob_refcnt;
if (type->tp_itemsize == 0)
PyObject_Init(obj, type);
else
PyObject_InitVar((PyVarObject *) obj, type, nitems);
// This line is required to deal with Python bug (GH-11661)
// Some versions of Python fail to increment the reference counter of
// heap types properly.
if (refcnt == ((PyObject*) type)->ob_refcnt)
Py_INCREF(type); // GCOVR_EXCL_LINE
if (PyType_IS_GC(type))
{
PyObject_GC_Track(obj);
}
return obj;
}
bool PyJPValue_hasJavaSlot(PyTypeObject* type)
{
if (type == NULL
|| type->tp_alloc != (allocfunc) PyJPValue_alloc
|| type->tp_finalize != (destructor) PyJPValue_finalize)
return false; // GCOVR_EXCL_LINE
return true;
}
// Check for a Java slot (must work on all object types)
Py_ssize_t PyJPValue_getJavaSlotOffset(PyObject* self)
{
PyTypeObject *type = Py_TYPE(self);
if (type == NULL
|| type->tp_alloc != (allocfunc) PyJPValue_alloc
|| type->tp_finalize != (destructor) PyJPValue_finalize)
return 0;
Py_ssize_t offset;
Py_ssize_t sz = Py_SIZE(self);
// I have no clue what negative sizes mean
if (sz < 0)
sz = -sz;
if (type->tp_itemsize == 0)
offset = _PyObject_VAR_SIZE(type, 1);
else
offset = _PyObject_VAR_SIZE(type, sz + 1);
return offset;
}
JPValue* PyJPValue_getJavaSlot(PyObject* self)
{
Py_ssize_t offset = PyJPValue_getJavaSlotOffset(self);
if (offset == 0)
return NULL;
JPValue* value = (JPValue*) (((char*) self) + offset);
if (value->getClass() == NULL)
return NULL;
return value;
}
void PyJPValue_free(void* obj)
{
// Normally finalize is not run on simple classes.
PyTypeObject *type = Py_TYPE(obj);
if (type->tp_finalize != NULL)
type->tp_finalize((PyObject*) obj);
if (type->tp_flags & Py_TPFLAGS_HAVE_GC)
PyObject_GC_Del(obj);
else
PyObject_Free(obj); // GCOVR_EXCL_LINE
}
void PyJPValue_finalize(void* obj)
{
JPValue* value = PyJPValue_getJavaSlot((PyObject*) obj);
... destroy the objects hidden slot if it exists
}
```
--
___
Python t
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment:
I am fairly sure this is a Python "bug" in the sense that there was some change
in undocumented change in requirements and the distutils pattern for building a
module no longer reflects that requirement. That said very likely JPype is
the only module to be affected and thus I will have to manually adjust to
account for the new requirement.
Unfortunately as far as developers, I am it so fixing it (with your assistance)
is going to have to fall on me. So lets do a run down of how this all working
so you can point me where to look.
1) JPype gets built as an ordinary setup.py CPython module. So it is possibly
a bug in the build pattern of the customizers in JPype that was exposed by
Python 3.9. I am just going to cut and paste so that it is easy to follow
along.
jpype/setup.py
```
...
from setuptools import Extension
...
import setupext
...
jpypeLib = Extension(name='_jpype', **setupext.platform.Platform(
include_dirs=[Path('native', 'common', 'include'),
Path('native', 'python', 'include'),
Path('native', 'embedded', 'include')],
sources=[Path('native', 'common', '*.cpp'),
Path('native', 'python', '*.cpp'),
Path('native', 'embedded', '*.cpp')], platform=platform,
))
```
We have two sets of customization in setup.py. Both are included from a module
called jpype/setupext/
The key one is the jpype/setupext/platform.py which defines the compiler flags.
There are two sections of interest...
jpype/setupext/platform.py contains these modifications for win32.
(So there is the Advapi32, not sure why it appears there because this section
is all before my time as this was started in 2001 and I joined in 2018)
```
static = True
if platform == 'win32':
distutils.log.info("Add windows settings")
platform_specific['libraries'] = ['Advapi32']
platform_specific['define_macros'] = [('WIN32', 1)]
if sys.version > '3':
platform_specific['extra_compile_args'] = [
'/Zi', '/EHsc', '/std:c++14']
else:
platform_specific['extra_compile_args'] = ['/Zi', '/EHsc']
platform_specific['extra_link_args'] = ['/DEBUG']
jni_md_platform = 'win32'
```
The second section is currently commented out. Though it is relevant because
JPype is exotic in one way. It is a module which is loaded in three ways.
When imported from Python it is an ordinary library (1) which will later pull
in Java which will then load library a second time (2) to bind Java native
methods. It can also be used to launch Python if Java starts the session (3).
In that case, it needs libpython.dll to be bound to module so that the Java
equivalent to LoadLibrary can work. When it does Java first manually loads
libpython.dll then loads the module and calls the hook to get Python started.
```
# This code is used to include python library in the build when starting Python
from
# within Java. It will be used in the future, but is not currently required.
#if static and sysconfig.get_config_var('BLDLIBRARY') is not None:
#
platform_specific['extra_link_args'].append(sysconfig.get_config_var('BLDLIBRARY'))
```
The actual buildext has a few minor patches so that Java libraries can run
through the normal process. But nothing seems like a good candidate
We have one section tweeking some of the build options.
```
def initialize_options(self, *args):
"""omit -Wstrict-prototypes from CFLAGS since its only valid for C
code."""
self.android = False
self.makefile = False
self.jar = False
import distutils.sysconfig
cfg_vars = distutils.sysconfig.get_config_vars()
replacement = {
'-Wstrict-prototypes': '',
'-Wimplicit-function-declaration': '',
}
tracing = self.distribution.enable_tracing
# Arguments to remove so we set debugging and optimization level
remove_args = ['-O0', '-O1', '-O2', '-O3', '-g']
for k, v in cfg_vars.items():
if not isinstance(v, str):
continue
if not k == "OPT" and not "FLAGS" in k:
continue
args = v.split()
for r in remove_args:
args = list(filter((r).__ne__, args))
cfg_vars[k] = " ".join(args)
super().initialize_options()
```
Then later we interrupt the build process for Java.
```
def build_extension(self, ext):
if ext.language == "java":
return self.build_java_ext(ext)
if self.jar:
return
print("Call build ext")
return super().build_extension(ext)
```
2) Next we have the module start. I am guessing this is not the source of the
error because adding a printf shows we never got to this poi
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Any progress on this item? I am seeing additional reports of this error in the conda stream tracker and elsewhere. As it only occurs when the debugger is not hooked up I don't have much traction to make progress myself. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42617] Enhancement request for PyType_FromSpecWIthBases add option for meta class
Karl Nelson added the comment: Perhaps just having PyType_InitHeapFromSpec which have just heap type linkage and slot copy section would help with the issue. The major problem I have is maintaining the slot code which may change at some point in the future. There would still be some risk of missing a portion of the init procedure but it would be a bit safer some of the process was in Python. -- title: Enhancement request for PyType_FromSpecWIthBases add option for meta class -> Enhancement request for PyType_FromSpecWIthBases add option for meta class ___ Python tracker <https://bugs.python.org/issue42617> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Oddly that was the only exception that I got. When I hit continue it proceeded without issue and the dll was loaded correctly. However, when I try without the debugger attached the error reappears. So this is one of those Schrodinger errors. I know the error is there because I see it when we run on the CI and on a local machine, but all attempts to isolate it say that is should not exist. There can't really be a path that is different when compiling then loading from a pyc unless this is some type of uninitialized variable issues where we just happened to have been unfortunate enough to hit it. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42618] Enhancement request for importing stacktraces from foreign sources
New submission from Karl Nelson :
In JPype, I am transfer stack information from Java into Python for diagnostics
and display purposed. Unfortunately, as the exception system is directly
accessing traceback structure elements this cannot be replicated without
creating traceback structures in C. I have thus been forced to create new
methods that create this using internal Python structures. It would be much
better if there were C API method to allow importing a foreign stack trace into
Python.
Here is an example of the code used in JPype for reference.
```
// Transfer list of filenames, functions and lines to Python.
PyObject* PyTrace_FromJPStackTrace(JPStackTrace& trace)
{
PyTracebackObject *last_traceback = NULL;
PyObject *dict = PyModule_GetDict(PyJPModule);
for (JPStackTrace::iterator iter = trace.begin(); iter != trace.end();
++iter)
{
last_traceback = tb_create(last_traceback, dict,
iter->getFile(),
iter->getFunction(), iter->getLine());
}
if (last_traceback == NULL)
Py_RETURN_NONE;
return (PyObject*) last_traceback;
}
PyTracebackObject *tb_create(
PyTracebackObject *last_traceback,
PyObject *dict,
const char* filename,
const char* funcname,
int linenum)
{
// Create a code for this frame.
PyCodeObject *code = PyCode_NewEmpty(filename, funcname, linenum);
// Create a frame for the traceback.
PyFrameObject *frame = (PyFrameObject*)
PyFrame_Type.tp_alloc(_Type, 0);
frame->f_back = NULL;
if (last_traceback != NULL)
{
frame->f_back = last_traceback->tb_frame;
Py_INCREF(frame->f_back);
}
frame->f_builtins = dict;
Py_INCREF(frame->f_builtins);
frame->f_code = (PyCodeObject*) code;
frame->f_executing = 0;
frame->f_gen = NULL;
frame->f_globals = dict;
Py_INCREF(frame->f_globals);
frame->f_iblock = 0;
frame->f_lasti = 0;
frame->f_lineno = 0;
frame->f_locals = NULL;
frame->f_localsplus[0] = 0;
frame->f_stacktop = NULL;
frame->f_trace = NULL;
frame->f_valuestack = 0;
#if PY_VERSION_HEX>=0x0307
frame->f_trace_lines = 0;
frame->f_trace_opcodes = 0;
#endif
// Create a traceback
PyTracebackObject *traceback = (PyTracebackObject*)
PyTraceBack_Type.tp_alloc(_Type, 0);
traceback->tb_frame = frame;
traceback->tb_lasti = frame->f_lasti;
traceback->tb_lineno = linenum;
traceback->tb_next = last_traceback;
return traceback;
}
```
--
components: C API
messages: 382852
nosy: Thrameos
priority: normal
severity: normal
status: open
title: Enhancement request for importing stacktraces from foreign sources
type: enhancement
___
Python tracker
<https://bugs.python.org/issue42618>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42617] Enhancement request for PyType_FromSpecWIthBases add option for meta class
New submission from Karl Nelson :
PyType_FromSpecWithBases is missing an argument for taking a meta class. As a
result it is necessary to replicate a large portion of Python code when I need
to create a new heap type with a specified meta class. This is a maintenance
issue as replicating Python code is likely to get broken in future.
I have replicated to code from JPype so that it is clear how the meta class is
coming into place. PyJPClass_Type is derived from a
PyHeapType with additional slots for Java to use and overrides allocation slots
to that it can add extra slots (needs true multiple inheritance as it needed to
add Java slots to types with mismatching memory layouts object, long, float,
exception, ...).
This code could be made much safer if there were a PyType_FromSpecWithBasesMeta
which used the meta class to allocate the memory (then I could safely override
the slots after creation).
```
PyObject* PyJPClass_FromSpecWithBases(PyType_Spec *spec, PyObject *bases)
{
JP_PY_TRY("PyJPClass_FromSpecWithBases");
// Python lacks a FromSpecWithMeta so we are going to have to fake it
here.
PyTypeObject* type = (PyTypeObject*)
PyJPClass_Type->tp_alloc(PyJPClass_Type, 0); // <== we need to use the meta
class here
PyHeapTypeObject* heap = (PyHeapTypeObject*) type;
type->tp_flags = spec->flags | Py_TPFLAGS_HEAPTYPE | Py_TPFLAGS_HAVE_GC;
type->tp_name = spec->name;
const char *s = strrchr(spec->name, '.');
if (s == NULL)
s = spec->name;
else
s++;
heap->ht_qualname = PyUnicode_FromString(s);
heap->ht_name = heap->ht_qualname;
Py_INCREF(heap->ht_name);
if (bases == NULL)
type->tp_bases = PyTuple_Pack(1, (PyObject*) &
PyBaseObject_Type);
else
{
type->tp_bases = bases;
Py_INCREF(bases);
}
type->tp_base = (PyTypeObject*) PyTuple_GetItem(type->tp_bases, 0);
Py_INCREF(type->tp_base);
type->tp_as_async = >as_async;
type->tp_as_buffer = >as_buffer;
type->tp_as_mapping = >as_mapping;
type->tp_as_number = >as_number;
type->tp_as_sequence = >as_sequence;
type->tp_basicsize = spec->basicsize;
if (spec->basicsize == 0)
type->tp_basicsize = type->tp_base->tp_basicsize;
type->tp_itemsize = spec->itemsize;
if (spec->itemsize == 0)
type->tp_itemsize = type->tp_base->tp_itemsize;
// <=== Replicated code from the meta class
type->tp_alloc = PyJPValue_alloc;
type->tp_free = PyJPValue_free;
type->tp_finalize = (destructor) PyJPValue_finalize;
// <= Replicated code from Python
for (PyType_Slot* slot = spec->slots; slot->slot; slot++)
{
switch (slot->slot)
{
case Py_tp_free:
type->tp_free = (freefunc) slot->pfunc;
break;
case Py_tp_new:
type->tp_new = (newfunc) slot->pfunc;
break;
case Py_tp_init:
type->tp_init = (initproc) slot->pfunc;
break;
case Py_tp_getattro:
type->tp_getattro = (getattrofunc) slot->pfunc;
break;
case Py_tp_setattro:
type->tp_setattro = (setattrofunc) slot->pfunc;
break;
case Py_tp_dealloc:
type->tp_dealloc = (destructor) slot->pfunc;
break;
case Py_tp_str:
type->tp_str = (reprfunc) slot->pfunc;
break;
case Py_tp_repr:
type->tp_repr = (reprfunc) slot->pfunc;
break;
case Py_tp_methods:
type->tp_methods = (PyMethodDef*) slot->pfunc;
break;
case Py_sq_item:
heap->as_sequence.sq_item = (ssizeargfunc)
slot->pfunc;
break;
case Py_sq_length:
heap->as_sequence.sq_length = (lenfunc)
slot->pfunc;
break;
case Py_mp_ass_subscript:
heap->as_mapping.mp_ass_subscri
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I managed to get the debugger attached (unfortunately I am not a windows programmer so I don't use these tools). It appears when loading from a pyc, it is attempting to open the directory as a Zip file which is throwing an exception resulting in a failure to load the internal _jpype module. Unfortunately this is outside my area as nothing in jpype nor _jpype is calling the zipimport module. So it must be something internal to Python. It is a major issue as I can't release to anaconda on windows until I can resolve the problem. Can you direct me how to proceed? Details: zipimport.ZipImportError Message=not a Zip file StackTrace: :88 in __init__ Replicating the problem is simple. Download jpype from git, use "python setup.py develop" to create the module, run python -c "import jpype", and repeat python -c "import jpype" Screen shot of the visual studio error is shown. -- Added file: https://bugs.python.org/file49665/Capture.PNG ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
New submission from Karl Nelson : While trying to use JPype on Windows Python 3.9.0, we are running into an bizarre issue with loading the internal module which is written in C. When running a python script the first time the internal module loads correctly. However, the second time that script is run the internal module reports "A dynamic link library (DLL) initialization routine failed." If you then erase the pyc cache file that is importing the internal module then it works again. This only occurs on Windows and was not present using the same source prior versions of Python 3.8. We investigate the byte codes from both version and they are doing the same series of actions so the problem appears to be calling the same opcode to execute an import. I make sure all required symbols were found in the libraries and only only copy of the internal DLL was the same with and without loading from a pyc. It may be a change in the requirements of module initialization, but I don't know how to proceed. There was one deprecation warning but correcting that did not alter the outcome. It appears that the execute path for importing a CPython module takes a different path when the script was imported from something compiled on the fly as opposed to loading from a pyc. ``` import: 'jpype' Traceback (most recent call last): File "D:\bld\jpype1_1605785280189\test_tmp\run_test.py", line 2, in import jpype File "D:\bld\jpype1_1605785280189\_test_env\lib\site-packages\jpype\__init__.py", line 18, in import _jpype ImportError: DLL load failed while importing _jpype: A dynamic link library (DLL) initialization routine failed. ``` -- components: Windows messages: 382275 nosy: Thrameos, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: CPython DLL initialization routine failed from PYC cache file type: behavior versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
