[issue47001] deadlock in ctypes?

[Rocco Matano]

Rocco Matano  added the comment:

I forgot to say thank you. I would like to make up for that:
Thank you, Eryk.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47001] deadlock in ctypes?

[Rocco Matano]

Rocco Matano  added the comment:

@Eryk I think you hit the nail on the head with your recommendation to avoid 
ctypes.c_wchar_p (i.e. wintypes.LPWSTR) as the parameter type when dealing 
resource type/name strings. Of course ctypes automatic conversion from a C 
character pointer to a Python string will cause problems when being confronted 
with a 16-bit ID that is NOT a valid pointer. 

Now that it is clear that the problem was in front of the monitor, I consider 
this case closed.

--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47001] deadlock in ctypes?

[Eryk Sun]

Eryk Sun  added the comment:

Pointers to resource type/name strings use the lower 16-bit range for integer 
identifiers such as RT_ICON (3) and RT_GROUP_ICON (14). C code checks for these 
cases using the IS_INTRESOURCE() macro.

It's incorrect to use a simple C string pointer type such as ctypes.c_wchar_p 
(i.e. wintypes.LPWSTR) for a resource type/name string. Simple types get 
automatically converted to the corresponding Python type (e.g. str), but a 
16-bit ID is not a valid pointer. Support for resource IDs can be implemented 
by a subclass of ctypes.c_wchar_p, since subclasses of simple types don't get 
converted automatically. For example:

class LPWSTR(ctypes.c_wchar_p):
@property
def value(self):
v = ctypes.c_void_p.from_buffer(self).value or 0
if v >> 16 == 0:
return v
return super().value

@value.setter
def value(self, v):
ctypes.c_wchar_p.value.__set__(self, v)

def __eq__(self, other):
if not isinstance(other, __class__):
return NotImplemented
return self.value == other.value

def __hash__(self):
return hash(self.value)


RT_ICON = LPWSTR(3)
RT_GROUP_ICON = LPWSTR(RT_ICON.value + 11)

> kernel32 = ctypes.windll.kernel32

I recommend using `kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)`, 
which creates function pointers that capture the last error value before they 
return. Call ctypes.get_last_error() to get the captured error value. To get an 
exception for the error value, use ctypes.WinError(ctypes.get_last_error()). 
This is more reliable since there's a chance that the thread executes some 
WinAPI code that modifies the last error before GetLastError() is called. This 
is particularly likely when working with ctypes calls in the interactive shell.

Using a separate WinDLL() instance also isolates a library from other 
libraries. Since ctypes.windll caches modules, which cache function pointers, 
it's a problem when multiple libraries require different function prototypes 
(i.e. restype, argtypes, errcheck). The last library to set the prototype wins. 
The other libraries will probably be broken in some way. Their function calls 
may raise a ctypes.ArgumentError, or their post-call errcheck() isn't used, or 
their required restype isn't returned.

> hmod = GetModuleHandle(sys.executable)

This is unnecessary and possibly wrong since there's no guarantee that 
sys.executable is a valid file, let alone a file that's loaded as a module in 
the current process. Pass hModule as NULL (None) to use the main module of the 
process (i.e. the process image / executable).

>  context structure that is used to use regular python functions as callbacks
>  where external C code expects C callbacks with a certain signature.

Generally there's no need to use C context parameters, which aren't necessary 
in object-oriented and functional programming paradigms. The callback function 
can defined as a nested function, with access to closure variables, or as a 
method, with access to instance attributes.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47001] deadlock in ctypes?

[Rocco Matano]

New submission from Rocco Matano :

When using ctypes under Windows, I have observed a baffling behavior which I am 
not sure if it is the result of a bug in ctypes, or is simply due to false 
expectations on my part.

Environment:
 - Windows 10 Pro, 21H1, 19043.1586, x64
 - Python versions 3.6 up to 3.10, x64
 
What I was trying to do was to enumerate all the icons in the resource section 
of an executable file. The Windows-API for such a case is EnumResourceNamesW, 
which requires the pointer to a callback function as an argument. I wanted to 
hide the details of the corresponding code so that a user of that code does not 
have to deal with ctypes directly and can supply a regular Python function for 
the callback. So I added an extra level of indirection.

When I ran my code, I was surprised to observe that the program was stuck. 
After several tries, I found a solution that differed from the original code 
only in one small detail: Use different types when describing the C callback.

Below is a self contained sample that can be used to show the successful case 
as well as to trigger the blocking. Unfortunately it is quit long. 


import sys
import ctypes
from ctypes.wintypes import HMODULE, LPVOID, LPWSTR, BOOL

# context structure that is used to use regular python functions as callbacks
# where external C code expects C callbacks with a certain signature.
class CallbackContext(ctypes.Structure):
_fields_ = (
("callback", ctypes.py_object),
("context", ctypes.py_object),
)

CallbackContextPtr = ctypes.POINTER(CallbackContext)

# quote from
# 
https://docs.microsoft.com/en-us/windows/win32/api/libloaderapi/nc-libloaderapi-enumresnameprocw
#
# BOOL Enumresnameprocw(
#   [in, optional] HMODULE hModule,
#  LPCWSTR lpType,
#  LPWSTR lpName,
#   [in]   LONG_PTR lParam
# )
# Note that the arguments lpType and lpName are declared as pointers to strings.


if len(sys.argv) > 1 and sys.argv[1].lower() == "fail":
# Declaring them as pointers to strings in the ctypes prototype does NOT 
work.
EnumResNameCallback_prototype = ctypes.WINFUNCTYPE(
BOOL,
HMODULE,
LPWSTR,
LPWSTR,
CallbackContextPtr
)
else:
# Declaring them as void pointers does work!
EnumResNameCallback_prototype = ctypes.WINFUNCTYPE(
BOOL,
HMODULE,
LPVOID,
LPVOID,
CallbackContextPtr
)

# this is the ctypes callback function that mimics the required C call signature
@EnumResNameCallback_prototype
def EnumResNameCallback(hmod, typ, name, ctxt):
cbc = ctxt.contents
return cbc.callback(hmod, typ, name, cbc.context)

kernel32 = ctypes.windll.kernel32

EnumResourceNames = kernel32.EnumResourceNamesW
EnumResourceNames.restype = BOOL
EnumResourceNames.argtypes = (
HMODULE,
LPWSTR,
EnumResNameCallback_prototype,
CallbackContextPtr
)

# Get a module handle for an executable that contains icons
GetModuleHandle = kernel32.GetModuleHandleW
GetModuleHandle.restype = HMODULE
GetModuleHandle.argtypes = (LPWSTR,)

hmod = GetModuleHandle(sys.executable)
if hmod == 0:
raise ctypes.WinError()

RT_GROUP_ICON = ctypes.cast(14, LPWSTR)

# this is the 'regular' callback function that does not have to care about
# the C call signature
def enum_callback(hmod, typ, name, unused_context):
print(hmod, typ, name)
return True

cbc = CallbackContext(enum_callback, None)
rcbc = ctypes.byref(cbc)

print("Trying to enumerate icons.")
print("In the case of failure, this WILL BLOCK indefinitely!")
EnumResourceNames(hmod, RT_GROUP_ICON, EnumResNameCallback, rcbc)

--
components: ctypes
messages: 415029
nosy: rocco.matano
priority: normal
severity: normal
status: open
title: deadlock in ctypes?
type: behavior
versions: Python 3.10, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com