[issue6069] casting error from ctypes array to structure
Vlad Riscutia riscutiav...@gmail.com added the comment: Meador, I believe this was the first issue on the tracker that got me looking into bitfield allocation. I agree that big-endian on MSVC doesn't make too much sense but you can disregard that - using default endianess will still yield different sizes of bitfields when compiled with GCC and MSVC. Basically bitfield allocation is compiler specific and patch in issue12528 implements a way to select which allocation strategy to be used at runtime instead of hardcoding the one with which Python is compiled. This should improve cross-compiler interop. I wanted to hyperlink that patch to all other bitfield bugs, that's why I followed up with link to the patch. Feel free to close this, either as not an issue or as a duplicate of issue12528. And yes, this bit about bitfield allocation should be documented and I was planning to look into it at some point after 12528 gets committed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6069 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6069] casting error from ctypes array to structure
Meador Inge mead...@gmail.com added the comment: On Thu, Sep 1, 2011 at 9:45 AM, Vlad Riscutia rep...@bugs.python.org wrote: Vlad Riscutia riscutiav...@gmail.com added the comment: Meador, I believe this was the first issue on the tracker that got me looking into bitfield allocation. I agree that big-endian on MSVC doesn't make too much sense but you can disregard that - using default endianess will still yield different sizes of bitfields when compiled with GCC and MSVC. Sure, but this particular issue is purporting that the layout of the structure is incorrect, not that the size is. Basically bitfield allocation is compiler specific and patch in issue12528 implements a way to select which allocation strategy to be used at runtime instead of hardcoding the one with which Python is compiled. This should improve cross-compiler interop. I wanted to hyperlink that patch to all other bitfield bugs, that's why I followed up with link to the patch. Yes, it is very compiler specific. I have some thoughts about making this configurable, but I will comment on issue12528 with those. Feel free to close this, either as not an issue or as a duplicate of issue12528. I will open a documentation bug and close this one out. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6069 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6069] casting error from ctypes array to structure
Vlad Riscutia riscutiav...@gmail.com added the comment: Sounds good. Please nosy me in the doc bug. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6069 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6069] casting error from ctypes array to structure
Changes by Meador Inge mead...@gmail.com: -- stage: test needed - committed/rejected ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6069 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6069] casting error from ctypes array to structure
Meador Inge mead...@gmail.com added the comment: I opened issue12880 for the doc bug. Closing this one out ... -- resolution: - invalid status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6069 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6069] casting error from ctypes array to structure
Meador Inge mead...@gmail.com added the comment:
Hmmm ... Assuming a native VC++ compiler on an x86 machine running Windows,
then it doesn't make sense to validate these test cases in such an environment.
All the tests are all big-endian.
'ctypes' can't be expected to behave the same as the native compiler that
compiled the Python interpreter for structures of non-native endianities
produced by 'ctypes'. That doesn't make sense. The best we can do is document
how 'ctypes' does handle non-native endianites on various platforms.
FWIW, I did try the first set of tests (http://bugs.python.org/msg88145) with
GCC for a 32-bit MIPS ELF target using the following test case:
#include stdio.h
struct T {
unsigned int x : 31;
unsigned int y : 32;
};
struct S {
unsigned long long x : 31;
unsigned long long y : 32;
};
int main (int argc, char **argv)
{
unsigned char buf[8] = {0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55};
struct T *t = (struct T*)buf;
struct S *s = (struct S*)buf;
printf (%X, %X\n, t-x, t-y);
printf (%X, %X\n, s-x, s-y);
}
The test output:
Data0 is 0x2AAA, Data1 is 0x for uint
Data0 is 0x2AAA, Data1 is 0x for ulonglong
is correct with respect to that environment.
The difference in the first case (uint) and the second case (ulonglong) is that
the first is placed into two 4-byte unsigned integer units where as the second
is placed into one 8-byte unsigned long long unit.
I am slightly confused how issue12528 is going to address this, when there
seems to be no bug; only what seems to be a test case problem. I think we
should close this issue out.
Another issue should be opened to enhance the documentation, though. We should
document exactly how 'ctypes' does the structure layout for different
endianities on different platforms. Something similar to how VC++ documents
this ( http://msdn.microsoft.com/en-us/library/ewwyfdbe(v=vs.71).aspx ).
--
assignee: theller -
nosy: +meadori -theller
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6069
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6069] casting error from ctypes array to structure
Vlad Riscutia riscutiav...@gmail.com added the comment: Opened http://bugs.python.org/issue12528 to address this. -- versions: +Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6069 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6069] casting error from ctypes array to structure
Vlad Riscutia riscutiav...@gmail.com added the comment:
I took a look at this and I believe behavior is correct on Windows, the issue
is with the test. For example this test is failing:
class closest_fit(ctypes.BigEndianStructure):
_pack_ = 1# aligned to 8 bits, not ctypes default of 32
_fields_= [
(Data0, ctypes.c_uint32, 32),
(Data1, ctypes.c_uint8, 3),
(Data2, ctypes.c_uint16, 12),
]
But you also have this assumption when generating the test data:
size_of_structures_in_bytes = 6
I verified and this does not hold with MSVC compiler. Using VC++ 2005, this code
typedef struct Test {
unsigned int x: 32; // uint_32 : 32
unsigned char y: 3; // uint_8 : 3
unsigned short int z: 12; // uint_16 : 12
} Test;
gives sizeof(Test) == 7. In Python, if you look at sizeof(closest_fit), it will
also be 7.
Looking at cfield.c, seems this was taken into account when creating bit fields:
if (bitsize /* this is a bitfield request */
*pfield_size /* we have a bitfield open */
#ifdef MS_WIN32
/* MSVC, GCC with -mms-bitfields */
dict-size * 8 == *pfield_size
#else
/* GCC */
dict-size * 8 = *pfield_size
#endif
(*pbitofs + bitsize) = *pfield_size) {
/* continue bit field */
fieldtype = CONT_BITFIELD;
#ifndef MS_WIN32
} else if (bitsize /* this is a bitfield request */
*pfield_size /* we have a bitfield open */
dict-size * 8 = *pfield_size
(*pbitofs + bitsize) = dict-size * 8) {
/* expand bit field */
fieldtype = EXPAND_BITFIELD;
#endif
} else if (bitsize) {
/* start new bitfield */
fieldtype = NEW_BITFIELD;
*pbitofs = 0;
*pfield_size = dict-size * 8;
Though I don't know this first-hand, above code plus sizeof experiment leads me
to believe that gcc packs bitfields differently than MSVC. Seems that gcc will
expand existing bitfield trying to pack structure more tightly so indeed on
Linux (or I assume Windows gcc build), size of this structure is 6 as gcc will
combine these seeing that an unsigned short can hold all 15 bits required but
with MSVC this won't work. MSVC will allocate both the c_uint8 and the c_uint16
once is sees that last 12 bits don't fit in remaining c_uint8.
As far as I can tell this is by design and Python matches expected MSVC
structure packing for this test case.
--
nosy: +vladris
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6069
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6069] casting error from ctypes array to structure
Terry J. Reedy tjre...@udel.edu added the comment: WinXP,32bit,3.1.2 result Data0 is 0x2AAA, Data1 is 0x for uint Data0 is 0x2AAA, Data1 is 0x for ulonglong Data1 failed but I do not understand ctypes enough to verify that test is correct. -- nosy: +tjreedy stage: - unit test needed type: - behavior versions: +Python 2.7, Python 3.1, Python 3.2 -Python 2.5, Python 2.6, Python 3.0 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6069 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6069] casting error from ctypes array to structure
higstar adr...@higstar.com added the comment:
Another example of this:
---
import ctypes
correct_data_dict = {
'Data0' : 0x,
'Data1' : 0x02,
'Data2' : 0x0AAA,
}
class closest_fit(ctypes.BigEndianStructure):
_pack_ = 1# aligned to 8 bits, not ctypes default of 32
_fields_= [
(Data0, ctypes.c_uint32, 32),
(Data1, ctypes.c_uint8, 3),
(Data2, ctypes.c_uint16, 12),
]
class closest_fit_min_16(ctypes.BigEndianStructure):
_pack_ = 1# aligned to 8 bits, not ctypes default of 32
_fields_= [
(Data0, ctypes.c_uint32, 32),
(Data1, ctypes.c_uint16, 3),
(Data2, ctypes.c_uint16, 12),
]
class closest_fit_min_32(ctypes.BigEndianStructure):
_pack_ = 1# aligned to 8 bits, not ctypes default of 32
_fields_= [
(Data0, ctypes.c_uint32, 32),
(Data1, ctypes.c_uint32, 3),
(Data2, ctypes.c_uint32, 12),
]
class uint32(ctypes.BigEndianStructure):
_pack_ = 1# aligned to 8 bits, not ctypes default of 32
_fields_= [
(Data0, ctypes.c_uint32, 32),
(Data1, ctypes.c_uint32, 3),
(Data2, ctypes.c_uint32, 12),
]
class uint64(ctypes.BigEndianStructure):
_pack_ = 1# aligned to 8 bits, not ctypes default of 32
_fields_= [
(Data0, ctypes.c_uint64, 32),
(Data1, ctypes.c_uint64, 3),
(Data2, ctypes.c_uint64, 12),
]
size_of_structures_in_bytes = 6
def castbytes(type):
buffer = (ctypes.c_byte * size_of_structures_in_bytes)()
for index in range(size_of_structures_in_bytes):
buffer[index] = 0x55
return ctypes.cast(ctypes.pointer(buffer),
ctypes.POINTER(type)).contents
def print_members(test):
print(Data0 is 0x%X, Data1 is 0x%X, Data2 is 0x%X for
%s%(test.Data0, test.Data1, test.Data2, test.__class__.__name__))
test_classes = [closest_fit, uint32, closest_fit_min_16,
closest_fit_min_32, uint64]
Test_Failed = False
tests = [castbytes(type) for type in test_classes]
for test in tests:
#print_members(test)
for data in correct_data_dict:
if not test.__getattribute__(data) == correct_data_dict[data]:
Test_Failed = True
print(%s failed for %s, value was 0x%X but should have been
0x%X%(data, test.__class__.__name__, test.__getattribute__(data),
correct_data_dict[data]))
if not Test_Failed:
print(Passed)
---
c:\python25\python.exe IssueNEW.py
Data2 failed for closest_fit, value was 0x550 but should have been 0xAAA
c:\python26\python.exe IssueNEW.py
Data2 failed for closest_fit, value was 0x550 but should have been 0xAAA
c:\python30\python.exe IssueNEW.py
Data2 failed for closest_fit, value was 0x550 but should have been 0xAAA
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6069
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6069] casting error from ctypes array to structure
New submission from higstar adr...@higstar.com: Structure fails to correctly cast from a 2 byte bitfield. From my very limited investigation, is looks like when using a member type of less than the total size of the structure (or at least the size of any byte boundaries) the casting is not done correctly? I created this test.py and appended the results below: --- import ctypes import time class closest_fit(ctypes.BigEndianStructure): #_pack_ = 1# aligned to 8 bits, not ctypes default of 32 _fields_= [ (Data0, ctypes.c_ubyte, 7), (Data1, ctypes.c_ubyte, 8), ] class all_ulong(ctypes.BigEndianStructure): #_pack_ = 1# aligned to 8 bits, not ctypes default of 32 _fields_= [ (Data0, ctypes.c_ulonglong, 7), (Data1, ctypes.c_ulonglong, 8), ] def castbytes(type): buffer = (ctypes.c_byte * 2)() buffer[0] = 0x55 buffer[1] = 0x55 return ctypes.cast(ctypes.pointer(buffer), ctypes.POINTER(type)).contents def print_members(test): print(Data0 is 0x%X, Data1 is 0x%X for %s%(test.Data0, test.Data1, test.__class__.__name__)) test_classes = [ closest_fit, all_ulonglong] Failed = False tests = [castbytes(type) for type in test_classes] for test in tests: print_members(test) if not tests[0].Data0 == tests[1].Data0: Failed = True if not tests[0].Data1 == tests[1].Data1: Failed = True if Failed: print(Failed) else: print(Passed) --- c:\python25\python.exe test.py Data0 is 0x2A, Data1 is 0x55 for closest_fit Data0 is 0x2A, Data1 is 0xAA for all_ulonglong Failed c:\python26\python.exe test.py Data0 is 0x2A, Data1 is 0x55 for closest_fit Data0 is 0x2A, Data1 is 0xAA for all_ulonglong Failed c:\python30\python.exe test.py Data0 is 0x2A, Data1 is 0x55 for closest_fit Data0 is 0x2A, Data1 is 0xAA for all_ulonglong Failed As you can see the second member Data1, should be 0xAA, however when using c_ubyte types for members the value is not offset by one bit. As you can see using c_ulonglong for all members avoids this issue, however this results in a read only structure (see Issue 6068). I am using structures to cast CAN messages which are 8 bytes, with very funky bit fields crossing all sorts of byte boundaries, so I essentially expected that ctypes would provide a method for an arbitrary bit field definition for use within python. Hopefully this is just my bad ctypes driving, or a simple fix. -- assignee: theller components: ctypes messages: 88111 nosy: higstar, theller severity: normal status: open title: casting error from ctypes array to structure versions: Python 2.5, Python 2.6, Python 3.0 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6069 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6069] casting error from ctypes array to structure
higstar adr...@higstar.com added the comment: After reading the documentation for ctypes (specifically Bit fields are only possible for integer fields from section 16.15.1.12) I've updated the test. --- import ctypes import time class uint(ctypes.BigEndianStructure): _pack_ = 1# aligned to 8 bits, not ctypes default of 32 _fields_= [ (Data0, ctypes.c_uint, 31), (Data1, ctypes.c_uint, 32), ] class ulonglong(ctypes.BigEndianStructure): _pack_ = 1# aligned to 8 bits, not ctypes default of 32 _fields_= [ (Data0, ctypes.c_ulonglong, 31), (Data1, ctypes.c_ulonglong, 32), ] size_of_structures_in_bytes = 8 def castbytes(type): buffer = (ctypes.c_byte * size_of_structures_in_bytes)() for index in range(size_of_structures_in_bytes): buffer[index] = 0x55 return ctypes.cast(ctypes.pointer(buffer), ctypes.POINTER(type)).contents def print_members(test): print(Data0 is 0x%X, Data1 is 0x%X for %s%(test.Data0, test.Data1, test.__class__.__name__)) test_classes = [ uint, ulonglong] Failed = False tests = [castbytes(type) for type in test_classes] for test in tests: print_members(test) if not tests[0].Data0 == tests[1].Data0 == 0x2AAA: Failed = True print(Data0 failed) if not tests[0].Data1 == tests[1].Data1 == 0x: Failed = True print(Data1 failed) if not Failed: print(Passed) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6069 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
