[issue11427] ctypes from_buffer no longer accepts bytes

[eryksun]

eryksun added the comment:

cast calls ctypes._cast(obj, obj, typ). _cast is a ctypes function pointer 
defined as follows:

_cast = PYFUNCTYPE(py_object, 
   c_void_p, py_object, py_object)(_cast_addr)

Since cast makes an FFI call that converts the first arg to c_void_p, you can 
directly cast bytes to a pointer type:

>>> from ctypes import *
>>> data = b'123\x00abc'

>>> ptr = cast(data, c_void_p)

string_at is defined similarly using c_void_p for the first arg:

_string_at = PYFUNCTYPE(py_object, c_void_p, c_int)(_string_at_addr)
 
>>> string_at(ptr, 8)
b'123\x00abc\x00'

Get around the from_buffer mutability requirement by casting to an array (i.e. 
to an array pointer followed by a dereference):

>>> arr = cast(data, POINTER(c_char * len(data)))[0]
>>> arr[:]
b'123\x00abc'

Then use byref to pass an offset into the array:

>>> from ctypes.util import find_library
>>> printf = CDLL(find_library('c')).printf

>>> printf(b'%s\n', byref(arr, 4))
abc
4

Since this doesn't copy the buffer, take care to ensure the function call won't 
modify the contents.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[Martin Panter]

Martin Panter added the comment:

For the record, I have been (ab?)using the “c_char_p” type to get the address 
of immutable byte strings without copying memory:

>>> from ctypes import *
>>> data = b"123\x00abc"
>>> pubyte = cast(c_char_p(data), POINTER(c_ubyte))
>>> address = addressof(pubyte.contents)
>>> string_at(address, 7)
b'123\x00abc'
>>> # 2nd instance has same address, rather than memory copy:
... cast(c_char_p(data), c_void_p).value == address
True

In to the documentation, “c_char_p” is only defined for zero-terminated 
strings, but it seems to also work for strings with embedded zero bytes. 
However it does not work for “bytearray” objects (“from_buffer” should work 
instead), nor memory views.

--
nosy: +vadmium

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[STINNER Victor]

STINNER Victor  added the comment:

> I just want to call my C function.

You can use something else than ctypes_array.from_buffer(buffer). 
ctypes_array.from_buffer() creates a read-write object which can be used to 
modify the buffer. You cannot pass a read-only object to 
ctypes_array.from_buffer().

> I'm pretty sure that I verified that this code worked in 3.1.3 before
> opening this bug, but it's been a while.

If it was possible, it was a bug. You cannot modify a read-only object, like 
bytes, because it would lead to inconsistent object and may lead your program 
to a crash, even if you don't call C functions.

Example in pure Python:

>>> import ctypes
>>> T=(ctypes.c_uint8*4)
>>> B=bytearray(4)
>>> x=T.from_buffer(B)
>>> B
bytearray(b'\x00\x00\x00\x00')
>>> x[0]=9
>>> x[2]=100
>>> B
bytearray(b'\t\x00d\x00')
>>> list(x) 
>>> 
[9, 0, 100, 0]

B is modified when x[index] is modified.

--

A bug tracker is not the right place to ask help with ctypes.

--
resolution:  -> invalid
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[benrg]

benrg  added the comment:

I am still interested in this for the same reason I was interested in this in 
the first place; nothing has changed. I guess I will reiterate, and try to 
expand.

The problem is that ctypes tries to enforce const correctness (inconsistently), 
but it has no way to declare its objects as const, and assumes they are all 
non-const. This is the worst possible combination. It would make sense to have 
no notion of const and never try to enforce it (like K&R C), and it would make 
sense to have constness and try to enforce it (like C++). But the present 
situation is like a compiler that treats string literals as const char[], and 
doesn't allow casting away const, but doesn't allow the use of the const 
keyword in user code, so there's no way to pass a string literal as a function 
argument because the necessary type can't be expressed. Instead you have to 
copy the literal to a char array and pass that.

Calling C functions is inherently unsafe. Calling C functions could always 
corrupt the Python heap or otherwise screw around with state that the Python 
environment thinks that it owns. All I want is some means to assert that my 
code is not going to do that, as a way of getting around the limited type 
system that can't provide those sorts of guarantees. More broadly, what I want 
from ctypes is a way to do the sorts of things that I can do in C. If I can 
call foo(&bar + 17) in C, I want to be able to make that call in Python.

I wasn't using (c_type*N).from_buffer because I wanted to. I was using it after 
wasting literally hours trying to find some other way to get ctypes to agree to 
pass a pointer into the buffer of a Python object to an external function 
(which was not going to alter it, I promise). This should be easy; instead it's 
nearly impossible. I don't want to wrestle with random un-overridable attempts 
to enforce correctness when calling a language where that can never be 
enforced. I just want to call my C function.

I'm pretty sure that I verified that this code worked in 3.1.3 before opening 
this bug, but it's been a while. I could try to reproduce it, but I think this 
functionality should be present regardless. You can call it a feature request 
instead of a regression if you want.

--
resolution: invalid -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[STINNER Victor]

STINNER Victor  added the comment:

.from_buffer() expects a modifiable buffer, bytes type is immutable. If 
.from_buffer() accepted an immutable buffer, it was a bug and the bug was fixed.

If you want to use bytes, use .from_buffer_copy() instead of .from_buffer().

--
resolution:  -> invalid
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[Meador Inge]

Meador Inge  added the comment:

benrg, any further interest in this?  If so, please comment on my last reply.  
Otherwise, I am closing this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[Meador Inge]

Meador Inge  added the comment:

Hmmm... I take back what I said before (not sure what I was thinking), I 
don't think this is a regression.  If this ever did work, it must have 
been a bug.  For one reason, Python byte strings are now immutable.  
Allowing their memory to be indirectly mutated in Python via buffers 
created from 'from_buffer' would be very bad.

Also, I can't reproduce the reported behavior with 3.1.3:

Python 3.1.3 (unknown, Nov  9 2011, 19:57:58) 
[GCC 4.6.1 20110908 (Red Hat 4.6.1-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> (ctypes.c_char*5).from_buffer(b'abcde')
Traceback (most recent call last):
  File "", line 1, in 
TypeError: expected an object with a writable buffer interface

Are you sure it was 3.1.3?  At first I thought you might be using
3.0a, since bytes are mutable in that version, but 'from_buffer'
didn't exist in those times.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[Santoso Wijaya]

Santoso Wijaya  added the comment:

>From what I understand, even in 2.7 the exception is expected. The doc for 
>`from_buffer` says:

> The source object must support the writeable buffer interface.

Is there any reason why `from_buffer_copy` cannot be used, instead? The latter 
only requires readable buffer interface.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[Santoso Wijaya]

Changes by Santoso Wijaya :


--
nosy: +santa4nt
versions: +Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[Meador Inge]

Meador Inge  added the comment:

I agree this is a regression.  I am looking more into it now.

--
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[Ezio Melotti]

Changes by Ezio Melotti :


--
nosy: +meador.inge

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[Georg Brandl]

Changes by Georg Brandl :


--
keywords: +3.2regression

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11427] ctypes from_buffer no longer accepts bytes

[benrg]

New submission from benrg :

In Python 3.1.3, (c_char*5).from_buffer(b'abcde') worked. In 3.2 it fails with 
"TypeError: expected an object with a writable buffer interface".

This seems to represent a significant decrease in the functionality of ctypes, 
since, if I understand correctly, it has no notion of a const array or a const 
char. I used from_buffer with a bytes argument in 3.1 and it was far from 
obvious how to port to 3.2 without introducing expensive copying. I understand 
the motivation behind requiring a writable buffer, but I think it's a bad idea. 
If you take this to its logical conclusion, it should not be possible to pass 
bytes or str values directly to C functions, since there's no way to be sure 
they won't write through the pointer.

--
assignee: theller
components: ctypes
messages: 130229
nosy: benrg, theller
priority: normal
severity: normal
status: open
title: ctypes from_buffer no longer accepts bytes
type: behavior
versions: Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com