[issue27743] Python 2 has a wrong artificial limit on the amount of memory that can be allocated in ctypes

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

It was fixed in issue16865.

--
nosy: +serhiy.storchaka
resolution:  -> out of date
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27743] Python 2 has a wrong artificial limit on the amount of memory that can be allocated in ctypes

[Eryk Sun]

Eryk Sun added the comment:

AFAIK this only affects Windows. It looks like a relatively simple fix. In 
PyCArrayType_new, change the declaration of `length` to Py_ssize_t to match the 
definition StgDictObject.length; ensure the _length_ attribute is an index via 
PyIndex_Check instead of PyInt_Check; and call PyNumber_AsSsize_t instead of 
PyInt_AS_LONG.

--
components: +Windows
nosy: +eryksun, paul.moore, tim.golden, zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27743] Python 2 has a wrong artificial limit on the amount of memory that can be allocated in ctypes

[tzickel]

New submission from tzickel:

Python 2 has a wrong artificial limit on the amount of memory that can be 
allocated in ctypes via sequence repeating (i.e. using create_string_buffer or 
c_char * )

The problem is practical in Windows 64 bit, when running python 64 bit, since 
in that platform the sys.maxint is 2GB and while sys.maxsize is as large as in 
other platforms, trying to allocate more than 2GB of memory results in a 
different exception than other platforms (where sys.maxint = sys.maxsize):

Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec  5 2015, 20:40:30) [MSC v.1500 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, ctypes
>>> ctypes.c_char * (sys.maxint + 1)
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: class must define a '_length_' attribute, which must be a 
positive integer

>>> ctypes.create_string_buffer(sys.maxint + 1)
Traceback (most recent call last):
  File "", line 1, in 
  File "c:\Python27-64\lib\ctypes\__init__.py", line 65, in create_string_buffer
buftype = c_char * init
AttributeError: class must define a '_length_' attribute, which must be a 
positive integer

on other platforms you get:

Traceback (most recent call last):
  File "", line 1, in 
OverflowError: cannot fit 'long' into an index-sized integer

Thus to allocate more than 2GB, you need to use other methods (numpy or 
something else).

>From my reading of the code, I assume the bug is this line:
https://github.com/python/cpython/blob/2.7/Modules/_ctypes/_ctypes.c#L1388

Where the code checks if _length_ is an integer (PyInt_Check). As soon as the 
number is bigger than sys.maxint, it's a long, and while it's a valid size for 
the platform (< sys.maxsize), it will bork there. Since this seems like an 
artificial limit, I think it should be fixed, since it's practical to allocate 
this sizes on 64 bit systems for some applications.

Python 3 has no issue, since it has no int type :)

--
components: ctypes
messages: 272510
nosy: Bob, Christoph Sarnowski, Patrick Stewart, doko, jpe, larry, 
mark.dickinson, mattip, meador.inge, python-dev, rkuska, steve.dower, tzickel, 
vinay.sajip
priority: normal
severity: normal
status: open
title: Python 2 has a wrong artificial limit on the amount of memory that can 
be allocated in ctypes
type: behavior
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com