[issue8990] array constructor and array.fromstring should accept bytearray.
Antoine Pitrou pit...@free.fr added the comment: The patches were committed in r84403. Thank you very much! -- resolution: - fixed stage: unit test needed - committed/rejected status: open - closed versions: -Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Antoine Pitrou pit...@free.fr added the comment: The patch shouldn't remove the tests for tostring()/fromstring() (they might be deprecated, but they are still supported). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Thomas Jollans tho...@jollans.com added the comment: That sounds reasonable. I've updated the patch to keep the old test_tofromstring testcase. I'll also attach another patch in a moment that removes what I'm reasonably sure is all the uses of array.tostring and .fromstring in the standard library and the other modules' tests. -- Added file: http://bugs.python.org/file18697/tofrombytes.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Changes by Thomas Jollans tho...@jollans.com: Added file: http://bugs.python.org/file18698/tostring_usage.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Changes by Thomas Jollans tho...@jollans.com: Removed file: http://bugs.python.org/file18607/tofrombytes.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Changes by Thomas Jollans tho...@jollans.com: Added file: http://bugs.python.org/file18606/tofrombytes.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Changes by Thomas Jollans tho...@jollans.com: Removed file: http://bugs.python.org/file18606/tofrombytes.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Thomas Jollans tho...@jollans.com added the comment: Hello again, sorry for the absense. Victor, thanks for the input. I've attached a new patch that checks the PyErr_WarnEx return value. -- Added file: http://bugs.python.org/file18607/tofrombytes.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Changes by Antoine Pitrou pit...@free.fr: Removed file: http://bugs.python.org/file17828/tofrombytes.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Antoine Pitrou pit...@free.fr added the comment: From a quick glance, the latest patch looks ok. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
STINNER Victor victor.stin...@haypocalc.com added the comment: I prefer the second solution (add to/frombytes, deprecate to/fromstring) because I prefer the new method names and it keeps backward compatibility (until we choose to remove the old methods, which should be in Python 3.3). About the patch (tofrombytes.diff). You should use PyErr_WarnEx() result: if it is not nul (eg. if the user choosed to raise exceptions on warnings), you have to exit. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Thomas Jollans tho...@jollans.com added the comment: Two more patches: Firstly, this patch (array_3.2_fromstring.diff) is nearly identical to array2.diff. y* would (again) have to be changed to s* to apply this to 3.1 -- Added file: http://bugs.python.org/file17826/array_3.2_fromstring.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Thomas Jollans tho...@jollans.com added the comment:
Secondly, this is my attempt to add the more sensibly named {to|from}bytes
methods, and to deprecate {to|from}string.
I doubt it's perfect, maybe there's some policy on deprecating methods that I
didn't find? This may be better discussed in a separate forum, eg a separate
issue or python-dev maybe? I wouldn't know.
The unpatched test suite passes with the rest of this patch applied. (unless
using -Werror)
--
Added file: http://bugs.python.org/file17827/tofrombytes.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8990
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Changes by Thomas Jollans tho...@jollans.com: Added file: http://bugs.python.org/file17828/tofrombytes.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Changes by Thomas Jollans tho...@jollans.com: Removed file: http://bugs.python.org/file17827/tofrombytes.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
STINNER Victor victor.stin...@haypocalc.com added the comment: array2.diff: - can you reindent the line Py_ssize_t old_size = Py_SIZE(self);?(even if it's not part of your patch) - you can avoid the char *str; variable (use directly buffer.buf) -- I changed s# to y*. This means that str arguments are no longer accepted by fromstring (...) it is an incompatible change It's maybe time to create .frombytes() and .tobytes() methods: - .tostring() will be a (deprecated?) alias to .tobytes() - .frombytes() only accepts bytes, bytearray and buffer compatible objects: use y* format - .fromstring() accepts str, bytes, bytearray and buffer compatible objects (encode str to utf-8): use s* format But I still don't understand why array.fromstring() accepts character strings. So an easier solution is to apply array2.diff to Python 3.2, and replace y* by s* when the patch is applied to 3.1. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Thomas Jollans tho...@jollans.com added the comment: OK, here's the new patch. I added tests for array(typecode, bytearray(b'abab')), a.extend(b'123') and a.extend(bytearray(b'123')). @Victor: int itemsize is the array's item size, buffer.itemsize is the strings' (and must be 1) PROBLEM with this patch: I changed s# to y*. This means that str arguments are no longer accepted by fromstring. I don't think they ever should have been in 3.x, but it is an incompatible change and this got the test suite, which (I assume the code hasn't changed since 2.x) used a str argument. (changed in patch). It might be best to use s* instead of y*, especially if this is applied to 3.1? -- Added file: http://bugs.python.org/file17749/array2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Thomas Jollans tho...@jollans.com added the comment:
Thanks for the input. I'm going to re-work the patch a bit (releasing buffers
and such) and add a test within the next few days.
The question remains whether or not to accept other buffers with itemsize == 1.
The way I understand it, fromstring already accepted any read-only buffer
object, no matter the item size / whether it actually makes sense to call it a
string. I don't think accepting a hypothetical read-only buffer with items
wider than 1 in fromstring (yes, bad naming) is desirable behaviour - I see a
few options on how to deal with input validation:
1. ignore the item size. This'd be similar to current behaviour, plus r/w
buffers
2. only accept byte-based buffers. (things that look like 'const char*') -
this is what I've been aiming at.
3. only accept bytes and bytearray, and let the user think about how to deal
with other objects. Question is - shouldn't array('B') be treated like
bytearray in this respect?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8990
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
STINNER Victor victor.stin...@haypocalc.com added the comment: About the patch: - Why not reading itemsize from buffer.itemsize? - A test would be nice in Lib/test/test_array.py (in test_tofromstring?). Please add also a.extend(bytearray(b'xjxjx')) to the test suite. I agree that it is a bugfix and so i can be applied to 3.1 (and 3.2). -- fromstring should become frombytes. I remember an issue proposing that. I also prefer .frombytes() name: we can add .frombytes() and keep .fromstring() as an alias to .frombytes() (and maybe mark it as deprecated). A message: http://bugs.python.org/issue3565#msg71205 An issue starting at: http://bugs.python.org/issue1023290#msg92069 Can someone open a issue about .frombytes()? fromunicode is still needed. Can someone open a new issue about .fromunicodes()? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Antoine Pitrou pit...@free.fr added the comment: On the principle the patch looks good. In practice, it lacks a call to `PyBuffer_Release(buffer)` in the various error cases (before returning NULL). It also lacks some tests in Lib/test/test_array.py. In 3.x unicode type was renamed to str and str to bytes, so fromunicode should become fromstring (or maybe fromstr to avoid confusion) and fromstring should become frombytes. This should be discussed separately, perhaps on python-dev first. -- nosy: +haypo, pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Terry J. Reedy tjre...@udel.edu added the comment:
Summary: real bug, needs test, patch provided, looks plausible but I am not
qualified to even read-review it, let alone apply and test.
(I am removing 3.3 because that is only listed for syntax feature requests that
cannot go into 3.2 because of its moratorium on such.)
I agree that this is a bug. For 3.1, the '8.6. array' doc says
class array.array(typecode[, initializer])
A new array whose items are restricted by typecode, and initialized from the
optional initializer value, which must be a list, object supporting the buffer
interface, or iterable over elements of the appropriate type.
I believe bytearray 'supports the buffer interface', whatever that means in
3.x. 'Readonly' is definitely *not* specified. Even if bytearray did not
qualify on that, instances are *definitely* iterable.
The doc goes on: If given a list or string, the initializer is passed to the
new array’s fromlist(), fromstring(), or fromunicode() method (see below) to
add initial items to the array. Otherwise, the iterable initializer is passed
to the extend() method.
For 3.x, this is pretty bad as it is copied without change from 2.x and uses
'string' in the 2.x sense. It seems to me that fromstring/fromunicode should
have been renamed to frombytes/fromstring with a 2to3 fixer. Too late now, I
suppose. However, if indeed 'fromstring' is intended to mean 'from bytes only
and not bytearrays', then bytearrays should be passed on to .extend, which
accepts them as well as anything else.
from array import array
a=array('h')
a.extend(bytearray(b'xjxjx'))
a
array('h', [120, 106, 120, 106, 120])
The two possible fixes:
1. Extend .fromstring to accept bytearrays. Its current doc Appends items from
the string, interpreting the string as an array of machine values to me
supports doing that. This is what Thomas's patch does.
2. If that is rejected, then bytearray initializers should be passed to .extend.
Thomas: a complete patch needs to include a update to the array unit test
suite. This can then move to the patch review stage.
--
nosy: +tjreedy
stage: - unit test needed
versions: -Python 3.3
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8990
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Alexander Belopolsky belopol...@users.sourceforge.net added the comment: Thomas' patch does more than just allow bytearray. It allows any object that can present itself as a buffer with byte-size items. It is a bit unfortunate that such method will end up being called fromstring in 3.x.With string being unicode, this is hopelessly ambiguous. Is the string interpreted as array of bytes, UTF-8 style, 16 or 32-bit integers? It would be much better to have frombuffer, and even better extendfrombuffer method with this functionality. I am +1 on Terry's option 2. It may be possible to optimize .extend to detect objects that support buffer interface and bypass iterator protocol. Of course, when receiving array is not type 'b', extend and fromxyz are not the same, so maybe we should just rename fromunicode to fromstring and fromstring to frombytes? -- nosy: +belopolsky ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Terry J. Reedy tjre...@udel.edu added the comment: Whatever is done, I think a bytearray should be handled the same as bytes. It must be that they give the same result. In basic operations, I believe that bytearrays can *always* substitute for bytes. Bytes and bytearray objects contain single bytes – the former is immutable while the latter is a mutable sequence. For example: b'abc'.capitalize() b'Abc' bytearray(b'abc').capitalize() bytearray(b'Abc') This, to me, implies that .fromstring should accept bytearray (though probably not general buffer objects). In 2.x, I understand .fromstring to initialize an array from machine bytes read into a string, but not .fromunicode. This is *not* a text method, and the result may vary for 2 and 4-byte unicode builds. So I can see that 3.x needs .fromstring(bytes) but not .fromunicode(string). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
Alexander Belopolsky belopol...@users.sourceforge.net added the comment: This, to me, implies that .fromstring should accept bytearray (though probably not general buffer objects). In 2.x, I understand .fromstring to initialize an array from machine bytes read into a string, but not .fromunicode. This is *not* a text method, and the result may vary for 2 and 4-byte unicode builds. So I can see that 3.x needs .fromstring(bytes) but not .fromunicode(string). No, fromunicode is still needed. It is used to fill type 'u' array with unicode wide characters. In 3.x unicode type was renamed to str and str to bytes, so fromunicode should become fromstring (or maybe fromstr to avoid confusion) and fromstring should become frombytes. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8990 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8990] array constructor and array.fromstring should accept bytearray.
New submission from Thomas Jollans tho...@jollans.com:
Currently, the array constructor, if given a bytearray, detects this with
PyByteArray_Check, and hands it on to array_fromstring, which does not support
bytearray (by using s# with PyArg_ParseTuple) and raises TypeError.
array('h', bytearray(b'xyxyxyxyxyxyxyxyxy'))
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: must be bytes or read-only buffer, not bytearray
I see no reason to insist on read-only buffers. I'm attaching a patch that I
think fixes this.
--
components: Extension Modules, Library (Lib)
files: array.diff
keywords: patch
messages: 107744
nosy: tjollans
priority: normal
severity: normal
status: open
title: array constructor and array.fromstring should accept bytearray.
type: behavior
versions: Python 3.1, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file17658/array.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8990
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
