STINNER Victor added the comment:
It looks like PyUnicode_FromUnicode() should accept invalid UTF-16 surrogates
because the array module indirectly relies on that:
On Windows (16-bit wchar_t/Py_UNICODE), len(array.array('u', '\U0010')) is
2 and array.array('u', '\U0010')[0] is
Roundup Robot added the comment:
New changeset 1fd165883a65 by Victor Stinner in branch '3.3':
Issue #17223: the test is specific to 32-bit wchar_t type
http://hg.python.org/cpython/rev/1fd165883a65
New changeset 42970cbfc982 by Victor Stinner in branch 'default':
(Merge 3.3) Issue #17223: the
STINNER Victor added the comment:
The test should now pass on Windows.
--
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17223
___
Ezio Melotti added the comment:
Tests are still failing on Windows:
http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/1558/steps/test/logs/stdio
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17223
STINNER Victor added the comment:
ezio.melotti: Tests are still failing on Windows
Oh, I read the PyUnicode_FromUnicode() twice and there is a bug :-( With 16-bit
wchar_t type (on Windows), find_maxchar_surrogates() doesn't fail if the
wchar_* string contains in invalid surrogate pair.
Roundup Robot added the comment:
New changeset 15190138d3f3 by Victor Stinner in branch 'default':
Issue #17223: Add another test to check that _PyUnicode_Ready() rejects
http://hg.python.org/cpython/rev/15190138d3f3
New changeset b9f7b1bf36aa by Victor Stinner in branch 'default':
Issue
STINNER Victor added the comment:
Changeset b9f7b1bf36aa should fix the test on Windows. My Windows VM is dead, I
cannot test myself. If the fix works, it must be backported in Python 3.3.
--
___
Python tracker rep...@bugs.python.org
STINNER Victor added the comment:
Changeset b9f7b1bf36aa should fix the test on Windows.
Oh, many tests are failing because of this change, so I reverted it.
==
ERROR: test_surrogatepass_handler (test.test_codecs.CP65001Test)
Richard Oudkerk added the comment:
The new test seems to be reliably failing on Windows:
==
FAIL: test_issue17223 (__main__.UnicodeTest)
--
Traceback (most
STINNER Victor added the comment:
On Windows, test_array can use an invalid surrogate pair to test this
issue: b'\xff\xdf\x61\x00' for example.
I don't know how to easily check the size of wchar_t.
ctypes.sizeof(ctypes.c_wchar) can be used, but ctypes is not always
available. sys.unicode is now
Roundup Robot added the comment:
New changeset 66e9d0185b0f by Victor Stinner in branch '3.3':
Issue #17223: Fix test_array on Windows (16-bit wchar_t/Py_UNICODE)
http://hg.python.org/cpython/rev/66e9d0185b0f
New changeset 5aaf6bc1d502 by Victor Stinner in branch 'default':
(Merge 3.3) Issue
STINNER Victor added the comment:
It was discussed to add new formats for UCS1, UCS2 and UCS4 formats to the
array module, but nobody implemented the idea. The u format is kept unchanged
(use Py_UNICODE / wchar_t) for backward compatibility with Python 3.2.
See also issue #13072 for this
STINNER Victor added the comment:
If the problem is that PyUnicode_FromUnicode() rejects character outside range
[U+; U+10], it would be better to use the byte string '\xff' *
sizeof_PY_UNICODE. U+66647361 may become valid in a future version of Unicode,
I don't thing that U+
Ezio Melotti added the comment:
If the problem is that PyUnicode_FromUnicode() rejects character
outside range [U+; U+10],
But this used to return two valid characters:
str(array('u', b'asdf'))
array('u', '獡晤')
so I think it still should -- unless the operation was already
Ezio Melotti added the comment:
We discussed this on IRC, and apparently the seemingly valid result I got on
3.2 was because I had a narrow build. On a wide 3.2 build I get:
str(array('u', b'asdf'))
array('u', '\\U66647361')
Since 3.3+ behaves like a wide build and since \U66647361 is not
Roundup Robot added the comment:
New changeset c354afedb866 by Victor Stinner in branch '3.3':
Issue #17223: Fix PyUnicode_FromUnicode() for string of 1 character outside
http://hg.python.org/cpython/rev/c354afedb866
New changeset a4295ab52427 by Victor Stinner in branch 'default':
(Merge 3.3)
STINNER Victor added the comment:
I think this should be updated to work with the PEP 393 implementation,
rather than raising an error.
It was discussed to add new formats for UCS1, UCS2 and UCS4 formats to the
array module, but nobody implemented the idea. The u format is kept unchanged
Roundup Robot added the comment:
New changeset ebeed44702ec by Victor Stinner in branch '3.3':
Issue #17223: array module: Fix a crasher when converting an array containing
http://hg.python.org/cpython/rev/ebeed44702ec
New changeset 381de621ff6a by Victor Stinner in branch 'default':
(Merge
Ezio Melotti added the comment:
Even if deprecated it should continue to work (if possible).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17223
___
Manuel Jacob added the comment:
http://docs.python.org/3/library/array.html states that the 'u' type code is
deprecated together with the rest of the Py_UNICODE API (which includes
PyUnicode_FromUnicode), so keeping this using PyUnicode_FromUnicode should be
fine.
--
Ezio Melotti added the comment:
Shouldn't this still work on 3.3/3.4?
In Modules/arraymodule.c:1562, in the array_tounicode function, there is:
return PyUnicode_FromUnicode((Py_UNICODE *) self-ob_item, Py_SIZE(self));
I think this should be updated to work with the PEP 393 implementation,
Changes by Jesús Cea Avión j...@jcea.es:
--
nosy: +jcea
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17223
___
___
Python-bugs-list mailing list
New submission from Manuel Jacob:
from array import array
str(array('u', b'asdf'))
[1]19411 segmentation fault (core dumped) python
This error occures with Python 3.3 and hg tip but not with Python 3.2.
--
components: Library (Lib), Unicode
messages: 182291
nosy: ezio.melotti,
Manuel Jacob added the comment:
The attached patch fixes the crash.
Output:
from array import array
str(array('u', b'asdf'))
Traceback (most recent call last):
File stdin, line 1, in module
ValueError: character U+66647361 is not in range [U+; U+10]
--
keywords: +patch
Changes by Ezio Melotti ezio.melo...@gmail.com:
--
nosy: +haypo
stage: - patch review
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17223
___
___
Ezio Melotti added the comment:
Thanks for the report and the patch.
Could you also include a test for this?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17223
___
Manuel Jacob added the comment:
I've attached a new patch with a test that segfaults on Python 3.3 and passes
on hg tip with the patch applied.
--
Added file: http://bugs.python.org/file29110/issue17223_with_test.diff
___
Python tracker
Changes by STINNER Victor victor.stin...@gmail.com:
--
versions: +Python 2.7, Python 3.2
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17223
___
STINNER Victor added the comment:
issue17223_with_test.diff looks good to me (we may just drop {...} around
return NULL).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17223
___
29 matches
Mail list logo
