[issue16335] Integer overflow in unicode-escape decoder

Changes by Serhiy Storchaka : -- status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://

[issue16335] Integer overflow in unicode-escape decoder

Serhiy Storchaka added the comment: The test passed in any case, but for different size it doesn't check that the bug is fixed. Due to the bug bytes b'\\N{SPACE...xxx'}' decoded as b'\\N{SPACE'}' if the number of x-es divisible by (UINT_MAX + 1). In this case unicode-escape decodin

[issue16335] Integer overflow in unicode-escape decoder

Antoine Pitrou added the comment: > Serhiy Storchaka added the comment: > > > Ah, yes, that's because you should have used `size` instead > > of `_testcapi.UINT_MAX` inside the test. > > This test has sense only if size % (_testcapi.UINT_MAX + 1) == 0. Why so? Does it fail otherwise?

[issue16335] Integer overflow in unicode-escape decoder

Serhiy Storchaka added the comment: Due to PEP 393 Python 3.3+ requires less memory for temporary output buffer. As for difference between ">" and ">=", the meaning of -M parameter a little differs in 2.7 and 3.x -- in 2.7 some overhead (5MiB) counted up. -- __

[issue16335] Integer overflow in unicode-escape decoder

Stefan Krah added the comment: The test is fixed here, thanks. The limits appear to be different in 2.7 and 3.4: In 2.7 the bigmem tests are executed with -M x > 16G, in 3.4 with -M x >= 12G. I don't know if that's deliberate, just mentioning it. -- __

[issue16335] Integer overflow in unicode-escape decoder

Serhiy Storchaka added the comment: > Ah, yes, that's because you should have used `size` instead > of `_testcapi.UINT_MAX` inside the test. This test has sense only if size % (_testcapi.UINT_MAX + 1) == 0. -- ___ Python tracker

[issue16335] Integer overflow in unicode-escape decoder

Antoine Pitrou added the comment: > Bigmem test in 2.7 ran even if -M option is not specified and this > causes the memory error. Ah, yes, that's because you should have used `size` instead of `_testcapi.UINT_MAX` inside the test. -- ___ Python track

[issue16335] Integer overflow in unicode-escape decoder

Serhiy Storchaka added the comment: Bigmem test in 2.7 ran even if -M option is not specified and this causes the memory error. But memuse parameter should be increased (I tested with smaller sizes and found that 1 + 4 // len(u'\U0001') is not enough, but 2 + 4 // len(u'\U0001') is eno

[issue16335] Integer overflow in unicode-escape decoder

Roundup Robot added the comment: New changeset fc21f8e83062 by Serhiy Storchaka in branch '2.7': Don't run the test for issue #16335 when -M is not specified. http://hg.python.org/cpython/rev/fc21f8e83062 New changeset e3d1b68d34e3 by Serhiy Storchaka in branch '3.2': Increase the memory limit i

[issue16335] Integer overflow in unicode-escape decoder

Stefan Krah added the comment: I just ran the 2.7 tests while dealing with another issue, and I'm getting a memory error or excessive swapping in test_ucn: The statement x = b'\\N{SPACE' + b'x' * int(_testcapi.UINT_MAX + 1) + b'}' uses over 8GB on my system, so I think that minsize=_test

[issue16335] Integer overflow in unicode-escape decoder

Roundup Robot added the comment: New changeset ec3a35ab3659 by Serhiy Storchaka in branch '2.7': Add bigmemtest decorator to test of issue #16335. http://hg.python.org/cpython/rev/ec3a35ab3659 New changeset 6e0c3e4136b1 by Serhiy Storchaka in branch '3.2': Add bigmemtest decorator to test of iss

[issue16335] Integer overflow in unicode-escape decoder

Roundup Robot added the comment: New changeset f84a6c89ccbc by Serhiy Storchaka in branch '3.2': Fix memory error in test_ucn. http://hg.python.org/cpython/rev/f84a6c89ccbc New changeset 7c2aae472b27 by Serhiy Storchaka in branch '3.3': Fix memory error in test_ucn. http://hg.python.org/cpython/

[issue16335] Integer overflow in unicode-escape decoder

Serhiy Storchaka added the comment: I rewrote the test in EAFP style. -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker ___

[issue16335] Integer overflow in unicode-escape decoder

Roundup Robot added the comment: New changeset f4d30d1a529e by Serhiy Storchaka in branch '2.7': Issue #16335: Fix integer overflow in unicode-escape decoder. http://hg.python.org/cpython/rev/f4d30d1a529e -- ___ Python tracker

[issue16335] Integer overflow in unicode-escape decoder

Roundup Robot added the comment: New changeset 7625866f8127 by Serhiy Storchaka in branch '3.2': Issue #16335: Fix integer overflow in unicode-escape decoder. http://hg.python.org/cpython/rev/7625866f8127 New changeset 494d341e9143 by Serhiy Storchaka in branch '3.3': Issue #16335: Fix integer o

[issue16335] Integer overflow in unicode-escape decoder

Changes by Serhiy Storchaka : -- assignee: -> serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue16335] Integer overflow in unicode-escape decoder

Serhiy Storchaka added the comment: I add tests. Victor ran the test and got MemoryError. This means that I incorrectly calculated the minimal memory size for bigmem. This is unacceptable, the test should skip or pass. Only someone with enough memory for test can measure a minimal memory requi

[issue16335] Integer overflow in unicode-escape decoder

Terry J. Reedy added the comment: Serhiy, please be more specific as to 'measure' and 'how much' for what effect. I ran two examples, one ran (with error), the other raised. -- ___ Python tracker _

[issue16335] Integer overflow in unicode-escape decoder

Serhiy Storchaka added the comment: Terry, can you measure how much memory tests really needed (3.2 and 3.3 should want different quantity)? Looks as I wrong in my assumptions. -- ___ Python tracker __

[issue16335] Integer overflow in unicode-escape decoder

Serhiy Storchaka added the comment: Here are patches for different Python versions. Test added. Victor, now you can try it on 12GB. Unfortunately, I can't run the tests. -- Added file: http://bugs.python.org/file27933/decode_unicode_escape_overflow-3.3.patch Added file: http://bugs.p

[issue16335] Integer overflow in unicode-escape decoder

Changes by Serhiy Storchaka : Removed file: http://bugs.python.org/file27741/decode_unicode_escape_overflow.patch ___ Python tracker ___ ___

[issue16335] Integer overflow in unicode-escape decoder

Ezio Melotti added the comment: Tests would be good. You could use test.support.bigmemtest. -- ___ Python tracker ___ ___ Python-bugs

[issue16335] Integer overflow in unicode-escape decoder

Terry J. Reedy added the comment: >>> x=(b'\\N{WHITE SMILING FACE' + b'x' * 2**16 + b'}') >>> y=x.decode('unicode-escape') Traceback (most recent call last): File "", line 1, in y=x.decode('unicode-escape') UnicodeDecodeError: 'unicodeescape' codec can't decode bytes in position 0-65557:

[issue16335] Integer overflow in unicode-escape decoder

Serhiy Storchaka added the comment: Wow! Do we need a test for this case? -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue16335] Integer overflow in unicode-escape decoder

Terry J. Reedy added the comment: I don't know what to make of this, but... Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32 Win7 pro, 24 gb mem >>> x=(b'\\N{WHITE SMILING FACE' + b'x' * 2**32 + b'}') >>> len(x) 4294967318 >>> y=x.decode('unicode-es

[issue16335] Integer overflow in unicode-escape decoder

Serhiy Storchaka added the comment: > MemoryError It's because you need >4GB for source bytes + at least >8GB (>12GB on Windows) for temporary UCS2 string. -- ___ Python tracker __

[issue16335] Integer overflow in unicode-escape decoder

STINNER Victor added the comment: I have 12 GB of RAM. Let's test. $ ./python Python 3.4.0a0 (default:8573a86c11b5+, Oct 31 2012, 22:17:00) [GCC 4.6.3 20120306 (Red Hat 4.6.3-2)] on linux >>> x=(b'\\N{WHITE SMILING FACE' + b'x' * 2**32 + b'}') >>> len(x) 4294967318 >>> y=x.decode('unicode-esca

[issue16335] Integer overflow in unicode-escape decoder

Serhiy Storchaka added the comment: (b'\\N{WHITE SMILING FACE' + b'x' * 2**32 + '}').decode('unicode-escape') may pass on platform with 32-bit int and more than 32-bit size_t if there is enough memory. I don't have so much memory. -- ___ Python tra

[issue16335] Integer overflow in unicode-escape decoder

STINNER Victor added the comment: If I understood correctly, (b'\\N{' + b'x' * (INT_MAX+1)) + '}').decode('unicode-decode') may crash? Did you try such string? -- ___ Python tracker ___

[issue16335] Integer overflow in unicode-escape decoder

New submission from Serhiy Storchaka: Size of parsed Unicode character name casted to int in unicode-escape decoder. This can cause integer overflow. -- components: Interpreter Core, Unicode files: decode_unicode_escape_overflow.patch keywords: patch messages: 173902 nosy: benjamin.pet