[issue17043] Invalid read in test_codecs

2013-02-09 Thread Serhiy Storchaka 

Changes by Serhiy Storchaka storch...@gmail.com:


--
resolution:  - fixed
stage: patch review - committed/rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17043
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17043] Invalid read in test_codecs

2013-02-07 Thread Roundup Robot 

Roundup Robot added the comment:

New changeset 498b54e0e856 by Serhiy Storchaka in branch '2.7':
Issue #17043: The unicode-internal decoder no longer read past the end of
http://hg.python.org/cpython/rev/498b54e0e856

New changeset 0f1c2e2b6bc2 by Serhiy Storchaka in branch '3.2':
Issue #17043: The unicode-internal decoder no longer read past the end of
http://hg.python.org/cpython/rev/0f1c2e2b6bc2

New changeset fec2976c8503 by Serhiy Storchaka in branch '3.3':
Issue #17043: The unicode-internal decoder no longer read past the end of
http://hg.python.org/cpython/rev/fec2976c8503

New changeset eb0370d4686c by Serhiy Storchaka in branch 'default':
Issue #17043: The unicode-internal decoder no longer read past the end of
http://hg.python.org/cpython/rev/eb0370d4686c

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17043
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17043] Invalid read in test_codecs

2013-02-05 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:

Ping.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17043
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17043] Invalid read in test_codecs

2013-01-31 Thread Serhiy Storchaka 

Changes by Serhiy Storchaka storch...@gmail.com:


--
assignee:  - serhiy.storchaka

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17043
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17043] Invalid read in test_codecs

2013-01-26 Thread Stefan Krah 

New submission from Stefan Krah:

Found this in test_codecs running under Valgrind (Python 3.3):


test_bug1251300 (test.test_codecs.UnicodeInternalTest) ... ==11511== Invalid 
read of size 1
==11511==at 0x44AF37: _PyUnicode_DecodeUnicodeInternal 
(unicodeobject.c:6133)
==11511==by 0x4DEB5C: unicode_internal_decode (_codecsmodule.c:251)
==11511==by 0x5093F6: PyObject_Call (abstract.c:2082)
==11511==by 0x47D7F2: PyEval_CallObjectWithKeywords (ceval.c:3942)
==11511==by 0x491C38: PyCodec_Decode (codecs.c:403)
==11511==by 0x459D7D: PyUnicode_Decode (unicodeobject.c:3129)
==11511==by 0x45A287: PyUnicode_FromEncodedObject (unicodeobject.c:3023)
==11511==by 0x519A45: bytes_decode (bytesobject.c:2320)
==11511==by 0x484AB8: PyEval_EvalFrameEx (ceval.c:4374)
==11511==by 0x485ACB: PyEval_EvalFrameEx (ceval.c:4150)
==11511==by 0x486779: PyEval_EvalCodeEx (ceval.c:3433)
==11511==by 0x4859CA: PyEval_EvalFrameEx (ceval.c:4160)
==11511==  Address 0x984a7e2 is 0 bytes after a block of size 34 alloc'd
==11511==at 0x4C27972: realloc (vg_replace_malloc.c:525)
==11511==by 0x51AC34: _PyBytes_Resize (bytesobject.c:2881)
==11511==by 0x51B1FA: PyBytes_FromObject (bytesobject.c:2732)
==11511==by 0x51C134: bytes_new (bytesobject.c:2594)
==11511==by 0x42A4E4: type_call (typeobject.c:723)
==11511==by 0x5093F6: PyObject_Call (abstract.c:2082)
==11511==by 0x4843D5: PyEval_EvalFrameEx (ceval.c:4282)
==11511==by 0x485ACB: PyEval_EvalFrameEx (ceval.c:4150)
==11511==by 0x486779: PyEval_EvalCodeEx (ceval.c:3433)
==11511==by 0x4859CA: PyEval_EvalFrameEx (ceval.c:4160)
==11511==by 0x486779: PyEval_EvalCodeEx (ceval.c:3433)
==11511==by 0x538EF8: function_call (funcobject.c:633)
==11511== 



_PyUnicode_DecodeUnicodeInternal (s=0x984a7e0 , size=value optimized out, 
errors=0x0)
at Objects/unicodeobject.c:6133
6133((char *) uch)[2] = s[2];




==




==11511== 
==11511== Debugger has detached.  Valgrind regains control.  We continue.
==11511== Invalid read of size 1
==11511==at 0x44AF3E: _PyUnicode_DecodeUnicodeInternal 
(unicodeobject.c:6134)
==11511==by 0x4DEB5C: unicode_internal_decode (_codecsmodule.c:251)
==11511==by 0x5093F6: PyObject_Call (abstract.c:2082)
==11511==by 0x47D7F2: PyEval_CallObjectWithKeywords (ceval.c:3942)
==11511==by 0x491C38: PyCodec_Decode (codecs.c:403)
==11511==by 0x459D7D: PyUnicode_Decode (unicodeobject.c:3129)
==11511==by 0x45A287: PyUnicode_FromEncodedObject (unicodeobject.c:3023)
==11511==by 0x519A45: bytes_decode (bytesobject.c:2320)
==11511==by 0x484AB8: PyEval_EvalFrameEx (ceval.c:4374)
==11511==by 0x485ACB: PyEval_EvalFrameEx (ceval.c:4150)
==11511==by 0x486779: PyEval_EvalCodeEx (ceval.c:3433)
==11511==by 0x4859CA: PyEval_EvalFrameEx (ceval.c:4160)
==11511==  Address 0x984a7e3 is 1 bytes after a block of size 34 alloc'd
==11511==at 0x4C27972: realloc (vg_replace_malloc.c:525)
==11511==by 0x51AC34: _PyBytes_Resize (bytesobject.c:2881)
==11511==by 0x51B1FA: PyBytes_FromObject (bytesobject.c:2732)
==11511==by 0x51C134: bytes_new (bytesobject.c:2594)
==11511==by 0x42A4E4: type_call (typeobject.c:723)
==11511==by 0x5093F6: PyObject_Call (abstract.c:2082)
==11511==by 0x4843D5: PyEval_EvalFrameEx (ceval.c:4282)
==11511==by 0x485ACB: PyEval_EvalFrameEx (ceval.c:4150)
==11511==by 0x486779: PyEval_EvalCodeEx (ceval.c:3433)
==11511==by 0x4859CA: PyEval_EvalFrameEx (ceval.c:4160)
==11511==by 0x486779: PyEval_EvalCodeEx (ceval.c:3433)
==11511==by 0x538EF8: function_call (funcobject.c:633)
==11511== 


Loaded symbols for /usr/lib/gconv/ISO8859-9.so
_PyUnicode_DecodeUnicodeInternal (s=0x8295790 , size=value optimized out, 
errors=0x0)
at Objects/unicodeobject.c:6134
6134((char *) uch)[3] = s[3];

--
messages: 180709
nosy: serhiy.storchaka, skrah
priority: normal
severity: normal
status: open
title: Invalid read in test_codecs
versions: Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17043
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17043] Invalid read in test_codecs

2013-01-26 Thread Stefan Krah 

Stefan Krah added the comment:

Same in test_codeccallbacks:

test_badhandlerresults (test.test_codeccallbacks.CodecCallbackTest) ... 
==11604== Invalid read of size 1
==11604==at 0x44AF37: _PyUnicode_DecodeUnicodeInternal 
(unicodeobject.c:6133)
==11604==by 0x4DEB5C: unicode_internal_decode (_codecsmodule.c:251)
==11604==by 0x5093F6: PyObject_Call (abstract.c:2082)
==11604==by 0x47D7F2: PyEval_CallObjectWithKeywords (ceval.c:3942)
==11604==by 0x491C38: PyCodec_Decode (codecs.c:403)
==11604==by 0x459D7D: PyUnicode_Decode (unicodeobject.c:3129)
==11604==by 0x45A287: PyUnicode_FromEncodedObject (unicodeobject.c:3023)
==11604==by 0x519A45: bytes_decode (bytesobject.c:2320)
==11604==by 0x484AB8: PyEval_EvalFrameEx (ceval.c:4374)
==11604==by 0x485ACB: PyEval_EvalFrameEx (ceval.c:4150)
==11604==by 0x486779: PyEval_EvalCodeEx (ceval.c:3433)
==11604==by 0x4859CA: PyEval_EvalFrameEx (ceval.c:4160)
==11604==  Address 0xfa1f8a2 is 0 bytes after a block of size 34 alloc'd
==11604==at 0x4C27972: realloc (vg_replace_malloc.c:525)
==11604==by 0x51AC34: _PyBytes_Resize (bytesobject.c:2881)
==11604==by 0x51C338: PyBytes_DecodeEscape (bytesobject.c:495)
==11604==by 0x56E871: ast_for_expr (ast.c:3837)
==11604==by 0x570562: ast_for_testlist (ast.c:1106)
==11604==by 0x56E859: ast_for_expr (ast.c:1881)
==11604==by 0x570562: ast_for_testlist (ast.c:1106)
==11604==by 0x56E859: ast_for_expr (ast.c:1881)
==11604==by 0x5715C4: ast_for_stmt (ast.c:3302)
==11604==by 0x5724F8: ast_for_suite (ast.c:3086)
==11604==by 0x5715E3: ast_for_stmt (ast.c:3305)
==11604==by 0x5724F8: ast_for_suite (ast.c:3086)

_PyUnicode_DecodeUnicodeInternal (s=0xfa1f8a0 , size=value optimized out, 
errors=
0xf652fa0 test.badhandler) at Objects/unicodeobject.c:6133
6133((char *) uch)[2] = s[2];



[...]
_PyUnicode_DecodeUnicodeInternal (s=0xfa1f8a0 , size=value optimized out, 
errors=
0xf652fa0 test.badhandler) at Objects/unicodeobject.c:6134
6134((char *) uch)[3] = s[3];

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17043
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17043] Invalid read in test_codecs

2013-01-26 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:

Here are patches for all 4 versions.

--
keywords: +patch
Added file: 
http://bugs.python.org/file28860/decodeunicodeinternal_overflow-2.7.patch
Added file: 
http://bugs.python.org/file28861/decodeunicodeinternal_overflow-3.2.patch
Added file: 
http://bugs.python.org/file28862/decodeunicodeinternal_overflow-3.3.patch
Added file: 
http://bugs.python.org/file28863/decodeunicodeinternal_overflow-3.4.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17043
___diff -r 523f309cf558 Objects/unicodeobject.c
--- a/Objects/unicodeobject.c   Sat Jan 26 13:31:44 2013 +0100
+++ b/Objects/unicodeobject.c   Sun Jan 27 00:05:19 2013 +0200
@@ -3399,37 +3399,34 @@
 end = s + size;
 
 while (s  end) {
+if (end-s  Py_UNICODE_SIZE) {
+endinpos = end-starts;
+reason = truncated input;
+goto error;
+}
 memcpy(p, s, sizeof(Py_UNICODE));
+#ifdef Py_UNICODE_WIDE
 /* We have to sanity check the raw data, otherwise doom looms for
some malformed UCS-4 data. */
-if (
-#ifdef Py_UNICODE_WIDE
-*p  unimax || *p  0 ||
+if (*p  unimax || *p  0) {
+endinpos = s - starts + Py_UNICODE_SIZE;
+reason = illegal code point ( 0x10);
+goto error;
+}
 #endif
-end-s  Py_UNICODE_SIZE
-)
-{
-startinpos = s - starts;
-if (end-s  Py_UNICODE_SIZE) {
-endinpos = end-starts;
-reason = truncated input;
-}
-else {
-endinpos = s - starts + Py_UNICODE_SIZE;
-reason = illegal code point ( 0x10);
-}
-outpos = p - PyUnicode_AS_UNICODE(v);
-if (unicode_decode_call_errorhandler(
-errors, errorHandler,
-unicode_internal, reason,
-starts, size, startinpos, endinpos, exc, s,
-v, outpos, p)) {
-goto onError;
-}
-}
-else {
-p++;
-s += Py_UNICODE_SIZE;
+p++;
+s += Py_UNICODE_SIZE;
+continue;
+
+  error:
+startinpos = s - starts;
+outpos = p - PyUnicode_AS_UNICODE(v);
+if (unicode_decode_call_errorhandler(
+errors, errorHandler,
+unicode_internal, reason,
+starts, size, startinpos, endinpos, exc, s,
+v, outpos, p)) {
+goto onError;
 }
 }
 
diff -r f7eda8165e6f Objects/unicodeobject.c
--- a/Objects/unicodeobject.c   Sat Jan 26 12:14:02 2013 +0200
+++ b/Objects/unicodeobject.c   Sat Jan 26 23:55:55 2013 +0200
@@ -4415,37 +4415,34 @@
 end = s + size;
 
 while (s  end) {
+if (end-s  Py_UNICODE_SIZE) {
+endinpos = end-starts;
+reason = truncated input;
+goto error;
+}
 memcpy(p, s, sizeof(Py_UNICODE));
+#ifdef Py_UNICODE_WIDE
 /* We have to sanity check the raw data, otherwise doom looms for
some malformed UCS-4 data. */
-if (
-#ifdef Py_UNICODE_WIDE
-*p  unimax || *p  0 ||
+if (*p  unimax || *p  0) {
+endinpos = s - starts + Py_UNICODE_SIZE;
+reason = illegal code point ( 0x10);
+goto error;
+}
 #endif
-end-s  Py_UNICODE_SIZE
-)
-{
-startinpos = s - starts;
-if (end-s  Py_UNICODE_SIZE) {
-endinpos = end-starts;
-reason = truncated input;
-}
-else {
-endinpos = s - starts + Py_UNICODE_SIZE;
-reason = illegal code point ( 0x10);
-}
-outpos = p - PyUnicode_AS_UNICODE(v);
-if (unicode_decode_call_errorhandler(
-errors, errorHandler,
-unicode_internal, reason,
-starts, end, startinpos, endinpos, exc, s,
-v, outpos, p)) {
-goto onError;
-}
-}
-else {
-p++;
-s += Py_UNICODE_SIZE;
+p++;
+s += Py_UNICODE_SIZE;
+continue;
+
+  error:
+startinpos = s - starts;
+outpos = p - PyUnicode_AS_UNICODE(v);
+if (unicode_decode_call_errorhandler(
+errors, errorHandler,
+unicode_internal, reason,
+starts, end, startinpos, endinpos, exc, s,
+v, outpos, p)) {
+goto onError;
 }
 }
 
diff -r 8c49dd8e4d22 Objects/unicodeobject.c
--- a/Objects/unicodeobject.c   Sat Jan 26 18:57:19 2013 +0100
+++ b/Objects/unicodeobject.c   Sat Jan 26 23:50:50 2013 +0200
@@ -6125,6 +6125,11 @@
 while (s  end) {
 Py_UNICODE uch;

[issue17043] Invalid read in test_codecs

2013-01-26 Thread Serhiy Storchaka 

Changes by Serhiy Storchaka storch...@gmail.com:


--
components: +Interpreter Core, Unicode
nosy: +ezio.melotti
stage:  - patch review
type:  - behavior
versions: +Python 2.7, Python 3.2, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17043
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com