[issue24462] bytearray.find Buffer Over-read

2015-06-22 Thread DmitryJ

DmitryJ added the comment:

Attached please find a patch against the 2.7 branch. CPython built with the 
patch passes the tests from the test suite. Unfortunately, as there is not much 
control over memory allocation, there is no 100% reliable test case that would 
allow for reproducing the reported issue.

Some notes on the patch.

I have studied possible ways of fixing the issue narrowing them to two options; 
the approaches considered were:
a. Patch bytearray methods so they use stringlib's functions with respect to 
the corner case of out-of-bounds access on m = n.
b. Patch fastsearch() avoiding the out-of-bounds access on m = n completely.

Of these two, approach a is less invasive as changes, in theory, would be 
contained in bytearray() code only and should not affect any other code. 
Approach b fixes all possible cases, but affects other code not related to 
bytearray.

Upon closer studying of both bytearray and stringlib code, I discovered that it 
might be impossible to patch bytearray code only as stringlib contains a few 
methods that make use of the affected fastsearch() function, see e.g. 
stringlib_partition() as used in bytearray_partition(). If the approach of 
fixing bytearray specific code only would be chosen, I have to incorporate at 
least some of code following the fastsearch() call in stringlib_partition(). 
Similar considerations apply to other bytearray methods that make use of 
stringlib; the amount of code duplication varies. The end result is, I chose to 
patch fastsearch() instead.

Performance wise, the change incurs a small penalty due to one extra branch 
when m != n and brings considerable gain in (potentially rare) case when m = n.

I would appreciate if someone could test and review the patch.

NB. I stand corrected for the comment in msg245457 -- there is a note I missed 
in the C code. My sincere apologies to the author.

--
keywords: +patch
Added file: http://bugs.python.org/file39772/issue24462-2.7.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24462
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24481] hotshot pack_string Heap Buffer Overflow

2015-06-22 Thread DmitryJ

Changes by DmitryJ ga...@tut.by:


--
nosy: +dev_zzo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24481
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24462] bytearray.find Buffer Over-read

2015-06-22 Thread DmitryJ

DmitryJ added the comment:

I am preparing a patch for this issue, then.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24462
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24467] bytearray pop and remove Buffer Over-read

2015-06-23 Thread DmitryJ

Changes by DmitryJ ga...@tut.by:


Added file: http://bugs.python.org/file39781/issue24467-3.2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24467
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24467] bytearray pop and remove Buffer Over-read

2015-06-23 Thread DmitryJ

DmitryJ added the comment:

Attached is a patch that fixes the reported issue.

Since there are no visible side effects in Python, I could not write a test for 
this.

--
keywords: +patch
Added file: http://bugs.python.org/file39780/issue24467-2.7.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24467
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24467] bytearray pop and remove Buffer Over-read

2015-06-23 Thread DmitryJ

Changes by DmitryJ ga...@tut.by:


Added file: http://bugs.python.org/file39783/issue24467-3.4.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24467
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24467] bytearray pop and remove Buffer Over-read

2015-06-23 Thread DmitryJ

Changes by DmitryJ ga...@tut.by:


Added file: http://bugs.python.org/file39784/issue24467-3.5.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24467
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24467] bytearray pop and remove Buffer Over-read

2015-06-23 Thread DmitryJ

Changes by DmitryJ ga...@tut.by:


Added file: http://bugs.python.org/file39782/issue24467-3.3.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24467
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24467] bytearray pop and remove Buffer Over-read

2015-06-23 Thread DmitryJ

DmitryJ added the comment:

Offending code in 2.7:

https://hg.python.org/cpython/file/20c9290a5de4/Objects/bytearrayobject.c#l2381
https://hg.python.org/cpython/file/20c9290a5de4/Objects/bytearrayobject.c#l2412

Let n = 16, where = 0; memmove() then attempts to copy (n - where) = 16 bytes 
where it should have copied 15, since we drop one. This appears to be a typical 
case of off-by-one. Changing (n - where) to (n - where - 1) should fix the 
issue. This underfows when (where + 1)  n, but this case is guarded against in 
bytearray_pop() and cannot occur in bytearray_remove().

The exact same memmove() invocation code is found in all 3.x branches as well.

--
nosy: +dev_zzo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24467
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24462] bytearray.find Buffer Over-read

2015-06-18 Thread DmitryJ

DmitryJ added the comment:

Quick analysis tells this can be attributed to the following code (in 2.7):

https://hg.python.org/cpython/file/a8e24d776e99/Objects/stringlib/fastsearch.h#l110
https://hg.python.org/cpython/file/a8e24d776e99/Objects/stringlib/fastsearch.h#l116

Suppose i = 0, then s[i+m] causes OOB access when m=n. Note only one iteration 
is possible in case of m=n due to loop condition of i = (w = n-m = 0). 
Theoretically, one can try disclosing one adjacent byte, but more likely 
results are nothing (or potentially invalid match result) or a potential crash 
in an unlucky case of s[m] hitting an unmapped page.

The same code lives in 3.2 (and likely any prior 3.x release), and 3.3 seems to 
be affected as well. 3.4 code has a modified version, but has the same problem 
(ss = s + m - 1; if (!STRINGLIB_BLOOM(mask, ss[i+1])) ...).

--
nosy: +dev_zzo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24462
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24462] bytearray.find Buffer Over-read

2015-06-18 Thread DmitryJ

DmitryJ added the comment:

From the author's page at http://effbot.org/zone/stringlib.htm

Note that the above Python code may access s[n], which would result in an 
IndexError exception. For the CPython implementation, this is not really a 
problem, since CPython adds trailing NULL entries to both 8-bit and Unicode 
strings.

Apparently, this flaw was known to the author, but was not documented in C code.

A possible quick-and-dirty solution is to treat m=n as a special case and 
resort to memcmp() or somesuch as there is no actual need to perform multiple 
match tries. This should fix things for bytearray and str in case str's 
implementation changes from appending a trailing NUL.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24462
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24467] bytearray pop and remove Buffer Over-read

2015-06-28 Thread DmitryJ

DmitryJ added the comment:

If this is the case, then issue24462 should be fixed by this patch as well.

I'm sorry about missing the root cause here.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24467
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com