Bugs item #1526585, was opened at 2006-07-21 10:18
Message generated for change (Comment added) made by nnorwitz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1526585&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 2.5
Status: Open
Resolution: Accepted
Priority: 6
Submitted By: Jp Calderone (kuran)
Assigned to: Armin Rigo (arigo)
Summary: Concatenation on a long string breaks
Initial Comment:
Consider this transcript:
[EMAIL PROTECTED]:~/Projects/python/trunk$ ./python
Python 2.5b2 (trunk:50698, Jul 18 2006, 10:08:36)
[GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for
more information.
>>> x = 'x' * (2 ** 31 - 1)
>>> x = x + 'x'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
SystemError: Objects/stringobject.c:4103: bad argument
to internal function
>>> len(x)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined
>>>
I would expect some exception other than SystemError
and for the locals namespace to not become corrupted.
----------------------------------------------------------------------
>Comment By: Neal Norwitz (nnorwitz)
Date: 2006-07-26 08:50
Message:
Logged In: YES
user_id=33168
You're correct that bigmem is primarily for testing
int/Py_ssize_t. But it doesn't have to be. It has support
for machines with largish amounts of memory (and limiting
test runs). I didn't know where else to put such a test. I
agree that this bug would only occur on 32-bit platforms.
Most machines can't run it, so about the only other option I
can think of would be to put it in it's own file and add a
-u option. That seemed like even more work.
I'm not tied to bigmem at all, but it would be nice to have
a test for this somewhere. I'm sure there are a bunch of
other places we have this sort of overflow and it would be
good to test them somewhere. Do whatever you think is best.
----------------------------------------------------------------------
Comment By: Armin Rigo (arigo)
Date: 2006-07-26 02:01
Message:
Logged In: YES
user_id=4771
I'm unsure about how the bigmem tests should be used.
I think that I am not supposed to set a >2G limit on
a 32-bit machine, right? When I do, I get 9 failures:
8 OverflowErrors and a stranger AssertionError in
test_hash. I think that these tests are meant to
test the int/Py_ssize_t difference on 64-bit
machines instead. The bug this tracker is about
only shows up on 32-bit machines.
My concern is that if we add a test for the current
bug in test_bigmem, then with a memory limit < 2GB
the test is essentially skipped, and with a memory
limit > 2GB then 9 other tests fail anyway.
----------------------------------------------------------------------
Comment By: Neal Norwitz (nnorwitz)
Date: 2006-07-25 21:19
Message:
Logged In: YES
user_id=33168
Armin, can you check in your patch and backport? Also a
news entry and a test in test_bigmem would be great.
Thanks. If not let me know (assign to me) and I'll finish
this up.
This fix should be backported as well.
----------------------------------------------------------------------
Comment By: Jp Calderone (kuran)
Date: 2006-07-25 13:06
Message:
Logged In: YES
user_id=366566
Tested Armin's patch, looks like it addresses the issue.
One thing I noticed though, ('x' * (2 ** 32 - 2) + 'x')
fails the same way ('x' * (2 ** 32 - 1) + 'x'). Don't
really understand why, just thought I'd mention it.
----------------------------------------------------------------------
Comment By: Armin Rigo (arigo)
Date: 2006-07-25 11:25
Message:
Logged In: YES
user_id=4771
The check should be done before the variable is removed
from the namespace, so that 'x' doesn't just disappear.
Attached a patch that does this. Also, let's produce
an exception consistent with what PyString_Concat()
raises in the same case, which is an OverflowError
instead of a MemoryError.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2006-07-24 20:57
Message:
Logged In: YES
user_id=31435
[Neal]
> Tim since we know both lens are >= 0, is this test
> sufficient for verifying overflow:
>
> Py_ssize_t new_len = v_len + w_len;
> if (new_len < 0) {
Right! That's all the new checking needed in
string_concatenate().
----------------------------------------------------------------------
Comment By: Neal Norwitz (nnorwitz)
Date: 2006-07-24 20:46
Message:
Logged In: YES
user_id=33168
Attached a patch against 2.5. The patch should work against
2.4 too, but you will need to change all Py_ssize_t to int.
Tim since we know both lens are >= 0, is this test
sufficient for verifying overflow:
Py_ssize_t new_len = v_len + w_len;
if (new_len < 0) {
Jp, can you test the patch?
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2006-07-22 07:11
Message:
Logged In: YES
user_id=31435
Part of the problem appears to be that ceval.c's
string_concatenate() doesn't check for overflow in the
second argument here:
if (_PyString_Resize(&v, v_len + w_len) != 0)
The Windows malloc on my box returns NULL (and so Python
raises MemoryError) on the initial:
x = 'x' * (2 ** 31 - 1)
attempt, so I never get that far. I'm afraid this is
muddier in strange ways on Linux because the Linux malloc()
is pathologically optimistic (can return a non-NULL value
pointing at "memory" that can't actually be used for anything).
----------------------------------------------------------------------
Comment By: Michael Hudson (mwh)
Date: 2006-07-22 02:00
Message:
Logged In: YES
user_id=6656
Confirmed with 2.4. Ouch.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1526585&group_id=5470
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
