Boya Sun boya@case.edu added the comment:
I am still a little bit confused. Can you explain a little more in
detail? What is the difference between the suspicious code and the ones
that are fixed?
--
___
Python tracker rep...@bugs.python.org
Martin v. Löwis mar...@v.loewis.de added the comment:
A problem can only occur if you preserve a pointer to the buffer,
and the object gets a chance to change its buffer underneath. This
can happen when there are user-defined callback, and when other
threads can get control. In the cases being
Boya Sun boya@case.edu added the comment:
Although the bug is fixed, the following three code segments seems
suspicious in _codecsmodule.c in the latest revision 74624, and they are
similar to the bug described here:
(1)
escape_decode(PyObject *self,
PyObject *args)
{
...
Amaury Forgeot d'Arc amaur...@gmail.com added the comment:
In theory, yes. But it happens that the GIL is not released before the
buffer is used.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3139
Travis Oliphant [EMAIL PROTECTED] added the comment:
I'm sorry that I was unavailable for comment during July and August as
it looks like a lot of decisions were made that have changed the
semantics a bit. I'm still trying to figure out why the decisions were
made that were made.
I get the
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Hi Travis,
Glad you're back!
I'm not convinced that Py_buffer should have grown a link to an object.
I think this is a shortcut solution due to misuse of the protocol that
may have unfortunate consequences.
What consequences are you
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Le samedi 16 août 2008 à 05:50 +, Ismail Donmez a écrit :
This seems to break test_unicode on MacOSX 10.5.4,
test_unicode
python(80320,0xa0659fa0) malloc: *** mmap(size=2147483648) failed (error
code=12)
*** error: can't allocate
Ismail Donmez [EMAIL PROTECTED] added the comment:
Can you run Lib/test/test_unicode.py directly to know which test
precisely crashes?
The latest py3k branch is fine now and the test passes.
___
Python tracker [EMAIL PROTECTED]
Ismail Donmez [EMAIL PROTECTED] added the comment:
This seems to break test_unicode on MacOSX 10.5.4,
test_unicode
python(80320,0xa0659fa0) malloc: *** mmap(size=2147483648) failed (error
code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Martin v. Löwis [EMAIL PROTECTED] added the comment:
I'm a bit confused. In the PyBuffer_Release implementation you
committed, there is no DECREF at all.
Oops, I meant to make the reference own by Py_buffer, but actually
forgot to implement that. Fixed in r65677, r65678.
Now, when I try to
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Le jeudi 14 août 2008 à 16:13 +, Martin v. Löwis a écrit :
Now, when I try to merge that into the 3k branch, test_unicode terribly
leaks memory :-( It's really frustrating.
If you don't have the time to work on it anymore, you can post
Martin v. Löwis [EMAIL PROTECTED] added the comment:
The patch is really trivial, and attached.
Added file: http://bugs.python.org/file4/refcount.diff
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Le jeudi 14 août 2008 à 18:52 +, Martin v. Löwis a écrit :
The patch is really trivial, and attached.
Added file: http://bugs.python.org/file4/refcount.diff
By the way, even without that patch, there is a memory leak:
Python
Martin v. Löwis [EMAIL PROTECTED] added the comment:
By the way, even without that patch, there is a memory leak:
With the patch, this memory leak goes away.
Regards,
Martin
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Le jeudi 14 août 2008 à 19:06 +, Martin v. Löwis a écrit :
Martin v. Löwis [EMAIL PROTECTED] added the comment:
By the way, even without that patch, there is a memory leak:
With the patch, this memory leak goes away.
But now:
30
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Sorry, the roundup e-mail interface ate some lines of code:
b = b''
sys.getrefcount(b)
30
m = memoryview(b)
sys.getrefcount(b)
32
del m
sys.getrefcount(b)
31
It doesn't happen with bytearrays, so I suspect it's that you only
DECREF when
Martin v. Löwis [EMAIL PROTECTED] added the comment:
It doesn't happen with bytearrays, so I suspect it's that you only
DECREF when releasebuffer method exists:
Thanks, that was indeed the problem; this is now fixed in r65683 and r65685.
My initial observation that test_unicode leaks a lot
Changes by Martin v. Löwis [EMAIL PROTECTED]:
--
resolution: - fixed
status: open - closed
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
___
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Le mardi 12 août 2008 à 16:15 +, Martin v. Löwis a écrit :
I also started working on porting it to 3.0, but couldn't complete that
port yet - the memoryview object doesn't play nicely.
I've seen your recent merge and I don't know if you
Martin v. Löwis [EMAIL PROTECTED] added the comment:
I've seen your recent merge and I don't know if you have finished with
it.
Indeed, I'm done, r65654. Unless there are actual bugs in these patches
(in the sense that they don't fix the reported problem, or introduce new
bugs), I'd like to
Antoine Pitrou [EMAIL PROTECTED] added the comment:
In any case, the corresponding DECREF *does* exist: in memory_dealloc,
PyBuffer_Release is invoked, which releases the reference.
I'm a bit confused. In the PyBuffer_Release implementation you
committed, there is no DECREF at all.
By the
Martin v. Löwis [EMAIL PROTECTED] added the comment:
I have now committed the patch to 2.6 as r65654, adding changes for the
bz2module.
I also decided to make the Py_buffer structure own its reference, as I
was running out of arguments why not to. In the process, I removed
Martin v. Löwis [EMAIL PROTECTED] added the comment:
I also started working on porting it to 3.0, but couldn't complete that
port yet - the memoryview object doesn't play nicely.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
Antoine Pitrou [EMAIL PROTECTED] added the comment:
The last beta is arriving soon. We need to go ahead on this, or retarget
it for 2.7/3.1.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
Martin v. Löwis [EMAIL PROTECTED] added the comment:
The last beta is arriving soon. We need to go ahead on this, or retarget
it for 2.7/3.1.
I'll look into it tomorrow.
Regards,
Martin
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
Marc-Andre Lemburg [EMAIL PROTECTED] added the comment:
Two comments:
* I like the new *-getarg parameters, but it would be better to test
for '#' first since this is still by far the most used getarg
parameter.
* Antoine, I think your codecs.c patch has a glitch:
+decoded =
Antoine Pitrou [EMAIL PROTECTED] added the comment:
You've missed the preceding line that says:
+consumed = pbuf.len;
If final is NULL, consumed isn't updated by the call to
PyUnicode_DecodeMBCSStateful and keeps its original value of pbuf.len.
(in all honesty, I didn't test under Windows
Marc-Andre Lemburg [EMAIL PROTECTED] added the comment:
On 2008-08-02 16:49, Antoine Pitrou wrote:
Antoine Pitrou [EMAIL PROTECTED] added the comment:
You've missed the preceding line that says:
+consumed = pbuf.len;
If final is NULL, consumed isn't updated by the call to
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Here is a new patch wrapping up Martin's patch with the following additions:
- getargs.c fixes
- codecs module fixes
- multiprocessing module fix
- fileobject readinto fix
- in bytearray deallocator, print out a SystemError if there are
Changes by Antoine Pitrou [EMAIL PROTECTED]:
Removed file: http://bugs.python.org/file11032/codecs.patch
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
Changes by Antoine Pitrou [EMAIL PROTECTED]:
Removed file: http://bugs.python.org/file11033/getargs.patch
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
Martin v. Löwis [EMAIL PROTECTED] added the comment:
Amaury, I think the issue of thread-safety of the io library should be
decoupled from this issue. I don't think it is release-critical that the
io library is thread-safe (and even if it is release-critical, the
problem should be tracked in a
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Selon Martin v. Löwis [EMAIL PROTECTED]:
Amaury, I think the issue of thread-safety of the io library should be
decoupled from this issue. I don't think it is release-critical that the
io library is thread-safe (and even if it is
Martin v. Löwis [EMAIL PROTECTED] added the comment:
I have now updated the patch to fix the socket bug, and the rejects
bytearray for t#.
As for making Py_buffer own a reference to the object: what should be
the semantics for PyObject_ReleaseBuffer? I see the following options:
- Drop
Changes by Martin v. Löwis [EMAIL PROTECTED]:
Removed file: http://bugs.python.org/file10969/s_star.diff
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Selon Martin v. Löwis [EMAIL PROTECTED]:
As for making Py_buffer own a reference to the object: what should be
the semantics for PyObject_ReleaseBuffer? I see the following options:
- Drop PyObject_ReleaseBuffer
- make it DECREF the
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Le vendredi 01 août 2008 à 17:53 +, Amaury Forgeot d'Arc a écrit :
There is a small issue with the patch: in the w# format handler,
bf_getwritebuffer(arg, 0, res) is wrong. The third argument should be
res (there is a compilation warning
Changes by Giampaolo Rodola' [EMAIL PROTECTED]:
--
nosy: -giampaolo.rodola
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
___
Python-bugs-list
Martin v. Löwis [EMAIL PROTECTED] added the comment:
I don't know, is there supposed to be a semantic difference between
PyObject_ReleaseBuffer and PyBuffer_Release? If not, I'd say drop it.
There are existing callers of it which would need to be changed, perhaps
outside the core also; plus
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Martin, sorry for noticing this now but on python-dev you had proposed
the following:
« I propose that new codes s*, t*, w* are added, and that s#,t#,w#
refuses objects which implement a releasebuffer procedure
(alternatively, s# etc might be
Martin v. Löwis [EMAIL PROTECTED] added the comment:
I don't see any changes to s# and t# in your patch. Do you plan to do it
later or do you think this part should be dropped?
It's implemented. When bf_releasebuffer is not NULL, convertbuffer will
complain string or read-only buffer.
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
Ok for s#.
But t# uses bf_getcharbuffer(), which does not seem to lock anything.
I wonder if this can lead to the same kind of problems, for example when
calling file_write with a binary file.
___
Martin v. Löwis [EMAIL PROTECTED] added the comment:
But t# uses bf_getcharbuffer(), which does not seem to lock anything.
Indeed. I think we can safely drop support for passing buffer objects
into t# which have getbuffer/releasebuffer, so the check for
releasebuffer being NULL should be added
Antoine Pitrou [EMAIL PROTECTED] added the comment:
The problem is that the fix for #3295 was committed in the py3k branch
(in r64751) rather thank on the trunk!
Once PyExc_BufferError is defined properly the crash disappears and
exceptions are printed instead.
Benjamin Peterson [EMAIL PROTECTED] added the comment:
Sorry, that was my oversight! I've backported the fix.
--
nosy: +benjamin.peterson
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
It's indeed better.
Now with when running my previous script I can see the exception ;-)
Exception in thread Thread-2:
Traceback (most recent call last):
File C:\dev\python\trunk1\lib\threading.py, line 523, in
__bootstrap_inner
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Le mercredi 30 juillet 2008 à 23:03 +, Amaury Forgeot d'Arc a
écrit :
Again, I think this is unfortunate for a simple script that prints from
several threads.
Yes, but it's an issue with the BufferedWriter implementation, since it
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
If it was rewritten to use a fixed-size bytearray
does such an object exist today?
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Le jeudi 31 juillet 2008 à 00:00 +, Amaury Forgeot d'Arc a écrit :
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
If it was rewritten to use a fixed-size bytearray
does such an object exist today?
Manually, yes :)
Just
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
With the patch the following script crashes the 2.6 interpreter on Windows:
import sys, io, threading
stdout2 = io.open(sys.stdout.fileno(), mode=w)
def f(i):
for i in range(10):
stdout2.write(unicode((x, i)) + '\n')
for x in
Martin v. Löwis [EMAIL PROTECTED] added the comment:
(1) are you sure it is safe not to INCREF the obj pointer in the
Py_buffer?
Yes, that's the semantics of the current buffer interface, and I cannot
see a flaw in it. When you call getbuffer, you certainly hold a
reference, and it is your
Antoine Pitrou [EMAIL PROTECTED] added the comment:
I haven't yet studied the patch in detail but I have a few questions:
(1) are you sure it is safe not to INCREF the obj pointer in the
Py_buffer? in the cases handled by your patch we still hold a reference
to the original args tuple and dict
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Travis, it would be really nice to have your input on this.
--
nosy: +teoliphant
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
Martin v. Löwis [EMAIL PROTECTED] added the comment:
Here is a patch adding the s* format, and changing files, sockets, and
fileio to use it. For bz2, the immediate effect is that you get a type
error (as an object providing bf_releasebuffer cannot be converted
through s#/w# anymore); it would
Changes by Barry A. Warsaw [EMAIL PROTECTED]:
--
priority: deferred blocker - release blocker
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
Barry A. Warsaw [EMAIL PROTECTED] added the comment:
Not blocking beta 2 because there's not enough time for the fix, but
this will definitely block beta 3.
--
nosy: +barry
priority: release blocker - deferred blocker
___
Python tracker [EMAIL
Martin v. Löwis [EMAIL PROTECTED] added the comment:
Fixing this in the methods themselves has the disadvantage that the
error reporting is not that good anymore.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
Changes by Ismail Donmez [EMAIL PROTECTED]:
--
nosy: +cartman
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
___
Python-bugs-list mailing list
Changes by Martin v. Löwis [EMAIL PROTECTED]:
--
priority: - release blocker
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
___
Antoine Pitrou [EMAIL PROTECTED] added the comment:
If I try to follow the chain the consequences:
- all PyArg_ParseTuple(s#) calls that release the GIL afterwards
should be re-written to use another API (which one I don't know exactly,
but hopefully the appropriate functions are already
Antoine Pitrou [EMAIL PROTECTED] added the comment:
By the way, here's a more reliable way to make it crash (on my Linux
machine):
import bz2, threading
bz2c = bz2.BZ2Compressor()
# Span at least a whole arena (256KB long)
junk_len = 512 * 1024
junk = ba * junk_len
buf = bytearray()
for x in
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Now I've just discovered there is the same problem with the
array.array() type (see following code).
import bz2, threading, array
bz2c = bz2.BZ2Compressor()
# Span at least a whole arena (256KB long)
junk_len = 512 * 1024
junk =
Martin v. Löwis [EMAIL PROTECTED] added the comment:
I believe the 2.6 s# processing works correctly; the error is in the
bytearray object. This either shouldn't support the buffer interface, or
it shouldn't reallocate the buffer after having returned a pointer to it.
For 3k, convertbuffer
Antoine Pitrou [EMAIL PROTECTED] added the comment:
For reference, here is a proof-of-concept patch which shows how to fix
the bytearray crasher above (it raises a BufferError instead).
Added file: http://bugs.python.org/file10822/bzcrash.py
___
Python
Changes by Antoine Pitrou [EMAIL PROTECTED]:
Removed file: http://bugs.python.org/file10822/bzcrash.py
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
Changes by Antoine Pitrou [EMAIL PROTECTED]:
Added file: http://bugs.python.org/file10823/bzcrash.patch
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
___
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
Probably, but this affects all PyArg_ParseTuple(s#) calls that release
the GIL afterwards. How many of them are there?
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3139
67 matches
Mail list logo
