[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2013-10-19 Thread Kristján Valur Jónsson 

Kristján Valur Jónsson added the comment:

Perhaps in debug builds the memory apis should verify consistency and matching 
useage.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2013-10-19 Thread STINNER Victor 

STINNER Victor added the comment:

 Kristján Valur Jónsson added the comment:

 Perhaps in debug builds the memory apis should verify consistency and
matching useage.

Python does check usage of apis in debug mode. Memory allocation failure
are almost never checked. See my pyfailmalloc module for that.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2013-10-18 Thread Roundup Robot 

Roundup Robot added the comment:

New changeset 6c9050ad1afc by Victor Stinner in branch 'default':
Issue #16742: My fix on PyOS_StdioReadline() was incomplete, PyMem_FREE() was
http://hg.python.org/cpython/rev/6c9050ad1afc

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2013-10-10 Thread Roundup Robot 

Roundup Robot added the comment:

New changeset 98dbe677dfe7 by Victor Stinner in branch 'default':
Close #16742: Fix misuse of memory allocations in PyOS_Readline()
http://hg.python.org/cpython/rev/98dbe677dfe7

--
nosy: +python-dev
resolution:  - fixed
stage: needs patch - committed/rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2013-06-19 Thread STINNER Victor 

Changes by STINNER Victor victor.stin...@gmail.com:


--
dependencies: +API for setting the memory allocator used by Python

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2013-06-14 Thread STINNER Victor 

STINNER Victor added the comment:

Updated patch for the final API of #3329. Update also the documentation. 
PyOS_ReadlineFunctionPointer must now use PyMem_RawMalloc() or 
PyMem_RawRealloc(), instead of PyMem_Malloc() or PyMem_Realloc().

--
Added file: http://bugs.python.org/file30593/readline_gil-2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2013-06-13 Thread STINNER Victor 

STINNER Victor added the comment:

I just found the readline/GIL issue while working on #18203. I created #18205 
but then I found this issue. I just closed #18205 as a duplicate. Here is a 
patch for Python 3.4.

--

Copy of the initial message (msg191089):

The callback PyOS_ReadlineFunctionPointer (used to read a line from the 
standard input) must return a buffer allocated by PyMem_Malloc(), but 
PyOS_Readline() releases the GIL before calling PyOS_ReadlineFunctionPointer.

Simplified extract of PyOS_Readline():

Py_BEGIN_ALLOW_THREADS
if (!isatty (fileno (sys_stdin)) || !isatty (fileno (sys_stdout)))
rv = PyOS_StdioReadline (sys_stdin, sys_stdout, prompt);
else
rv = (*PyOS_ReadlineFunctionPointer)(sys_stdin, sys_stdout,
 prompt);
Py_END_ALLOW_THREADS

tok_nextc() calls PyOS_Readline() and calls PyMem_FREE() to release its result.

PyOS_ReadlineFunctionPointer should allocate memory using malloc(), not using 
PyMem_Malloc(). But PyOS_Readline() should copy the line into a buffer 
allocated by PyMem_Malloc() to keep backward compatibility.

See also issue #18203 and #3329.

--
keywords: +patch
nosy: +haypo
versions:  -Python 3.2
Added file: http://bugs.python.org/file30578/readline_gil.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2013-06-13 Thread STINNER Victor 

STINNER Victor added the comment:

See the following thread on python-dev, the root problem is that PyMem_Malloc() 
cannot be called with the GIL held. This is a bug in my opinion, and it should 
be fixed.
http://mail.python.org/pipermail/python-dev/2013-June/126822.html

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2013-05-06 Thread Kristján Valur Jónsson 

Kristján Valur Jónsson added the comment:

My quick and dirty fix is simple:

_PyOS_ReadlineTState = PyThreadState_GET();
/* CCP change, cannot release the GIL here because PyOS_StdioReadline uses
 * the regular MALLOC
 */
/*
Py_BEGIN_ALLOW_THREADS
*/
#ifdef WITH_THREAD
PyThread_acquire_lock(_PyOS_ReadlineLock, 1);
#endif

/* This is needed to handle the unlikely case that the
 * interpreter is in interactive mode *and* stdin/out are not
 * a tty.  This can happen, for example if python is run like
 * this: python -i  test1.py
 */
if (!isatty (fileno (sys_stdin)) || !isatty (fileno (sys_stdout)))
rv = PyOS_StdioReadline (sys_stdin, sys_stdout, prompt);
else
rv = (*PyOS_ReadlineFunctionPointer)(sys_stdin, sys_stdout,
 prompt);
/*
Py_END_ALLOW_THREADS
*/

#ifdef WITH_THREAD
PyThread_release_lock(_PyOS_ReadlineLock);
#endif

Basically, we just comment out the lock release since we don't need it.  The 
reason we found this was that we were using GIL a custom mallocator which 
should have been run with the GIL but wasn´t.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2013-05-04 Thread Antoine Pitrou 

Antoine Pitrou added the comment:

So, could you propose a patch?

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2013-01-02 Thread Meador Inge 

Changes by Meador Inge mead...@gmail.com:


--
nosy: +meador.inge

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2012-12-25 Thread Kristján Valur Jónsson 

Changes by Kristján Valur Jónsson krist...@ccpgames.com:


--
nosy: +kristjan.jonsson

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2012-12-23 Thread Gregory P. Smith 

Changes by Gregory P. Smith g...@krypto.org:


--
nosy: +gregory.p.smith
priority: normal - critical

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16742] PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe

2012-12-21 Thread Trent Nelson 

New submission from Trent Nelson:

Relevant thread: 
http://mail.python.org/pipermail/python-dev/2012-December/123225.html

PyOS_StdioReadline features numerous calls that require the GIL to be held.  
Ideally, the GIL drop-take should be moved closer to the actual underlying read 
system call.

--
assignee: trent
components: Interpreter Core
messages: 177874
nosy: trent
priority: normal
severity: normal
stage: needs patch
status: open
title: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread 
safe
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16742
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com