[issue31282] C APIs called without GIL in PyOS_Readline

[STINNER Victor]

STINNER Victor  added the comment:

I fixed this issue in bpo-40826, especially with this change:

commit c353764fd564e401cf47a5d9efab18c72c60014e
Author: Victor Stinner 
Date:   Mon Jun 1 20:59:35 2020 +0200

bpo-40826: Fix GIL usage in PyOS_Readline() (GH-20579)

Fix GIL usage in PyOS_Readline(): lock the GIL to set an exception.

Pass tstate to my_fgets() and _PyOS_WindowsConsoleReadline(). Cleanup
these functions.

--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed
versions: +Python 3.10, Python 3.8, Python 3.9 -Python 2.7, Python 3.6, Python 
3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31282] C APIs called without GIL in PyOS_Readline

[Xiang Zhang]

Xiang Zhang added the comment:

>>> I fixed Python 3, no?

Yes. In Python3 they are replaced by PyMem_RawMalloc. But it's not only 
PyMem_Malloc, there are also PyErr_SetString, PyErr_NoMemory, even in Python3.

BTW, even in Python3, when memory allocators are in debug mode, it finally 
calls bumpserialno, which IIUC, is not thread safe. But of course it's another 
issue.

>>> For Python 2, in practice you can call PyMem_Malloc() without holding the 
>>> GIL, it's just malloc() which is thread safe.

Hmm, I know it. But it's not stated in the doc they are thread safe, I am not 
sure assuming this is suitable. An example is 
https://github.com/psycopg/psycopg2/issues/110.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31282] C APIs called without GIL in PyOS_Readline

[STINNER Victor]

STINNER Victor added the comment:

You must hold the GIL to call PyMem_Malloc(). A debug assert should be
raised if you don't hold the GIL since Python 3.6 with PYTHONMALLOC=debug.

Call PyMem_RawMalloc().

I fixed Python 3, no?

For Python 2, in practice you can call PyMem_Malloc() without holding the
GIL, it's just malloc() which is thread safe.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31282] C APIs called without GIL in PyOS_Readline

[Xiang Zhang]

Xiang Zhang added the comment:

In pymem.h, it's stated that "The GIL must be held when using these APIs". I 
think the reason is when PYMALLOC_DEBUG is defined, they finally delegate to 
PyObject_Malloc.

Victor must be familiar with them since once he proposed to remote the 
restriction: https://mail.python.org/pipermail/python-dev/2013-June/126822.html.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31282] C APIs called without GIL in PyOS_Readline

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

AFAIK PyMem_Malloc/MALLOC can be used with released GIL.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31282] C APIs called without GIL in PyOS_Readline

[Xiang Zhang]

Changes by Xiang Zhang :


--
stage:  -> needs patch
versions: +Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31282] C APIs called without GIL in PyOS_Readline

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +haypo, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31282] C APIs called without GIL in PyOS_Readline

[Xiang Zhang]

New submission from Xiang Zhang:

When debugging our project I find something interesting. In PyOS_Readline it 
releases the GIL and delegate its job to PyOS_ReadlineFunctionPointer, which 
could be call_readline or PyOS_StdioReadline(I don't find where 
vms__StdioReadline is defined). But in the two functions, they use some C APIs 
like PyMem_Malloc/MALLOC, PyErr_SetString which need guarded by GIL. I don't 
understand why not doing find-grained lock control in call_readline or 
PyOS_StdioReadline since the code is ancient. :-(

I find this because our project makes test_cmd_line fail with `echo "Timer\n" | 
python -i -m timeit -n 1`.

--
components: Interpreter Core
messages: 300862
nosy: xiang.zhang
priority: normal
severity: normal
status: open
title: C APIs called without GIL in PyOS_Readline
type: behavior
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com