[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Jeethu Rao]

Change by Jeethu Rao :


--
nosy: +jeethu

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Ned Deily]

Ned Deily added the comment:


New changeset c794b643c9172d69afa46f85982befd82511d9df by Ned Deily (Victor 
Stinner) in branch '3.6':
bpo-30704, bpo-30604: Fix memleak in code_dealloc() (#2455) (#2456)
https://github.com/python/cpython/commit/c794b643c9172d69afa46f85982befd82511d9df


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

STINNER Victor added the comment:


New changeset 26daad4ee14693381d84a5235709d22aed1c22ed by Victor Stinner in 
branch '3.6':
bpo-30704, bpo-30604: Fix memleak in code_dealloc() (#2455) (#2456)
https://github.com/python/cpython/commit/26daad4ee14693381d84a5235709d22aed1c22ed


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2514

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

STINNER Victor added the comment:


New changeset 23e7944eba1968bb8432fdc4cc96d4fdd2c1a230 by Victor Stinner in 
branch 'master':
bpo-30704, bpo-30604: Fix memleak in code_dealloc() (#2455)
https://github.com/python/cpython/commit/23e7944eba1968bb8432fdc4cc96d4fdd2c1a230


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

STINNER Victor added the comment:

> _PyCode_SetExtra() uses two memory block for code extras. By changing how 
> memory is accessed and allocated, it would be possible to use a single memory 
> block. Was it on purpose to use two memory blocks?

I discussed with Yury who is not opposed to such change in Python 3.7, so I 
created bpo-30789.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

STINNER Victor added the comment:

_PyCode_SetExtra() uses two memory block for code extras. By changing how 
memory is accessed and allocated, it would be possible to use a single memory 
block. Was it on purpose to use two memory blocks?

See for example PyTupleObject which uses a single memory block vs PyListObject 
which uses two memory blocks.

typedef struct {
PyObject_VAR_HEAD
PyObject *ob_item[1];

/* ob_item contains space for 'ob_size' elements.
 * Items must normally not be NULL, except during construction when
 * the tuple is not yet visible outside the function that builds it.
 */
} PyTupleObject;

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2512

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

STINNER Victor added the comment:


New changeset cea2174ab7cce01c420b2770562be4c91f1f4e35 by Victor Stinner in 
branch '3.6':
bpo-30604: Skip CoExtra tests if ctypes is missing (#2356) (#2358)
https://github.com/python/cpython/commit/cea2174ab7cce01c420b2770562be4c91f1f4e35


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

STINNER Victor added the comment:

The test_code is fixed again, so I close the issue.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2405

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

STINNER Victor added the comment:


New changeset a4b091e135ccf345cfafdd8477aef897c5214f82 by Victor Stinner in 
branch 'master':
bpo-30604: Skip CoExtra tests if ctypes is missing (#2356)
https://github.com/python/cpython/commit/a4b091e135ccf345cfafdd8477aef897c5214f82


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2403

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

STINNER Victor added the comment:

If a test requires ctypes, please skip your test if ctypes is missing. The new 
test fails on the "x86 Ubuntu Shared 3.x" buildbot which lacks the _ctypes 
module (for an unknown reason, but does it really matter here? ;-)).

http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/924/steps/test/logs/stdio

test test_code crashed -- Traceback (most recent call last):
  File 
"/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py",
 line 156, in runtest_inner
the_module = importlib.import_module(abstest)
  File 
"/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/importlib/__init__.py", 
line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 978, in _gcd_import
  File "", line 961, in _find_and_load
  File "", line 950, in _find_and_load_unlocked
  File "", line 655, in _load_unlocked
  File "", line 679, in exec_module
  File "", line 205, in _call_with_frames_removed
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_code.py", 
line 218, in 
import ctypes
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/ctypes/__init__.py", 
line 7, in 
from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Yury Selivanov]

Yury Selivanov added the comment:

Closing the issue. Thank you Dino for working on this!

--
resolution:  -> fixed
stage: backport needed -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Yury Selivanov]

Yury Selivanov added the comment:

It doesn't need to be, it's only for 3.6

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Mariatta Wijaya]

Mariatta Wijaya added the comment:

PR 2152 is not yet ported to master.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Brett Cannon]

Brett Cannon added the comment:

Should this be closed since the all PRs got merged?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Yury Selivanov]

Yury Selivanov added the comment:


New changeset f3cffd2b7879d209f982de899b782fb89cfc410a by Yury Selivanov (Dino 
Viehland) in branch 'master':
bpo-30604: clean up co_extra support (#2144)
https://github.com/python/cpython/commit/f3cffd2b7879d209f982de899b782fb89cfc410a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Mariatta Wijaya]

Mariatta Wijaya added the comment:

I'm setting the stage to 'backport needed', but it really is a 'porting needed' 
stage :)

The two PRs merged PRs here were made against 3.6.

--
nosy: +Mariatta
stage: patch review -> backport needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

STINNER Victor added the comment:


New changeset 932946ca14168e556293d2508c8eebb23a56a2b2 by Victor Stinner in 
branch '3.6':
bpo-30604: Fix __PyCodeExtraState_Get() prototype (#2152)
https://github.com/python/cpython/commit/932946ca14168e556293d2508c8eebb23a56a2b2


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2204

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Avoid using double underscores in C code. C compiler uses names with double 
underscores for its own needs, and this can lead to conflicts.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Dino Viehland]

Changes by Dino Viehland :


--
pull_requests: +2194

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Ned Deily]

Ned Deily added the comment:


New changeset 2997fec01ee7300c6d5940e6c55e4ccf9f56f1b5 by Ned Deily (Dino 
Viehland) in branch '3.6':
[3.6] bpo-30604: Move co_extra_freefuncs to interpreter state to avoid crashes 
in threads (#2015)
https://github.com/python/cpython/commit/2997fec01ee7300c6d5940e6c55e4ccf9f56f1b5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

User code shouldn't allocate PyInterpreterState and PyThreadState structures, 
it only uses structures created by the interpreter. Changing the size of 
PyInterpreterState should be safe. The only possible breaking compatibility is 
if user code directly access co_extra_user_count, co_extra_freefuncs, 
async_gen_firstiter or async_gen_finalizer rather than using the API: 
_PyEval_RequestCodeExtraIndex(), _PyCode_GetExtra(), _PyCode_SetExtra(), 
_PyEval_GetAsyncGenFirstiter(), _PyEval_SetAsyncGenFirstiter(), 
_PyEval_GetAsyncGenFinalizer() and _PyEval_SetAsyncGenFinalizer().

Nick's idea about _preserve_36_ABI_1 and _preserve_36_ABI_2 should   address 
concerns about direct access to async_gen_firstiter and async_gen_finalizer. 
Direct access to co_extra_user_count and co_extra_freefuncs obviously can't be 
preserved.

PyInterpreterState and PyThreadState are not in the stable ABI. They are opaque 
types when use limited API. May be they should be made opaque for all user code 
(in 3.7+).

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Ned Deily]

Ned Deily added the comment:

See review comments on PR 2015.

--
nosy: +ned.deily
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Dino Viehland]

Changes by Dino Viehland :


--
pull_requests: +2080

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30604] co_extra_freefuncs is stored thread locally and can lead to crashes

[Dino Viehland]

New submission from Dino Viehland:

The co_extra_freefuncs are stored in PyThreadState.  When calling 
_PyEval_RequestCodeExtraIndex you are given a thread specific index.  The code 
object can then lose it's last reference on a different thread, and the wrong 
free function can be called if users of the extra space have made calls to get 
their index in different orders.  

This can also lead to crashes if the extra thread hasn't yet requested extra 
indexes either.

--
assignee: dino.viehland
components: Interpreter Core
keywords: patch
messages: 295467
nosy: brett.cannon, dino.viehland, yselivanov
priority: normal
severity: normal
stage: patch review
status: open
title: co_extra_freefuncs is stored thread locally and can lead to crashes
type: crash
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com