[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[Fabio Zadrozny]

Fabio Zadrozny  added the comment:

>> I.e.: something as adding a thread_id to sys.settrace -- 
>> sys.settrace(trace_func, thread_id=None).

> What is the use case for this feature?

The use case is having the user attach the debugger (either programmatically or 
by doing an attach to process) and be able to debug all threads, not just the 
current thread.

> It seems quite complicated to implement the thread_id for 
> sys.settrace(trace_func, thread_id=None).

Humm, isn't it just a matter of passing the proper tstate to _PyEval_SetTrace? 
It seems reasonably simple to do at C (i.e.: iterate the existing thread states 
to get the thread id and then pass the proper tsate to _PyEval_SetTrace -- 
which is roughly what is done in the debugger, although it's a bit more 
complicated because it supports Python 2.7 up to Python 3.8...).

> At the C level, Python doesn't maintain a list of thread. There is only 
> threading.enumerate() which is implemented in Python.

The tstate does contain the thread id, so, iterating the available tstates 
should be enough for that.

> PyDev.Debugger seems to use the C API. Can it continue to use the C API?

It can for CPython, but it can't for other Python implementations (and ideally 
I'd like to rely less on the CPython C-API -- because there's too much 
implementation details on it, things seem to break at each new version).

> Note: There is threading.setprofile() and threading.settrace() which set a 
> profile/trace function when *new* threads are spawned

Yes, I know about those, but it's not enough if the user attaches the debugger 
after the process is already running.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

STINNER Victor  added the comment:

> I.e.: something as adding a thread_id to sys.settrace -- 
> sys.settrace(trace_func, thread_id=None).

What is the use case for this feature?

It seems quite complicated to implement the thread_id for 
sys.settrace(trace_func, thread_id=None).

Currently, there is no way for a thread to execute code directly in another 
thread. In asyncio, it has to through call_soon_threadsafe() which queues the 
function calls and the function is called "later" and the caller doesn't get 
the result.

https://docs.python.org/dev/library/asyncio-eventloop.html#asyncio.loop.call_soon_threadsafe

sys.setprofile() and sys.settrace() use the current Python thread state 
(tstate). The threading.enumerate() function returns threading.Thread 
instances, but it's not currently possible to get the Python thread state (C 
structure PyThreadState) from a Python threading.Thread object.

At the C level, Python doesn't maintain a list of thread. There is only 
threading.enumerate() which is implemented in Python.

PyDev.Debugger seems to use the C API. Can it continue to use the C API?

Note: There is threading.setprofile() and threading.settrace() which set a 
profile/trace function when *new* threads are spawned.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[Fabio Zadrozny]

Fabio Zadrozny  added the comment:

>> As a note, the original request was for a Python-level tracing function (so 
>> that in the future other Python implementations also provide that function) 
>> -- does this need a PEP?

> What do you mean by a Python-level tracing function?

I mean that it's a function to be called from Python (not only from C) -- which 
hopefully could be adopted by other Python implementations in the long run. 

I.e.: something as adding a thread_id to sys.settrace -- 
sys.settrace(trace_func, thread_id=None).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 046255c40fc0d9c5a4c528eb5955792fa08df66f by Victor Stinner in 
branch '3.8':
bpo-35370: PyEval_SetTrace() logs unraisable error (GH-18977) (GH-19029)
https://github.com/python/cpython/commit/046255c40fc0d9c5a4c528eb5955792fa08df66f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

STINNER Victor  added the comment:

> As a note, the original request was for a Python-level tracing function (so 
> that in the future other Python implementations also provide that function) 
> -- does this need a PEP?

What do you mean by a Python-level tracing function?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18378
pull_request: https://github.com/python/cpython/pull/19029

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

STINNER Victor  added the comment:


New changeset f6a58507820c67e8d0fb07875cd1b1d9f5e510a8 by Victor Stinner in 
branch 'master':
bpo-35370: PyEval_SetTrace() logs unraisable error (GH-18977)
https://github.com/python/cpython/commit/f6a58507820c67e8d0fb07875cd1b1d9f5e510a8


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[Fabio Zadrozny]

Fabio Zadrozny  added the comment:

Holding the GIL is a reasonable constraint.

As a note, the original request was for a Python-level tracing function (so 
that in the future other Python implementations also provide that function) -- 
does this need a PEP?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

STINNER Victor  added the comment:

Fabio: I added _PyEval_SetProfile() and _PyEval_SetTrace() which take a tstate 
parameter. These functions have a constraint: the caller must hold the GIL. Is 
it an acceptable constraint for you?

That's not something new, it's already the code in Python 3.8, it's just that 
it wasn't documented. In Python 3.7, it was less important: Python 3.8 added a 
call to PySys_Audit() which can execute arbitrary Python code.

Anyway, touching Python internals without holding the GIL is risky: see 
bpo-1021318 for example.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

STINNER Victor  added the comment:

> PyEval_SetTrace() and PyEval_SetProfile() function have no return value but 
> can raise an exception. I'm not comfortable about these functions. Maybe one 
> option would be to call PyErr_WriteUnraisable() on PySys_Audit() error. It 
> might be better than "leaking" an exception which is unexpected to 
> PyEval_SetTrace() and PyEval_SetProfile() callers.

I wrote PR 18977 to implement this idea.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18325
pull_request: https://github.com/python/cpython/pull/18977

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

STINNER Victor  added the comment:

PyEval_SetTrace() and PyEval_SetProfile() function have no return value but can 
raise an exception. I'm not comfortable about these functions. Maybe one option 
would be to call PyErr_WriteUnraisable() on PySys_Audit() error. It might be 
better than "leaking" an exception which is unexpected to PyEval_SetTrace() and 
PyEval_SetProfile() callers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 309d7cc5df4e2bf3086c49eb2b1b56b929554500 by Victor Stinner in 
branch 'master':
bpo-35370: Add _PyEval_SetTrace() function (GH-18975)
https://github.com/python/cpython/commit/309d7cc5df4e2bf3086c49eb2b1b56b929554500


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[Fabio Zadrozny]

Fabio Zadrozny  added the comment:

I'm iterating on all threads and getting its thread id to find out the thread 
state (in my use case) and then doing what you just did there...

So, while this solution does work for me, if the idea is making tstate opaque, 
then having (an optional) thread id in settrace (which iterates to find the 
proper thread if given) could solve it without having to rely on any CPython 
internals on my side (although it should probably return a bool to say if it 
did work then).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

STINNER Victor  added the comment:

> Maybe better would be the thread id so that the tstate structure is not 
> needed there.

PyInterpreterState.tstate_head allows to iterate on all thread states, but I'm 
not aware of an API to get a PyThreadState from its identifier (tstate->id).

--

See also bpo-1021318: "PyThreadState_Next not thread safe".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

STINNER Victor  added the comment:

Attached PR 18975 adds _PyEval_SetTrace(tstate, func, arg) function. It 
requires that the caller holds the GIL. The function calls PySys_Audit() in the 
context of the current thread state (not in "tstate").

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Add _PyEval_SetTrace(tstate, func, arg) function

[STINNER Victor]

Change by STINNER Victor :


--
title: Provide API to set the tracing function to be used for running threads. 
-> Add _PyEval_SetTrace(tstate, func, arg) function
versions: +Python 3.9 -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com