[issue34968] loop.call_soon_threadsafe should be documented to be re-entrant-safe too

[Nathaniel Smith]

Nathaniel Smith  added the comment:

> What would make it not reentrant-safe?

Probably the most obvious example of a non-reentrant-safe operation is taking a 
lock. It's very natural to write code like:

def call_soon(...):
with self._call_soon_lock:
...

but now imagine that inside the body of that 'with' statement, a signal 
arrives, so the interpreter pauses what it's doing to invoke the signal 
handler, and the signal handler turns around and invokes call_soon, which tries 
to acquire the same lock that it already holds → instant deadlock.

And this rules out quite a few of the tools you might normally expect to use in 
thread-safe code, like queue.Queue, since they use locks internally.

The reason I think the stdlib's call_soon is OK is that it doesn't perform any 
blocking operations, and the critical operation is simply 
'self._ready.append(...)', which in CPython is atomic with respect to 
threads/signals/GC.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34968] loop.call_soon_threadsafe should be documented to be re-entrant-safe too

[Yury Selivanov]

Yury Selivanov  added the comment:

> AFAICT the stdlib's implementation of call_soon_threadsafe is already 
> reentrant-safe

What would make it not reentrant-safe?  We'll need to document that for the 
benefit of asyncio and third-party maintainers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34968] loop.call_soon_threadsafe should be documented to be re-entrant-safe too

[Andrew Svetlov]

Andrew Svetlov  added the comment:

Agree.
Docs update sounds good.
Nathaniel, would you create a docs patch?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34968] loop.call_soon_threadsafe should be documented to be re-entrant-safe too

[Nathaniel Smith]

New submission from Nathaniel Smith :

Asyncio needs a way to schedule work from other threads; and it also needs a 
way to scheduler work from code that can run at arbitrary times in the same 
thread, such as signal handlers or object finalizers ("reentrant contexts").

Currently loop.call_soon_threadsafe is documented to be the way to schedule 
work from other threads, but there is no documented way to schedule work from 
reentrant contexts. These are not quite the same thing, because reentrant 
contexts block the main thread while they're running. Generally, making code 
safe to call from __del__ or signal handlers is strictly harder than making it 
safe to call from other threads. (See bpo-14976 for an example of stdlib code 
being thread-safe but not reentrant-safe.)

Technically speaking, this means that right now, if you need to call an asyncio 
API from a __del__ method, then the only officially supported way to do that is 
to write something like:

def __del__(self):
def actual_cleanup_code():
...
def thread_dispatcher():
loop.call_soon_threadsafe(actual_cleanup_code)
thread = threading.Thread(target=thread_dispatcher)
thread.start()

But this is kind of silly. There should be some equivalent of loop.call_soon 
that *is* safe to call from reentrant contexts, so we could just write:

def __del__(self):
def actual_cleanup_code():
...
loop.call_soon_reentrant_safe(actual_cleanup_code)

But... it doesn't really make sense to add a new method for this, since the 
desired semantics are strictly more powerful than the current 
loop.call_soon_threadsafe. Instead, we should tighten the guarantees on 
call_soon_threadsafe, by documenting that it's safe to use from reentrant 
contexts.

Also, AFAICT the stdlib's implementation of call_soon_threadsafe is already 
reentrant-safe, so this wouldn't require any changes to stdlib code, only to 
the docs. But it would provide an additional formal guarantee that user-level 
code could take advantage of, and impose an additional constraint on developers 
of third-party loops.

(I don't think the constraint is *too* onerous, fortunately. It's quite tricky 
to implement a version of call_soon that's thread-safe, reentrant-safe, *and* 
guarantees that the callback will eventually be invoked, even if call_soon 
races with loop shutdown. But in asyncio, all variants of call_soon are allowed 
to silently drop callbacks at loop shutdown, which makes this much easier.)

--
components: asyncio
messages: 327618
nosy: asvetlov, njs, yselivanov
priority: normal
severity: normal
status: open
title: loop.call_soon_threadsafe should be documented to be re-entrant-safe too
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com