On Wed, Feb 25, 2015 at 8:43 AM, Victor Stinner <[email protected]>
wrote:
> Hi,
>
> On IRC, someone told me that it took him hours to understand that
> asyncio.Queue is not thread-safe and he expected asyncio.Queue to be
> thread-safe. I modified the asyncio documentation to mention in almost
> all classes that asyncio classes are not thread-safe.
>
> I didn't touch the doc of lock classes (ex: asyncio.Lock), because I
> don't know if they are thread safe or not.
Whether the current implementations happen to be thread-safe or not, I
don't think they are intended to guarantee thread-safety.
> Since locks should be used
> with "yield from lock", it's not easy to combine them with
> loop.call_soon_threadsafe().
>
Queues are also called with "yield from" in many cases (put() to a bounded
queue, or get() from any queue); the only queue methods that could
trivially be made thread-safe are put_nowait() and get_nowait() (and the
latter is not very useful without some way to wait for updates).
Supporting both "yield from" and threaded-style use of a queue would
require either separate methods for use from other threads or a more
complicated Future implementation that could be either yielded from or
blocked on (this would have performance implications and also remove one of
the safeguards that prevents misuse of asyncio.Future today).
>
> Maybe we should modify asyncio.Queue and asyncio.Lock to make them
> thread-safe?
>
-1 on modifying asyncio.Lock. You can use a threading.Lock in an asyncio
program to protect data that is used by other threads (and if you're
concerned about blocking, make your critical sections smaller).
A hybrid thread-safe and asynchronous queue is more interesting. I'm on the
fence about whether it makes sense to modify asyncio.Queue or make it a
separate class, but it does seem like something that would be useful to
have. It might be as simple as adding a couple of methods like this:
def blocking_get(self):
"""Remove and return an item from the queue. If queue is empty, wait
until an item is available.
This method is **not** a coroutine and should not be called from the
event loop thread.
It is safe to call from any other thread.
""""
future = concurrent.futures.Future()
@asyncio.coroutine
def cb():
result = yield from self.get()
future.set_result(result)
self._loop.call_soon_threadsafe(cb)
return future.result()
In fact, this wrapper could be made generic to let you call any asyncio
coroutine from another thread and block waiting for the results.
-Ben
>
> What do you think?
>
> Victor
>
