A small update on this, since I've been playing with it.
I'm trying to implement a websocket proxy, since it's an example of a toy
project that needs to juggle two long-lived asyncio connections at once.
I'm using Starlette/Uvicorn for the server part (the part that accepts the
connection) and aiohttp's client functionality to connect to an echo server
on the Internet.
I've settled on the async iterator interface as a good building block for
this, not queues. Turns out an async iterator looks very much as the read
interface for a Go channel (they spit out values and signal their closure).
Aiohttp already provides an async iterator interface for reading from the
upstream server. Starlette doesn't, but an adapter is very easy to write.
Now that we've settled on a suitable interface, a helper function (wait_on)
is easy to write. (Code omitted for brevity.) This is the entire request
handler:
async def ws_proxy(client: WebSocket):
await client.accept()
async with ClientSession() as session:
async with session.ws_connect("wss://echo.websocket.org") as s:
c = starlette_websocket_iterator(client)
async for r in wait_on(c, s):
match r:
case (src, None):
print(f"{src} closed the connection")
break
case (src, msg) if src is c:
print(f"CLIENT: {msg}")
await s.send_str(msg)
case (src, msg) if src is s:
print(f"SERVER: {msg}")
await client.send_text(msg.data)
So yeah, the helper yields tuples of the source and message, using None as
a sentinel for closure. Guards are used to match on the source, using
iterator identity. My first version just used `case (s, msg):` hoping to
match on the identity of s, but that doesn't work since s is not a literal.
I'd say this is pretty cool. With the ecosystem moving to async iterators
for streams of data (or just writing adapters), I'd say this style of
programming is quite readable and understandable.
On Tue, Oct 27, 2020 at 1:16 AM Tin Tvrtković <tinches...@gmail.com> wrote:
> Hello,
>
> Go channels are indeed very similar to asyncio Queues, with some added
> features like channels being closable. (There is also special syntax in the
> select statement, `val, ok <- chan`, that will set the `ok` variable to
> false if the channel has been closed.) A larger difference, I think, is
> that in Go channels are used practically everywhere, more so than asyncio
> Queues. They are an abstraction the vast majority of Go concurrency is
> built upon.
>
> Building this for asyncio tasks, instead of just queues, would be much
> more useful in Python.
>
> Contemplating this some more, I would agree we don't need an async match.
> A function and some types to match on would probably be enough to get us
> close to a select statement in a PEP634 Python. I guess the challenge is
> designing these matchable types for ease of use now, and I need to study
> the pattern matching PEPs in more detail to be able to contribute here.
>
> On one hand, this means this problem can be solved by a third party
> library. On the other hand, I feel like this would be very useful so it
> might be worth it to have it somewhere in the stdlib asyncio namespace.
>
> Since `asyncio.wait` can yield multiple tasks in the completed set, this
> would probably have to be wrapped in an `async for`.
>
>
>
> On Mon, Oct 26, 2020 at 12:33 PM Gustavo Carneiro <gjcarne...@gmail.com>
> wrote:
>
>> It's true that asyncio.wait provides the tools that you need, but it's a
>> bit clunky to use correctly.
>>
>> Maybe would be something along the lines of:
>>
>> ------
>> queue1 = asyncio.Queue()
>> queue2 = asyncio.Queue()
>> ...
>> get1 = asyncio.create_task(queue1.get())
>> get2 = asyncio.create_task(queue2.get())
>> await asyncio.wait({get1, get2}, return_when=asyncio.FIRST_COMPLETED)
>> match [task.done() for task in (get1, get2)]:
>> case [True, False]: get2.cancel(); item1 = await get1; ....
>> case [False, True]: get1.cancel(); item2 = await get2; ....
>> case [True, True]: item1 = await get1; ....; item2 = await get2;
>> ....
>> ------
>>
>> If asyncio.Queue() is the equivalent of Go channels, perhaps it would be
>> worth designing a new API for asyncio.Queue, one that is better suited to
>> the match statement:
>>
>> class Queue:
>> async def read_wait(self) -> 'Queue':
>> """
>> Waits until the queue has at least one item ready to read, without
>> actually consuming the item.
>> """
>>
>> Then we could more easily use match statement with multiple queues, thus:
>>
>> ------
>> async def ready_queue(*queues: asyncio.Queue) -> asyncio.Queue:
>> """
>> Take multiple queue parameters and waits for at least one of them to
>> have items pending to read, returning that queue.
>> """
>> await asyncio.wait({queue.read_wait() for queue in queues},
>> return_when=asyncio.FIRST_COMPLETED)
>> for queue in queues:
>> if queue.qsize() > 0:
>> return queue
>>
>> ...
>>
>> queue1 = asyncio.Queue()
>> queue2 = asyncio.Queue()
>>
>> ...
>>
>> match await ready_queue(queue1, queue2):
>> case queue1: item1 = queue1.get_nowait(); ....
>> case queue2: item2 = queue2.get_nowait(); ....
>> ------
>>
>> Which is less clunky, maybe?...
>>
>> The above is not 100% bug free. I think those queue.get_nowait() calls
>> may still end up raising QueueEmpty exceptions, in case there is another
>> concurrent reader for those queues. This code would need more work, most
>> likely.
>>
>> In any case, perhaps it's not the match statement that needs to change,
>> but rather asyncio API that needs to be enhanced.
>>
>>
>> On Sun, 25 Oct 2020 at 01:14, Nick Coghlan <ncogh...@gmail.com> wrote:
>>
>>> On Sat., 24 Oct. 2020, 4:21 am Guido van Rossum, <gu...@python.org>
>>> wrote:
>>>
>>>> On Fri, Oct 23, 2020 at 6:19 AM Tin Tvrtković <tinches...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> first of all, I'm a big fan of the changes being proposed here since
>>>>> in my code I prefer the 'union' style of logic over the OO style.
>>>>>
>>>>> I was curious, though, if there are any plans for the match operator
>>>>> to support async stuff. I'm interested in the problem of waiting on
>>>>> multiple asyncio tasks concurrently, and having a branch of code execute
>>>>> depending on the task.
>>>>>
>>>>> Currently this can be done by using asyncio.wait, looping over the
>>>>> done set and executing an if-else chain there, but this is quite tiresome.
>>>>> Go has a select statement (https://tour.golang.org/concurrency/5)
>>>>> that looks like this:
>>>>>
>>>>> select {
>>>>> case <-ch1:
>>>>> fmt.Println("Received from ch1")
>>>>> case <-ch2:
>>>>> fmt.Println("Received from ch2")
>>>>> }
>>>>>
>>>>> Speaking personally, this is a Go feature I miss a lot when writing
>>>>> asyncio code. The syntax is similar to what's being proposed here.
>>>>> Although
>>>>> it could be a separate thing added later, async match, I guess.
>>>>>
>>>>
>>>> Hadn't seen this before. You could propose this as a follow-up for
>>>> 3.11. But aren't Go channels more like asyncio Queues? I guess we'd need
>>>> way more in terms of a worked-out example (using asyncio code, not Go
>>>> code).
>>>>
>>>
>>> I think we'd also want to see how far folks get with using guard clauses
>>> for this kind of "where did the data come from?" check - the only
>>> specifically asynchronous bit would be the "await multiple tasks"
>>> operation, and you can already tell asyncio.wait() to return on the first
>>> completed task rather than waiting for all the results.
>>>
>>> Cheers,
>>> Nick.
>>>
>>>
>>>
>>>> _______________________________________________
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/NQWYLFLGLLCEHAXYHUOXQ3M7IOEL65ET/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>>
>> --
>> Gustavo J. A. M. Carneiro
>> Gambit Research
>> "The universe is always one step beyond logic." -- Frank Herbert
>>
>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/BJYJNK5NPHH3ZAPER4NWQ6Q4OWDYVYFH/
Code of Conduct: http://python.org/psf/codeofconduct/
