The real problem I'm playing with is implementing "happy eyeballs", where I may have several sockets attempting to connect simultaneously, and the first one to successfully connect gets used. I had the idea of preparing all of the loop.sock_connect() coroutine objects in advance, and scheduling them one by one on the loop, but wanted to make double sure that the sockets won't start connecting before the coroutines are scheduled. I wanted to write something like this:
successful_socket = await staggered_start([loop.sock_connect(socket.socket(), addr) for addr in addresses]) where async def staggered_start(coros) is some kind of reusable scheduling logic. As it turns out, I can't actually depend on loop.sock_connect() doing the Right Thing (TM) if I want to support Python 3.4. On Fri, May 4, 2018 at 12:37 AM, Andrew Svetlov <andrew.svet...@gmail.com> wrote: > What real problem do you want to solve? > Correct code should always use `await loop.sock_connect(sock, addr)`, it > this case the behavior difference never hurts you. > > On Thu, May 3, 2018 at 7:04 PM twisteroid ambassador > <twisteroid.ambassa...@gmail.com> wrote: >> >> Hi, >> >> tl;dr: coroutine functions and regular functions returning Futures >> behave differently: the latter may start running immediately without >> being scheduled on a loop, or even with no loop running. This might be >> bad since the two are sometimes advertised to be interchangeable. >> >> >> I find that sometimes I want to construct a coroutine object, store it >> for some time, and run it later. Most times it works like one would >> expect: I call a coroutine function which gives me a coroutine object, >> I hold on to the coroutine object, I later await it or use >> loop.create_task(), asyncio.gather(), etc. on it, and only then it >> starts to run. >> >> However, I have found some cases where the "coroutine" starts running >> immediately. The first example is loop.run_in_executor(). I guess this >> is somewhat unsurprising since the passed function don't actually run >> in the event loop. Demonstrated below with strace and the interactive >> console: >> >> $ strace -e connect -f python3 >> Python 3.6.5 (default, Apr 4 2018, 15:01:18) >> [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import asyncio >> >>> import socket >> >>> s = socket.socket() >> >>> loop = asyncio.get_event_loop() >> >>> coro = loop.sock_connect(s, ('127.0.0.1', 80)) >> >>> loop.run_until_complete(asyncio.sleep(1)) >> >>> task = loop.create_task(coro) >> >>> loop.run_until_complete(asyncio.sleep(1)) >> connect(3, {sa_family=AF_INET, sin_port=htons(80), >> sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection >> refused) >> >>> s.close() >> >>> s = socket.socket() >> >>> coro2 = loop.run_in_executor(None, s.connect, ('127.0.0.1', 80)) >> strace: Process 13739 attached >> >>> [pid 13739] connect(3, {sa_family=AF_INET, sin_port=htons(80), >> >>> sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection >> >>> refused) >> >> >>> coro2 >> <Future pending cb=[_chain_future.<locals>._call_check_cancel() at >> /usr/lib64/python3.6/asyncio/futures.py:403]> >> >>> loop.run_until_complete(asyncio.sleep(1)) >> >>> coro2 >> <Future finished exception=ConnectionRefusedError(111, 'Connection >> refused')> >> >>> >> >> Note that with loop.sock_connect(), the connect syscall is only run >> after loop.create_task() is called on the coroutine AND the loop is >> running. On the other hand, as soon as loop.run_in_executor() is >> called on socket.connect, the connect syscall gets called, without the >> event loop running at all. >> >> Another such case is with Python 3.4.2, where even loop.sock_connect() >> will run immediately: >> >> $ strace -e connect -f python3 >> Python 3.4.2 (default, Oct 8 2014, 10:45:20) >> [GCC 4.9.1] on linux >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import socket >> >>> import asyncio >> >>> loop = asyncio.get_event_loop() >> >>> s = socket.socket() >> >>> c = loop.sock_connect(s, ('127.0.0.1', 82)) >> connect(7, {sa_family=AF_INET, sin_port=htons(82), >> sin_addr=inet_addr("127.0.0.1")}, 16) = -1ECONNREFUSED (Connection >> refused) >> >>> c >> <Future finished exception=ConnectionRefusedError(111, 'Connection >> refused')> >> >>> >> >> In both these cases, the misbehaving "coroutine" aren't actually >> defined as coroutine functions, but regular functions returning a >> Future, which is probably why they don't act like coroutines. However, >> coroutine functions and regular functions returning Futures are often >> used interchangeably: Python docs Section 18.5.3.1 even says: >> >> > Note: In this documentation, some methods are documented as coroutines, >> > even if they are plain Python functions returning a Future. This is >> > intentional to have a freedom of tweaking the implementation of these >> > functions in the future. >> >> In particular, both run_in_executor() and sock_connect() are >> documented as coroutines. >> >> If an asyncio API may change from a function returning Future to a >> coroutine function and vice versa any time, then one cannot rely on >> the behavior of creating the "coroutine object" not running the >> coroutine immediately. This seems like an important Gotcha waiting to >> bite someone. >> >> Back to the scenario in the beginning. If I want to write a function >> that takes coroutine objects and schedule them to run later, and some >> coroutine objects turn out to be misbehaving like above, then they >> will run too early. To avoid this, I could either 1. pass the >> coroutine functions and their arguments separately "callback style", >> 2. use functools.partial or lambdas, or 3. always pass in real >> coroutine objects returned from coroutine functions defined with >> "async def". Does this sound right? >> >> Thanks, >> >> twistero >> _______________________________________________ >> Async-sig mailing list >> Async-sig@python.org >> https://mail.python.org/mailman/listinfo/async-sig >> Code of Conduct: https://www.python.org/psf/codeofconduct/ > > -- > Thanks, > Andrew Svetlov _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/