Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Koos Zevenhoven]

On Wed, Mar 1, 2017 at 7:42 AM, Nick Coghlan  wrote:
> Short version:
>
> - there are some reasonable requests for async variants of contextlib APIs
> for 3.7
> - prompted by Raymond, I'm thinking it actually makes more sense to add
> these in a new `asyncio.contextlib` module than it does to add them directly
> to the existing module
> - would anyone object strongly to my asking authors of the affected PRs to
> take their changes in that direction?
>

Related to this, here's a post from two years ago in attempt to tackle
the cause of this problem (of needing async and non-async variants)
and solve it in the long term.

https://mail.python.org/pipermail/python-ideas/2015-May/033267.html

You can read the details in that thread, but in short, the idea is
that all functionality that may have to wait for something (IO etc.)
should be explicitly awaited, regardless of whether the code takes
advantage of concurrency or not. This solution is an attempt to do
this without enforcing a specific async framework.

In the post, I made up the terms "Y end" and "L end", because I did
not know what to call them. This was when the draft PEP492 was being
discussed.

L is the end that 'drives' the (chain of) coroutines, usually an event
loop. Y is the other end, the most inner co-routine in the
calling/awaiting chain that does the yields. The L and Y end together
could hide the need of two variants, as explained in the above link.

—Koos

> Longer version:
>
> There are a couple of open issues requesting async variants of some
> contextlib APIs (asynccontextmanager and AsyncExitStack). I'm inclined to
> accept both of them, but Raymond raised a good question regarding our
> general design philosophy for these kinds of additions: would it make more
> sense to put these in an "asyncio.contextlib" module than it would to add
> them directly to contextlib itself?
>
> The main advantage I see to the idea is that if someone proposed adding an
> "asyncio" dependency to contextlib, I'd say no. For the existing
> asynccontextmanager PR, I even said no to adding that dependency to the
> standard contextlib test suite, and instead asked that the new tests be
> moved out to a separate file, so the existing tests could continue to run
> even if asyncio was unavailable for some reason.
>
> While rejecting the idea of an asyncio dependency isn't a problem for
> asyncontextmanager specifically (it's low level enough for it not to
> matter), it's likely to be more of a concern for the AsyncExitStack API,
> where the "asyncio.iscoroutinefunction" introspection API is likely to be
> quite helpful, as are other APIs like `asyncio.ensure_future()`.
>
> So would folks be OK with my asking the author of the PR for
> https://bugs.python.org/issue29679 (adding asynccontextmanager) to rewrite
> the patch to add it as asyncio.contextlib.asyncontextmanager (with a
> cross-reference from the synchronous contextlib docs), rather than the
> current approach of adding it directly to contextlib?
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Nick Coghlan]

On 8 March 2017 at 04:15, Ethan Furman  wrote:

> On 03/07/2017 09:41 AM, Brett Cannon wrote:
>
> I don't think a common practice has bubbled up yet for when there's both
>> synchronous and asynchronous versions of an API
>> (closest I have seen is appending an "a" to the async version but that
>> just looks like a spelling mistake to me most of
>> the time). This is why the question of whether separate modules are a
>> better idea is coming up.
>>
>
> I'm undoubtedly going to show my ignorance with this question, but is it
> feasible to have both sync and async support in the same object?
>

As Jelle says, it depends on the API. For contextlib, we've already decided
that 'asynccontextmanager' and 'AsyncExitStack' are going to be parallel
APIs, as even though they *could* be the same object, they're much easier
to document if they're separate, and you get a form of "verb agreement" at
both definition time and at runtime that lets us be confident of the
developer's intent.

For simpler APIs like "closing" though, I'm leaning more towards the "just
make it work everywhere" approach, where the async protocol methods use
"await obj.aclose()" if the latter is defined, and a synchronous
"obj.close()" otherwise.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Nathaniel Smith]

On Tue, Mar 7, 2017 at 9:41 AM, Brett Cannon  wrote:
> I don't think a common practice has bubbled up yet for when there's both
> synchronous and asynchronous versions of an API (closest I have seen is
> appending an "a" to the async version but that just looks like a spelling
> mistake to me most of the time). This is why the question of whether
> separate modules are a better idea is coming up.

For the CSV case, it might be sensible to factor out the io. Like,
provide an API that looks like:

pushdictreader = csv.PushDictReader()
while pushdictreader:
chunk = read_some(...)
pushdictreader.push(chunk)
for row in pushdictreader:
...

This API can now straightforwardly be used with sync and async code.
Of course you'd want to wrap it up in a nicer interface, somewhere in
the ballpark of:

def sync_rows(read_some):
pushdictreader = csv.PushDictReader()
while pushdictreader:
chunk = read_some(...)
pushdictreader.push(chunk)
for row in pushdictreader:
yield row

async def async_rows(read_some):
pushdictreader = csv.PushDictReader()
while pushdictreader:
chunk = await read_some(...)
pushdictreader.push(chunk)
for row in pushdictreader:
yield row

So there'd still be a bit of code duplication, but much much less.

Essentially the idea here is to convert the csv module to sans-io
style (http://sans-io.readthedocs.io/).

Another option is to make it all-async internally, and then offer a
sync facade around it. So like start with the natural all-async
interface:

class AsyncFileLike(ABC):
async def async_read(...):
...

class AsyncDictReader:
def __init__(self, async_file_like):
self._async_file_like = async_file_like

async def __anext__(self):
...

And (crucially!) let's assume that the only way AsyncDictReader
interacts with the coroutine runner is by calls to
self._async_file_like.async_read. Now we can pass in a
secretly-actually-synchronous AsyncFileLike and make a synchronous
facade around the whole thing:

class AsyncSyncAdapter(AsyncFileLike):
def __init__(self, sync_file_like):
self._sync_file_like = sync_file_like

# Technically an async function, but guaranteed to never yield
async def read(self, *args, **kwargs):
return self._sync_file_like.read(*args, **kwargs)

# Minimal coroutine supervisor: runs async_fn(*args, **kwargs), which
must never yield
def syncify(async_fn, *args, **kwargs):
coro = async_fn(*args, **kwargs)
it = coro.__await__()
return next(it)

class DictReader:
def __init__(self, sync_file_like):
# Technically an AsyncDictReader, but guaranteed to never yield
self._async_dict_reader =
AsyncDictReader(AsyncSyncAdapter(sync_file_like))

def __next__(self):
return syncify(self._async_dict_reader.__anext__)

So here we still have some goo around the edges of the module, but the
actual CSV logic only has to be written once, and can still be written
in a "pull" style where it does its own I/O, just like it is now.

This is basically another approach to writing sans-io protocols, with
the annoying trade-off that it means even your synchronous version
requires Python 3.5+. But for a stdlib module that's no big deal...

-n


> On Tue, 7 Mar 2017 at 02:24 Michel Desmoulin 
> wrote:
>>
>> Last week I had to download a CSV from an FTP and push any update on it
>> using websocket so asyncio was a natural fit and the network part went
>> well.
>>
>> The surprise was that the CSV part would not work as expected. Usually I
>> read csv doing:
>>
>> import csv
>>
>> file_like_object = csv_crawler.get_file()
>> for row in csv.DictReader(file_like_object)
>>
>> But it didn't work because file_like_object.read() was a coroutine which
>> the csv module doesn't handle.
>>
>> So I had to do:
>>
>> import csv
>> import io
>>
>> raw_bytes = await stream.read(1000)
>> wrapped_bytes = io.BytesIO(raw_bytes)
>> text = io.TextIOWrapper(wrapped_bytes, encoding=encoding,
>> errors='replace')
>>
>> for i, row in enumerate(csv.DictReader(text)):
>>
>> Turns out I used asyncio a bit, and I now the stdlib, the io AIP, etc.
>> But for somebody that doesn't, it's not very easy to figure out. Plus
>> it's not as elegant as traditional Python. Not to mention it loads the
>> entire CSV in memory.
>>
>> So I wondered if I could fix the csv module so it accept async. But the
>> question arised. Where should I put it ?
>>
>> - Create AsyncDictReader and AsyncReader ?
>> - Add inspect.iscoroutine calls widh it in the regular Readers and some
>> __aiter__ and __aenter__ ?
>> - add a csv.async namespace ?
>>
>> What API design are we recommanding for expose both sync and async
>> behaviors ?
>>
>>
>> Le 07/03/2017 à 03:08, Guido van Rossum a écrit :
>> > On Mon, Mar 6, 2017 at 5:57 PM, Raymond Hettinger
>> > 

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Sven R. Kunze]

On 07.03.2017 19:37, Jelle Zijlstra wrote:



2017-03-07 10:15 GMT-08:00 Ethan Furman >:


On 03/07/2017 09:41 AM, Brett Cannon wrote:

I don't think a common practice has bubbled up yet for when
there's both synchronous and asynchronous versions of an API
(closest I have seen is appending an "a" to the async version
but that just looks like a spelling mistake to me most of
the time). This is why the question of whether separate
modules are a better idea is coming up.


I'm undoubtedly going to show my ignorance with this question, but
is it feasible to have both sync and async support in the same object?

It's possible, but it quickly gets awkward and will require a lot of 
code duplication.


Correct me if I'm wrong, but we would get the code duplication anyway.

async intrinsically does the same thing (just a little bit different) as 
its sync counterpart. Otherwise, you wouldn't use it.


For example, we could make @contextmanager work for async functions by 
making the _GeneratorContextManager class implement both enter/exit 
and aenter/aexit, but then you'd get an obscure error if you used with 
on an async contextmanager or async with on a non-async contextmanager.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Jelle Zijlstra]

2017-03-07 10:15 GMT-08:00 Ethan Furman :

> On 03/07/2017 09:41 AM, Brett Cannon wrote:
>
> I don't think a common practice has bubbled up yet for when there's both
>> synchronous and asynchronous versions of an API
>> (closest I have seen is appending an "a" to the async version but that
>> just looks like a spelling mistake to me most of
>> the time). This is why the question of whether separate modules are a
>> better idea is coming up.
>>
>
> I'm undoubtedly going to show my ignorance with this question, but is it
> feasible to have both sync and async support in the same object?
>
> It's possible, but it quickly gets awkward and will require a lot of code
duplication. For example, we could make @contextmanager work for async
functions by making the _GeneratorContextManager class implement both
enter/exit and aenter/aexit, but then you'd get an obscure error if you
used with on an async contextmanager or async with on a non-async
contextmanager.


> --
> ~Ethan~
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jelle.
> zijlstra%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Ethan Furman]

On 03/07/2017 09:41 AM, Brett Cannon wrote:


I don't think a common practice has bubbled up yet for when there's both 
synchronous and asynchronous versions of an API
(closest I have seen is appending an "a" to the async version but that just 
looks like a spelling mistake to me most of
the time). This is why the question of whether separate modules are a better 
idea is coming up.


I'm undoubtedly going to show my ignorance with this question, but is it feasible to have both sync and async support in 
the same object?


--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Brett Cannon]

I don't think a common practice has bubbled up yet for when there's both
synchronous and asynchronous versions of an API (closest I have seen is
appending an "a" to the async version but that just looks like a spelling
mistake to me most of the time). This is why the question of whether
separate modules are a better idea is coming up.

On Tue, 7 Mar 2017 at 02:24 Michel Desmoulin 
wrote:

> Last week I had to download a CSV from an FTP and push any update on it
> using websocket so asyncio was a natural fit and the network part went
> well.
>
> The surprise was that the CSV part would not work as expected. Usually I
> read csv doing:
>
> import csv
>
> file_like_object = csv_crawler.get_file()
> for row in csv.DictReader(file_like_object)
>
> But it didn't work because file_like_object.read() was a coroutine which
> the csv module doesn't handle.
>
> So I had to do:
>
> import csv
> import io
>
> raw_bytes = await stream.read(1000)
> wrapped_bytes = io.BytesIO(raw_bytes)
> text = io.TextIOWrapper(wrapped_bytes, encoding=encoding,
> errors='replace')
>
> for i, row in enumerate(csv.DictReader(text)):
>
> Turns out I used asyncio a bit, and I now the stdlib, the io AIP, etc.
> But for somebody that doesn't, it's not very easy to figure out. Plus
> it's not as elegant as traditional Python. Not to mention it loads the
> entire CSV in memory.
>
> So I wondered if I could fix the csv module so it accept async. But the
> question arised. Where should I put it ?
>
> - Create AsyncDictReader and AsyncReader ?
> - Add inspect.iscoroutine calls widh it in the regular Readers and some
> __aiter__ and __aenter__ ?
> - add a csv.async namespace ?
>
> What API design are we recommanding for expose both sync and async
> behaviors ?
>
>
> Le 07/03/2017 à 03:08, Guido van Rossum a écrit :
> > On Mon, Mar 6, 2017 at 5:57 PM, Raymond Hettinger
> > >
> wrote:
> >
> > Of course, it makes sense that anything not specific to asyncio
> > should go outside of asyncio.
> >
> > What I'm more concerned about is what the other places actually
> > are.   Rather than putting async variants of everything sprinkled
> > all over the standard library, I suggest collecting them all
> > together, perhaps in a new asynctools module.
> >
> >
> > That's a tough design choice. I think neither extreme is particularly
> > attractive -- having everything in an asynctools package might also
> > bundle together thing that are entirely unrelated. In the extreme it
> > would be like proposing that all metaclasses should go in a new
> > "metaclasstools" package. I think we did a reasonable job with ABCs:
> > core support goes in abc.py, support for collections ABCs goes into the
> > collections package (in a submodule), and other packages and modules
> > sometimes define ABCs for their own users.
> >
> > Also, in some cases I expect we'll have to create a whole new module
> > instead of updating some ancient piece of code with newfangled async
> > variants to its outdated APIs.
> >
> > --
> > --Guido van Rossum (python.org/~guido )
> >
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.com
> >
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Michel Desmoulin]

Last week I had to download a CSV from an FTP and push any update on it
using websocket so asyncio was a natural fit and the network part went well.

The surprise was that the CSV part would not work as expected. Usually I
read csv doing:

import csv

file_like_object = csv_crawler.get_file()
for row in csv.DictReader(file_like_object)

But it didn't work because file_like_object.read() was a coroutine which
the csv module doesn't handle.

So I had to do:

import csv
import io

raw_bytes = await stream.read(1000)
wrapped_bytes = io.BytesIO(raw_bytes)
text = io.TextIOWrapper(wrapped_bytes, encoding=encoding,
errors='replace')

for i, row in enumerate(csv.DictReader(text)):

Turns out I used asyncio a bit, and I now the stdlib, the io AIP, etc.
But for somebody that doesn't, it's not very easy to figure out. Plus
it's not as elegant as traditional Python. Not to mention it loads the
entire CSV in memory.

So I wondered if I could fix the csv module so it accept async. But the
question arised. Where should I put it ?

- Create AsyncDictReader and AsyncReader ?
- Add inspect.iscoroutine calls widh it in the regular Readers and some
__aiter__ and __aenter__ ?
- add a csv.async namespace ?

What API design are we recommanding for expose both sync and async
behaviors ?


Le 07/03/2017 à 03:08, Guido van Rossum a écrit :
> On Mon, Mar 6, 2017 at 5:57 PM, Raymond Hettinger
> > wrote:
> 
> Of course, it makes sense that anything not specific to asyncio
> should go outside of asyncio.
> 
> What I'm more concerned about is what the other places actually
> are.   Rather than putting async variants of everything sprinkled
> all over the standard library, I suggest collecting them all
> together, perhaps in a new asynctools module.
> 
> 
> That's a tough design choice. I think neither extreme is particularly
> attractive -- having everything in an asynctools package might also
> bundle together thing that are entirely unrelated. In the extreme it
> would be like proposing that all metaclasses should go in a new
> "metaclasstools" package. I think we did a reasonable job with ABCs:
> core support goes in abc.py, support for collections ABCs goes into the
> collections package (in a submodule), and other packages and modules
> sometimes define ABCs for their own users.
> 
> Also, in some cases I expect we'll have to create a whole new module
> instead of updating some ancient piece of code with newfangled async
> variants to its outdated APIs.
> 
> -- 
> --Guido van Rossum (python.org/~guido )
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.com
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Guido van Rossum]

On Mon, Mar 6, 2017 at 5:57 PM, Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

> Of course, it makes sense that anything not specific to asyncio should go
> outside of asyncio.
>
> What I'm more concerned about is what the other places actually are.
>  Rather than putting async variants of everything sprinkled all over the
> standard library, I suggest collecting them all together, perhaps in a new
> asynctools module.
>

That's a tough design choice. I think neither extreme is particularly
attractive -- having everything in an asynctools package might also bundle
together thing that are entirely unrelated. In the extreme it would be like
proposing that all metaclasses should go in a new "metaclasstools" package.
I think we did a reasonable job with ABCs: core support goes in abc.py,
support for collections ABCs goes into the collections package (in a
submodule), and other packages and modules sometimes define ABCs for their
own users.

Also, in some cases I expect we'll have to create a whole new module
instead of updating some ancient piece of code with newfangled async
variants to its outdated APIs.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Raymond Hettinger]

> On Mar 1, 2017, at 8:47 AM, Yury Selivanov  wrote:
> 
>> IMHO this is a good idea*iff*  the new APIs really are bound to
>> asyncio, rather than being generic across all uses of async/await.
> 
> I agree.  There is no need to make asynccontextmanager and
> AsyncExitStack dependent on asyncio or specific to asyncio.
> 
> They should both stay framework agnostic (use only protocols
> defined by PEP 492 and PEP 525) and both shouldn't be put
> into asyncio package.

Of course, it makes sense that anything not specific to asyncio should go 
outside of asyncio.

What I'm more concerned about is what the other places actually are.   Rather 
than putting async variants of everything sprinkled all over the standard 
library, I suggest collecting them all together, perhaps in a new asynctools 
module.


Raymond
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Nick Coghlan]

On 2 March 2017 at 02:29, Barry Warsaw  wrote:

> On Mar 01, 2017, at 10:55 AM, Victor Stinner wrote:
>
> >I suggest to create 3rd party modules on PyPI. It became easy to pull
> >dependencies using pip and virtualenv.
> >
> >It seems like https://github.com/aio-libs is the home of many asyncio
> >libraries.
>
> This is what we did for aiosmtpd, an asyncio-based replacement for smtpd.
> It's worked out great on all fronts so far (good community contributions,
> rapid development, API flexibility as we move toward 1.0, good visibility
> inside the more general aio-libs umbrella).
>

While I agree with this approach for higher level stuff, it's specifically
the lower level pieces that just interact with the async/await language
features rather than the event loop itself where I needed some discussion
to clarify my own thoughts :)

My conclusion from the thread is:

- if it needs to depend on asyncio, it should either go in asyncio, or be
published as a third party aio-lib
- if it *doesn't* need to depend on asyncio, then it's a candidate for
stdlib inclusion (e.g. the coroutine support in inspect)
- both asynccontextmanager and AsyncExitStack actually fall into the latter
category
- other contextlib APIs like closing() should be able to transparently
support both the sync and async variants of the CM protocol without
negatively affecting the synchronous version
- so for the specific case of contextlib, supporting both synchronous and
asynchronous contexts in the one module makes sense
- I still plan to keep the test cases separate, since the async test cases
need more infrastructure than the synchronous ones

What we shouldn't do is take this design decision as setting a binding
precedent for any other modules like itertools - the trade-offs there are
going to be different, and there are already third party modules like
https://github.com/asyncdef/aitertools that provide equivalent APIs for the
asynchronous programming model.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Barry Warsaw]

On Mar 01, 2017, at 10:55 AM, Victor Stinner wrote:

>I suggest to create 3rd party modules on PyPI. It became easy to pull
>dependencies using pip and virtualenv.
>
>It seems like https://github.com/aio-libs is the home of many asyncio
>libraries.

This is what we did for aiosmtpd, an asyncio-based replacement for smtpd.
It's worked out great on all fronts so far (good community contributions,
rapid development, API flexibility as we move toward 1.0, good visibility
inside the more general aio-libs umbrella).

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Paul Moore]

On 1 March 2017 at 15:34, Yury Selivanov  wrote:
> +1 to put both in contextlib.

With the proviso that the implementation shouldn't depend on asyncio.
As Yury says, it should be framework agnostic, let's be careful to
make that the case and not rely on helpers from asyncio, either
deliberately or accidentally.

If writing framework-agnostic versions is difficult, maybe that
implies that some framework-agnostic helpers need to be moved out of
asyncio?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Yury Selivanov]

On 2017-03-01 2:16 AM, Nathaniel Smith wrote:

On Tue, Feb 28, 2017 at 9:42 PM, Nick Coghlan  wrote:

Short version:

- there are some reasonable requests for async variants of contextlib APIs
for 3.7
- prompted by Raymond, I'm thinking it actually makes more sense to add
these in a new `asyncio.contextlib` module than it does to add them directly
to the existing module
- would anyone object strongly to my asking authors of the affected PRs to
take their changes in that direction?

IMHO this is a good idea*iff*  the new APIs really are bound to
asyncio, rather than being generic across all uses of async/await.


I agree.  There is no need to make asynccontextmanager and
AsyncExitStack dependent on asyncio or specific to asyncio.

They should both stay framework agnostic (use only protocols
defined by PEP 492 and PEP 525) and both shouldn't be put
into asyncio package.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Yury Selivanov]

On 2017-03-01 12:42 AM, Nick Coghlan wrote:

Short version:

- there are some reasonable requests for async variants of contextlib APIs
for 3.7
- prompted by Raymond, I'm thinking it actually makes more sense to add
these in a new `asyncio.contextlib` module than it does to add them
directly to the existing module
- would anyone object strongly to my asking authors of the affected PRs to
take their changes in that direction?


Both asynccontextmanager and AsyncExitStack do not require asyncio is 
their implementations.  Using asyncio as a helper to write tests is 
totally OK.  For example, I use asyncio to test asynchronous generators 
(PEP 525).


async/await is a generic language feature; asyncio is a framework that 
uses it.  Things like asynccontextmanager are framework agnostic, they 
can be used in programs built with asyncio, Twisted, Tornado, etc.


+1 to put both in contextlib.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Victor Stinner]

Please don't put code using asyncio in Python stdlib yet. The Python
language is still changing rapidly to get new async features
(async/await keywords, async generators, etc.), and asyncio also
evolved quickly.

I suggest to create 3rd party modules on PyPI. It became easy to pull
dependencies using pip and virtualenv.

It seems like https://github.com/aio-libs is the home of many asyncio libraries.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Nathaniel Smith]

On Tue, Feb 28, 2017 at 9:42 PM, Nick Coghlan  wrote:
> Short version:
>
> - there are some reasonable requests for async variants of contextlib APIs
> for 3.7
> - prompted by Raymond, I'm thinking it actually makes more sense to add
> these in a new `asyncio.contextlib` module than it does to add them directly
> to the existing module
> - would anyone object strongly to my asking authors of the affected PRs to
> take their changes in that direction?

IMHO this is a good idea *iff* the new APIs really are bound to
asyncio, rather than being generic across all uses of async/await.

It sounds like that's not the case, though? There are definitely use
cases for acontextmanager in programs that don't use asyncio at all
(but rather use twisted, curio, ...). Guido's even suggested that he'd
like to see a PEP for an "asyncio2" within the 3.7/3.8 timeframe:
https://mail.python.org/pipermail/async-sig/2016-November/000175.html

asyncio is an important use case for async/await, but it's definitely
not the only one. In cases where it's possible to write generic
machinery in terms of async/await semantics, without assuming any
particular coroutine runner's semantics, then I strongly urge you to
do so.

> Longer version:
>
> There are a couple of open issues requesting async variants of some
> contextlib APIs (asynccontextmanager and AsyncExitStack). I'm inclined to
> accept both of them, but Raymond raised a good question regarding our
> general design philosophy for these kinds of additions: would it make more
> sense to put these in an "asyncio.contextlib" module than it would to add
> them directly to contextlib itself?
>
> The main advantage I see to the idea is that if someone proposed adding an
> "asyncio" dependency to contextlib, I'd say no. For the existing
> asynccontextmanager PR, I even said no to adding that dependency to the
> standard contextlib test suite, and instead asked that the new tests be
> moved out to a separate file, so the existing tests could continue to run
> even if asyncio was unavailable for some reason.

asyncio is a stable, non-provisional part of the standard library;
it's not going anywhere. Personally I wouldn't be bothered about
depending on it for tests. (My async_generator library is in a similar
position: it isn't tied to any particular framework, and I don't even
use asyncio myself, but the test suite depends on asyncio because hey,
whatever, everyone already has it and it plays the role of generic
coroutine runner as well as anything else does.)

OTOH if you don't need to do any I/O then it's actually pretty easy to
write a trivial coroutine runner. I think something like this should
be sufficient to write any test you might want:

@types.coroutine
def send_me(value):
return yield ("value", value)

@types.coroutine
def throw_me(exc):
yield ("error", exc)

async def yield_briefly():
await send_me(None)

def run(async_fn, *args, **kwargs):
coro = async_fn(*args, **kwargs)
next_msg = ("value", None)
try:
while True:
if next_msg[0] == "value":
next_msg = coro.send(next_msg[1])
else:
next_msg = coro.throw(next_msg[1])
except StopIteration as exc:
return exc.value

> While rejecting the idea of an asyncio dependency isn't a problem for
> asyncontextmanager specifically (it's low level enough for it not to
> matter), it's likely to be more of a concern for the AsyncExitStack API,
> where the "asyncio.iscoroutinefunction" introspection API is likely to be
> quite helpful, as are other APIs like `asyncio.ensure_future()`.

FYI FWIW, every time I've tried to use iscoroutinefunction so far I've
ended up regretting it and ripping it out again :-). The problem is
that people will do things like apply a decorator to a coroutine
function, and get a wrapped function that returns a coroutine object
and which is interchangeable with a real coroutine function in every
way except that iscoroutinefunction returns False. And there's no
collections.abc.CoroutineFunction (not sure how that would even work).
Better to just call the thing and then do an isinstance(...,
collections.abc.Coroutine) on the return value.

I haven't had a reason to try porting ExitStack to handle async
context managers yet, so I can't speak to it beyond that :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Ethan Furman]

On 02/28/2017 09:42 PM, Nick Coghlan wrote:


So would folks be OK with my asking the author of the PR for 
https://bugs.python.org/issue29679 (adding
asynccontextmanager) to rewrite the patch to add it as 
asyncio.contextlib.asyncontextmanager (with a cross-reference
from the synchronous contextlib docs), rather than the current approach of 
adding it directly to contextlib?


I like the idea of keep the asyncio stuff in one general location.

--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Nick Coghlan]

Short version:

- there are some reasonable requests for async variants of contextlib APIs
for 3.7
- prompted by Raymond, I'm thinking it actually makes more sense to add
these in a new `asyncio.contextlib` module than it does to add them
directly to the existing module
- would anyone object strongly to my asking authors of the affected PRs to
take their changes in that direction?

Longer version:

There are a couple of open issues requesting async variants of some
contextlib APIs (asynccontextmanager and AsyncExitStack). I'm inclined to
accept both of them, but Raymond raised a good question regarding our
general design philosophy for these kinds of additions: would it make more
sense to put these in an "asyncio.contextlib" module than it would to add
them directly to contextlib itself?

The main advantage I see to the idea is that if someone proposed adding an
"asyncio" dependency to contextlib, I'd say no. For the existing
asynccontextmanager PR, I even said no to adding that dependency to the
standard contextlib test suite, and instead asked that the new tests be
moved out to a separate file, so the existing tests could continue to run
even if asyncio was unavailable for some reason.

While rejecting the idea of an asyncio dependency isn't a problem for
asyncontextmanager specifically (it's low level enough for it not to
matter), it's likely to be more of a concern for the AsyncExitStack API,
where the "asyncio.iscoroutinefunction" introspection API is likely to be
quite helpful, as are other APIs like `asyncio.ensure_future()`.

So would folks be OK with my asking the author of the PR for
https://bugs.python.org/issue29679 (adding asynccontextmanager) to rewrite
the patch to add it as asyncio.contextlib.asyncontextmanager (with a
cross-reference from the synchronous contextlib docs), rather than the
current approach of adding it directly to contextlib?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com