Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Nick Coghlan]

On 8 March 2017 at 04:15, Ethan Furman  wrote:

> On 03/07/2017 09:41 AM, Brett Cannon wrote:
>
> I don't think a common practice has bubbled up yet for when there's both
>> synchronous and asynchronous versions of an API
>> (closest I have seen is appending an "a" to the async version but that
>> just looks like a spelling mistake to me most of
>> the time). This is why the question of whether separate modules are a
>> better idea is coming up.
>>
>
> I'm undoubtedly going to show my ignorance with this question, but is it
> feasible to have both sync and async support in the same object?
>

As Jelle says, it depends on the API. For contextlib, we've already decided
that 'asynccontextmanager' and 'AsyncExitStack' are going to be parallel
APIs, as even though they *could* be the same object, they're much easier
to document if they're separate, and you get a form of "verb agreement" at
both definition time and at runtime that lets us be confident of the
developer's intent.

For simpler APIs like "closing" though, I'm leaning more towards the "just
make it work everywhere" approach, where the async protocol methods use
"await obj.aclose()" if the latter is defined, and a synchronous
"obj.close()" otherwise.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 3.6.1 release status and plans

[Ned Deily]

An update on the 3.6.1 release:  As you probably noticed, 3.6.1 release 
candidate 1 was made available (finally!) two days ago.  Thank you for your 
patience as we worked though the details of producing a release using our new 
GitHub-based development workflow.  As we've noted, it's really important for 
all segments of the community to try using 3.6.1rc1 to help make sure something 
didn't break along the way.  Please report any potential problems via the 
bugs.python.org tracker and mark them as "release blocker". 

Because rc1 was delayed a week, I've moved the planned release date for 3.6.1 
final back a week as well, now 2017-03-20.  That gives two weeks of exposure 
for rc1.  The plan is to, if at all possible, not ship any additional changes 
in the final beyond what is already in rc1 unless we discover any 
release-blocking critical problems in rc1.  The 3.6 branch remains open for new 
cherry-pick PRs etc but you should expect that any PRs that are merged into the 
3.6 branch since the v3.6.1rc1 tag will first be released in 3.6.2, expected 
before the end of June (about 3 months).

Again, if something critical comes up that you feel needs to be in 3.6.1, you 
need to make sure the issue is marked as a "release blocker" and you should 
make sure I am aware of it.  If any such release blockers do arise, we will 
discuss them and decide whether they should go into 3.6.1 and whether a second 
release candidate is needed.

Thanks for your support!

--Ned

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Nathaniel Smith]

On Tue, Mar 7, 2017 at 9:41 AM, Brett Cannon  wrote:
> I don't think a common practice has bubbled up yet for when there's both
> synchronous and asynchronous versions of an API (closest I have seen is
> appending an "a" to the async version but that just looks like a spelling
> mistake to me most of the time). This is why the question of whether
> separate modules are a better idea is coming up.

For the CSV case, it might be sensible to factor out the io. Like,
provide an API that looks like:

pushdictreader = csv.PushDictReader()
while pushdictreader:
chunk = read_some(...)
pushdictreader.push(chunk)
for row in pushdictreader:
...

This API can now straightforwardly be used with sync and async code.
Of course you'd want to wrap it up in a nicer interface, somewhere in
the ballpark of:

def sync_rows(read_some):
pushdictreader = csv.PushDictReader()
while pushdictreader:
chunk = read_some(...)
pushdictreader.push(chunk)
for row in pushdictreader:
yield row

async def async_rows(read_some):
pushdictreader = csv.PushDictReader()
while pushdictreader:
chunk = await read_some(...)
pushdictreader.push(chunk)
for row in pushdictreader:
yield row

So there'd still be a bit of code duplication, but much much less.

Essentially the idea here is to convert the csv module to sans-io
style (http://sans-io.readthedocs.io/).

Another option is to make it all-async internally, and then offer a
sync facade around it. So like start with the natural all-async
interface:

class AsyncFileLike(ABC):
async def async_read(...):
...

class AsyncDictReader:
def __init__(self, async_file_like):
self._async_file_like = async_file_like

async def __anext__(self):
...

And (crucially!) let's assume that the only way AsyncDictReader
interacts with the coroutine runner is by calls to
self._async_file_like.async_read. Now we can pass in a
secretly-actually-synchronous AsyncFileLike and make a synchronous
facade around the whole thing:

class AsyncSyncAdapter(AsyncFileLike):
def __init__(self, sync_file_like):
self._sync_file_like = sync_file_like

# Technically an async function, but guaranteed to never yield
async def read(self, *args, **kwargs):
return self._sync_file_like.read(*args, **kwargs)

# Minimal coroutine supervisor: runs async_fn(*args, **kwargs), which
must never yield
def syncify(async_fn, *args, **kwargs):
coro = async_fn(*args, **kwargs)
it = coro.__await__()
return next(it)

class DictReader:
def __init__(self, sync_file_like):
# Technically an AsyncDictReader, but guaranteed to never yield
self._async_dict_reader =
AsyncDictReader(AsyncSyncAdapter(sync_file_like))

def __next__(self):
return syncify(self._async_dict_reader.__anext__)

So here we still have some goo around the edges of the module, but the
actual CSV logic only has to be written once, and can still be written
in a "pull" style where it does its own I/O, just like it is now.

This is basically another approach to writing sans-io protocols, with
the annoying trade-off that it means even your synchronous version
requires Python 3.5+. But for a stdlib module that's no big deal...

-n


> On Tue, 7 Mar 2017 at 02:24 Michel Desmoulin 
> wrote:
>>
>> Last week I had to download a CSV from an FTP and push any update on it
>> using websocket so asyncio was a natural fit and the network part went
>> well.
>>
>> The surprise was that the CSV part would not work as expected. Usually I
>> read csv doing:
>>
>> import csv
>>
>> file_like_object = csv_crawler.get_file()
>> for row in csv.DictReader(file_like_object)
>>
>> But it didn't work because file_like_object.read() was a coroutine which
>> the csv module doesn't handle.
>>
>> So I had to do:
>>
>> import csv
>> import io
>>
>> raw_bytes = await stream.read(1000)
>> wrapped_bytes = io.BytesIO(raw_bytes)
>> text = io.TextIOWrapper(wrapped_bytes, encoding=encoding,
>> errors='replace')
>>
>> for i, row in enumerate(csv.DictReader(text)):
>>
>> Turns out I used asyncio a bit, and I now the stdlib, the io AIP, etc.
>> But for somebody that doesn't, it's not very easy to figure out. Plus
>> it's not as elegant as traditional Python. Not to mention it loads the
>> entire CSV in memory.
>>
>> So I wondered if I could fix the csv module so it accept async. But the
>> question arised. Where should I put it ?
>>
>> - Create AsyncDictReader and AsyncReader ?
>> - Add inspect.iscoroutine calls widh it in the regular Readers and some
>> __aiter__ and __aenter__ ?
>> - add a csv.async namespace ?
>>
>> What API design are we recommanding for expose both sync and async
>> behaviors ?
>>
>>
>> Le 07/03/2017 à 03:08, Guido van Rossum a écrit :
>> > On Mon, Mar 6, 2017 at 5:57 PM, Raymond Hettinger
>> > 

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Sven R. Kunze]

On 07.03.2017 19:37, Jelle Zijlstra wrote:



2017-03-07 10:15 GMT-08:00 Ethan Furman >:


On 03/07/2017 09:41 AM, Brett Cannon wrote:

I don't think a common practice has bubbled up yet for when
there's both synchronous and asynchronous versions of an API
(closest I have seen is appending an "a" to the async version
but that just looks like a spelling mistake to me most of
the time). This is why the question of whether separate
modules are a better idea is coming up.


I'm undoubtedly going to show my ignorance with this question, but
is it feasible to have both sync and async support in the same object?

It's possible, but it quickly gets awkward and will require a lot of 
code duplication.


Correct me if I'm wrong, but we would get the code duplication anyway.

async intrinsically does the same thing (just a little bit different) as 
its sync counterpart. Otherwise, you wouldn't use it.


For example, we could make @contextmanager work for async functions by 
making the _GeneratorContextManager class implement both enter/exit 
and aenter/aexit, but then you'd get an obscure error if you used with 
on an async contextmanager or async with on a non-async contextmanager.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Jelle Zijlstra]

2017-03-07 10:15 GMT-08:00 Ethan Furman :

> On 03/07/2017 09:41 AM, Brett Cannon wrote:
>
> I don't think a common practice has bubbled up yet for when there's both
>> synchronous and asynchronous versions of an API
>> (closest I have seen is appending an "a" to the async version but that
>> just looks like a spelling mistake to me most of
>> the time). This is why the question of whether separate modules are a
>> better idea is coming up.
>>
>
> I'm undoubtedly going to show my ignorance with this question, but is it
> feasible to have both sync and async support in the same object?
>
> It's possible, but it quickly gets awkward and will require a lot of code
duplication. For example, we could make @contextmanager work for async
functions by making the _GeneratorContextManager class implement both
enter/exit and aenter/aexit, but then you'd get an obscure error if you
used with on an async contextmanager or async with on a non-async
contextmanager.


> --
> ~Ethan~
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jelle.
> zijlstra%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Ethan Furman]

On 03/07/2017 09:41 AM, Brett Cannon wrote:


I don't think a common practice has bubbled up yet for when there's both 
synchronous and asynchronous versions of an API
(closest I have seen is appending an "a" to the async version but that just 
looks like a spelling mistake to me most of
the time). This is why the question of whether separate modules are a better 
idea is coming up.


I'm undoubtedly going to show my ignorance with this question, but is it feasible to have both sync and async support in 
the same object?


--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Brett Cannon]

I don't think a common practice has bubbled up yet for when there's both
synchronous and asynchronous versions of an API (closest I have seen is
appending an "a" to the async version but that just looks like a spelling
mistake to me most of the time). This is why the question of whether
separate modules are a better idea is coming up.

On Tue, 7 Mar 2017 at 02:24 Michel Desmoulin 
wrote:

> Last week I had to download a CSV from an FTP and push any update on it
> using websocket so asyncio was a natural fit and the network part went
> well.
>
> The surprise was that the CSV part would not work as expected. Usually I
> read csv doing:
>
> import csv
>
> file_like_object = csv_crawler.get_file()
> for row in csv.DictReader(file_like_object)
>
> But it didn't work because file_like_object.read() was a coroutine which
> the csv module doesn't handle.
>
> So I had to do:
>
> import csv
> import io
>
> raw_bytes = await stream.read(1000)
> wrapped_bytes = io.BytesIO(raw_bytes)
> text = io.TextIOWrapper(wrapped_bytes, encoding=encoding,
> errors='replace')
>
> for i, row in enumerate(csv.DictReader(text)):
>
> Turns out I used asyncio a bit, and I now the stdlib, the io AIP, etc.
> But for somebody that doesn't, it's not very easy to figure out. Plus
> it's not as elegant as traditional Python. Not to mention it loads the
> entire CSV in memory.
>
> So I wondered if I could fix the csv module so it accept async. But the
> question arised. Where should I put it ?
>
> - Create AsyncDictReader and AsyncReader ?
> - Add inspect.iscoroutine calls widh it in the regular Readers and some
> __aiter__ and __aenter__ ?
> - add a csv.async namespace ?
>
> What API design are we recommanding for expose both sync and async
> behaviors ?
>
>
> Le 07/03/2017 à 03:08, Guido van Rossum a écrit :
> > On Mon, Mar 6, 2017 at 5:57 PM, Raymond Hettinger
> > >
> wrote:
> >
> > Of course, it makes sense that anything not specific to asyncio
> > should go outside of asyncio.
> >
> > What I'm more concerned about is what the other places actually
> > are.   Rather than putting async variants of everything sprinkled
> > all over the standard library, I suggest collecting them all
> > together, perhaps in a new asynctools module.
> >
> >
> > That's a tough design choice. I think neither extreme is particularly
> > attractive -- having everything in an asynctools package might also
> > bundle together thing that are entirely unrelated. In the extreme it
> > would be like proposing that all metaclasses should go in a new
> > "metaclasstools" package. I think we did a reasonable job with ABCs:
> > core support goes in abc.py, support for collections ABCs goes into the
> > collections package (in a submodule), and other packages and modules
> > sometimes define ABCs for their own users.
> >
> > Also, in some cases I expect we'll have to create a whole new module
> > instead of updating some ancient piece of code with newfangled async
> > variants to its outdated APIs.
> >
> > --
> > --Guido van Rossum (python.org/~guido )
> >
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.com
> >
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Michel Desmoulin]

Last week I had to download a CSV from an FTP and push any update on it
using websocket so asyncio was a natural fit and the network part went well.

The surprise was that the CSV part would not work as expected. Usually I
read csv doing:

import csv

file_like_object = csv_crawler.get_file()
for row in csv.DictReader(file_like_object)

But it didn't work because file_like_object.read() was a coroutine which
the csv module doesn't handle.

So I had to do:

import csv
import io

raw_bytes = await stream.read(1000)
wrapped_bytes = io.BytesIO(raw_bytes)
text = io.TextIOWrapper(wrapped_bytes, encoding=encoding,
errors='replace')

for i, row in enumerate(csv.DictReader(text)):

Turns out I used asyncio a bit, and I now the stdlib, the io AIP, etc.
But for somebody that doesn't, it's not very easy to figure out. Plus
it's not as elegant as traditional Python. Not to mention it loads the
entire CSV in memory.

So I wondered if I could fix the csv module so it accept async. But the
question arised. Where should I put it ?

- Create AsyncDictReader and AsyncReader ?
- Add inspect.iscoroutine calls widh it in the regular Readers and some
__aiter__ and __aenter__ ?
- add a csv.async namespace ?

What API design are we recommanding for expose both sync and async
behaviors ?


Le 07/03/2017 à 03:08, Guido van Rossum a écrit :
> On Mon, Mar 6, 2017 at 5:57 PM, Raymond Hettinger
> > wrote:
> 
> Of course, it makes sense that anything not specific to asyncio
> should go outside of asyncio.
> 
> What I'm more concerned about is what the other places actually
> are.   Rather than putting async variants of everything sprinkled
> all over the standard library, I suggest collecting them all
> together, perhaps in a new asynctools module.
> 
> 
> That's a tough design choice. I think neither extreme is particularly
> attractive -- having everything in an asynctools package might also
> bundle together thing that are entirely unrelated. In the extreme it
> would be like proposing that all metaclasses should go in a new
> "metaclasstools" package. I think we did a reasonable job with ABCs:
> core support goes in abc.py, support for collections ABCs goes into the
> collections package (in a submodule), and other packages and modules
> sometimes define ABCs for their own users.
> 
> Also, in some cases I expect we'll have to create a whole new module
> instead of updating some ancient piece of code with newfangled async
> variants to its outdated APIs.
> 
> -- 
> --Guido van Rossum (python.org/~guido )
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.com
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com