Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Nick Coghlan]

On 2 March 2017 at 02:29, Barry Warsaw  wrote:

> On Mar 01, 2017, at 10:55 AM, Victor Stinner wrote:
>
> >I suggest to create 3rd party modules on PyPI. It became easy to pull
> >dependencies using pip and virtualenv.
> >
> >It seems like https://github.com/aio-libs is the home of many asyncio
> >libraries.
>
> This is what we did for aiosmtpd, an asyncio-based replacement for smtpd.
> It's worked out great on all fronts so far (good community contributions,
> rapid development, API flexibility as we move toward 1.0, good visibility
> inside the more general aio-libs umbrella).
>

While I agree with this approach for higher level stuff, it's specifically
the lower level pieces that just interact with the async/await language
features rather than the event loop itself where I needed some discussion
to clarify my own thoughts :)

My conclusion from the thread is:

- if it needs to depend on asyncio, it should either go in asyncio, or be
published as a third party aio-lib
- if it *doesn't* need to depend on asyncio, then it's a candidate for
stdlib inclusion (e.g. the coroutine support in inspect)
- both asynccontextmanager and AsyncExitStack actually fall into the latter
category
- other contextlib APIs like closing() should be able to transparently
support both the sync and async variants of the CM protocol without
negatively affecting the synchronous version
- so for the specific case of contextlib, supporting both synchronous and
asynchronous contexts in the one module makes sense
- I still plan to keep the test cases separate, since the async test cases
need more infrastructure than the synchronous ones

What we shouldn't do is take this design decision as setting a binding
precedent for any other modules like itertools - the trade-offs there are
going to be different, and there are already third party modules like
https://github.com/asyncdef/aitertools that provide equivalent APIs for the
asynchronous programming model.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help requested with Python 2.7 performance regression

[Louis Bouchard]

Hello,

Le 01/03/2017 à 20:40, Antoine Pitrou a écrit :
> On Wed, 1 Mar 2017 20:24:03 +0100
> Louis Bouchard  wrote:
>>
>> Indeed, this is something that is in the history of the LP bug so here is the
>> URL where those comparison can be found :
>>
>> https://docs.google.com/spreadsheets/d/1MyNBPVZlBeic1OLqVKe_bcPk2deO_pQs9trIfOFefM0/edit#gid=2034603487
> 
> Some more questions:
> * what does "faster" or "slower" mean (that is, which one is faster)?
> * is it possible to have actual performance differences in percent?
>   being 2% slower is not the same as being 30% slower...
> 

This means that the second element of the test is slower than the first. For
instance if the test is Trusty stock .vs. Xenial stock and it shows slower, it
means that Xenial stock is slower than Trusty stock.

This is directly taken from the output of "pyperformance compare". The third
column of each comparison (1.x) gives the proportion figure of the test.

A test that shows slower 1.14 is 14% slower.

HTH,

Kind regards,

...Louis


-- 
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer   Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63  B9C4 8B3D 867C 823E 7A61
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help requested with Python 2.7 performance regression

[Victor Stinner]

Hi,

Your document doesn't explain how you configured the host to run
benchmarks. Maybe you didn't tune Linux or anything else? Be careful
with modern hardware which can make funny (or not) surprises. See my
recent talk at FOSDEM (last month):

"How to run a stable benchmark"
https://fosdem.org/2017/schedule/event/python_stable_benchmark/

Factors impacting Python benchmarks:

* Linux Address Space Layout Randomization (ASRL),
/proc/sys/kernel/randomize_va_space
* Python random hash function: PYTHONHASHSEED
* Command line arguments and environmnet variables: enabling ASLR helps here (?)
* CPU power saving and performance features: disable Intel Turbo Boost
and/or use a fixed CPU frequency.
* Temperature: temperature has a limited impact on benchmarks. If the
CPU is below 95°C, Intel CPUs still run at full speed. With a correct
cooling system, temperature is not an issue.
* Linux perf probes: /proc/sys/kernel/perf_event_max_sample_rate
* Code locality, CPU L1 instruction cache (L1c): Profiled Guided
Optimization (PGO) helps here
* Other processes and the kernel, CPU isolation (CPU pinning) helps
here: use isolcpus=cpu_list and rcu_nocbs=cpu_list on the * Linux
kernel command line
* ... Reboot? Sadly, other unknown factors may still impact
benchmarks. Sometimes, it helps to reboot to restore standard
performances.

https://haypo-notes.readthedocs.io/microbenchmark.html#factors-impacting-benchmarks

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help requested with Python 2.7 performance regression

[Antoine Pitrou]

On Wed, 1 Mar 2017 20:24:03 +0100
Louis Bouchard  wrote:
> 
> Indeed, this is something that is in the history of the LP bug so here is the
> URL where those comparison can be found :
> 
> https://docs.google.com/spreadsheets/d/1MyNBPVZlBeic1OLqVKe_bcPk2deO_pQs9trIfOFefM0/edit#gid=2034603487

Some more questions:
* what does "faster" or "slower" mean (that is, which one is faster)?
* is it possible to have actual performance differences in percent?
  being 2% slower is not the same as being 30% slower...

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help requested with Python 2.7 performance regression

[Louis Bouchard]

Hello,

Le 01/03/2017 à 18:51, Antoine Pitrou a écrit :
> On Wed, 1 Mar 2017 12:28:24 -0500
> Barry Warsaw  wrote:
>>
>> Louis (Cc'd here) has done a ton of work to measure and analyze the problem,
>> but we've more or less hit a roadblock, so we're taking the issue public to
>> see if anybody on this mailing list has further ideas.  A detailed analysis 
>> is
>> available in this Google doc:
>>
>> https://docs.google.com/document/d/1zrV3OIRSo99fd2Ty4YdGk_scmTRDmVauBprKL8eij6w/edit
>>
>> The document should be public for comments and editing.
> 
> I may be misunderstanding the document, but this lacks at least a
> comparison of the *same* interpreter version with different compiler
> versions.
> 
> As for the high level: what if the training set used for PGO in Xenial
> has become skewed or inadequate?  Just a thought, as it would imply
> that PGO+LTO uses wrong information for code placement and other
> optimizations.
> 
> Regards
> 
> Antoine.
> 

Indeed, this is something that is in the history of the LP bug so here is the
URL where those comparison can be found :

https://docs.google.com/spreadsheets/d/1MyNBPVZlBeic1OLqVKe_bcPk2deO_pQs9trIfOFefM0/edit#gid=2034603487

Hope it can help,

Kind regards,

...Louis


-- 
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer   Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63  B9C4 8B3D 867C 823E 7A61
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help requested with Python 2.7 performance regression

[Antoine Pitrou]

On Wed, 1 Mar 2017 19:58:14 +0100
Matthias Klose  wrote:
> On 01.03.2017 18:51, Antoine Pitrou wrote:
> > As for the high level: what if the training set used for PGO in Xenial
> > has become skewed or inadequate?  
> 
> running the testsuite

I did some tests a year or two ago, and running the whole test suite is
not a good idea, as coverage varies wildly from one functionality to the
other, so PGO will not infer the right information from it.  You don't
get very good benchmark results from it.

(for example, decimal has an extensive test suite which might lead PGO
to believe that code paths exercised by the decimal module are the
hottest ones)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help requested with Python 2.7 performance regression

[Matthias Klose]

On 01.03.2017 18:51, Antoine Pitrou wrote:
> As for the high level: what if the training set used for PGO in Xenial
> has become skewed or inadequate?

running the testsuite

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python-3.6.0 compilation error

[Brett Cannon]

This mailing list is for the development **of** Python, not **with** it.
For build failures like this it's best to ask on python-list (and if you
search for the missing libintl_* symbols you will find they come from
gettext).

On Wed, 1 Mar 2017 at 08:10 Porel, Subrata  wrote:

> Hi Team,
>
>
>
> While trying to compile the Python-3.6.0 in our Solaris 10 server to
> install this beside python 2.6.4 we have got below error: Please help on
> this.
>
>
>
> root@xxyyzz:/opt/Python-3.6.0#python
>
> Python 2.6.4 (r264:75706, Jun 27 2012, 05:45:50) [C] on sunos5
>
> Type "help", "copyright", "credits" or "license" for more information.
>
> >>> ^D
>
> root@xxyyzz:/opt/Python-3.6.0#make test
>
> gcc   -o Programs/_freeze_importlib Programs/_freeze_importlib.o
> Modules/getbuildinfo.o  Parser/acceler.o  Parser/grammar1.o
> Parser/listnode.o  Parser/node.o  Parser/parser.o  Parser/bitset.o
> Parser/metagrammar.o  Parser/firstsets.o  Parser/grammar.o  Parser/pgen.o
> Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o
> Objects/abstract.o  Objects/accu.o  Objects/boolobject.o
> Objects/bytes_methods.o  Objects/bytearrayobject.o  Objects/bytesobject.o
> Objects/cellobject.o  Objects/classobject.o  Objects/codeobject.o
> Objects/complexobject.o  Objects/descrobject.o  Objects/enumobject.o
> Objects/exceptions.o  Objects/genobject.o  Objects/fileobject.o
> Objects/floatobject.o  Objects/frameobject.o  Objects/funcobject.o
> Objects/iterobject.o  Objects/listobject.o  Objects/longobject.o
> Objects/dictobject.o  Objects/odictobject.o  Objects/memoryobject.o
> Objects/methodobject.o  Objects/moduleobject.o  Objects/namespaceobject.o
> Objects/object.o  Objects/obmalloc.o  Objects/capsule.o
> Objects/rangeobject.o  Objects/setobject.o  Objects/sliceobject.o
> Objects/structseq.o  Objects/tupleobject.o  Objects/typeobject.o
> Objects/unicodeobject.o  Objects/unicodectype.o  Objects/weakrefobject.o
> Python/_warnings.o  Python/Python-ast.o  Python/asdl.o  Python/ast.o
> Python/bltinmodule.o  Python/ceval.o  Python/compile.o  Python/codecs.o
> Python/dynamic_annotations.o  Python/errors.o  Python/frozenmain.o
> Python/future.o  Python/getargs.o  Python/getcompiler.o
> Python/getcopyright.o  Python/getplatform.o  Python/getversion.o
> Python/graminit.o  Python/import.o  Python/importdl.o  Python/marshal.o
> Python/modsupport.o  Python/mystrtoul.o  Python/mysnprintf.o
> Python/peephole.o  Python/pyarena.o  Python/pyctype.o  Python/pyfpe.o
> Python/pyhash.o  Python/pylifecycle.o  Python/pymath.o  Python/pystate.o
> Python/pythonrun.o  Python/pytime.o  Python/random.o
> Python/structmember.o  Python/symtable.o  Python/sysmodule.o
> Python/traceback.o  Python/getopt.o  Python/pystrcmp.o  Python/pystrtod.o
> Python/pystrhex.o  Python/dtoa.o  Python/formatter_unicode.o
> Python/fileutils.o  Python/dynload_shlib.o  Python/thread.o
> Modules/config.o  Modules/getpath.o  Modules/main.o  Modules/gcmodule.o
> Modules/_threadmodule.o  Modules/posixmodule.o  Modules/errnomodule.o
>  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o
> Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/_operator.o
> Modules/_collectionsmodule.o  Modules/itertoolsmodule.o
> Modules/atexitmodule.o  Modules/signalmodule.o  Modules/_stat.o
> Modules/timemodule.o  Modules/_localemodule.o  Modules/_iomodule.o
> Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o
> Modules/textio.o Modules/stringio.o  Modules/zipimport.o
> Modules/faulthandler.o  Modules/_tracemalloc.o Modules/hashtable.o
> Modules/symtablemodule.o  Modules/xxsubtype.o -lsocket -lnsl -lintl -lrt
> -ldl -lsendfile   -lm
>
> Undefined   first referenced
>
> symbol in file
>
> libintl_bind_textdomain_codeset Modules/_localemodule.o
>
> libintl_gettext Modules/_localemodule.o
>
> libintl_textdomain  Modules/_localemodule.o
>
> libintl_dcgettext   Modules/_localemodule.o
>
> libintl_bindtextdomain  Modules/_localemodule.o
>
> libintl_dgettextModules/_localemodule.o
>
> ld: fatal: symbol referencing errors. No output written to
> Programs/_freeze_importlib
>
> collect2: ld returned 1 exit status
>
> *** Error code 1
>
> make: Fatal error: Command failed for target `Programs/_freeze_importlib'
>
> root@xxyyzz:/opt/Python-3.6.0#gcc
>
> gcc: no input files
>
> root@xxyyzz:/opt/Python-3.6.0#which gcc
>
> /usr/sfw/bin/gcc
>
> root@fcudrpt01:/opt/Python-3.6.0#gcc --version
>
> gcc (GCC) 3.4.3 (csl-sol210-3_4-branch+sol_rpath)
>
> Copyright (C) 2004 Free Software Foundation, Inc.
>
> This is free software; see the source for copying conditions.  There is NO
>
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
>
>
>
>
> Regards,
>
> Subrata Porel
>
> Tronc TCS UNIX Admin
>
> TCS Gitanjali Park, Kolkata, India
> Email: spo...@tronc.com
> Mobile: +91 798 036 7240 <+91%2079803%2067240>
> 

Re: [Python-Dev] Help requested with Python 2.7 performance regression

[Antoine Pitrou]

On Wed, 1 Mar 2017 12:28:24 -0500
Barry Warsaw  wrote:
> 
> Louis (Cc'd here) has done a ton of work to measure and analyze the problem,
> but we've more or less hit a roadblock, so we're taking the issue public to
> see if anybody on this mailing list has further ideas.  A detailed analysis is
> available in this Google doc:
> 
> https://docs.google.com/document/d/1zrV3OIRSo99fd2Ty4YdGk_scmTRDmVauBprKL8eij6w/edit
> 
> The document should be public for comments and editing.

I may be misunderstanding the document, but this lacks at least a
comparison of the *same* interpreter version with different compiler
versions.

As for the high level: what if the training set used for PGO in Xenial
has become skewed or inadequate?  Just a thought, as it would imply
that PGO+LTO uses wrong information for code placement and other
optimizations.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Help requested with Python 2.7 performance regression

[Barry Warsaw]

Hello all,

Over in Ubuntu, we've gotten reports about some performance regressions in
Python 2.7 when moving from Trusty (14.04 LTS) to Xenial (16.04 LTS).
Trusty's version is based on 2.7.6 while Xenial's version is based on 2.7.12
with bits of .13 cherry picked.

We've not been able to identify any change in Python itself (or the
Debian/Ubuntu deltas) which could account for this, so the investigation has
led to various gcc compiler options and version differences.  In particular
disabling LTO (link-time optimization) seems to have a positive impact, but
doesn't completely regain the loss.

Louis (Cc'd here) has done a ton of work to measure and analyze the problem,
but we've more or less hit a roadblock, so we're taking the issue public to
see if anybody on this mailing list has further ideas.  A detailed analysis is
available in this Google doc:

https://docs.google.com/document/d/1zrV3OIRSo99fd2Ty4YdGk_scmTRDmVauBprKL8eij6w/edit

The document should be public for comments and editing.

If you have any thoughts, or other lines of investigation you think are
worthwhile pursuing, please add your comments to the document.

Cheers,
-Barry


pgpbIf0pajZds.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Barry Warsaw]

On Mar 01, 2017, at 10:55 AM, Victor Stinner wrote:

>I suggest to create 3rd party modules on PyPI. It became easy to pull
>dependencies using pip and virtualenv.
>
>It seems like https://github.com/aio-libs is the home of many asyncio
>libraries.

This is what we did for aiosmtpd, an asyncio-based replacement for smtpd.
It's worked out great on all fronts so far (good community contributions,
rapid development, API flexibility as we move toward 1.0, good visibility
inside the more general aio-libs umbrella).

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python-3.6.0 compilation error

[Porel, Subrata]

Hi Team,

While trying to compile the Python-3.6.0 in our Solaris 10 server to install 
this beside python 2.6.4 we have got below error: Please help on this.

root@xxyyzz:/opt/Python-3.6.0#python
Python 2.6.4 (r264:75706, Jun 27 2012, 05:45:50) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
root@xxyyzz:/opt/Python-3.6.0#make test
gcc   -o Programs/_freeze_importlib Programs/_freeze_importlib.o 
Modules/getbuildinfo.o  Parser/acceler.o  Parser/grammar1.o  Parser/listnode.o  
Parser/node.o  Parser/parser.o  Parser/bitset.o  Parser/metagrammar.o  
Parser/firstsets.o  Parser/grammar.o  Parser/pgen.o Parser/myreadline.o 
Parser/parsetok.o Parser/tokenizer.o  Objects/abstract.o  Objects/accu.o  
Objects/boolobject.o  Objects/bytes_methods.o  Objects/bytearrayobject.o  
Objects/bytesobject.o  Objects/cellobject.o  Objects/classobject.o  
Objects/codeobject.o  Objects/complexobject.o  Objects/descrobject.o  
Objects/enumobject.o  Objects/exceptions.o  Objects/genobject.o  
Objects/fileobject.o  Objects/floatobject.o  Objects/frameobject.o  
Objects/funcobject.o  Objects/iterobject.o  Objects/listobject.o  
Objects/longobject.o  Objects/dictobject.o  Objects/odictobject.o  
Objects/memoryobject.o  Objects/methodobject.o  Objects/moduleobject.o  
Objects/namespaceobject.o  Objects/object.o  Objects/obmalloc.o  
Objects/capsule.o  Objects/rangeobject.o  Objects/setobject.o  
Objects/sliceobject.o  Objects/structseq.o  Objects/tupleobject.o  
Objects/typeobject.o  Objects/unicodeobject.o  Objects/unicodectype.o  
Objects/weakrefobject.o  Python/_warnings.o  Python/Python-ast.o  Python/asdl.o 
 Python/ast.o  Python/bltinmodule.o  Python/ceval.o  Python/compile.o  
Python/codecs.o  Python/dynamic_annotations.o  Python/errors.o  
Python/frozenmain.o  Python/future.o  Python/getargs.o  Python/getcompiler.o  
Python/getcopyright.o  Python/getplatform.o  Python/getversion.o  
Python/graminit.o  Python/import.o  Python/importdl.o  Python/marshal.o  
Python/modsupport.o  Python/mystrtoul.o  Python/mysnprintf.o  Python/peephole.o 
 Python/pyarena.o  Python/pyctype.o  Python/pyfpe.o  Python/pyhash.o  
Python/pylifecycle.o  Python/pymath.o  Python/pystate.o  Python/pythonrun.o  
Python/pytime.o  Python/random.o  Python/structmember.o  Python/symtable.o  
Python/sysmodule.o  Python/traceback.o  Python/getopt.o  Python/pystrcmp.o  
Python/pystrtod.o  Python/pystrhex.o  Python/dtoa.o  Python/formatter_unicode.o 
 Python/fileutils.o  Python/dynload_shlib.o  Python/thread.o
Modules/config.o  Modules/getpath.o  Modules/main.o  Modules/gcmodule.o  
Modules/_threadmodule.o  Modules/posixmodule.o  Modules/errnomodule.o  
Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  
Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/_operator.o  
Modules/_collectionsmodule.o  Modules/itertoolsmodule.o  Modules/atexitmodule.o 
 Modules/signalmodule.o  Modules/_stat.o  Modules/timemodule.o  
Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o 
Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  
Modules/zipimport.o  Modules/faulthandler.o  Modules/_tracemalloc.o 
Modules/hashtable.o  Modules/symtablemodule.o  Modules/xxsubtype.o -lsocket 
-lnsl -lintl -lrt -ldl -lsendfile   -lm
Undefined   first referenced
symbol in file
libintl_bind_textdomain_codeset Modules/_localemodule.o
libintl_gettext Modules/_localemodule.o
libintl_textdomain  Modules/_localemodule.o
libintl_dcgettext   Modules/_localemodule.o
libintl_bindtextdomain  Modules/_localemodule.o
libintl_dgettextModules/_localemodule.o
ld: fatal: symbol referencing errors. No output written to 
Programs/_freeze_importlib
collect2: ld returned 1 exit status
*** Error code 1
make: Fatal error: Command failed for target `Programs/_freeze_importlib'
root@xxyyzz:/opt/Python-3.6.0#gcc
gcc: no input files
root@xxyyzz:/opt/Python-3.6.0#which gcc
/usr/sfw/bin/gcc
root@fcudrpt01:/opt/Python-3.6.0#gcc --version
gcc (GCC) 3.4.3 (csl-sol210-3_4-branch+sol_rpath)
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


Regards,
Subrata Porel
Tronc TCS UNIX Admin
TCS Gitanjali Park, Kolkata, India
Email: spo...@tronc.com
Mobile: +91 798 036 7240
Work Phone: +1 321 270 7685

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Paul Moore]

On 1 March 2017 at 15:34, Yury Selivanov  wrote:
> +1 to put both in contextlib.

With the proviso that the implementation shouldn't depend on asyncio.
As Yury says, it should be framework agnostic, let's be careful to
make that the case and not rely on helpers from asyncio, either
deliberately or accidentally.

If writing framework-agnostic versions is difficult, maybe that
implies that some framework-agnostic helpers need to be moved out of
asyncio?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Yury Selivanov]

On 2017-03-01 2:16 AM, Nathaniel Smith wrote:

On Tue, Feb 28, 2017 at 9:42 PM, Nick Coghlan  wrote:

Short version:

- there are some reasonable requests for async variants of contextlib APIs
for 3.7
- prompted by Raymond, I'm thinking it actually makes more sense to add
these in a new `asyncio.contextlib` module than it does to add them directly
to the existing module
- would anyone object strongly to my asking authors of the affected PRs to
take their changes in that direction?

IMHO this is a good idea*iff*  the new APIs really are bound to
asyncio, rather than being generic across all uses of async/await.


I agree.  There is no need to make asynccontextmanager and
AsyncExitStack dependent on asyncio or specific to asyncio.

They should both stay framework agnostic (use only protocols
defined by PEP 492 and PEP 525) and both shouldn't be put
into asyncio package.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants ofexisting stdlib APIs?

[Steve Dower]

Big +1 here, and an implicit -1 on the other suggestions. 

asyncio != async/await

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Nathaniel Smith" 
Sent: ‎2/‎28/‎2017 23:19
To: "Nick Coghlan" 
Cc: "python-dev@python.org" 
Subject: Re: [Python-Dev] API design: where to add async variants ofexisting 
stdlib APIs?

On Tue, Feb 28, 2017 at 9:42 PM, Nick Coghlan  wrote:
> Short version:
>
> - there are some reasonable requests for async variants of contextlib APIs
> for 3.7
> - prompted by Raymond, I'm thinking it actually makes more sense to add
> these in a new `asyncio.contextlib` module than it does to add them directly
> to the existing module
> - would anyone object strongly to my asking authors of the affected PRs to
> take their changes in that direction?

IMHO this is a good idea *iff* the new APIs really are bound to
asyncio, rather than being generic across all uses of async/await.

It sounds like that's not the case, though? There are definitely use
cases for acontextmanager in programs that don't use asyncio at all
(but rather use twisted, curio, ...). Guido's even suggested that he'd
like to see a PEP for an "asyncio2" within the 3.7/3.8 timeframe:
https://mail.python.org/pipermail/async-sig/2016-November/000175.html

asyncio is an important use case for async/await, but it's definitely
not the only one. In cases where it's possible to write generic
machinery in terms of async/await semantics, without assuming any
particular coroutine runner's semantics, then I strongly urge you to
do so.

> Longer version:
>
> There are a couple of open issues requesting async variants of some
> contextlib APIs (asynccontextmanager and AsyncExitStack). I'm inclined to
> accept both of them, but Raymond raised a good question regarding our
> general design philosophy for these kinds of additions: would it make more
> sense to put these in an "asyncio.contextlib" module than it would to add
> them directly to contextlib itself?
>
> The main advantage I see to the idea is that if someone proposed adding an
> "asyncio" dependency to contextlib, I'd say no. For the existing
> asynccontextmanager PR, I even said no to adding that dependency to the
> standard contextlib test suite, and instead asked that the new tests be
> moved out to a separate file, so the existing tests could continue to run
> even if asyncio was unavailable for some reason.

asyncio is a stable, non-provisional part of the standard library;
it's not going anywhere. Personally I wouldn't be bothered about
depending on it for tests. (My async_generator library is in a similar
position: it isn't tied to any particular framework, and I don't even
use asyncio myself, but the test suite depends on asyncio because hey,
whatever, everyone already has it and it plays the role of generic
coroutine runner as well as anything else does.)

OTOH if you don't need to do any I/O then it's actually pretty easy to
write a trivial coroutine runner. I think something like this should
be sufficient to write any test you might want:

@types.coroutine
def send_me(value):
return yield ("value", value)

@types.coroutine
def throw_me(exc):
yield ("error", exc)

async def yield_briefly():
await send_me(None)

def run(async_fn, *args, **kwargs):
coro = async_fn(*args, **kwargs)
next_msg = ("value", None)
try:
while True:
if next_msg[0] == "value":
next_msg = coro.send(next_msg[1])
else:
next_msg = coro.throw(next_msg[1])
except StopIteration as exc:
return exc.value

> While rejecting the idea of an asyncio dependency isn't a problem for
> asyncontextmanager specifically (it's low level enough for it not to
> matter), it's likely to be more of a concern for the AsyncExitStack API,
> where the "asyncio.iscoroutinefunction" introspection API is likely to be
> quite helpful, as are other APIs like `asyncio.ensure_future()`.

FYI FWIW, every time I've tried to use iscoroutinefunction so far I've
ended up regretting it and ripping it out again :-). The problem is
that people will do things like apply a decorator to a coroutine
function, and get a wrapped function that returns a coroutine object
and which is interchangeable with a real coroutine function in every
way except that iscoroutinefunction returns False. And there's no
collections.abc.CoroutineFunction (not sure how that would even work).
Better to just call the thing and then do an isinstance(...,
collections.abc.Coroutine) on the return value.

I haven't had a reason to try porting ExitStack to handle async
context managers yet, so I can't speak to it beyond that :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Yury Selivanov]

On 2017-03-01 12:42 AM, Nick Coghlan wrote:

Short version:

- there are some reasonable requests for async variants of contextlib APIs
for 3.7
- prompted by Raymond, I'm thinking it actually makes more sense to add
these in a new `asyncio.contextlib` module than it does to add them
directly to the existing module
- would anyone object strongly to my asking authors of the affected PRs to
take their changes in that direction?


Both asynccontextmanager and AsyncExitStack do not require asyncio is 
their implementations.  Using asyncio as a helper to write tests is 
totally OK.  For example, I use asyncio to test asynchronous generators 
(PEP 525).


async/await is a generic language feature; asyncio is a framework that 
uses it.  Things like asynccontextmanager are framework agnostic, they 
can be used in programs built with asyncio, Twisted, Tornado, etc.


+1 to put both in contextlib.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Victor Stinner]

Please don't put code using asyncio in Python stdlib yet. The Python
language is still changing rapidly to get new async features
(async/await keywords, async generators, etc.), and asyncio also
evolved quickly.

I suggest to create 3rd party modules on PyPI. It became easy to pull
dependencies using pip and virtualenv.

It seems like https://github.com/aio-libs is the home of many asyncio libraries.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com