Re: [Python-Dev] Migrate python-dev to Mailman 3?

[Terry Reedy]

On 11/1/2017 11:06 PM, Guido van Rossum wrote:
Another one is core-mentorship, which satisfies the same criteria; and 
in my view this has the added and useful property that its beneficiaries 
are non-core members. After that I'd do core-workflow. Honestly I'd 
leave python-committers alone for a while, we're a curmudgeonly group. :-)


As an idledev admin, I also volunteer it.  It has an archive but is 
currently dormant, so it could be shut down for even a couple of weeks 
to practice archive conversion and test messages.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reminder: 12 weeks to 3.7 feature code cutoff

[Lele Gaifax]

Hi,

I'd like to know what should I do wrt to issue #27645 [1] and the related PR
#377 [2]: I think I fulfilled every requested item, and a month ago I rebased
the work [3] to solve the NEWS conflicts.

I'm not sure if the rebase should have been done on the original branch
instead of creating a new one, or instead if I should open a new PR (and close
the original one?).

Thanks for any hint,
ciao, lele.

[1] https://bugs.python.org/issue27645
[2] https://github.com/python/cpython/pull/377
[3] https://github.com/lelit/cpython/compare/sqlite-backup-api-v3
-- 
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
l...@metapensiero.it  | -- Fortunato Depero, 1929.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 511 (code transformers) rejected

[Victor Stinner]

(Email resent, I first sent it to Nick privately by mistake.)

2017-11-02 2:53 GMT+01:00 Nick Coghlan :
> The piece that we're currently missing to make such workflows easier to
> manage is an equivalent of JavaScript's source maps (...)

Code objects already have a instruction pointer => line number mapping
table: the code.co_lnotab field. It's documented at:
https://github.com/python/cpython/blob/master/Objects/lnotab_notes.txt

This table is built from the line number information of the AST tree.

The mapping table is optimized to be small. Before Python 3.5, line
number had to be monotonic. Since Python 3.6, you "move" instructions
at the AST level, and so have "non-monotonic" line numbers (ex: line
1, line 3, line 2, line 4).

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [edk2] Official port of Python on EDK2

[Thiebaud Weksteen via Python-Dev]

Christian, Antoine, Brett: Thanks for the clarification on what an official
support would require. As Christian mentioned, sending simple headers
patches is an obvious starting point, no matter if the support becomes
official or not.

Brian: Thanks for your email. As I suggested, by having the support
directly within the Python community, you would avoid having to maintain a
separate port. I don't think that having a new Python3 port as part of EDK2
is a good idea. What I am suggesting is that Intel should contribute
directly to the Python repository by sending your modifications upstream
and not expect someone to re-import Python into EDK2. That is, bringing
your UEFI experience to Python and not the opposite. This would be a much
better use of anyone's time.

Thanks,
Thiebaud

On Thu, Nov 2, 2017 at 2:36 AM, Richardson, Brian <
brian.richard...@intel.com> wrote:

> Thiebaud:
>
> Thank you. I have started discussions within Intel for updating the UEFI
> CPython implementation to Python 3.x. The TianoCore community would
> appreciate contributions by people with Python experience to bring this
> code up to current standards.
>
> Please review the contribution guidelines for TianoCore and let me know if
> you have any questions.
> http://www.tianocore.org/contrib/
>
> Thanks ... br
> ---
> Brian Richardson, Senior Technical Marketing Engineer, Intel Software
> brian.richard...@intel.com -- @intel_brian (Twitter & WeChat)
> https://software.intel.com/en-us/meet-the-developers/evangel
> ists/team/brian-richardson
>
> -Original Message-
> From: edk2-devel [mailto:edk2-devel-boun...@lists.01.org] On Behalf Of
> Thiebaud Weksteen
> Sent: Wednesday, November 1, 2017 5:07 AM
> To: python-dev@python.org
> Cc: edk2-de...@lists.01.org
> Subject: [edk2] Official port of Python on EDK2
>
> Hi,
>
> UEFI has become the standard for firmware (BIOS) interface. Intel has
> provided an open source implementation under the name EDK2 (part of the
> TianoCore initiative) [1] for some time. This implementation has evolved
> significantly and now provides the functionalities of a small OS with a
> standard library similar to POSIX.
>
> In 2011, a port of Python 2.7.1 was added to the EDK2 repository [2].
> This port then evolved to 2.7.2 which is still defined as the reference
> port [3]. In 2015, another port was added of Python 2.7.10 in parallel of
> 2.7.2 [4]. Since then, both implementations have diverged from upstream and
> know vulnerabilities have not been fixed.
>
> I would like to bring support for edk2 in the official Python repository
> to remediate this situation, that is officially support
> edk2 as a platform. Technically, there would be three main aspects for the
> on-boarding work:
>
> 1) Fix headers and source to resolve definition conflicts, similarly to
> ABS definition in [5];
> 2) Add the edk2module.c [6] to handle platform-specific functionalities,
> similarly to the posixmodule.c;
> 3) Add the build configuration file [7] and necessary modifications within
> Python to handle the edk2 toolchain;
>
> This work would target the master branch (that is Python 3). I would be
> interested in hearing your thoughts on this idea.
>
> Thanks,
> Thiebaud
>
> [1] https://github.com/tianocore/edk2
> [2] https://github.com/tianocore/edk2/commit/006fecd5a177b4b7b6b
> 36fab6690bf2b2fa11829
> [3] https://github.com/tianocore/edk2/blob/master/AppPkg/Applica
> tions/Python/PythonReadMe.txt
> [4] https://github.com/tianocore/edk2/commit/c8042e10763bca064df
> 257547d04ae3dfcdfaf91
> [5] https://gist.github.com/tweksteen/ed516ca7ab7dfa8d18428f59d9c22a3e
> [6] https://github.com/tianocore/edk2/blob/master/AppPkg/Applica
> tions/Python/Efi/edk2module.c
> [7] https://github.com/tianocore/edk2/blob/master/AppPkg/Applica
> tions/Python/PythonCore.inf
> ___
> edk2-devel mailing list
> edk2-de...@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 564: Add new time functions with nanosecond resolution

[Victor Stinner]

Thank you Guido for your review and approval.

I just implemented the PEP 564 and so changed the PEP status to Final.

FYI I also added 3 new clock identifiers to the time module in Python
3.7: CLOCK_BOOTTIME, CLOCK_PROF and CLOCK_UPTIME.

So you can now get your Linux uptime with a resolution of 1 nanosecond :-D

haypo@selma$ ./python -c 'import time;
print(time.clock_gettime_ns(time.CLOCK_BOOTTIME))'
232172588663888

Don't do that at home, it's just for educational purpose only! ;-)

Victor

2017-10-30 18:18 GMT+01:00 Guido van Rossum :
> I have read PEP 564 and (mostly) followed the discussion in this thread, and
> I am happy with the PEP. I am hereby approving PEP 564. Congratulations
> Victor!
> --
> --Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Steven D'Aprano]

On Wed, Nov 01, 2017 at 03:48:00PM -0700, Lukasz Langa wrote:

> PEP: 563
> Title: Postponed Evaluation of Annotations

> This PEP proposes changing function annotations and variable annotations
> so that they are no longer evaluated at function definition time.
> Instead, they are preserved in ``__annotations__`` in string form.

This means that now *all* annotations, not just forward references, are 
no longer validated at runtime and will allow arbitrary typos and 
errors:

def spam(n:itn):  # now valid
...

Up to now, it has been only forward references that were vulnerable to 
that sort of thing. Of course running a type checker should pick those 
errors up, but the evaluation of annotations ensures that they are 
actually valid (not necessarily correct, but at least a valid name), 
even if you happen to not be running a type checker. That's useful.

Are we happy to live with that change?


> Rationale and Goals
> ===
> 
> PEP 3107 added support for arbitrary annotations on parts of a function
> definition.  Just like default values, annotations are evaluated at
> function definition time.  This creates a number of issues for the type
> hinting use case:
> 
> * forward references: when a type hint contains names that have not been
>   defined yet, that definition needs to be expressed as a string
>   literal;

After all the discussion, I still don't see why this is an issue. 
Strings makes perfectly fine forward references. What is the problem 
that needs solving? Is this about people not wanting to type the leading 
and trailing ' around forward references?


> * type hints are executed at module import time, which is not
>   computationally free.

True; but is that really a performance bottleneck? If it is, that should 
be stated in the PEP, and state what typical performance improvement 
this change should give.

After all, if we're going to break people's code in order to improve 
performance, we should at least be sure that it improves performance :-)


> Postponing the evaluation of annotations solves both problems.

Actually it doesn't. As your PEP says later:

> This PEP is meant to solve the problem of forward references in type
> annotations.  There are still cases outside of annotations where
> forward references will require usage of string literals.  Those are
> listed in a later section of this document.

So the primary problem this PEP is designed to solve, isn't actually 
solved by this PEP.

(See Guido's comments, quoted later.)



> Implementation
> ==
> 
> In Python 4.0, function and variable annotations will no longer be
> evaluated at definition time.  Instead, a string form will be preserved
> in the respective ``__annotations__`` dictionary.  Static type checkers
> will see no difference in behavior,

Static checkers don't see __annotations__ at all, since that's not 
available at edit/compile time. Static checkers see only the source 
code. The checker (and the human reader!) will no longer have the useful 
clue that something is a forward reference:

# before
class C:
def method(self, other:'C'): 
...

since the quotes around C will be redundant and almost certainly left 
out. And if they aren't left out, then what are we to make of the 
annotation? Will the quotes be stripped out, or left in?

In other words, will method's __annotations__ contain 'C' or "'C'"? That 
will make a difference when the type hint is eval'ed.


> If an annotation was already a string, this string is preserved
> verbatim.

That's ambiguous. See above.


> Annotations can only use names present in the module scope as postponed
> evaluation using local names is not reliable (with the sole exception of
> class-level names resolved by ``typing.get_type_hints()``).

Even if you call get_type_hints from inside the function defining the 
local names?

def function():
A = something()
def inner(x:A)->int:
...
d = typing.get_type_hints(inner)
return (d, inner)

I would expect that should work. Will it?


> For code which uses annotations for other purposes, a regular
> ``eval(ann, globals, locals)`` call is enough to resolve the
> annotation.

Let's just hope nobody doing that has allowed any tainted strings to 
be stuffed into __annotations__.


> * modules should use their own ``__dict__``.

Which is better written as ``vars()`` with no argument, I believe. Or 
possibly ``globals()``.


> If a function generates a class or a function with annotations that
> have to use local variables, it can populate the given generated
> object's ``__annotations__`` dictionary directly, without relying on
> the compiler.

I don't understand this paragraph.


> The biggest controversy on the issue was Guido van Rossum's concern
> that untokenizing annotation expressions back to their string form has
> no precedent in the Python programming language and feels like a hacky
> workaround.  He said:
> 
> One thing that comes to mind is that i

Re: [Python-Dev] [edk2] Official port of Python on EDK2

[Jayaprakash, N]

Would you consider adding thread support in this port of Python for EDK2 shell?

Regards,
JP

-Original Message-
From: edk2-devel [mailto:edk2-devel-boun...@lists.01.org] On Behalf Of 
Richardson, Brian
Sent: Thursday, November 2, 2017 7:07 AM
To: Thiebaud Weksteen ; python-dev@python.org
Cc: edk2-de...@lists.01.org
Subject: Re: [edk2] Official port of Python on EDK2

Thiebaud:

Thank you. I have started discussions within Intel for updating the UEFI 
CPython implementation to Python 3.x. The TianoCore community would appreciate 
contributions by people with Python experience to bring this code up to current 
standards.

Please review the contribution guidelines for TianoCore and let me know if you 
have any questions.
http://www.tianocore.org/contrib/ 

Thanks ... br
---
Brian Richardson, Senior Technical Marketing Engineer, Intel Software 
brian.richard...@intel.com -- @intel_brian (Twitter & WeChat) 
https://software.intel.com/en-us/meet-the-developers/evangelists/team/brian-richardson
 

-Original Message-
From: edk2-devel [mailto:edk2-devel-boun...@lists.01.org] On Behalf Of Thiebaud 
Weksteen
Sent: Wednesday, November 1, 2017 5:07 AM
To: python-dev@python.org
Cc: edk2-de...@lists.01.org
Subject: [edk2] Official port of Python on EDK2

Hi,

UEFI has become the standard for firmware (BIOS) interface. Intel has provided 
an open source implementation under the name EDK2 (part of the TianoCore 
initiative) [1] for some time. This implementation has evolved significantly 
and now provides the functionalities of a small OS with a standard library 
similar to POSIX.

In 2011, a port of Python 2.7.1 was added to the EDK2 repository [2].
This port then evolved to 2.7.2 which is still defined as the reference port 
[3]. In 2015, another port was added of Python 2.7.10 in parallel of 2.7.2 [4]. 
Since then, both implementations have diverged from upstream and know 
vulnerabilities have not been fixed.

I would like to bring support for edk2 in the official Python repository to 
remediate this situation, that is officially support
edk2 as a platform. Technically, there would be three main aspects for the 
on-boarding work:

1) Fix headers and source to resolve definition conflicts, similarly to ABS 
definition in [5];
2) Add the edk2module.c [6] to handle platform-specific functionalities, 
similarly to the posixmodule.c;
3) Add the build configuration file [7] and necessary modifications within 
Python to handle the edk2 toolchain;

This work would target the master branch (that is Python 3). I would be 
interested in hearing your thoughts on this idea.

Thanks,
Thiebaud

[1] https://github.com/tianocore/edk2
[2] 
https://github.com/tianocore/edk2/commit/006fecd5a177b4b7b6b36fab6690bf2b2fa11829
[3] 
https://github.com/tianocore/edk2/blob/master/AppPkg/Applications/Python/PythonReadMe.txt
[4] 
https://github.com/tianocore/edk2/commit/c8042e10763bca064df257547d04ae3dfcdfaf91
[5] https://gist.github.com/tweksteen/ed516ca7ab7dfa8d18428f59d9c22a3e
[6] 
https://github.com/tianocore/edk2/blob/master/AppPkg/Applications/Python/Efi/edk2module.c
[7] 
https://github.com/tianocore/edk2/blob/master/AppPkg/Applications/Python/PythonCore.inf
___
edk2-devel mailing list
edk2-de...@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel
___
edk2-devel mailing list
edk2-de...@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Brett Cannon]

On Thu, 2 Nov 2017 at 08:46 Steven D'Aprano  wrote:

> On Wed, Nov 01, 2017 at 03:48:00PM -0700, Lukasz Langa wrote:
>
> > PEP: 563
> > Title: Postponed Evaluation of Annotations
>
> > This PEP proposes changing function annotations and variable annotations
> > so that they are no longer evaluated at function definition time.
> > Instead, they are preserved in ``__annotations__`` in string form.
>
> This means that now *all* annotations, not just forward references, are
> no longer validated at runtime and will allow arbitrary typos and
> errors:
>
> def spam(n:itn):  # now valid
> ...
>
> Up to now, it has been only forward references that were vulnerable to
> that sort of thing. Of course running a type checker should pick those
> errors up, but the evaluation of annotations ensures that they are
> actually valid (not necessarily correct, but at least a valid name),
> even if you happen to not be running a type checker. That's useful.
>
> Are we happy to live with that change?
>

I would say "yes" for two reasons. One, if you're bothering to provide type
hints then you should be testing those type hints. So as you pointed out,
Steve, that will be caught at that point.

Two, code editors with auto-completion will help prevent this kind of typo.
Now I would never suggest that we design Python with expectations of what
sort of tooling people have available, but in this instance it will help.
It also feeds into a question you ask below...


>
>
> > Rationale and Goals
> > ===
> >
> > PEP 3107 added support for arbitrary annotations on parts of a function
> > definition.  Just like default values, annotations are evaluated at
> > function definition time.  This creates a number of issues for the type
> > hinting use case:
> >
> > * forward references: when a type hint contains names that have not been
> >   defined yet, that definition needs to be expressed as a string
> >   literal;
>
> After all the discussion, I still don't see why this is an issue.
> Strings makes perfectly fine forward references. What is the problem
> that needs solving? Is this about people not wanting to type the leading
> and trailing ' around forward references?
>

I think it's mainly about the next point you ask about...


>
>
> > * type hints are executed at module import time, which is not
> >   computationally free.
>
> True; but is that really a performance bottleneck? If it is, that should
> be stated in the PEP, and state what typical performance improvement
> this change should give.
>
> After all, if we're going to break people's code in order to improve
> performance, we should at least be sure that it improves performance :-)
>

The cost of constructing some of the objects used as type hints can be very
expensive and make importing really expensive (this has been pointed out by
Lukasz previously as well as Inada-san). By making Python itself not have
to construct objects from e.g. the 'typing' module at runtime, you then
don't pay a runtime penalty for something you're almost never going to use
at runtime anyway.


>
>
> > Postponing the evaluation of annotations solves both problems.
>
> Actually it doesn't. As your PEP says later:
>
> > This PEP is meant to solve the problem of forward references in type
> > annotations.  There are still cases outside of annotations where
> > forward references will require usage of string literals.  Those are
> > listed in a later section of this document.
>
> So the primary problem this PEP is designed to solve, isn't actually
> solved by this PEP.
>

I think the performance bit is really the big deal here.

And as I mentioned earlier, if you turn all of your type hints into
strings, you lose auto-completion/intellisense which is a shame.

I think there's also a benefit here of promoting the fact that type hints
are not a runtime thing, they are a static analysis thing. By requiring the
extra step to convert from a string to an actual object, it helps get the
point across that type hints are just bits of metadata for tooling and not
something you're expected really interact with at runtime unless you have a
really good reason to.

So I'm +1 on the idea, but the __future__ statement is a bit too generic
for me. I would prefer something like `from __future__ import
annotation_strings` or `annotations_as_strings`.

-Brett


>
> (See Guido's comments, quoted later.)
>
>
>
> > Implementation
> > ==
> >
> > In Python 4.0, function and variable annotations will no longer be
> > evaluated at definition time.  Instead, a string form will be preserved
> > in the respective ``__annotations__`` dictionary.  Static type checkers
> > will see no difference in behavior,
>
> Static checkers don't see __annotations__ at all, since that's not
> available at edit/compile time. Static checkers see only the source
> code. The checker (and the human reader!) will no longer have the useful
> clue that something is a forward reference:
>
> # before
> class C:
>

[Python-Dev] Interesting what version of python should I start with

[Bob Woolsey]

Sent from my iPhone
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reminder: 12 weeks to 3.7 feature code cutoff

[Ronald Oussoren]

> On 1 Nov 2017, at 22:47, Ned Deily  wrote:
> 
> Happy belated Halloween to those who celebrate it; I hope it wasn't too 
> scary!  Also possibly scary: we have just a little over 12 weeks remaining 
> until Python 3.7's feature code cutoff, 2018-01-29.  Those 12 weeks include a 
> number of traditional holidays around the world so, if you are planning on 
> writing another PEP for 3.7 or working on getting an existing one approved or 
> getting feature code reviewed, please plan accordingly.If you have 
> something in the pipeline, please either let me know or, when implemented, 
> add the feature to PEP 537, the 3.7 Release Schedule PEP.

I’d still like to finish PEP 447, but don’t know if I can manage to find enough 
free time to do so.

Ronald
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 511 (code transformers) rejected

[Brett Cannon]

On Wed, 1 Nov 2017 at 16:17 Lukasz Langa  wrote:

> I find this sad. In the JavaScript community the existence of Babel is
> very important for the long-term evolution of the language independently
> from the runtime. With Babel, JavaScript programmers can utilize new
> language syntax while being able to deploy on dated browsers. While there's
> always some experimentation, I doubt our community would abuse the new
> syntactic freedom that the PEP provided.
>
> Then again, maybe we should do what Babel did, e.g. release a tool like it
> totally separately from the runtime.
>

I think the trick here would be getting people more comfortable with
ahead-of-time compilation and then adding the appropriate support to
bytecode files to load other "optimization" levels/tags. Then you load the
.pyc files and rely on co_lnotab as Victor pointed out to get your source
mapping by compiling your source code explicitly instead of as a
side-effect of import. And since this approach would then just be about
generalizing how to specify different tags to match against in .pyc file
names it's easier to get accepted.

-Brett


>
> - Ł
>
> > On Oct 17, 2017, at 1:23 PM, Victor Stinner 
> wrote:
> >
> > Hi,
> >
> > I rejected my own PEP 511 "API for code transformers" that I wrote in
> > January 2016:
> >
> >
> https://github.com/python/peps/commit/9d8fd950014a80324791d7dae3c130b1b64fdace
> >
> > Rejection Notice:
> >
> > """
> > This PEP was rejected by its author.
> >
> > This PEP was seen as blessing new Python-like programming languages
> > which are close but incompatible with the regular Python language. It
> > was decided to not promote syntaxes incompatible with Python.
> >
> > This PEP was also seen as a nice tool to experiment new Python features,
> > but it is already possible to experiment them without the PEP, only with
> > importlib hooks. If a feature becomes useful, it should be directly part
> > of Python, instead of depending on an third party Python module.
> >
> > Finally, this PEP was driven was the FAT Python optimization project
> > which was abandonned in 2016, since it was not possible to show any
> > significant speedup, but also because of the lack of time to implement
> > the most advanced and complex optimizations.
> > """
> >
> > Victor
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/lukasz%40langa.pl
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Jukka Lehtosalo]

On Thu, Nov 2, 2017 at 3:45 PM, Steven D'Aprano  wrote:

> On Wed, Nov 01, 2017 at 03:48:00PM -0700, Lukasz Langa wrote:
>
> > This PEP proposes changing function annotations and variable annotations
> > so that they are no longer evaluated at function definition time.
> > Instead, they are preserved in ``__annotations__`` in string form.
>
> This means that now *all* annotations, not just forward references, are
> no longer validated at runtime and will allow arbitrary typos and
> errors:
>
> def spam(n:itn):  # now valid
> ...
>
> Up to now, it has been only forward references that were vulnerable to
> that sort of thing. Of course running a type checker should pick those
> errors up, but the evaluation of annotations ensures that they are
> actually valid (not necessarily correct, but at least a valid name),
> even if you happen to not be running a type checker. That's useful.
>
> Are we happy to live with that change?
>

Within functions misspellings won't be caught until you invoke a function:

def spam(s):
return itn(s)  # no error unless spam() is called

We've lived with this for a long time and generally people seem to be happy
with it. The vast majority of code in non-trivial programs (where type
annotations are useful) tends to be within functions, so this will only
slightly increase the number of things that won't be caught without running
tests (or running the type checker).

As type checking has become the main use case for annotations, using
annotations without a type checker is fast becoming a marginal use case.
Type checkers can easily and reliably validate that names in annotations aren't
misspelled.

> * forward references: when a type hint contains names that have not been
> >   defined yet, that definition needs to be expressed as a string
> >   literal;
>
> After all the discussion, I still don't see why this is an issue.
> Strings makes perfectly fine forward references. What is the problem
> that needs solving? Is this about people not wanting to type the leading
> and trailing ' around forward references?
>

Let's make a thought experiment. What if every forward reference would
require special quoting? Would Python programmers be happy with this? Say,
let's use ! as a suffix to mark a forward reference. They make perfectly
fine forward references. They are visually pretty unobtrusive (I'm not
suggesting $ or other ugly perlisms):

def main():
args = parse_args!()  # A forward reference
do_stuff!(args)  # Explicit is better than implicit

def parse_args():
...

def do_stuff(args):
...

Of course, I'm not seriously proposing this, but this highlights the fact
that in normal code forward references "just work" (at least usually), and
if we'd require a special quoting mechanism to use them anywhere, Python
would look uglier and more inconsistent. Nobody would be happy with this
change, even though you'd only have to type a single ! character extra --
that's not a lot work, right?

I think that the analogy is reasonable. In type checked code annotations
are one of most widely used language features -- it's quite possible to
have annotations for almost every function in a code base. This is not a
marginal feature, and people expect commonly used features to feel polished
and usable, not inconsistent and hacky. It's quite possible that the first
type annotated experiment a user writes requires the use of forward
references, and this gives a pretty bad first impression -- not unlike how
the ! forward reference would make the first impression of using
non-type-checked Python pretty awkward.

Here are more arguments why literal escapes are a usability problem:

1) It's hard to predict when string quotes are needed. Real-world large
code bases tend to have a lot of import cycles, and string literal escapes
are often needed within import cycles. However, they aren't always needed.
To annotate code correctly, you frequently need to understand how the file
you are editing is related to other modules in terms of import cycle
structure. In large code bases this can be very difficult to keep in your
head, so basically adding forward references becomes a matter of
tweak-until-it-works. So either each time you write an annotation, you can
look at how imports are structured -- to see whether a particular type
needs to be quoted -- or you can guess and hope for the best. This is a
terrible user experience and increases cognitive load significantly. Our
goal should not be to just have something that technically 'works', as this
is a very low standard. I want Python to be easy to use, intuitive and
elegant. I don't expect that anybody who has annotated large code bases
could consider string literal forward references to be any of those.

2) It's one of the top complaints from users. Even a non-user with a basic
understanding of mypy told me what amounts to "Python doesn't have real
static typing; forward references make it obvious that types are just an
afterthought".

3) It's not

Re: [Python-Dev] Interesting what version of python should I start with

[Ryan Gonzalez]

This list is for development of Python itself, not development with
Python. You want python-list:
https://mail.python.org/mailman/listinfo/python-list

That being said: use the latest version (3.6 as of right now).

On Thu, Nov 2, 2017 at 12:52 PM, Bob Woolsey  wrote:
>
>
> Sent from my iPhone
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com



-- 
Ryan (ライアン)
Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else
https://refi64.com/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Ivan Levkivskyi]

On 2 November 2017 at 18:00, Brett Cannon  wrote:

>
>
> On Thu, 2 Nov 2017 at 08:46 Steven D'Aprano  wrote:
>
>> On Wed, Nov 01, 2017 at 03:48:00PM -0700, Lukasz Langa wrote:
>> [...snip...]
>
> I think the performance bit is really the big deal here.
>
>
I don't think so. Although subscripting generics is indeed very expensive,
this is heavily optimised
by various caches. I think PEP 560 might actually have comparable (or even
bigger) performance effects.

IIUC performance is listed second in the PEP for a reason. I am OK with
using quotes for forward references, but I have seen
many people complaining about this (especially novices). I think Jukka is
right here.

We can't allow all unquoted forward references but those that will still be
quoted appear in more advanced situations
like bounded type variables and derived generics. But in these cases it is
clear that a forward reference appears
in runtime context.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Ivan Levkivskyi]

On 1 November 2017 at 23:48, Lukasz Langa  wrote:

> Runtime annotation resolution and class decorators
> --
>
> Metaclasses and class decorators that need to resolve annotations for
> the current class will fail for annotations that use the name of the
> current class.  Example::
>
> def class_decorator(cls):
> annotations = get_type_hints(cls)  # raises NameError on 'C'
> print(f'Annotations for {cls}: {annotations}')
> return cls
>
> @class_decorator
> class C:
> singleton: 'C' = None
>
> This was already true before this PEP.  The class decorator acts on
> the class before it's assigned a name in the current definition scope.
>
>
Just a random idea: maybe this can be resolved by just updating the localns
before calling get_type_hints() like this:

  localns = locals().copy()
  localns.update({cls.__name__: cls})


In general I like how the PEP is written now. I maybe would add examples
for this

> the cases listed above might be worked around by placing the usage
> in a if TYPE_CHECKING: block.
> ...
> For named tuples, using the new class definition syntax introduced in
Python 3.6 solves the issue.

actually showing something like

  if TYPE_CHECKING:
  Alias = List[Tuple[int, SomeClass]]

  class NT(NamedTuple):
  one: SomeClass
  other: Alias

  class SomeClass:
  ...

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reminder: 12 weeks to 3.7 feature code cutoff

[Terry Reedy]

On 11/2/2017 4:54 AM, Lele Gaifax wrote:


I'm not sure if the rebase should have been done on the original branch
instead of creating a new one, or instead if I should open a new PR (and close
the original one?).


It is normal to 'git merge upstream/master' after updating the master 
branch and checking out the pr branch, and then push the updated branch. 
 I sometimes have to do that when reviewing a pr.  Often, for me, the 
problem is not merge conflicts, but re version conflicts.  However, if 
there are merge conflicts that make it easier to reproduce the patch 
from the current master, closing and opening a new PR is ok too.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Stéfane Fermigier]

On Thu, Nov 2, 2017 at 7:39 PM, Jukka Lehtosalo 
wrote:

>
> As type checking has become the main use case for annotations, using
> annotations without a type checker is fast becoming a marginal use case.
> Type checkers can easily and reliably validate that names in annotations 
> aren't
> misspelled.
>

Another common use case is dependency injection / IoC:

Examples include:

- Injector (https://github.com/alecthomas/injector):

>>> class Outer:
... @inject
... def __init__(self, inner: Inner):
... self.inner = inner


- Flsk-Injector (ok it's the same underlying injector):

# Route with injection
@app.route("/foo")
def foo(db: sqlite3.Connection):
users = db.execute('SELECT * FROM users').all()
return render("foo.html")

- Apistar components (https://github.com/encode/apistar#components ):

def say_hello(user: User):
return {'hello': user.username}

=> In each of the examples, the type annotation are used at runtime by the
IoC container to inject an object of the appropriate type, based on some
specifications.

They may or may not be used by a typechecker too, but that's secondary.

  S.

-- 
Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier -
http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software -
http://www.abilian.com/
Chairman, Free&OSS Group / Systematic Cluster -
http://www.gt-logiciel-libre.org/
Co-Chairman, National Council for Free & Open Source Software (CNLL) -
http://cnll.fr/
Founder & Organiser, PyData Paris - http://pydata.fr/
---
“You never change things by fighting the existing reality. To change
something, build a new model that makes the existing model obsolete.” — R.
Buckminster Fuller
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [edk2] Official port of Python on EDK2

[Blibbet]

On 11/02/2017 09:41 AM, Jayaprakash, N wrote:
> Would you consider adding thread support in this port of Python for
EDK2 shell?

FYI, this library adds thread support to UEFI:

https://github.com/Openwide-Ingenierie/GreenThreads-UEFI

Note that the library is GPLv2, ...but the author (a 1-person project)
could be asked to relicense to BSD to fit into Tianocore.

Note that library is currently Intel x64-centric, and contains a bit of
assembly. Will need some ARM/RISC-V/x86 contributions.

HTH,
Lee Fisher
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

> On Nov 2, 2017, at 1:13 PM, Stéfane Fermigier  wrote:
> 
> Another common use case is dependency injection / IoC:
> 
> - Injector (https://github.com/alecthomas/injector 
> ):
> - Flask-Injector (ok it's the same underlying injector):

Pretty cool! This is already using `get_type_hints()` so it's perfectly 
compatible with PEP 563:

https://github.com/alecthomas/injector/blob/master/injector.py#L915

> - Apistar components (https://github.com/encode/apistar#components 
>  ):

This is using `inspect` directly so will have to migrate to `get_type_hints()` 
later. But, as the PEP mentions, it should already be using `get_type_hints()` 
since people are free to use forward references even today.

Thanks for flagging those use cases!


- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reminder: 12 weeks to 3.7 feature code cutoff

[Eric V. Smith]

On 11/1/2017 5:47 PM, Ned Deily wrote:

Happy belated Halloween to those who celebrate it; I hope it wasn't too scary!  
Also possibly scary: we have just a little over 12 weeks remaining until Python 
3.7's feature code cutoff, 2018-01-29.  Those 12 weeks include a number of 
traditional holidays around the world so, if you are planning on writing 
another PEP for 3.7 or working on getting an existing one approved or getting 
feature code reviewed, please plan accordingly.If you have something in 
the pipeline, please either let me know or, when implemented, add the feature 
to PEP 537, the 3.7 Release Schedule PEP.


I hope to be able to free up some time to complete PEP 557, Data Classes.

Eric.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reminder: 12 weeks to 3.7 feature code cutoff

[Ivan Levkivskyi]

I will be happy to see PEP 544 and PEP 560 in Python 3.7, and maybe also
PEP 562 (if we decide on it and I will have time).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[INADA Naoki]

I'm 100% agree with Łukasz and Brett.
+1 and thanks for writing this PEP.
INADA Naoki  


On Fri, Nov 3, 2017 at 2:00 AM, Brett Cannon  wrote:
>
>
> On Thu, 2 Nov 2017 at 08:46 Steven D'Aprano  wrote:
>>
>> On Wed, Nov 01, 2017 at 03:48:00PM -0700, Lukasz Langa wrote:
>>
>> > PEP: 563
>> > Title: Postponed Evaluation of Annotations
>>
>> > This PEP proposes changing function annotations and variable annotations
>> > so that they are no longer evaluated at function definition time.
>> > Instead, they are preserved in ``__annotations__`` in string form.
>>
>> This means that now *all* annotations, not just forward references, are
>> no longer validated at runtime and will allow arbitrary typos and
>> errors:
>>
>> def spam(n:itn):  # now valid
>> ...
>>
>> Up to now, it has been only forward references that were vulnerable to
>> that sort of thing. Of course running a type checker should pick those
>> errors up, but the evaluation of annotations ensures that they are
>> actually valid (not necessarily correct, but at least a valid name),
>> even if you happen to not be running a type checker. That's useful.
>>
>> Are we happy to live with that change?
>
>
> I would say "yes" for two reasons. One, if you're bothering to provide type
> hints then you should be testing those type hints. So as you pointed out,
> Steve, that will be caught at that point.
>
> Two, code editors with auto-completion will help prevent this kind of typo.
> Now I would never suggest that we design Python with expectations of what
> sort of tooling people have available, but in this instance it will help. It
> also feeds into a question you ask below...
>
>>
>>
>>
>> > Rationale and Goals
>> > ===
>> >
>> > PEP 3107 added support for arbitrary annotations on parts of a function
>> > definition.  Just like default values, annotations are evaluated at
>> > function definition time.  This creates a number of issues for the type
>> > hinting use case:
>> >
>> > * forward references: when a type hint contains names that have not been
>> >   defined yet, that definition needs to be expressed as a string
>> >   literal;
>>
>> After all the discussion, I still don't see why this is an issue.
>> Strings makes perfectly fine forward references. What is the problem
>> that needs solving? Is this about people not wanting to type the leading
>> and trailing ' around forward references?
>
>
> I think it's mainly about the next point you ask about...
>
>>
>>
>>
>> > * type hints are executed at module import time, which is not
>> >   computationally free.
>>
>> True; but is that really a performance bottleneck? If it is, that should
>> be stated in the PEP, and state what typical performance improvement
>> this change should give.
>>
>> After all, if we're going to break people's code in order to improve
>> performance, we should at least be sure that it improves performance :-)
>
>
> The cost of constructing some of the objects used as type hints can be very
> expensive and make importing really expensive (this has been pointed out by
> Lukasz previously as well as Inada-san). By making Python itself not have to
> construct objects from e.g. the 'typing' module at runtime, you then don't
> pay a runtime penalty for something you're almost never going to use at
> runtime anyway.
>
>>
>>
>>
>> > Postponing the evaluation of annotations solves both problems.
>>
>> Actually it doesn't. As your PEP says later:
>>
>> > This PEP is meant to solve the problem of forward references in type
>> > annotations.  There are still cases outside of annotations where
>> > forward references will require usage of string literals.  Those are
>> > listed in a later section of this document.
>>
>> So the primary problem this PEP is designed to solve, isn't actually
>> solved by this PEP.
>
>
> I think the performance bit is really the big deal here.
>
> And as I mentioned earlier, if you turn all of your type hints into strings,
> you lose auto-completion/intellisense which is a shame.
>
> I think there's also a benefit here of promoting the fact that type hints
> are not a runtime thing, they are a static analysis thing. By requiring the
> extra step to convert from a string to an actual object, it helps get the
> point across that type hints are just bits of metadata for tooling and not
> something you're expected really interact with at runtime unless you have a
> really good reason to.
>
> So I'm +1 on the idea, but the __future__ statement is a bit too generic for
> me. I would prefer something like `from __future__ import
> annotation_strings` or `annotations_as_strings`.
>
> -Brett
>
>>
>>
>> (See Guido's comments, quoted later.)
>>
>>
>>
>> > Implementation
>> > ==
>> >
>> > In Python 4.0, function and variable annotations will no longer be
>> > evaluated at definition time.  Instead, a string form will be preserved
>> > in the respective ``__annotations__`` dictionary.  Static type checkers
>> > will see no difference in behavior,

Re: [Python-Dev] PEP 564: Add new time functions with nanosecond resolution

[Guido van Rossum]

Yay! Record time from acceptance to implementation. :-)

On Thu, Nov 2, 2017 at 8:16 AM, Victor Stinner 
wrote:

> Thank you Guido for your review and approval.
>
> I just implemented the PEP 564 and so changed the PEP status to Final.
>
> FYI I also added 3 new clock identifiers to the time module in Python
> 3.7: CLOCK_BOOTTIME, CLOCK_PROF and CLOCK_UPTIME.
>
> So you can now get your Linux uptime with a resolution of 1 nanosecond :-D
>
> haypo@selma$ ./python -c 'import time;
> print(time.clock_gettime_ns(time.CLOCK_BOOTTIME))'
> 232172588663888
>
> Don't do that at home, it's just for educational purpose only! ;-)
>
> Victor
>
> 2017-10-30 18:18 GMT+01:00 Guido van Rossum :
> > I have read PEP 564 and (mostly) followed the discussion in this thread,
> and
> > I am happy with the PEP. I am hereby approving PEP 564. Congratulations
> > Victor!
> > --
> > --Guido van Rossum (python.org/~guido)
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 511 (code transformers) rejected

[Nick Coghlan]

On 2 November 2017 at 23:42, Victor Stinner  wrote:
> (Email resent, I first sent it to Nick privately by mistake.)

Oops, I didn't even notice that. Reposting my reply below.

> 2017-11-02 2:53 GMT+01:00 Nick Coghlan :
>> The piece that we're currently missing to make such workflows easier to
>> manage is an equivalent of JavaScript's source maps (...)
>
> Code objects already have a instruction pointer => line number mapping
> table: the code.co_lnotab field. It's documented at:
> https://github.com/python/cpython/blob/master/Objects/lnotab_notes.txt
>
> This table is built from the line number information of the AST tree.
>
> The mapping table is optimized to be small. Before Python 3.5, line
> number had to be monotonic. Since Python 3.6, you "move" instructions
> at the AST level, and so have "non-monotonic" line numbers (ex: line
> 1, line 3, line 2, line 4).

Right, and linecache knows how to read that, However, it can only do
so if the source files are on the running system with the bytecode
files, *and* the source code we're interested in is the source code
that was actually compiled by the interpreter.

Source code transformers fail that second precondition (since the
interpreter only sees the post-transformation code), and this is one
of the big reasons folks ended up writing actual single source 2/3
compatible code bases rather than running 2to3 as a source code
transformer when building packages: with transformed source,
conventional tracebacks quote the line from the transformed source
code, *not* the line in the original pre-transformation source code.

However, if the code transformer were to emit a JavaScript style
source map in addition to emitting the transformed code, then
automated tooling could take a traceback that referenced lines in the
transformed code, and work out the equivalent traceback for the
pre-transformation code. (I believe Cython has something like that in
order to provide it's HTML annotation mode, and PyPy's JIT can trace
from machine code back to the related Python source lines, but we
don't have anything that's independent of a particular toolchain the
way source maps are)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 511 (code transformers) rejected

[Nick Coghlan]

On 3 November 2017 at 03:19, Brett Cannon  wrote:
> On Wed, 1 Nov 2017 at 16:17 Lukasz Langa  wrote:
>>
>> I find this sad. In the JavaScript community the existence of Babel is
>> very important for the long-term evolution of the language independently
>> from the runtime. With Babel, JavaScript programmers can utilize new
>> language syntax while being able to deploy on dated browsers. While there's
>> always some experimentation, I doubt our community would abuse the new
>> syntactic freedom that the PEP provided.
>>
>> Then again, maybe we should do what Babel did, e.g. release a tool like it
>> totally separately from the runtime.
>
> I think the trick here would be getting people more comfortable with
> ahead-of-time compilation and then adding the appropriate support to
> bytecode files to load other "optimization" levels/tags. Then you load the
> .pyc files and rely on co_lnotab as Victor pointed out to get your source
> mapping by compiling your source code explicitly instead of as a side-effect
> of import. And since this approach would then just be about generalizing how
> to specify different tags to match against in .pyc file names it's easier to
> get accepted.

I'm not sure it's quite that simple, as you still need to define:

- how does the import system know that a given input file is a
"cache-only" import?
- how do linecache and similar tools know what source file the pyc maps back to?

 Right now, the source-file/cache-file relationship is hardcoded in
two functions:

* 
https://docs.python.org/3/library/importlib.html#importlib.util.cache_from_source;
and
* 
https://docs.python.org/3/library/importlib.html#importlib.util.source_from_cache

If we look at the code from hylang's custom importer for ".hy" files
[1] we can see that the "cache_from_source" implementation has a
convenient property: it ignores the source extension entirely, which
means it works for input paths with arbitrary file extensions, not
just Python source files.

This means that hy's import system integration can use that helper,
but if you have a "foo.hy" source file and a
"__pycache__/foo-.pyc" ouptut file, the regular import
machinery will *ignore* the latter file, and you have to register Hy's
customer importer in order for Python to acknowledge that the cached
file exists.

The reverse lookup, by contrast, always assumes that the source suffix
is a ".py" file (which is already broken for "pyw" source files on
Windows). Correcting for that at the standard library level would
require changing the cache filename format to include an optional
additional element: the source file extension (cache_to_source doesn't
assume it has access to the pyc file itself - only the filename).

So if we went down that path, then the import system level additions
we'd want would probably be along the lines of:

- an enhancement to the cache file naming scheme to allow source file
extensions to be saved in PYC filenames
- an update to the SourceFileLoader to use that new naming scheme when
implicitly compiling source files with the pyw extension
- a new "CacheOnlyLoader" together with a new CACHE_ONLY_SUFFIXES list
- a new ".pyb" suffix (for "Python backport") as the sole default
entry in CACHE_ONLY_SUFFIXES (awful pun alert: you could also argue
that this suffix makes sense because "pyb files come before pyc
files")

To make this syntactic polyfill approach usable with older Python
versions (including 2.7), importlib2 could be resynced to the first
importlib version that supported this (importlib2 is currently up to
date with Python 3.5's multi-phase initialisation support, since that
was the last major functional change in importlib).

Cheers,
Nick.


[1] https://github.com/hylang/hy/blob/master/hy/importer.py

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 3 November 2017 at 03:00, Brett Cannon  wrote:
> The cost of constructing some of the objects used as type hints can be very
> expensive and make importing really expensive (this has been pointed out by
> Lukasz previously as well as Inada-san). By making Python itself not have to
> construct objects from e.g. the 'typing' module at runtime, you then don't
> pay a runtime penalty for something you're almost never going to use at
> runtime anyway.

Another point worth noting is that merely importing the typing module
is expensive:

$ python -m perf timeit -s "from importlib import reload; import
typing" "reload(typing)"
.
Mean +- std dev: 10.6 ms +- 0.6 ms

10 ms is a *big* chunk out of a CLI application's startup time budget.

So I think to be truly effective in achieving its goals, the PEP will
also need to promote TYPE_CHECKING to a builtin, such that folks can
write:

from __future__ import lazy_annotations # More self-explanatory name
if TYPE_CHECKING:
import typing

and be able to express their type annotations in a way that a static
type checker will understand, while incurring near-zero runtime
overhead.

[snip]

Regarding the thunk idea: the compiler can pretty easily rewrite all
annotations from being "" to "lambda: ".

This has a couple of very nice properties:

* rendering them later is purely a matter of calling them, since the
compiler will take care of capturing all the right namespaces and name
references (including references from method declarations to the type
defining them)
* quoted and unquoted annotations will reliably render differently
(since one will return a string when called, and the other won't)

The big downside to this approach is that it makes simple annotations
*more* expensive rather than less expensive.

Baseline (mentioning a builtin):
$ python -m perf timeit "str"
.
Mean +- std dev: 27.1 ns +- 1.4 ns

String constant (~4x speedup):
$ python -m perf timeit "'str'"
.
Mean +- std dev: 7.84 ns +- 0.22 ns

Lambda expression (~2x slowdown):
$ python -m perf timeit "lambda: str"
.
Mean +- std dev: 62.3 ns +- 1.4 ns

That said, I'll also point out the following:

* for application startup purposes, if you save 10 ms by not importing
the typing module, then that buys you time for around 285 *thousand*
implicit lambda declarations before your startup actually gets slower
(assuming the relative timings on my machine are typical)
* for nested functions, the overhead of the function call is enough
that the dramatic 4x speedup vs 2x slowdown ratio disappears (see P.S.
for numbers)
* I don't believe we've really invested much time in optimising the
creation of zero-argument lambdas yet, so there may be options for
bringing the numbers down for the lambda based approach

The other key downside to the lambda based approach is that it hits
the same backwards compatibility problem we hit when list
comprehensions were given their own nested scope: if we push
annotations down into a new scope, they won't be able to see class
level attributes any more. For comprehensions, we could partially
mitigate that by evaluating the outermost iterable expression in the
class scope, but there's no equivalent to that available for
annotations (since the annotation's lambda expression may never be
called at all).

Cheers,
Nick.

P.S. Relative performance of the annotation styles in a nested
function definition

Baseline:
$ python -m perf timeit -s "def f(): str" "f()"
.
Mean +- std dev: 103 ns +- 3 ns

String constant (~1.25x speedup):
$ python -m perf timeit -s "def f(): 'str'" "f()"
.
Mean +- std dev: 77.0 ns +- 1.7 ns

Lambda expression (~1.5x slowdown):
$ python -m perf timeit -s "def f(): (lambda: str)" "f()"
.
Mean +- std dev: 149 ns +- 6 ns

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 3 November 2017 at 04:39, Jukka Lehtosalo  wrote:
>> > * forward references: when a type hint contains names that have not been
>> >   defined yet, that definition needs to be expressed as a string
>> >   literal;
>>
>> After all the discussion, I still don't see why this is an issue.
>> Strings makes perfectly fine forward references. What is the problem
>> that needs solving? Is this about people not wanting to type the leading
>> and trailing ' around forward references?
>
>
> Let's make a thought experiment. What if every forward reference would
> require special quoting? Would Python programmers be happy with this? Say,
> let's use ! as a suffix to mark a forward reference. They make perfectly
> fine forward references. They are visually pretty unobtrusive (I'm not
> suggesting $ or other ugly perlisms):
>
> def main():
> args = parse_args!()  # A forward reference
> do_stuff!(args)  # Explicit is better than implicit
>
> def parse_args():
> ...
>
> def do_stuff(args):
> ...
>
> Of course, I'm not seriously proposing this, but this highlights the fact
> that in normal code forward references "just work" (at least usually), and
> if we'd require a special quoting mechanism to use them anywhere, Python
> would look uglier and more inconsistent. Nobody would be happy with this
> change, even though you'd only have to type a single ! character extra --
> that's not a lot work, right?
>
> I think that the analogy is reasonable.

I think it also makes a pretty decent argument that pushing function
annotations into implicit lambda expressions will be easier to explain
to people than converting them into strings, and then having to
explain an entirely new complex set of name resolution rules.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com