Re: [Python-ideas] Delay evaluation of annotations

2016-09-23 Thread Chris Angelico 
On Fri, Sep 23, 2016 at 11:58 PM, אלעזר  wrote:
> What other context you see where the result of an expression is not intended
> to be used at all? Well there's Expression statements, which are evaluated
> for side effect. There's docstrings, which are a kind of annotations. What
> else? The only other that comes to mind is reveal_type(exp)... surely I
> don't need evaluation there.

Function annotations ARE used. They're stored as function attributes,
just as default argument values and docstrings are. (It's not the
language's problem if you never use them.)

>> The PEP that introduced them describes them as expressions:
>
> Syntactically, yes. Just like X in "a = lambda: X" is an expression, but you
> don't see it evaluated, do you? And this is an _actual_ expression,
> undeniably so, that is intended to be evaluated and used at runtime.

And the X in "if False: X" is a statement, but you don't see it
evaluated either. This is an actual expression that has to be
evaluated and used just like any other does.

>> I think its time to give up arguing that annotations aren't expressions.
>>
>
> I don't care if you call them expressions, delayed-expressions, or flying
> monkeys. The allowed syntax is exactly that of an expression (like inside a
> lambda). The time of binding of names to scope is the same (again like a
> lambda) but the evaluation time is unknown to the non-reflecting-developer.
> Decorators may promise time of evaluation, if they want to.

Thing is, literally every other expression in Python is evaluated at
the point where it's hit. You can guard an expression with control
flow statements or operators, but other than that, it will be hit when
execution reaches its line:

def func(x):
expr # evaluated when function called

if cond:
expr # evaluated if cond is true

[expr for x in range(n)] # evaluated if n > 0
(expr for x in [1]) # evaluated when genexp nexted
expr if cond else "spam" # evaluated if cond is true
lambda: expr # evaluated when function called

def func(x=expr): pass # evaluated when function defined
def func(x: expr): pass # evaluated when function defined

Default arguments trip some people up because they expect them to be
evaluated when the function's called, but it can easily be explained.
Function annotations are exactly the same. Making them magically
late-evaluate would have consequences for the grokkability of the
language - they would be special. Now, that can be done, but as
Rumplestiltskin keeps reminding us, all magic comes with a price, so
it has to be strongly justified. (For instance, the no-arg form of
super() is most definitely magical, but its justification is obvious
when you compare Py2 inheritance with Py3.)

> "Unknown evaluation time" is scary. _for expressions_, which might have side
> effects (one of which is running time). But annotations must be pure by
> convention (and tools are welcome to warn about it). I admit that I propose
> breaking the following code:
>
> def foo(x: print("defining foo!")): pass
>
> Do you know anyone who would dream about writing such code?

Yes, side effects make evaluation time scary. But so do rebindings,
and any other influences on expression evaluation. Good, readable code
generally follows the rule that the first instance of a name is its
definition. That's why we put imports up the top of the script, and so
on. Making annotations not work that way isn't going to improve
readability; you'd have to search the entire project for the class
being referenced. And since you can't probe them at definition time,
you have to wait until, uhh, SOME time, to do that search - you never
know where the actual name binding will come from. (It might even get
injected from another file, so you can't statically search the one
file.)

>> You want to make them fancy, give them super-powers, in order to solve
>> the forward reference problem. I don't think that the problem is serious
>> enough to justify changing the semantics of annotation evaluation and
>> make them non-standard, fancy, lazy-evaluated expressions.
>>
>
> My proposal solves the forward reference problem, but I believe in it
> because I believe it is aligned with what the programmer see.

This is on par with a proposal to make default argument values
late-bind, which comes up every now and then. It's just not worth
making these expressions magical.

>> > > class MyClass:
>> > > pass
>> > >
>> > > def function(arg: MyCalss):
>> > > ...
>> > >
>> > > I want to see an immediate NameError here, thank you very much
>> >
>> > Two things to note here:
>> > A. IDEs will point at this NameError
>>
>> Some or them might. Not everyone uses an IDE, it is not a requirement
>> for Python programmers. Runtime exceptions are still, and always will
>> be, the primary way of detecting such errors.
>
> How useful is the detection of this error at production?

The sooner you catch an error, the better. Always.

> Can you repeat that? NameError indeed happens at

Re: [Python-ideas] Delay evaluation of annotations

2016-09-23 Thread אלעזר 
On Fri, Sep 23, 2016 at 3:11 PM Steven D'Aprano  wrote:

> On Fri, Sep 23, 2016 at 10:17:15AM +, אלעזר wrote:
> > On Fri, Sep 23, 2016 at 6:06 AM Steven D'Aprano 
> wrote:
> > > On Thu, Sep 22, 2016 at 07:21:18PM +, אלעזר wrote:
> > > > On Thu, Sep 22, 2016 at 9:43 PM Steven D'Aprano wrote:
> > > > > On Thu, Sep 22, 2016 at 05:19:12PM +, אלעזר wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > Annotations of function parameters and variables are evaluated
> when
> > > > > > encountered.
> > > > >
> > > > > Right, like all other Python expressions in general, and
> specifically
> > > > > like function parameter default arguments.
> > > > >
> > > >
> > > > Just because you call it "expression", when for most purposes it
> isn't -
> > > > it is an annotation.
> > >
> > > It is *both*. It's an expression, because it's not a statement or a
> > > block.
> >
> >
> > Did you just use a false-trichotomy argument? :)
>
> No.
>
> You are the one trying to deny that annotations are expressions -- I'm
> saying that they are both annotations and expressions at the same time.
> There's no dichotomy here, since the two are not mutually exclusive.
> (The word here is dichotomy, not trichotomy, since there's only two
> things under discussion, not three.)
>
>
The argument "It's an expression, because it's not a statement or a block"
assumes that things must an expression, a statement or a block. Hence
"trichotomy". And it is false.

But I think we are getting lost in the terminology. Since I propose no
change in what is considered valid syntax,


> > > You cannot write:
> > >
> > > def func(arg1: while flag: sleep(1), arg2: raise ValueError):
> > > ...
> >
> > > because the annotation must be a legal Python expression, not a code
> > > block or a statement.
> >
> >
> > This is the situation I'm asking to change
>
> That's a much bigger change than what you suggested earlier, changing
> function annotations to lazy evaluation instead of eager.
>
> Supporting non-expressions as annotations -- what's your use-case? Under
> what circumstances would you want to annotate an function parameter with
> a code block instead of an expression?
>
>
It indeed came out different than I meant. I don't suggest allowing
anything that is not already allowed, syntactically. I only propose giving
the current syntax a slightly different meaning, in a way that I'm sure
matches how Python coders already understand the code.

> > It's an annotation because that's the
> > > specific *purpose* of the expression in that context.
> >
> > Exactly! Ergo, this is an annotation.
>
> I've never denied that annotations are annotations, or that annotations
> are used to annotate function parameters. I'm not sure why you are
> giving a triumphant cry of "Exactly!" here -- it's not under dispute
> that annotations are annotations.
>
>
:( this kind of fighting over terminology takes us nowhere indeed.

What other context you see where the result of an expression is not
intended to be used at all? Well there's Expression statements, which are
evaluated for side effect. There's docstrings, which are a kind of
annotations. What else? The only other that comes to mind is
reveal_type(exp)... surely I don't need evaluation there.

And it shouldn't be under dispute that annotations are expressions.
> They're not code blocks. They're not statements. What else could they be
> apart from expressions?
>
> Now it is a false dichotomy as question. The answer is "annotation" as an
independent concept, closely related to expressions, but not evaluated when
encountered. Very similar to E in `lambda: E` except that lambda are there
mainly for the resulting value (hence "expression") and annotations are
there mainly for being there. In the code.


> The PEP that introduced them describes them as expressions:
>
> Function annotations are nothing more than a way of associating
> arbitrary Python EXPRESSIONS with various parts of a function at
> compile-time. [Emphasis added.]
>
> https://www.python.org/dev/peps/pep-3107/
>
>
Syntactically, yes. Just like X in "a = lambda: X" is an expression, but
you don't see it evaluated, do you? And this is an _actual_ expression,
undeniably so, that is intended to be evaluated and used at runtime.


> and they are documented as an expression:
>
> parameter ::=  identifier [":" expression]
>
> Parameters may have annotations of the form “: expression” following
> the parameter name. ... These annotations can be any valid Python
> expression
>
>
> https://docs.python.org/3/reference/compound_stmts.html#function-definitions
>
> I think its time to give up arguing that annotations aren't expressions.
>
>
I don't care if you call them expressions, delayed-expressions, or flying
monkeys. The allowed syntax is exactly that of an expression (like inside a
lambda). The time of binding of names to scope is the same (again like a
lambda) but the evaluation time is

Re: [Python-ideas] Delay evaluation of annotations

2016-09-23 Thread Steven D'Aprano 
On Fri, Sep 23, 2016 at 10:17:15AM +, אלעזר wrote:
> On Fri, Sep 23, 2016 at 6:06 AM Steven D'Aprano  wrote:
> > On Thu, Sep 22, 2016 at 07:21:18PM +, אלעזר wrote:
> > > On Thu, Sep 22, 2016 at 9:43 PM Steven D'Aprano wrote:
> > > > On Thu, Sep 22, 2016 at 05:19:12PM +, אלעזר wrote:
> > > > > Hi all,
> > > > >
> > > > > Annotations of function parameters and variables are evaluated when
> > > > > encountered.
> > > >
> > > > Right, like all other Python expressions in general, and specifically
> > > > like function parameter default arguments.
> > > >
> > >
> > > Just because you call it "expression", when for most purposes it isn't -
> > > it is an annotation.
> >
> > It is *both*. It's an expression, because it's not a statement or a
> > block.
> 
> 
> Did you just use a false-trichotomy argument? :)

No.

You are the one trying to deny that annotations are expressions -- I'm 
saying that they are both annotations and expressions at the same time. 
There's no dichotomy here, since the two are not mutually exclusive. 
(The word here is dichotomy, not trichotomy, since there's only two 
things under discussion, not three.)



> > You cannot write:
> >
> > def func(arg1: while flag: sleep(1), arg2: raise ValueError):
> > ...
> 
> > because the annotation must be a legal Python expression, not a code
> > block or a statement.
> 
> 
> This is the situation I'm asking to change

That's a much bigger change than what you suggested earlier, changing 
function annotations to lazy evaluation instead of eager.

Supporting non-expressions as annotations -- what's your use-case? Under 
what circumstances would you want to annotate an function parameter with 
a code block instead of an expression?


> > It's an annotation because that's the
> > specific *purpose* of the expression in that context.
>
> Exactly! Ergo, this is an annotation.

I've never denied that annotations are annotations, or that annotations 
are used to annotate function parameters. I'm not sure why you are 
giving a triumphant cry of "Exactly!" here -- it's not under dispute 
that annotations are annotations.

And it shouldn't be under dispute that annotations are expressions. 
They're not code blocks. They're not statements. What else could they be 
apart from expressions?

The PEP that introduced them describes them as expressions:

Function annotations are nothing more than a way of associating 
arbitrary Python EXPRESSIONS with various parts of a function at 
compile-time. [Emphasis added.]

https://www.python.org/dev/peps/pep-3107/

and they are documented as an expression:

parameter ::=  identifier [":" expression]

Parameters may have annotations of the form “: expression” following 
the parameter name. ... These annotations can be any valid Python 
expression

https://docs.python.org/3/reference/compound_stmts.html#function-definitions

I think its time to give up arguing that annotations aren't expressions.


> > As an analogy: would you argue that it is wrong to call the for-loop
> > iterable an expression?
> >
> > for  in :
> > block
> >
> > I trust that you understand that the loop iterable can be any expression
> > that evaluates to an iterable. Well, annotations can be any expression
> > that evaluates to anything at all, but for the purposes of type
> > checking, are expected to evaluate to a string or a type object.
> >
> >
> for-loop iterable is an expression, evaluated at runtime, _for_ the
> resulting value to be used in computation. A perfectly standard expression.
> Nothing fancy.

Right. And so are annotations.

You want to make them fancy, give them super-powers, in order to solve 
the forward reference problem. I don't think that the problem is serious 
enough to justify changing the semantics of annotation evaluation and 
make them non-standard, fancy, lazy-evaluated expressions.


> > In the case of function annotations, remember that they can be any
> > legal Python expression. They're not even guaranteed to be type
> > annotations. Guido has expressed a strong preference that they are only
> > used as type annotations, but he hasn't yet banned other uses (and I
> > hope he doesn't), so any "solution" for a type annotation problem must
> > not break other uses.
> >
> >
> Must *allow* other use cases. My proposal allows: just evaluate them at the
> time of their use, instead at definition time.

I meant what I said. Changing the evaluation model for annotations is a 
big semantic change, a backwards-incompatible change. It's not just 
adding new syntax for something that was a syntax error before, it would 
be changing the meaning of existing Python code.

The transition from 3.6 to 3.7 is not like that from 2.x to 3.0 -- 
backwards compatibility is a hard requirement. Code that works a certain 
way in 3.6 is expected to work the same way in 3.7 onwards, unless we go 
through a deprecation period of at least one full release, and probably 
with

Re: [Python-ideas] Delay evaluation of annotations

2016-09-23 Thread אלעזר 
On Fri, Sep 23, 2016 at 6:24 AM Nick Coghlan  wrote:

> On 23 September 2016 at 13:06, Steven D'Aprano 
> wrote:
> > On Thu, Sep 22, 2016 at 07:21:18PM +, אלעזר wrote:
> >> "Expression" is something that you need its value right
> >> now, and "annotation" is something that, well, annotates the code you
> see
> >> right now.
> >
> > Right. In the case of Python, function annotations **do** have a runtime
> > effect: the expressions are evaluated, and the evaluated results are
> > assigned in function.__annotations__ and made available for runtime
> > introspection.
> >
> > Don't think that function annotations are **only** for the static type
> > checker. Python is a much richer language than that!
>
> If folks are after a simple non-type-checking related example of
> annotation usage, the "begins" CLI library is a decent one:
> https://pypi.python.org/pypi/begins
>
> That lets you supply command line help for parameters as annotations:
>
> 
> In Python3, any function annotations for a parameter become the
> command line option help. For example:
>
>   >>> import begin
>   >>> @begin.start # doctest: +SKIP
>   ... def run(name: 'What, is your name?',
>   ... quest: 'What, is your quest?',
>   ... colour: 'What, is your favourite colour?'):
>   ... pass
>
> Will generate command help like:
>
> usage: holygrail_py3.py [-h] -n NAME -q QUEST -c COLOUR
>
> optional arguments:
>   -h, --helpshow this help message and exit
>   -n NAME, --name NAME  What, is your name?
>   -q QUEST, --quest QUEST
> What, is your quest?
>   -c COLOUR, --colour COLOUR
> What, is your favourite colour?
> 
>
> It's not a substitute for something like click or argparse when it
> comes to more complex argument parsing, but it's a good example of the
> kind of simple pseudo-DSL folks have long been able to create with
> annotations independently of the type hinting use case.
>
>
That's a very nice use, and I was wrong - I did know it; I've found it not
long ago when I wanted to implement it myself...

And guess what? It does not require eager evaluation _at all_. No
decorator-helped-annotation mechanism require eager evaluation built into
the language. Lazy evaluation is more general than eager, in that it can
always be forced (and not the other way around).

def eager_annotation(f):
f.__annotations__ = {k:v() for k, v in f.__annotations__}
return f

Use @eager_annotation wherever you like, or collapse it into other
decorators.

You don't need @eager_annotation for type annotations, or any other form of
annotation without runtime semantics. On the other hand - if you do want
side effect in this function's annotations, well there's better be some
nice big @EAGER! decorator above it.

Elazar
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Delay evaluation of annotations

2016-09-23 Thread אלעזר 
On Fri, Sep 23, 2016 at 6:06 AM Steven D'Aprano  wrote:

> On Thu, Sep 22, 2016 at 07:21:18PM +, אלעזר wrote:
> > On Thu, Sep 22, 2016 at 9:43 PM Steven D'Aprano 
> wrote:
> >
> > > On Thu, Sep 22, 2016 at 05:19:12PM +, אלעזר wrote:
> > > > Hi all,
> > > >
> > > > Annotations of function parameters and variables are evaluated when
> > > > encountered.
> > >
> > > Right, like all other Python expressions in general, and specifically
> > > like function parameter default arguments.
> > >
> >
> > Just because you call it "expression", when for most purposes it isn't -
> it
> > is an annotation.
>
> It is *both*. It's an expression, because it's not a statement or a
> block.


Did you just use a false-trichotomy argument? :)


> You cannot write:
>
> def func(arg1: while flag: sleep(1), arg2: raise ValueError):
> ...

because the annotation must be a legal Python expression, not a code
> block or a statement.


This is the situation I'm asking to change


> It's an annotation because that's the
> specific *purpose* of the expression in that context.
>
>
Exactly! Ergo, this is an annotation.


> As an analogy: would you argue that it is wrong to call the for-loop
> iterable an expression?
>
> for  in :
> block
>
> I trust that you understand that the loop iterable can be any expression
> that evaluates to an iterable. Well, annotations can be any expression
> that evaluates to anything at all, but for the purposes of type
> checking, are expected to evaluate to a string or a type object.
>
>
for-loop iterable is an expression, evaluated at runtime, _for_ the
resulting value to be used in computation. A perfectly standard expression.
Nothing fancy.


> In the case of function annotations, remember that they can be any
> legal Python expression. They're not even guaranteed to be type
> annotations. Guido has expressed a strong preference that they are only
> used as type annotations, but he hasn't yet banned other uses (and I
> hope he doesn't), so any "solution" for a type annotation problem must
> not break other uses.
>
>
Must *allow* other use cases. My proposal allows: just evaluate them at the
time of their use, instead at definition time.


>
> > "Expression" is something that you need its value right
> > now, and "annotation" is something that, well, annotates the code you see
> > right now.
>
> Right. In the case of Python, function annotations **do** have a runtime
> effect: the expressions are evaluated, and the evaluated results are
> assigned in function.__annotations__ and made available for runtime
> introspection.
>
> Don't think that function annotations are **only** for the static type
> checker. Python is a much richer language than that!
>
>
function.__annotations__  can have the delayed value, be it a lambda, ast
or string. It can also be computed at the time of access as suggested
earlier.


>
> > > > It is
> > > > also easy to forget, and the result might be a (very uninteresting)
> > > > exception in certain untested paths, e.g. inside functions.
> > >
> > > Unlikely, unless you're talking about functions nested inside other
> > > functions, or unusual (but legal and sometimes useful) conditional
> > > definitions:
> >
> > I was thinking about the former, but yeah, uncovered code will fail at
> > runtime, possibly in production, for *no* real reason. I do not claim
> that
> > this is common, but it is definitely unnecessary - unlike initialization
> > expressions.
>
> Unnecessary?
>
> class MyClass:
> pass
>
>
> def function(arg: MyCalss):
> ...
>
> I want to see an immediate NameError here, thank you very much
>

Two things to note here:
A. IDEs will point at this NameError
B. Type checkers catch this NameError
C. Even the compiler can be made to catch this name error, since the name
MyCalss is bound to builtins where it does not exist - you see, name lookup
does happen at compile time anyway. I'm not really suggesting the compiler
should make it error though.
D. Really, where's the error here? if no tool looks at this signature,
there's nothing wrong with it - As a human I understand perfectly. If a
tool will look at it, it will warn or fail, exactly as I would liked it too.

function.__annotations__['arg']()
>
> to see whether or not the annotation is valid.
>
> I accept that using strings as forward annotations is not a foolproof
> solution either:
>
>
> def function(arg: 'MyCalss'):
> ...
>
but let's not jump into a "fix" that actually makes things worse.
>
>
>
That's not a "fix". I suggest always using the last form - which is already
in common use - with a nicer syntax and semantics, since there's nothing
wrong about it. It is there for a very natural reason.


> > > if condition:
> > > # forward reference to MyClass
> > > def f(arg:'MyClass'): ...
> > > else:
> > > # oops, untested path
> > > def f(arg:MyClass): ...
> > >
> > > class MyClass: ...
> > >
> > >
> > > But generally speaking,

Re: [Python-ideas] Delay evaluation of annotations

2016-09-23 Thread אלעזר 
On Fri, Sep 23, 2016 at 5:54 AM Chris Angelico  wrote:

> On Fri, Sep 23, 2016 at 12:35 PM, Steven D'Aprano 
> wrote:
> > The straight-forward and simple way of writing a recursive spam()
> > function surprises beginners, but they might go years or their entire
> > career without running into a situation where they are caught by
> > surprise. After all, it is rare for productuon code to rename functions,
> > and rarer still to do it to recursive functions:
> >
> > func = spam
> > spam = something_else()
> > func()  # why does the recursion not work???
> >
> > In production code, that sort of thing almost never happens.
>
> There's actually one very common technique involving rebinding functions.
>
> @count_calls
> def mergesort(lst):
> mid = len(lst) // 2
> if not mid: return lst
> return merge(mergesort(lst[..mid]), mergesort(lst[mid..]))
>
> *Obviously* this is recursive. But if you used some magic that said
> "call the function that's currently being called", you'd be bypassing
> the count_calls decoration (which would presumably work by creating a
> wrapper function). Yes, it may defeat some potential optimizations (eg
> tail recursion optimization), but it enables all this flexibility.
>
> So we _need_ to have this kind of rebind available, and not just for
> experts.
>
>
I think you are mixing levels of abstraction because you know how this is
implemented. The user only sees "A function named mergesort decorated by
count_calls". She does not see "A function named mergesort passed to a
higher order function named count_calls whose result is bound into the
variable mergesort". Even if the latter is exactly what happens,
declaratively the former is more accurate by intention.

Ideally, the calls to mergesort will rebind to this _decorated_ function.
not to the mutable global variable. Again, the argument that it will be
very hard to implement it in a different way, or that is will break things,
is a very strong argument, and I am not confronting it.

> In the meantime, I'll usually just write my recursive functions the
> > old-fashioned normal way.
>
> As will I. Of course, people are welcome to work differently, just as
> long as I never have to write tests for their code, or refactor
> anything into a decorator, or anything like that. I want the
> POWAH! :)
>

As will I, simply because the old-fashioned way is more readable. And I
will sadly accept the fact that I can't be 100% sure what's function is
called at runtime. But _some_ people (medium-level, Steven, whose main
language is probably not Python) will not even know this is the case.

Tests are important and could have reworked into the system (through
inspect, or by using a special import which allow monkey patching). I can't
see why the ability to test must remain in production.

Elazar
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Delay evaluation of annotations

2016-09-23 Thread Nick Coghlan 
On 23 September 2016 at 15:50, Greg Ewing  wrote:
> אלעזר wrote:
>>
>> it feels like a
>> placeholder for this meaning would be better.  E.g.:
>>
>> class A:
>> def __add__(self, other: CLS) -> CLS: ...
>
>
> That's fine for a class that refers to itself, but
> what about classes that refer to each other? This
> only addresses a small part of the problem.

Same answer as with any other circular dependency: the code smell is
the circular dependency itself, not the awkwardness of the syntax for
spelling it. If the string based "circular reference here!" spelling
really bothers you, refactor to eliminate the circularity (e.g. by
extracting a base class or an implementation independent interface
definition), rather than advocating to make the spelling less
obnoxious.

The difference between that and the "methods referring to the class
they're defined in" case is that it's likely to be pretty normal to
want to do the latter, so it may prove worthwhile to provide a cleaner
standard spelling for it. The counter-argument is the general
circularity one above: do you *really* need instances of the
particular class being defined? Or is there a more permissive
interface based type signature you could specify instead? Or perhaps
no type signature at all, and let ducktyping sort it out implicitly at
runtime?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/