subject:"\[Python\-Dev\] Re\: PEP 649\: Deferred Evaluation Of Annotations"

I don't think I've seen code like this. It would be incredibly useful to have a 
cpython-primer (like mypy-primer and black-primer) that ran as many test suites 
as possible from pypi with every cPython commit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QEGUZ5GOKA4CI5ELA5BBDSS76UJN26BI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations


On 8/11/21 5:15 AM, Chris Angelico wrote:

On Wed, Aug 11, 2021 at 10:03 PM Larry Hastings  wrote:

This approach shouldn't break reasonable existing code.  That said, this change 
would be observable from Python, and pathological code could notice and break.  
For example:

def ensure_Foo_is_a_class(o):
 assert isinstance(Foo, type)
 return o

class Foo:
 ...

@ensure_Foo_is_a_class
def Foo():
 ...

This terrible code currently would not raise an assertion.  But if we made the 
proposed change to the implementation of decorators, it would.  I doubt anybody 
does this sort of nonsense, I just wanted to fully flesh out the topic.


You would be here declaring that a @monkeypatch decorator is terrible
code. I'm not sure whether you're right or wrong. You may very well be
right.

def monkeypatch(cls):
 basis = globals()[cls.__name__]
 for attr in dir(cls): setattr(basis, attr, getattr(cls, attr))
 return basis

@monkeypatch
class SomeClass:
 def new_method(self): ...

Currently this works, since SomeClass doesn't get assigned yet. This
could be made to work across versions by writing it as
@monkeypatch(SomeClass) instead (and then the actual class name would
become immaterial).



Golly!  I've never seen that.  Is that a common technique?

If we need to preserve that behavior, then this idea is probably a 
non-starter.



//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QNHESUROGJ7V4Q5LFUNXCU2FUWNOW7OE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-11 Thread Terry Reedy


On 8/11/2021 7:56 AM, Larry Hastings wrote:

So, here's an idea, credit goes to Eric V. Smith.  What if we tweak how 
decorators work, /jst slghtly/, so that they work like the 
workaround code above?


Specifically: currently, decorators are called just after the function 
or class object is created, before it's bound to a variable.  But we 
could change it so that we first bind the variable to the initial value, 
then call the decorator, then rebind.  That is, this code:


@dekor8
class C:
     ...

would become equivalent to this code:

class C:
     ...
C = dekorate(C)


This is how function decorators were originally defined. Before the 2016 
(3.5) revision of

https://docs.python.org/3/reference/compound_stmts.html#function-definitions
by https://bugs.python.org/issue26576

---
   @f1(arg)
   @f2
   def func(): pass

is equivalent to

   def func(): pass
   func = f1(arg)(f2(func))
---

After
---

@f1(arg)
@f2
def func(): pass

is roughly equivalent to

def func(): pass
func = f1(arg)(f2(func))

except that the original function is not temporarily bound to the name func.
---

I questioned on the issue whether the non-binding optimization "should 
it be documented as a guaranteed language feature or as just an optional 
optimization?"


--
Terry Jan Reedy


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PG4UIVT3XEWUS5TVO4MJIQV47UOVV3TD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-11 Thread Steve Holden

On Wed, Aug 11, 2021 at 7:09 AM Larry Hastings  wrote:

> On 8/10/21 11:15 AM, Thomas Grainger wrote:
>
> Although the co_annoations code could intercept the NameError and replace 
> return a ForwardRef object instead of the resolved name
>
>
> No, it should raise a NameError, just like any other Python code.
> Annotations aren't special enough to break the rules.
>
> I worry about Python-the-language enshrining design choices made by the
> typing module.  Python is now on its fourth string interpolation
> technology, and it ships with three command-line argument parsing
> libraries; in each of these cases, we were adding a New Thing that was
> viewed at the time as an improvement over the existing thing(s).  It'd be
> an act of hubris to assert that the current "typing" module is the
> ultimate, final library for expressing type information in Python.  But if
> we tie the language too strongly to the typing module, I fear we could
> strangle its successors in their cribs.
>
>
> Which would be unfortunate given the (explicit?) assurances that
annotations would be optional; they are casting their shadow over the whole
language.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XFR3KNKDYYXDXVQNDP6XPTGBBHA32AVI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-11 Thread Steve Holden

On Wed, Aug 11, 2021 at 2:27 PM Guido van Rossum  wrote:

> As it happens, I have a working prototype of lazy in marshaling that would
> work well for this.
>
> Nice to see you keep the time machine in working order ...

Kind regards,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PIMCZCDZ2SN2QDAFE3ORZTURIURHYF4K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-11 Thread Jukka Lehtosalo

On Wed, Aug 11, 2021 at 2:56 PM Thomas Grainger  wrote:

> Would:
> ```
> @dataclass
> class Node:
> global_node: __class__ | None
> ```
>
> "Just work" with co_annotations?
>

This feels too specialized to me.

It would be great to also handle forward references to other classes and
cyclic references, which are also somewhat common:

@dataclass
class NodeA:
component: NodeB | None

@dataclass
class NodeB:
component: NodeA | None

Another, slightly more complex example would be cyclic references within
two modules in an import cycle. For example, NodeA and NodeB could be
defined in different modules. The common thing is that the dependency cycle
can only be fully resolved after we have created both type objects. The
import cycle case is probably less common but I've seen it in real-world
code.

Jukka
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XFS6VSXGV6QMEY5AB7VO64OZMHAYSDSZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

Would:
```
@dataclass
class Node:
global_node: __class__ | None
```

"Just work" with co_annotations?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ONQM6PTLD5QFC6UAJIYH437XFMAEEOAD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-11 Thread Guido van Rossum

Oh, I agree it shouldn’t reference the typing module. But it should not
raise NameError. This whole thing already is a special case. We can debate
what else it should, e.g. skip the name, return a fixed error token, return
an error token that includes the name that failed (this is part if the
NameError), return a string, etc…

FWIW the “tiny” change to decorator syntax looks untenable.

On Tue, Aug 10, 2021 at 23:09 Larry Hastings  wrote:

> On 8/10/21 11:15 AM, Thomas Grainger wrote:
>
> Although the co_annoations code could intercept the NameError and replace 
> return a ForwardRef object instead of the resolved name
>
>
> No, it should raise a NameError, just like any other Python code.
> Annotations aren't special enough to break the rules.
>
> I worry about Python-the-language enshrining design choices made by the
> typing module.  Python is now on its fourth string interpolation
> technology, and it ships with three command-line argument parsing
> libraries; in each of these cases, we were adding a New Thing that was
> viewed at the time as an improvement over the existing thing(s).  It'd be
> an act of hubris to assert that the current "typing" module is the
> ultimate, final library for expressing type information in Python.  But if
> we tie the language too strongly to the typing module, I fear we could
> strangle its successors in their cribs.
>
>
> */arry*
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/5NUVYOLIHEHS373WEK3D2ZGXSWPI5XV7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CJRLXDSPFEP3V2PN2YIGQHXTQ6A262ZN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-11 Thread Guido van Rossum

As it happens, I have a working prototype of lazy in marshaling that would
work well for this.

On Wed, Aug 11, 2021 at 06:07 Larry Hastings  wrote:

> On 8/11/21 5:21 AM, Inada Naoki wrote:
>
> But memory footprint and GC time is still an issue.
> Annotations in PEP 649 semantics can be much heavier than docstrings.
>
>
> I'm convinced that, if we accept PEP 649 (or something like it), we can
> reduce its CPU and memory consumption.
>
> Here's a slightly crazy idea I had this morning: what if we didn't
> unmarshall the code object for co_annotation during the initial import, but
> instead lazily loaded it on demand?  The annotated object would retain
> knowledge of what .pyc file to load, and what offset the co_annotation code
> object was stored at.  (And, if the co_annotations function had to be a
> closure, a reference to the closure tuple.)  If the user requested
> __annotations__ (or __co_annotations__), the code would open the .pyc file,
> unmarshall it, bind it, etc.  Obviously this would only work for code
> loaded from .pyc (etc) files.  To go even crazier, the runtime could LRU
> cache N (maybe == 1) open .pyc file handles as a speed optimization,
> perhaps closing them after some wall-clock timeout.
>
> I doubt we'd do exactly this--it's easy to find problems with the
> approach.  But maybe this idea will lead to a better one?
>
>
> */arry*
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/JMLXOEV6GRBVPNHQD6CYEW63I7WJBWZC/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6I4WN5IAPXLIP4JHGWIJ6MIOFUZQU62E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

Larry Hastings wrote:
> On 8/11/21 2:48 AM, Jukka Lehtosalo wrote:
> > On Wed, Aug 11, 2021 at 10:32 AM Thomas Grainger  > mailto:tagr...@gmail.com> wrote:
> > Larry Hastings wrote:
> > > On 8/11/21 12:02 AM, Thomas Grainger wrote:
> > > > I think as long as there's a test case for something like
> > > > @dataclass
> > > > class Node:
> > > >      global_node: ClassVar[Node | None]
> > > >      left: InitVar[Node | None]
> > > >      right: InitVar[None | None]
> > > >
> > > > the bug https://bugs.python.org/issue33453
> >  and the current
> > implementation
> > https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6.
> > ..
> > shows this is a tricky problem
> > > > The most straightforward workaround for this is to skip the
> > decorator
> > > syntax.  With PEP 649 active, this code should work:
> > > class Node:
> > >          global_node: ClassVar[Node | None]
> > >          left: InitVar[Node | None]
> > >          right: InitVar[None | None]
> > >     Node = dataclass(Node)
> > > //arry/
> > 
> > the decorator version simply has to work
> > 
> > 
> > 
> > I also think that it would be unfortunate if the decorator version 
> > wouldn't work. This is a pretty basic use case.
> > So, here's an idea, credit goes to Eric V. Smith.  What if we tweak how 
> decorators work, /jst slghtly/, so that they work like the 
> workaround code above?
> Specifically: currently, decorators are called just after the function 
> or class object is created, before it's bound to a variable.  But we 
> could change it so that we first bind the variable to the initial value, 
> then call the decorator, then rebind.  That is, this code:
> @dekor8
> class C:
>      ...
> would become equivalent to this code:
> class C:
>      ...
> C = dekorate(C)
> This seems like it would solve the class self-reference problem--the 
> "Node" example above--when PEP 649 is active.
> This approach shouldn't break reasonable existing code.  That said, this 
> change would be observable from Python, and pathological code could 
> notice and break.  For example:
> def ensure_Foo_is_a_class(o):
>      assert isinstance(Foo, type)
>      return o
> class Foo:
>      ...
> @ensure_Foo_is_a_class
> def Foo():
>      ...
> This terrible code currently would not raise an assertion.  But if we 
> made the proposed change to the implementation of decorators, it would.  
> I doubt anybody does this sort of nonsense, I just wanted to fully flesh 
> out the topic.
> If this approach seems interesting, here's one wrinkle to iron out.  
> When an object has multiple decorators, would we want to re-bind after 
> each decorator call?  That is, would
> @dekor1
> @dekor2
> @dekor3
> class C:
>      ...
> turn into approach A:
> class C:
>      ...
> C = dekor1(dekor2(dekor3(C)))
> or approach B:
> class C:
>      ...
> C = dekor3(C)
> C = dekor2(C)
> C = dekor1(C)
> I definitely think "approach B" makes more sense.
> //arry/


You mention that you wanted this to work also for non-type hint usage of 
annotations, and so a ForwardRef won't work. As such, would you also change 
this for function decorators so you can do this?

```
@decorator
@typing.no_type_check
def ham(spam: ham):
...

So it means:

```
def ham(spam: ham):
...
ham = typing.no_type_check(ham)
ham = decorator(ham)
```

Obviously it's meaningless as a type hint, hence the no_type_check opt out
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OJKTIGCHNFTW224O6OM4NIZ3OZBK2YQI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations


On 8/11/21 5:21 AM, Inada Naoki wrote:

But memory footprint and GC time is still an issue.
Annotations in PEP 649 semantics can be much heavier than docstrings.



I'm convinced that, if we accept PEP 649 (or something like it), we can 
reduce its CPU and memory consumption.


Here's a slightly crazy idea I had this morning: what if we didn't 
unmarshall the code object for co_annotation during the initial import, 
but instead lazily loaded it on demand?  The annotated object would 
retain knowledge of what .pyc file to load, and what offset the 
co_annotation code object was stored at. (And, if the co_annotations 
function had to be a closure, a reference to the closure tuple.)  If the 
user requested __annotations__ (or __co_annotations__), the code would 
open the .pyc file, unmarshall it, bind it, etc.  Obviously this would 
only work for code loaded from .pyc (etc) files.  To go even crazier, 
the runtime could LRU cache N (maybe == 1) open .pyc file handles as a 
speed optimization, perhaps closing them after some wall-clock timeout.


I doubt we'd do exactly this--it's easy to find problems with the 
approach.  But maybe this idea will lead to a better one?



//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JMLXOEV6GRBVPNHQD6CYEW63I7WJBWZC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-11 Thread Inada Naoki


> 2021/08/11 19:22、Paul Moore のメール:
> 
> Also, I don't think that improving performance is a
> justification for a non-trivial backward compatibility break (I don't
> recall a case where we've taken that view in the past) so "PEP 649
> solves forward references without a backward compatibility impact, and
> performance is a small issue in the face of that" is a reasonable
> position to take.

OK. I will stop talking about import time.

But memory footprint and GC time is still an issue.
Annotations in PEP 649 semantics can be much heavier than docstrings.

So I want to measure them before deprecating 563.

IMHO, I don't think we can make PEP 563 default anymore.
FastAPI ecosystem changed how annotations are used after PEP 563 was accepted.

So I think two or three options are on the table:

a. Keep Python 3.10 state. PEP 563 is opt-in in foreseeable future.
b. Accept PEP 649 and deprecate both of old semantics and PEP 563.
c?. Accept PEP 649 and deprecate old semantics, but keep PEP 563 as opt-in.

(I exclude "Accept PEP 649 but keep old semantics" option because backward 
compatibility is not a problem if we keep old semantics.)

Of course, (b) is the most simple solution.
But I am not sure that (b) don't have significant memory usage and GC time 
regression.

We can keep (a) in Python 3.11 so we don't have to harry about accepting PEP 
649.

Regards,

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QTAXVWFNF632TPCCLNFJTAXPRMV45SO6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-11 Thread Chris Angelico

On Wed, Aug 11, 2021 at 10:03 PM Larry Hastings  wrote:
> Specifically: currently, decorators are called just after the function or 
> class object is created, before it's bound to a variable.  But we could 
> change it so that we first bind the variable to the initial value, then call 
> the decorator, then rebind.  That is, this code:
>
> @dekor8
> class C:
> ...
>
> would become equivalent to this code:
>
> class C:
> ...
> C = dekorate(C)
>
> This seems like it would solve the class self-reference problem--the "Node" 
> example above--when PEP 649 is active.

Critically, the decorator itself would have to be evaluated *before*
that assignment. So it would actually have to work out more like:

_tmp = dekor8
class C:
...
C = _tmp(C)

Otherwise things like "@foo.setter" would break.

> This approach shouldn't break reasonable existing code.  That said, this 
> change would be observable from Python, and pathological code could notice 
> and break.  For example:
>
> def ensure_Foo_is_a_class(o):
> assert isinstance(Foo, type)
> return o
>
> class Foo:
> ...
>
> @ensure_Foo_is_a_class
> def Foo():
> ...
>
> This terrible code currently would not raise an assertion.  But if we made 
> the proposed change to the implementation of decorators, it would.  I doubt 
> anybody does this sort of nonsense, I just wanted to fully flesh out the 
> topic.
>

You would be here declaring that a @monkeypatch decorator is terrible
code. I'm not sure whether you're right or wrong. You may very well be
right.

def monkeypatch(cls):
basis = globals()[cls.__name__]
for attr in dir(cls): setattr(basis, attr, getattr(cls, attr))
return basis

@monkeypatch
class SomeClass:
def new_method(self): ...

Currently this works, since SomeClass doesn't get assigned yet. This
could be made to work across versions by writing it as
@monkeypatch(SomeClass) instead (and then the actual class name would
become immaterial).

> If this approach seems interesting, here's one wrinkle to iron out.  When an 
> object has multiple decorators, would we want to re-bind after each decorator 
> call?  That is, would
>
> @dekor1
> @dekor2
> @dekor3
> class C:
> ...
>
> turn into approach A:
>
> class C:
> ...
> C = dekor1(dekor2(dekor3(C)))
>
> or approach B:
>
> class C:
> ...
> C = dekor3(C)
> C = dekor2(C)
> C = dekor1(C)
>
> I definitely think "approach B" makes more sense.
>

If you're going to do it at all, best go the whole way. Each decorator
functions independently, and a decorated class is a class like any
other, so approach B makes far more sense to me.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KSXPPKYUPWYOGW2MROMQJRXPWBEH5BIW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations



On 8/11/21 2:48 AM, Jukka Lehtosalo wrote:
On Wed, Aug 11, 2021 at 10:32 AM Thomas Grainger > wrote:


Larry Hastings wrote:
> On 8/11/21 12:02 AM, Thomas Grainger wrote:
> > I think as long as there's a test case for something like
> > @dataclass
> > class Node:
> >      global_node: ClassVar[Node | None]
> >      left: InitVar[Node | None]
> >      right: InitVar[None | None]
> >
> > the bug https://bugs.python.org/issue33453
 and the current
implementation

https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6.

..
shows this is a tricky problem
> > The most straightforward workaround for this is to skip the
decorator
> syntax.  With PEP 649 active, this code should work:
> class Node:
>          global_node: ClassVar[Node | None]
>          left: InitVar[Node | None]
>          right: InitVar[None | None]
>     Node = dataclass(Node)
> //arry/

the decorator version simply has to work


I also think that it would be unfortunate if the decorator version 
wouldn't work. This is a pretty basic use case.



So, here's an idea, credit goes to Eric V. Smith.  What if we tweak how 
decorators work, /jst slghtly/, so that they work like the 
workaround code above?


Specifically: currently, decorators are called just after the function 
or class object is created, before it's bound to a variable.  But we 
could change it so that we first bind the variable to the initial value, 
then call the decorator, then rebind.  That is, this code:


   @dekor8
   class C:
    ...

would become equivalent to this code:

   class C:
    ...
   C = dekorate(C)

This seems like it would solve the class self-reference problem--the 
"Node" example above--when PEP 649 is active.


This approach shouldn't break reasonable existing code.  That said, this 
change would be observable from Python, and pathological code could 
notice and break.  For example:


   def ensure_Foo_is_a_class(o):
    assert isinstance(Foo, type)
    return o

   class Foo:
    ...

   @ensure_Foo_is_a_class
   def Foo():
    ...

This terrible code currently would not raise an assertion.  But if we 
made the proposed change to the implementation of decorators, it would.  
I doubt anybody does this sort of nonsense, I just wanted to fully flesh 
out the topic.



If this approach seems interesting, here's one wrinkle to iron out.  
When an object has multiple decorators, would we want to re-bind after 
each decorator call?  That is, would


   @dekor1
   @dekor2
   @dekor3
   class C:
    ...

turn into approach A:

   class C:
    ...
   C = dekor1(dekor2(dekor3(C)))

or approach B:

   class C:
    ...
   C = dekor3(C)
   C = dekor2(C)
   C = dekor1(C)

I definitely think "approach B" makes more sense.


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OA4J23TU3XACE5NAUUQNQQ52BXGNHUIS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-11 Thread Paul Moore

On Wed, 11 Aug 2021 at 07:37, Larry Hastings  wrote:
>
[...]
> I think PEP 649 should be considered in the same way.  In my opinion, the 
> important thing is to figure out what semantics we want for the language.

I think this is a very important point. As originally implemented,
annotations allowed attaching arbitrary data to function arguments and
return values. That was a simple language capability, with no
particular nuances, although the intended use case was very much type
annotations. Later, the decision was made that typing would be the
primary and "official" use for annotations - but even at that point,
the semantics were still impartial.

String annotations are changing the fundamental semantics of
annotations (introspection returns the source code string, not a
Python object) and there's even discussion of how this allows the
possibility for annotations to not even conform to standard Python
syntax. Surely, before we start changing the fundamental syntax and
semantics of a language feature for a single (albeit officially
blessed) use case, we should look at what capabilities *in the
abstract* are missing from the current behaviour, and work out how or
if we can implement them. Proposers on python-ideas are routinely
asked to demonstrate how their suggestion affects cases other than
their particular use case. Why shouldn't we hold typing to a similar

PEP 563 states that it's addressing two issues:

* Forward references for type hints
* Module import time cost of having type hints at all

Both of those are real and valid problems that deserve a solution.

PEP 563 is asserting that no solution is possible within the existing
semantics, and it's necessary to significantly break backward
compatibility in terms of semantics, and also opens up the possibility
of annotations not even being normal Python syntax in future (the PEP
says that any such change would need a further PEP, but definitely
does *not* say that the possibility is out of the question). PEP 649,
in my opinion, is exploring the option of addressing these two issues
*within Python's existing syntax and semantics*. How we look at it IMO
depends significantly on how we view the whole question of typing and
what it means for typing to be "optional".

Personally, my experience is that typing is indeed optional for small
or personal projects. But for larger projects, where there's a team of
people involved, it's very hard to argue against typing. Even if
you're not convinced personally that the benefits justify the costs,
it's hard to make that case persuasive, so typing pretty much always
wins.

With that in mind, if we *are* saying (as PEP 563 is, in effect) that
we're at the breaking point where we cannot support the typing use
case with existing Python syntax (note that PEP 563 essentially makes
annotations a new syntax for string literals[1]) then I would far
prefer it if Python gave up on this ambivalent attitude towards
annotations, and made a proper commitment to having a way of
expressing types within the language. That means defining type syntax
as Python *language* syntax and treating type syntax as subject to all
of the compatibility guarantees that the rest of the language adheres
to. It may still mean that we define new, type-specific syntax, but we
can put it in the language definition, and say that it's only valid in
annotations. That's not a showstopper. And it means that we don't open
ourselves up to tool-specific interpretation of core Python syntax.

What that would mean is:

1. What is valid as an annotation would probably need to be specified
once and for all at the language level. That's a compatibility break,
but we'd be making it with our eyes open, having found that the "a
single Python expression" approach isn't sufficient for the typing use
case.
2. The forward reference issue is treated as a *python language*
issue, not just a typing issue. It may be that we decide to only
*solve* it for typing, but we can still look to the wider context for
ideas and inspiration.
3. The idea of having "special syntax" for type expressions would be
dropped - it's not "special syntax" if it's part of the language, any
more than assignment expressions or conditional expressions are
"special syntax".

It's possible that this is something that's already been discussed in
the typing community. But if so, then that's a discussion that should
have involved python-dev, as it's fairly basic to how Python as a
language is evolving.

On the other hand, PEP 647 is saying that we think that typing can
still be supported with existing syntax, and we don't need to (yet)
have that discussion. Although maybe the typing experts have further
plans (the whole "support non-Python syntax" hint suggests that they
might) that mean new syntax is an inevitability - but if so, let's get
that proposal out in the open where it can be discussed, and not use
PEP 563 as a "back door" to avoid that discussion.

[...]
>  Once we figure out what semantics we

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-11 Thread Jukka Lehtosalo

On Wed, Aug 11, 2021 at 10:32 AM Thomas Grainger  wrote:

> Larry Hastings wrote:
> > On 8/11/21 12:02 AM, Thomas Grainger wrote:
> > > I think as long as there's a test case for something like
> > > @dataclass
> > > class Node:
> > >  global_node: ClassVar[Node | None]
> > >  left: InitVar[Node | None]
> > >  right: InitVar[None | None]
> > >
> > > the bug https://bugs.python.org/issue33453 and the current
> implementation
> https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6...
> shows this is a tricky problem
> > > The most straightforward workaround for this is to skip the decorator
> > syntax.  With PEP 649 active, this code should work:
> > class Node:
> >  global_node: ClassVar[Node | None]
> >  left: InitVar[Node | None]
> >  right: InitVar[None | None]
> > Node = dataclass(Node)
> > //arry/
>
> the decorator version simply has to work
>

I also think that it would be unfortunate if the decorator version wouldn't
work. This is a pretty basic use case. If we go through a lot of trouble to
redesign how type annotations behave, it would be great if we could address
as many common pain points as possible. Otherwise I see a risk that we'll
have another (third!) redesign in Python 3.12 or 3.13.

Jukka
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6U27SVAIHIRTFFJVKJWCNIXEJM6YFO6H/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

Larry Hastings wrote:
> On 8/11/21 12:02 AM, Thomas Grainger wrote:
> > I think as long as there's a test case for something like
> > @dataclass
> > class Node:
> >  global_node: ClassVar[Node | None]
> >  left: InitVar[Node | None]
> >  right: InitVar[None | None]
> > 
> > the bug https://bugs.python.org/issue33453 and the current implementation 
> > https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6...
> >  shows this is a tricky problem
> > The most straightforward workaround for this is to skip the decorator 
> syntax.  With PEP 649 active, this code should work:
> class Node:
>  global_node: ClassVar[Node | None]
>  left: InitVar[Node | None]
>  right: InitVar[None | None]
> Node = dataclass(Node)
> //arry/

the decorator version simply has to work
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2SHPHLZ2QV7YVCIC4PFPNLIHTW2WZP3A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations


On 8/11/21 12:02 AM, Thomas Grainger wrote:

I think as long as there's a test case for something like

```
@dataclass
class Node:
 global_node: ClassVar[Node | None]
 left: InitVar[Node | None]
 right: InitVar[None | None]
```

the bug https://bugs.python.org/issue33453 and the current implementation 
https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff61a2/Lib/dataclasses.py#L658-L714
 shows this is a tricky problem



The most straightforward workaround for this is to skip the decorator 
syntax.  With PEP 649 active, this code should work:


   class Node:
global_node: ClassVar[Node | None]
left: InitVar[Node | None]
right: InitVar[None | None]
   Node = dataclass(Node)


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZWTTKKIOVA7UYKX4SFWJ6DZ5X72Z2O4P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

I think as long as there's a test case for something like

```
@dataclass
class Node:
global_node: ClassVar[Node | None]
left: InitVar[Node | None]
right: InitVar[None | None]
```

the bug https://bugs.python.org/issue33453 and the current implementation 
https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff61a2/Lib/dataclasses.py#L658-L714
 shows this is a tricky problem
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FACVXQOACURN5PJROQLFADLCFCXUMP7K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations


On 8/9/21 8:25 PM, Inada Naoki wrote:

Currently, reference implementation of PEP 649 has been suspended.
We need to revive it and measure performance/memory impact.


Perhaps this sounds strange--but I don't actually agree.

PEP 563 was proposed to solve a forward-references problem for the 
typing community.  If you read the PEP, you'll note it contains no 
discussion about its impact on CPU performance or memory consumption.  
The performance of its prototype was fine, and in any case, the 
important thing was to solve the problem in the language.


I think PEP 649 should be considered in the same way.  In my opinion, 
the important thing is to figure out what semantics we want for the 
language.  Once we figure out what semantics we want, we should 
implement them, and only /then/ should we start worrying about 
performance.  Fretting about performance at this point is premature and 
a distraction.


I assert PEP 649's performance and memory use is already acceptable, 
particularly for a prototype.  And I'm confident that if PEP 649 is 
accepted, the core dev community will find endless ways to optimize the 
implementation.




As far as I remember, the reference implementation created a function
object for each methods.


First, it's only for methods with annotations.  Second, it only stores a 
code object on the annotated function; the co_annotations function is 
lazily created.  The exception to this is when co_annotations needs a 
closure; in that case, it currently creates and binds the co_annotation 
function non-lazily.  Obviously this could be done lazily too, I didn't 
bother for the prototype.



//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R6QP2SQCJ3KBKUFXXA5HZW7GY2XAJQRD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

What about using a coroutine of `v = yield (name, scope)` so the caller can 
choose how and when names are resolved?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RDQXLGRLCF6OWLU5ANDUSXJURNMKCIBL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations


On 8/10/21 11:15 AM, Thomas Grainger wrote:

Although the co_annoations code could intercept the NameError and replace 
return a ForwardRef object instead of the resolved name



No, it should raise a NameError, just like any other Python code.  
Annotations aren't special enough to break the rules.


I worry about Python-the-language enshrining design choices made by the 
typing module.  Python is now on its fourth string interpolation 
technology, and it ships with three command-line argument parsing 
libraries; in each of these cases, we were adding a New Thing that was 
viewed at the time as an improvement over the existing thing(s).  It'd 
be an act of hubris to assert that the current "typing" module is the 
ultimate, final library for expressing type information in Python.  But 
if we tie the language too strongly to the typing module, I fear we 
could strangle its successors in their cribs.



//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5NUVYOLIHEHS373WEK3D2ZGXSWPI5XV7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations


On 8/10/21 11:09 AM, Damian Shaw wrote:
Could PEP 649 be modified to say that if a NameError is raised the 
result is not cached and therefore you can inspect it later at runtime 
to get the real type once it is defined? Wouldn't that then allow 
users to write code that allows for all use cases under this scenario?



The PEP doesn't say so explicitly, but that was the intent of the 
design, yes.  If you look at the pseudo-code in the "__co_annotations__" 
section:


   https://www.python.org/dev/peps/pep-0649/#co-annotations

you'll see that it doesn't catch NameError, allowing it to bubble up to 
user code.  The prototype intentionally behaves the same way.  Certainly 
it wouldn't hurt to mention that explicitly in the PEP.



//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Q4QHBAXUUV3KVD2ZLMNZW3K64A6CQV7B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-10 Thread Guido van Rossum

On Tue, Aug 10, 2021 at 11:41 AM Jelle Zijlstra 
wrote:

>
>
> El mar, 10 ago 2021 a las 11:20, Thomas Grainger ()
> escribió:
>
>> Although the co_annoations code could intercept the NameError and replace
>> return a ForwardRef object instead of the resolved name
>>
> I implemented a version of this in
> https://github.com/larryhastings/co_annotations/pull/3 but Larry didn't
> like it.
>

However, I do like it, despite all Larry's arguing against it.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OHXA3BUUB4WVHXKLNG4I2WKRPCZIEW2U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

Jelle Zijlstra wrote:
> El mar, 10 ago 2021 a las 11:20, Thomas Grainger (tagr...@gmail.com)
> escribió:
> > Although the co_annoations code could intercept the NameError and replace
> > return a ForwardRef object instead of the resolved name
> > I implemented a version of this in
> https://github.com/larryhastings/co_annotations/pull/3 but Larry didn't
> like it.
> > 
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> > https://mail.python.org/archives/list/python-dev@python.org/message/46OW5FQP...
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
I'd like the updated PEP to mention this approach and describe if/why it wasn't 
included
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UMDSJGZZFUCO3LC6EPXU76ZTCRBYID57/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-10 Thread Jelle Zijlstra

El mar, 10 ago 2021 a las 11:20, Thomas Grainger ()
escribió:

> Although the co_annoations code could intercept the NameError and replace
> return a ForwardRef object instead of the resolved name
>
I implemented a version of this in
https://github.com/larryhastings/co_annotations/pull/3 but Larry didn't
like it.


> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/46OW5FQPHG54VIVKY2OQGLNITTPNCJXV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R2EKO6OMVRQ3OJERDHRPD3SG4FQSA4PA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

Although the co_annoations code could intercept the NameError and replace 
return a ForwardRef object instead of the resolved name
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/46OW5FQPHG54VIVKY2OQGLNITTPNCJXV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-10 Thread Damian Shaw

> dataclasses need to check for ClassVar

Interesting, so the use case we are talking about is: 1) You are using
annotations to mean actual types, 2) But you also have to inspect them at
runtime, 3) For some of the types the name might not be defined at runtime
yet

In this example doesn't the current behavior, PEP 649, and PEP 563 (using
get_type_hints) all throw an exception?

Could PEP 649 be modified to say that if a NameError is raised the result
is not cached and therefore you can inspect it later at runtime to get the
real type once it is defined? Wouldn't that then allow users to write code
that allows for all use cases under this scenario?

- Damian (he/him)


On Tue, Aug 10, 2021 at 1:55 PM Thomas Grainger  wrote:

> dataclasses need to check for ClassVar
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WOTCUBAO62WRRCCQAO6ZJNASSMK36MDW/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M3XEKV7KKAGG5QYR2XZSC35S7V4OALJS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

Damian Shaw wrote:
> > dataclasses need to check for ClassVar
> > Interesting, so the use case we are talking about is: 1) You are using
> annotations to mean actual types, 2) But you also have to inspect them at
> runtime, 3) For some of the types the name might not be defined at runtime
> yet
> In this example doesn't the current behavior, PEP 649, and PEP 563 (using
> get_type_hints) all throw an exception?
> Could PEP 649 be modified to say that if a NameError is raised the result
> is not cached and therefore you can inspect it later at runtime to get the
> real type once it is defined? Wouldn't that then allow users to write code
> that allows for all use cases under this scenario?
> - Damian (he/him)
> On Tue, Aug 10, 2021 at 1:55 PM Thomas Grainger tagr...@gmail.com wrote:
> > dataclasses need to check for ClassVar
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> > https://mail.python.org/archives/list/python-dev@python.org/message/WOTCUBAO...
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
Yep it would need a note on the pep. It's a narrow usecase and everyone who 
does use the feature (attrs, pydantic etc) are watching this PEP carefully so 
it shouldn't be too much of a comparability concern
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TVFMEVEUUQSDQLEN5COUVXLYDXMWFMXM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

dataclasses need to check for ClassVar
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WOTCUBAO62WRRCCQAO6ZJNASSMK36MDW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-10 Thread Carl Meyer

On Mon, Aug 9, 2021 at 9:31 PM Inada Naoki  wrote:
> Currently, reference implementation of PEP 649 has been suspended.
> We need to revive it and measure performance/memory impact.

I volunteered to check performance impact in practice on the Instagram
codebase, which is almost fully annotated. However, when I tried, I
found that the reference implementation of PEP 649 wasn't even able to
import its own test file without a crash. I detailed the problem in
https://github.com/larryhastings/co_annotations/issues/12 a couple
months ago, but haven't gotten any response. Still willing to do this
testing on the IG codebase, but it seems like the implementation needs
some additional work before that will be possible.

Carl
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BOHZB2NBTSKMK5RXIJ75JLFIR6GVYY57/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-10 Thread Damian Shaw

> In this case PEP 649 doesn't help.

Sorry for the naive question but why doesn't PEP 649 help here? Is there
something fundamental about the dataclass that needs to inspect the type of
C.a to create the dataclass?

- Damian (he/him)

On Tue, Aug 10, 2021 at 1:10 PM Łukasz Langa  wrote:

>
> > On 10 Aug 2021, at 13:05, Eric V. Smith  wrote:
> >
> > If 649 is accepted, there will be few places where stringified
> annotations will be needed. For example, forward references that are
> defined before __annotations__ is examined will not need to be specified as
> strings. From the PEP: "Actually, annotations would become much easier to
> use, as they would now also handle forward references.
>
> In general, yes, agreed. However, a popular special case that you're
> intimately familiar with are class decorators. Since those are executed
> quite eagerly, you'll have to account for failing evaluation of forward
> references. Example:
>
> from dataclasses import dataclass
>
> @dataclass
> class C:
> a: A
>
> class A:
> ...
>
>
> In this case PEP 649 doesn't help. I don't see evaluation failures
> discussed explicitly in the PEP but I assume it would be a NameError. It
> would also be interesting to see if we can somehow make it handle
> annotations that refer to the class they're used in. Example:
>
> @dataclass
> class Node:
> parent: Node
>
> - Ł
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/FVJAUJ4TOYLAFJJS7UUPLFETXIBFBBEE/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6S5VOQQGEL74O7WCBAJECIZTQ5KNCTJD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-10 Thread Łukasz Langa

> On 10 Aug 2021, at 13:05, Eric V. Smith  wrote:
> 
> If 649 is accepted, there will be few places where stringified annotations 
> will be needed. For example, forward references that are defined before 
> __annotations__ is examined will not need to be specified as strings. From 
> the PEP: "Actually, annotations would become much easier to use, as they 
> would now also handle forward references.

In general, yes, agreed. However, a popular special case that you're intimately 
familiar with are class decorators. Since those are executed quite eagerly, 
you'll have to account for failing evaluation of forward references. Example:

from dataclasses import dataclass

@dataclass
class C:
a: A

class A:
...

In this case PEP 649 doesn't help. I don't see evaluation failures discussed 
explicitly in the PEP but I assume it would be a NameError. It would also be 
interesting to see if we can somehow make it handle annotations that refer to 
the class they're used in. Example:

@dataclass
class Node:
parent: Node

- Ł

signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FVJAUJ4TOYLAFJJS7UUPLFETXIBFBBEE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-10 Thread Inada Naoki

On Tue, Aug 10, 2021 at 5:11 PM Mark Shannon  wrote:
>
> >>
> >
> > Currently, reference implementation of PEP 649 has been suspended.
> > We need to revive it and measure performance/memory impact.
> >
> > As far as I remember, the reference implementation created a function
> > object for each methods.
>
> No function object is created under normal circumstances.
> __annotations__ is a property that calls the underlying
> __co_annotations__ property, which lazily creates a callable.
>
> I'll leave it to Larry to explain why __co_annotations__ isn't just a
> code object.
>

I am talking about methods.
As far as I remember, function objects are created for each method to
keep class namespace.
Larry explained it and possible optimization. That's what I am waiting for.
https://mail.python.org/archives/list/python-dev@python.org/message/2OOCEE6OMBQYEIJXEGFWIBE62VPIJHP5/

Regards,

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ROPYNKOG5GJIM233LEESA5AH75W7G2YI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-10 Thread Eric V. Smith

On 8/9/2021 11:25 PM, Inada Naoki wrote:

On Tue, Aug 10, 2021 at 12:30 AM Eric V. Smith wrote:

Personally, I'd like to see PEP 649 accepted. There are a number of
issues on bpo where people expect dataclass's field.type to be an actual
Python object representing a type, and not a string. This field is
copied directly from __annotations__. If we stick with PEP 563's string
annotations, I'll probably just close these issues as "won't fix", and
tell the caller they need to convert the strings to Python objects
themselves. If 649 is accepted, then they'll all get their wish, and in
addition I can remove some ugly logic from dataclasses.

I don't think there is much difference.

PEP 563 is not default.
But the intent is/was to make it the default. That was the case in 3.10,
up until right before the release of beta 1.

And PEP 563 is not the only source of
stringified annotation.
So Accepting PEP 649 doesn't mean "they'll all get their wish". We
need to say "won't fix" anyway.

If 649 is accepted, there will be few places where stringified
annotations will be needed. For example, forward references that are
defined before __annotations__ is examined will not need to be specified
as strings. From the PEP: "Actually, annotations would become much
easier to use, as they would now also handle forward references.

Do we need to do anything here to move forward on this issue? I've
chatted with Larry and Mark Shannon, who have some additional thoughts
and I'm sure will chime in.

Currently, reference implementation of PEP 649 has been suspended.
We need to revive it and measure performance/memory impact.

Agreed.

--
Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/CDWMGUEUW4OCVATVGSFO7BX7CPOI2L6S/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-10 Thread Mark Shannon


On 10/08/2021 4:25 am, Inada Naoki wrote:

On Tue, Aug 10, 2021 at 12:30 AM Eric V. Smith  wrote:


Personally, I'd like to see PEP 649 accepted. There are a number of
issues on bpo where people expect dataclass's field.type to be an actual
Python object representing a type, and not a string. This field is
copied directly from __annotations__. If we stick with PEP 563's string
annotations, I'll probably just close these issues as "won't fix", and
tell the caller they need to convert the strings to Python objects
themselves. If 649 is accepted, then they'll all get their wish, and in
addition I can remove some ugly logic from dataclasses.



I don't think there is much difference.

PEP 563 is not default. And PEP 563 is not the only source of
stringified annotation.
So Accepting PEP 649 doesn't mean "they'll all get their wish". We
need to say "won't fix" anyway.



Do we need to do anything here to move forward on this issue? I've
chatted with Larry and Mark Shannon, who have some additional thoughts
and I'm sure will chime in.


My only comment concerned performance.

I won't claim that the cost of PEP 649 will be zero, but it won't be 
significantly more than PEP 563 if annotations are unused.
The cost of unmarshalling a code object will be greater than a string, 
but it won't be significant. Not only that, but we are actively looking 
to reduce startup in 3.11 which will reduce the overhead further.


If annotations *are* used, then PEP 649 should be cheaper as it relies 
on the interpreter to do the evaluation in an efficient fashion.
For users of dataclasses and Pydantic, I expect PEP 649 to outperform 
PEP 563.






Currently, reference implementation of PEP 649 has been suspended.
We need to revive it and measure performance/memory impact.

As far as I remember, the reference implementation created a function
object for each methods.


No function object is created under normal circumstances.
__annotations__ is a property that calls the underlying 
__co_annotations__ property, which lazily creates a callable.


I'll leave it to Larry to explain why __co_annotations__ isn't just a 
code object.



It means doubles function objects. It has major impact to memory
usage, startup time, and GC time.


Only if __annotations__ are widely used, in which case PEP 563 is 
probably worse.


Cheers,
Mark.



There was an idea to avoid creating function objects for most cases.
But it was not implemented.

Regards,


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BIM5KHILDVFVTIL6SUIY6TGGV5SHG5MQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-09 Thread Inada Naoki

On Tue, Aug 10, 2021 at 12:30 AM Eric V. Smith  wrote:
>
> Personally, I'd like to see PEP 649 accepted. There are a number of
> issues on bpo where people expect dataclass's field.type to be an actual
> Python object representing a type, and not a string. This field is
> copied directly from __annotations__. If we stick with PEP 563's string
> annotations, I'll probably just close these issues as "won't fix", and
> tell the caller they need to convert the strings to Python objects
> themselves. If 649 is accepted, then they'll all get their wish, and in
> addition I can remove some ugly logic from dataclasses.
>

I don't think there is much difference.

PEP 563 is not default. And PEP 563 is not the only source of
stringified annotation.
So Accepting PEP 649 doesn't mean "they'll all get their wish". We
need to say "won't fix" anyway.

> Do we need to do anything here to move forward on this issue? I've
> chatted with Larry and Mark Shannon, who have some additional thoughts
> and I'm sure will chime in.
>

Currently, reference implementation of PEP 649 has been suspended.
We need to revive it and measure performance/memory impact.

As far as I remember, the reference implementation created a function
object for each methods.
It means doubles function objects. It has major impact to memory
usage, startup time, and GC time.

There was an idea to avoid creating function objects for most cases.
But it was not implemented.

Regards,
-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QNZE4GX6RD4QLTIM4U247G5RNWFX6BOQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-09 Thread Brett Cannon

On Mon, Aug 9, 2021 at 8:31 AM Eric V. Smith  wrote:

> I'd like to revive the discussion of PEP 649
> [https://www.python.org/dev/peps/pep-0649/] that started shortly before
> development of 3.10 features was closed. This is Larry's PEP to defer
> evaluation of __annotations__ until it's actually accessed. During the
> discussion the decision was made to back out the change that made "from
> __future__ import annotations" (PEP 563
> [https://www.python.org/dev/peps/pep-0563/]) the default behavior for
> 3.10.
>
> My understanding is that PEP 649 is currently in front of the SC.


Correct.


> But do
> we need to have any additional discussion here?


I think it's worth discussing whether PEP 649 is the solution we want to
see as the solution to this problem or not?


> My recollection is that
> we backed out the PEP 563 change because we didn't feel we had enough
> time to come to a good decision one way or the other before 3.10.
>

Correct.


>
> Personally, I'd like to see PEP 649 accepted. There are a number of
> issues on bpo where people expect dataclass's field.type to be an actual
> Python object representing a type, and not a string. This field is
> copied directly from __annotations__. If we stick with PEP 563's string
> annotations, I'll probably just close these issues as "won't fix", and
> tell the caller they need to convert the strings to Python objects
> themselves. If 649 is accepted, then they'll all get their wish, and in
> addition I can remove some ugly logic from dataclasses.
>
> Do we need to do anything here to move forward on this issue? I've
> chatted with Larry and Mark Shannon, who have some additional thoughts
> and I'm sure will chime in.
>

I think the question is whether we have general consensus around PEP 649?

-Brett


>
> --
> Eric
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/2MEOWHCVDLPABOBLYUGRXVOOOBYOLLU6/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OQN6ALRTEDMRYVLKQIG3YHTNSKYTKWEJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

2021-08-09 Thread Eric V. Smith

Some prior discussions:

"PEP 563 in light of PEP 649":
https://mail.python.org/archives/list/python-dev@python.org/message/ZBJ7MD6CSGM6LZAOTET7GXAVBZB7O77O/

"In support of PEP 649":
https://mail.python.org/archives/list/python-dev@python.org/message/7VMJWFGHVXDSRQFHMXVJKDDOVT47B54T/

"PEP 563 and 649: The Great Compromise":
https://mail.python.org/archives/list/python-dev@python.org/message/WUZGTGE43T7XV3EUGT6AN2N52OD3U7AE/

I'm sure there are other threads I missed.

--
Eric

On 8/9/2021 11:27 AM, Eric V. Smith wrote:
I'd like to revive the discussion of PEP 649
[https://www.python.org/dev/peps/pep-0649/] that started shortly
before development of 3.10 features was closed. This is Larry's PEP to
defer evaluation of __annotations__ until it's actually accessed.
During the discussion the decision was made to back out the change
that made "from __future__ import annotations" (PEP 563
[https://www.python.org/dev/peps/pep-0563/]) the default behavior for
3.10.

My understanding is that PEP 649 is currently in front of the SC. But
do we need to have any additional discussion here? My recollection is
that we backed out the PEP 563 change because we didn't feel we had
enough time to come to a good decision one way or the other before 3.10.

Personally, I'd like to see PEP 649 accepted. There are a number of
issues on bpo where people expect dataclass's field.type to be an
actual Python object representing a type, and not a string. This field
is copied directly from __annotations__. If we stick with PEP 563's
string annotations, I'll probably just close these issues as "won't
fix", and tell the caller they need to convert the strings to Python
objects themselves. If 649 is accepted, then they'll all get their
wish, and in addition I can remove some ugly logic from dataclasses.

Do we need to do anything here to move forward on this issue? I've
chatted with Larry and Mark Shannon, who have some additional thoughts
and I'm sure will chime in.

--
Eric V. Smith

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/H2RZWUHNG4OROL66KIOYHV5PXKILZIAM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-17 Thread Nick Coghlan

On Sun, 18 Apr 2021, 1:59 am Jelle Zijlstra, 
wrote:

> El sáb, 17 abr 2021 a las 8:30, Nick Coghlan ()
> escribió:.
>
>>
>> Metaclass __prepare__ methods can inject names into the class namespace
>> that the compiler doesn't know about, so yeah, it unfortunately has to be
>> conservative and use LOAD_NAME in class level code.
>>
>> But of course, most metaclasses don't. I wonder if there are cases where
> the compiler can statically figure out that there are no metaclass
> shenanigans going on, and emit LOAD_GLOBAL anyway. It seems safe at least
> when the class has no base classes and no metaclass=.
>

Aye, that particular case is one the symtable pass could at least
theoretically identify.

As soon as there is a name to resolve in the class header, though, it's no
longer safe for the compiler to make assumptions :(

Cheers,
Nick.

>


>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NDYKJTWUN4HT2BPNJVBUKYVD6HOTIBMY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-17 Thread Jelle Zijlstra

El sáb, 17 abr 2021 a las 8:30, Nick Coghlan ()
escribió:

>
>
> On Fri, 16 Apr 2021, 3:14 pm Larry Hastings,  wrote:
>
>>
>> Anyway I assume it wasn't "fixable".  The compiler would presumably
>> already prefer to generate LOAD_GLOBAL vs LOAD_NAME, because LOAD_GLOBAL
>> would be cheaper every time for a global or builtin.  The fact that it
>> already doesn't do so implies that it can't.
>>
>
> Metaclass __prepare__ methods can inject names into the class namespace
> that the compiler doesn't know about, so yeah, it unfortunately has to be
> conservative and use LOAD_NAME in class level code.
>
> But of course, most metaclasses don't. I wonder if there are cases where
the compiler can statically figure out that there are no metaclass
shenanigans going on, and emit LOAD_GLOBAL anyway. It seems safe at least
when the class has no base classes and no metaclass=.


> Cheers,
> Nick.
>
>>
>> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/IZJYDHWJNMMMICUE32M3O7DGMSMVIOQ3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W4LYALS5RMLTGIU5PE5YMF4L3MWL2HXY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-17 Thread Nick Coghlan

On Fri, 16 Apr 2021, 3:14 pm Larry Hastings,  wrote:

>
> Anyway I assume it wasn't "fixable".  The compiler would presumably
> already prefer to generate LOAD_GLOBAL vs LOAD_NAME, because LOAD_GLOBAL
> would be cheaper every time for a global or builtin.  The fact that it
> already doesn't do so implies that it can't.
>

Metaclass __prepare__ methods can inject names into the class namespace
that the compiler doesn't know about, so yeah, it unfortunately has to be
conservative and use LOAD_NAME in class level code.

Cheers,
Nick.

>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IZJYDHWJNMMMICUE32M3O7DGMSMVIOQ3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-15 Thread Larry Hastings



On 4/15/21 9:24 PM, Inada Naoki wrote:

Unlike simple function case, PEP 649 creates function object instead
of code object for __co_annotation__ of methods.
It cause this overhead.  Can we avoid creating functions for each annotation?



As the implementation of PEP 649 currently stands, there are two reasons 
why the compiler might pre-bind the __co_annotations__ code object to a 
function, instead of simply storing the code object:


 * If the annotations refer to a closure ("freevars" is nonzero), or
 * If the annotations /possibly/ refer to a class variable (the
   annotations code object contains either LOAD_NAME or LOAD_CLASSDEREF).

If the annotations refer to a closure, then the code object also needs 
to be bound with the "closure" tuple.  If the annotations possibly refer 
to a class variable, then the code object also needs to be bound with 
the current "f_locals" dict.  (Both could be true.)


Unfortunately, when generating annotations on a method, references to 
builtins (e.g. "int", "str") seem to generate LOAD_NAME instructions 
instead of LOAD_GLOBAL.  Which means pre-binding the function happens 
pretty often for methods.  I believe in your benchmark it will happen 
every time.  There's a lot of code, and a lot of runtime data 
structures, inside compile.c and symtable.c behind the compiler's 
decision about whether something is NAME vs GLOBAL vs DEREF etc, and I 
wasn't comfortable with seeing if I could fix it.


Anyway I assume it wasn't "fixable".  The compiler would presumably 
already prefer to generate LOAD_GLOBAL vs LOAD_NAME, because LOAD_GLOBAL 
would be cheaper every time for a global or builtin.  The fact that it 
already doesn't do so implies that it can't.



At the moment I have only one idea for a possible optimization, as 
follows.  Instead of binding the function object immediately, it /might/ 
be cheaper to write the needed values into a tuple, then only actually 
bind the function object on demand (like normal).


I haven't tried this because I assumed the difference at runtime would 
be negligible.  On one hand, you're creating a function object; on the 
other you're creating a tuple.  Either way you're creating an object at 
runtime, and I assumed that bound functions weren't /that/ much more 
expensive than tuples.  Of course I could be very wrong about that.


The other thing is, it would be a lot of work to even try the 
experiment.  Also, it's an optimization, and I was more concerned with 
correctness... and getting it done and getting this discussion underway.



What follows are my specific thoughts about how to implement this 
optimization.


In this scenario, the logic in the compiler that generates the code 
object would change to something like this:


   has_closure = co.co_freevars != 0
   has_load_name = co.co_code does not contain LOAD_NAME or
   LOAD_CLASSDEREF bytecodes
   if not (has_closure or has_load_name):
    co_ann = co
   elif has_closure and (not has_load_name):
    co_ann = (co, freevars)
   elif (not has_closure) and has_load_name:
    co_ann = (co, f_locals)
   else:
    co_ann = (co, freevars, f_locals)
   setattr(o, "__co_annotations__", co_ann)

(The compiler would have to generate instructions creating the tuple and 
setting its members, then storing the resulting object on the object 
with the annotations.)


Sadly, we can't pre-create this "co_ann" tuple as a constant and store 
it in the .pyc file, because the whole point of the tuple is to contain 
one or more objects only created at runtime.



The code implementing __co_annotations__ in the three objects (function, 
class, module) would examine the object it got.  If it was a code 
object, it would bind it; if it was a tuple, it would unpack the tuple 
and use the values based on their type:


   // co_ann = internal storage for __co_annotations__
   if isinstance(co_ann, FunctionType) or (co_ann == None):
    return co_ann
   co, freevars, locals = None
   if isinstance(co_ann, CodeType):
    co = co_ann
   else:
    assert isinstance(co_ann, tuple)
    assert 1 <= len(co_ann) <= 3
    for o in co_ann:
        if isinstance(o, CodeObject):
            assert not co
    co = o
        elif isinstance(o, tuple):
            assert not freevars
    freevars = o
        elif isinstance(o, dict):
            assert not locals
    locals = o
        else:
            raise ValueError(f"illegal value in co_annotations
   tuple: {o!r}")
   co_ann = make_function(co, freevars=freevars, locals=locals)
   return co_ann


If you experiment with this approach, I'd be glad to answer questions 
about it, either here or on Github, etc.



Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-15 Thread Inada Naoki

>
> I will read PEP 649 implementation to find missing optimizations other
> than GH-25419 and GH-23056.
>

I found each "__co_annotation__" has own name like "func0.__co_annotation__".
It increased pyc size a little.
I created a draft pull request for cherry-picking GH-25419 and
GH-23056 and using just "__co_annotation__" as a name.
https://github.com/larryhastings/co_annotations/pull/9/commits/48a99e0aafa2dd006d72194bc1d7d47443900502

```
# previous result
$ ./python ~/ann_test.py 3
code size: 204963 bytes
memory: 209257 bytes
unmarshal: avg: 587.336ms +/-2.073ms
exec: avg: 97.056ms +/-0.046ms

# Use single name
$ ./python ~/ann_test.py 3
code size: 182088 bytes
memory: 209257 bytes
unmarshal: avg: 539.841ms +/-0.227ms
exec: avg: 98.351ms +/-0.064ms
```

It reduced code size and unmarshal time by 10%.
I confirmed GH-25419 and GH-23056 works very well. All same constants
are shared.

Unmarshal time is still slow. It is caused by unmarshaling code object.
But I don't know where is the bottleneck: parsing marshal file, or
creating code object.

---

Then, I tried to measure method annotation overhead.
Code: 
https://gist.github.com/methane/abb509e5f781cc4a103cc450e1e7925d#file-ann_test_method-py
Result:

```
# No annotation
$ ./python ~/ann_test_method.py 0
code size: 113019 bytes
memory: 256008 bytes
unmarshal: avg: 336.665ms +/-6.185ms
exec: avg: 176.791ms +/-3.067ms

# PEP 563
$ ./python ~/ann_test_method.py 2
code size: 120532 bytes
memory: 269004 bytes
unmarshal: avg: 348.285ms +/-0.102ms
exec: avg: 176.933ms +/-4.343ms

# PEP 649 (all optimization included)
$ ./python ~/ann_test_method.py 3
code size: 196135 bytes
memory: 436565 bytes
unmarshal: avg: 579.680ms +/-0.147ms
exec: avg: 259.781ms +/-7.087ms
```

PEP 563 vs 649
* code size: +63%
* memory: +62%
* import time: +60%

PEP 649 annotation overhead (compared with no annotation):
* code size: +83 byte/method
* memory: +180 byte/method
* import time: +326 us/method

It is disappointing because having thousands methods is very common
for web applications.

Unlike simple function case, PEP 649 creates function object instead
of code object for __co_annotation__ of methods.
It cause this overhead.  Can we avoid creating functions for each annotation?

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4KUXC373ZOP7YHZI6N4NKOOOB3DCL7NW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-15 Thread Larry Hastings


On 4/15/21 2:02 PM, Sebastián Ramírez wrote:

## Questions

I'm not very familiar with the internals of Python, and I'm not sure how the new syntax for 
`Union`s using the vertical bar character ("pipe", "|") work.

But would PEP 649 still support things like this?:

def run(arg: int | str = 0): pass

And would it be inspectable at runtime?



As far as I can tell, absolutely PEP 649 would support this feature.  
Under the covers, all PEP 649 is really doing is changing the 
destination that annotation expressions get compiled to.  So anything 
that works in an annotation with "stock" semantics would work fine with 
PEP 649 semantics too, barring the exceptions specifically listed in the 
PEP (e.g. annotations defined in conditionals, walrus operator, etc).



Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RD4X7EGVTY6PIT76TWQCSWC6PLHMIL6Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-15 Thread Sebastián Ramírez

Thanks Brett Cannon for suggesting to get Samuel Colvin (Pydantic) and me,
Sebastián Ramírez (FastAPI and Typer) involved in this.

TL;DR: it seems to me PEP 649 would be incredibly important/useful for
Pydantic, FastAPI, Typer, and similar tools, and their communities.

## About FastAPI, Pydantic, Typer

Some of you probably don't know what these tools are, so, in short, FastAPI is
a web API framework based on Pydantic. It has had massive growth and adoption,
and very positive feedback.

FastAPI was included for the first time in the last Python developers survey
and it's already the third most used web framework, and apparently the fastest
growing one: https://www.jetbrains.com/lp/python-developers-survey-2020/.

It was also recently recommended by ThoughtWorks for the enterprises:
https://www.thoughtworks.com/radar/languages-and-frameworks?blipid=202104087

And it's currently used in lots of organizations, some widely known, chances
are your orgs already use it in some way.

Pydantic, in very short, is a library that looks a lot like dataclasses (and
also supports them), but it uses the same type annotations not only for type
hints, but also for data validation, serialization (e.g. to JSON) and
documentation (JSON Schema).

The key feature of FastAPI (thanks to Pydantic) is using the same type
annotations for _more_ than just type hinting: data validation, serialization,
and documentation. All those features are provided by default when building an
API with FastAPI and Pydantic.

Typer is a library for building CLIs, based on Click, but using the same ideas
from type annotations, from FastAPI and Pydantic.

## Why PEP 649

You can read Samuel's message to this mailing list here:
https://mail.python.org/archives/list/python-dev@python.org/thread/7VMJWFGHVXDSRQFHMXVJKDDOVT47B54T/

And a longer discussion of how PEP 563 affects Pydantic here:
https://github.com/samuelcolvin/pydantic/issues/2678

He has done most of the work to support these additional features from type
annotations. So he would have the deepest insight into the tradeoffs/issues.

>From my point of view, just being able to use local variables in Pydantic
>models would be enough to justify PEP 649. With PEP 563, if a developer
>decides to create a Pydantic model inside a function (say a factory function)
>they would probably get an error. And it would probably not be obvious that
>they have to declare the model in the top level of the module.

The main feature of FastAPI and Pydantic is that they are very easy/intuitive
to use. People from many backgrounds are now quickly and efficiently building
APIs with best practices.

I've read some isolated comments of people that were against type annotations
in general, saying that these tools justify adopting them. And I've also seen
comments from people coming from other languages and fields, and adopting
Python just to be able to use these tools.

Many of these developers are not Python experts, and supporting them and their
intuition as much as possible when using these tools would help towards the PSF
goal to:

> [...] support and facilitate the growth of a diverse and international
> community of Python programmers.

## Community support

To avoid asking people to spam here, Samuel and I are collecting "likes" in:

* This tweet: https://twitter.com/tiangolo/status/1382800928982642692
* This issue: https://github.com/samuelcolvin/pydantic/issues/2678

I just sent that tweet, I expect/hope it will collect some likes in support by
the time you see it.

## Questions

I'm not very familiar with the internals of Python, and I'm not sure how the
new syntax for `Union`s using the vertical bar character ("pipe", "|") work.

But would PEP 649 still support things like this?:

def run(arg: int | str = 0): pass

And would it be inspectable at runtime?

## Additional comments

The original author of PEP 563 was Łukasz Langa.

I was recently chatting with him about Typer and annotations. And he showed
interest, support, and help.

I think he probably knows the benefits of the way these libraries use type
annotations and I would like/hope to read his opinion on all this.

Or alternatively, any possible ideas for how to handle these things in tools
like Pydantic.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/J7YKCCCV2GL7HUDKG7DX4INAT65SXYRF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-15 Thread Inada Naoki

I updated the benchmark little:

* Added no annotation mode for baseline performance.
* Better stats output.

https://gist.github.com/methane/abb509e5f781cc4a103cc450e1e7925d

```
# No annotation (master + GH-25419)
$ ./python ~/ann_test.py 0
code size: 102967 bytes
memory: 181288 bytes
unmarshal: avg: 299.301ms +/-1.257ms
exec: avg: 104.019ms +/-0.038ms

# PEP 563 (master + GH-25419)
$ ./python ~/ann_test.py 2
code size: 110488 bytes
memory: 193572 bytes
unmarshal: avg: 313.032ms +/-0.068ms
exec: avg: 108.456ms +/-0.048ms

# PEP 649 (co_annotations + GH-25419 + GH-23056)
$ ./python ~/ann_test.py 3
code size: 204963 bytes
memory: 209257 bytes
unmarshal: avg: 587.336ms +/-2.073ms
exec: avg: 97.056ms +/-0.046ms

# Python 3.9
$ python3 ann_test.py 0
code size: 108955 bytes
memory: 173296 bytes
unmarshal: avg: 333.527ms +/-1.750ms
exec: avg: 90.810ms +/-0.347ms

$ python3 ann_test.py 1
code size: 121011 bytes
memory: 385200 bytes
unmarshal: avg: 334.929ms +/-0.055ms
exec: avg: 400.260ms +/-0.249ms
```

## Rough estimation of annotation overhead

Python 3.9 w/o PEP 563
code (pyc) size: +11%
memory usage: +122%  (211bytes / function)
import time: +73% (*)

PEP 563
code (pyc) size: +7.3%
memory usage: +0.68%  (13.3bytes / function)
import time: +4.5%

PEP 649
code (pyc) size: +99%
memory usage: +15%  (28 bytes / function)
import time: +70%

(*) import time can be much more slower for complex annotations.

## Conclusion

* PEP 563 is close to "zero overhead" in memory consumption. And
import time overhead is ~5%. Users can write type annotations without
caring overhead.

* PEP 649 is much better than old semantics for memory usage and
import time. But import time is still longer than no annotation code.

  * The import time overhead is coming from unmarshal, not from
eval().  If we implement a "lazy load" mechanizm for docstrings and
annotations, overhead will become cheaper.
  * pyc file become bigger (but who cares?)

I will read PEP 649 implementation to find missing optimizations other
than GH-25419 and GH-23056.

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/K5PFGS6DQUHUG63UWRXYNLLMXAVELP32/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Inada Naoki

On Thu, Apr 15, 2021 at 11:09 AM Larry Hastings  wrote:
>
> Thanks for doing this!  I don't think PEP 649 is going to be accepted or 
> rejected based on either performance or memory usage, but it's nice to see 
> you confirmed that its performance and memory impact is acceptable.
>
>
> If I run "ann_test.py 1", the annotations are already turned into strings.  
> Why do you do it that way?  It makes stock semantics look better, because 
> manually stringized annotations are much faster than evaluating real 
> expressions.
>

Because `if TYPE_CHECKING` and manually stringified annotation is used
in real world applications.
I want to mix both use cases.

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MKAGFBTVZ4LXYJC5X6P3LXXXRD7P2WH5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2




Thanks for doing this!  I don't think PEP 649 is going to be accepted or 
rejected based on either performance or memory usage, but it's nice to 
see you confirmed that its performance and memory impact is acceptable.



If I run "ann_test.py 1", the annotations are already turned into 
strings.  Why do you do it that way?  It makes stock semantics look 
better, because manually stringized annotations are much faster than 
evaluating real expressions.


It seems to me that the test would be more fair if test 1 used real 
annotations.  So I added this to "lines":


   from types import SimpleNamespace
   foo = SimpleNamespace()
   foo.bar = SimpleNamespace()
   foo.bar.baz = float

I also changed quote(t) so it always returned t unchanged.  When I ran 
it that way, stock semantics "exec" time got larger.



Cheers,


//arry/

On 4/14/21 6:44 PM, Inada Naoki wrote:

I added memory usage data by tracemalloc.

```
# Python 3.9 w/ old semantics
$ python3 ann_test.py 1
code size: 121011
memory: (385200, 385200)
unmarshal: avg: 0.3341682574478909 +/- 3.700437551781949e-05
exec: avg: 0.4067857594229281 +/- 0.0006858555167675445

# Python 3.9 w/ PEP 563 semantics
$ python3 ann_test.py 2
code size: 121070
memory: (398675, 398675)
unmarshal: avg: 0.3352349083404988 +/- 7.749102039824168e-05
exec: avg: 0.24610224328935146 +/- 0.0008628035427956459

# master + optimization w/ PEP 563 semantics
$ ./python ~/ann_test.py 2
code size: 110488
memory: (193572, 193572)
unmarshal: avg: 0.31316645480692384 +/- 0.00011766086337841035
exec: avg: 0.11456295938696712 +/- 0.0017481202239372398

# co_annotations + optimization w/ PEP 649 semantics
$ ./python ~/ann_test.py 3
code size: 204963
memory: (208273, 208273)
unmarshal: avg: 0.597023528907448 +/- 0.00016614519056599577
exec: avg: 0.09546191191766411 +/- 0.00018099485135812695
```

Summary:

* Both of PEP 563 and PEP 649 has low memory consumption than Python 3.9.
* Importing time (unmarshal+exec) is about 0.7sec on old semantics and
PEP 649, 0.43sec on PEP 563.

On Thu, Apr 15, 2021 at 10:31 AM Inada Naoki  wrote:

I created simple benchmark:
https://gist.github.com/methane/abb509e5f781cc4a103cc450e1e7925d

This benchmark creates 1000 annotated functions and measure time to
load and exec.
And here is the result. All interpreters are built without --pydebug,
--enable-optimization, and --with-lto.

```
# Python 3.9 w/ stock semantics

$ python3 ~/ann_test.py 1
code size: 121011
unmarshal: avg: 0.33605549649801103 +/- 0.007382938279889738
exec: avg: 0.395090194279328 +/- 0.001004608380122509

# Python 3.9 w/ PEP 563 semantics

$ python3 ~/ann_test.py 2
code size: 121070
unmarshal: avg: 0.3407619891455397 +/- 0.0011833618746421965
exec: avg: 0.24590165729168803 +/- 0.0003123404336687428

# master branch w/ PEP 563 semantics

$ ./python ~/ann_test.py 2
code size: 149086
unmarshal: avg: 0.45410854648798704 +/- 0.00107521956753799
exec: avg: 0.11281821667216718 +/- 0.00011939747308270317

# master branch + optimization (*) w/ PEP 563 semantics
$ ./python ~/ann_test.py 2
code size: 110488
unmarshal: avg: 0.3184352931333706 +/- 0.0015278719180908732
exec: avg: 0.11042822999879717 +/- 0.00018108884723599264

# co_annotatins reference implementation w/ PEP 649 semantics

$ ./python ~/ann_test.py 3
code size: 229679
unmarshal: avg: 0.6402394526172429 +/- 0.0006400500128250688
exec: avg: 0.09774857209995388 +/- 9.275466265195788e-05

# co_annotatins reference implementation + optimization (*) w/ PEP 649 semantics

$ ./python ~/ann_test.py 3
code size: 204963
unmarshal: avg: 0.5824743471574039 +/- 0.007219086642131638
exec: avg: 0.09641968684736639 +/- 0.0001416784753249878
```

(*) I found constant folding creates new tuple every time even though
same tuple is in constant table.
See https://github.com/python/cpython/pull/25419
For co_annotations, I cherry-pick
https://github.com/python/cpython/pull/23056  too.


--
Inada Naoki  





___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PV3TWXK3WFITD2AGFQKQZEK3S26FQHRC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Inada Naoki

I added memory usage data by tracemalloc.

```
# Python 3.9 w/ old semantics
$ python3 ann_test.py 1
code size: 121011
memory: (385200, 385200)
unmarshal: avg: 0.3341682574478909 +/- 3.700437551781949e-05
exec: avg: 0.4067857594229281 +/- 0.0006858555167675445

# Python 3.9 w/ PEP 563 semantics
$ python3 ann_test.py 2
code size: 121070
memory: (398675, 398675)
unmarshal: avg: 0.3352349083404988 +/- 7.749102039824168e-05
exec: avg: 0.24610224328935146 +/- 0.0008628035427956459

# master + optimization w/ PEP 563 semantics
$ ./python ~/ann_test.py 2
code size: 110488
memory: (193572, 193572)
unmarshal: avg: 0.31316645480692384 +/- 0.00011766086337841035
exec: avg: 0.11456295938696712 +/- 0.0017481202239372398

# co_annotations + optimization w/ PEP 649 semantics
$ ./python ~/ann_test.py 3
code size: 204963
memory: (208273, 208273)
unmarshal: avg: 0.597023528907448 +/- 0.00016614519056599577
exec: avg: 0.09546191191766411 +/- 0.00018099485135812695
```

Summary:

* Both of PEP 563 and PEP 649 has low memory consumption than Python 3.9.
* Importing time (unmarshal+exec) is about 0.7sec on old semantics and
PEP 649, 0.43sec on PEP 563.

On Thu, Apr 15, 2021 at 10:31 AM Inada Naoki  wrote:
>
> I created simple benchmark:
> https://gist.github.com/methane/abb509e5f781cc4a103cc450e1e7925d
>
> This benchmark creates 1000 annotated functions and measure time to
> load and exec.
> And here is the result. All interpreters are built without --pydebug,
> --enable-optimization, and --with-lto.
>
> ```
> # Python 3.9 w/ stock semantics
>
> $ python3 ~/ann_test.py 1
> code size: 121011
> unmarshal: avg: 0.33605549649801103 +/- 0.007382938279889738
> exec: avg: 0.395090194279328 +/- 0.001004608380122509
>
> # Python 3.9 w/ PEP 563 semantics
>
> $ python3 ~/ann_test.py 2
> code size: 121070
> unmarshal: avg: 0.3407619891455397 +/- 0.0011833618746421965
> exec: avg: 0.24590165729168803 +/- 0.0003123404336687428
>
> # master branch w/ PEP 563 semantics
>
> $ ./python ~/ann_test.py 2
> code size: 149086
> unmarshal: avg: 0.45410854648798704 +/- 0.00107521956753799
> exec: avg: 0.11281821667216718 +/- 0.00011939747308270317
>
> # master branch + optimization (*) w/ PEP 563 semantics
> $ ./python ~/ann_test.py 2
> code size: 110488
> unmarshal: avg: 0.3184352931333706 +/- 0.0015278719180908732
> exec: avg: 0.11042822999879717 +/- 0.00018108884723599264
>
> # co_annotatins reference implementation w/ PEP 649 semantics
>
> $ ./python ~/ann_test.py 3
> code size: 229679
> unmarshal: avg: 0.6402394526172429 +/- 0.0006400500128250688
> exec: avg: 0.09774857209995388 +/- 9.275466265195788e-05
>
> # co_annotatins reference implementation + optimization (*) w/ PEP 649 
> semantics
>
> $ ./python ~/ann_test.py 3
> code size: 204963
> unmarshal: avg: 0.5824743471574039 +/- 0.007219086642131638
> exec: avg: 0.09641968684736639 +/- 0.0001416784753249878
> ```
>
> (*) I found constant folding creates new tuple every time even though
> same tuple is in constant table.
> See https://github.com/python/cpython/pull/25419
> For co_annotations, I cherry-pick
> https://github.com/python/cpython/pull/23056  too.
>
>
> --
> Inada Naoki  



-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JIZ6PCV5SSIL7BUKZUCVF45OFNO4H26I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Inada Naoki

I created simple benchmark:
https://gist.github.com/methane/abb509e5f781cc4a103cc450e1e7925d

This benchmark creates 1000 annotated functions and measure time to
load and exec.
And here is the result. All interpreters are built without --pydebug,
--enable-optimization, and --with-lto.

```
# Python 3.9 w/ stock semantics

$ python3 ~/ann_test.py 1
code size: 121011
unmarshal: avg: 0.33605549649801103 +/- 0.007382938279889738
exec: avg: 0.395090194279328 +/- 0.001004608380122509

# Python 3.9 w/ PEP 563 semantics

$ python3 ~/ann_test.py 2
code size: 121070
unmarshal: avg: 0.3407619891455397 +/- 0.0011833618746421965
exec: avg: 0.24590165729168803 +/- 0.0003123404336687428

# master branch w/ PEP 563 semantics

$ ./python ~/ann_test.py 2
code size: 149086
unmarshal: avg: 0.45410854648798704 +/- 0.00107521956753799
exec: avg: 0.11281821667216718 +/- 0.00011939747308270317

# master branch + optimization (*) w/ PEP 563 semantics
$ ./python ~/ann_test.py 2
code size: 110488
unmarshal: avg: 0.3184352931333706 +/- 0.0015278719180908732
exec: avg: 0.11042822999879717 +/- 0.00018108884723599264

# co_annotatins reference implementation w/ PEP 649 semantics

$ ./python ~/ann_test.py 3
code size: 229679
unmarshal: avg: 0.6402394526172429 +/- 0.0006400500128250688
exec: avg: 0.09774857209995388 +/- 9.275466265195788e-05

# co_annotatins reference implementation + optimization (*) w/ PEP 649 semantics

$ ./python ~/ann_test.py 3
code size: 204963
unmarshal: avg: 0.5824743471574039 +/- 0.007219086642131638
exec: avg: 0.09641968684736639 +/- 0.0001416784753249878
```

(*) I found constant folding creates new tuple every time even though
same tuple is in constant table.
See https://github.com/python/cpython/pull/25419
For co_annotations, I cherry-pick
https://github.com/python/cpython/pull/23056  too.


-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JEP4MXAHCVGQF7AI5OUUSGOENXAOR43O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Jim J. Jewett

Larry Hastings wrote:
> On 4/14/21 1:42 PM, Baptiste Carvello wrote:
> > Are there specific annoyances associated with quoting always, apart 
> > from the 2 more characters?

> Yes.  Since the quoted strings aren't parsed by Python, syntax errors in 
> these strings go undetected until somebody does parse them (e.g. your 
> static type analyzer).

This is a real problem.  But in theory, your code editor (and python) *could* 
parse the strings.  They generally don't, but I'm not sure asking them to do 
that is much harder than asking them to deal with new syntax.

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JR2RKEEUI3WSIUBZOJ4J3EVRTVYGLXSU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Jim J. Jewett

Baptiste Carvello wrote:
> Le 14/04/2021 à 19:44, Guido van Rossum a écrit :

> > No, what I heard is that, since in *most* cases the string quotes are
> > not needed, people are surprised and annoyed when they encounter cases
> > where they are needed.

> Well, I had assumed quotes would be used in all cases for consistency.

That does seem like a reasonable solution.  Redundant, ugly, and annoying, but 
safe and consistent.  Sort of like using type constraints in the first place.  
:D

> > ... the rule for finding the end of
> > an annotation would be very simple -- just skip words until the next
> > comma, close paren or colon, skipping matching brackets etc.

> ... But the hypothetic "def foo(prec:
> --precision int):" is already less readable. Will finding the closing
> comma or colon always be obvious to the human reader?

Nope.  "--" sometimes means "ignore the rest of the line, including the ")".  
At the moment, I can't remember where I've seen this outside of SQL, but I can 
guarantee that if I read it late enough at night, the *best* case would be that 
I notice the ambiguity, guess correctly and am only annoyed.

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MV3HQASUSAEKC332UFUDWQNONLCXZDRZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2


On 4/14/21 1:42 PM, Baptiste Carvello wrote:
Are there specific annoyances associated with quoting always, apart 
from the 2 more characters?



Yes.  Since the quoted strings aren't parsed by Python, syntax errors in 
these strings go undetected until somebody does parse them (e.g. your 
static type analyzer).  Having the Python compiler de-compile them back 
into strings means they got successfully parsed.  Though this doesn't 
rule out other errors, e.g. NameError.


I thought this was discussed in PEP 563, but now I can't find it, so 
unfortunately I can't steer you towards any more info on the subject.



Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UKO7SMXHUK5KLTWVTMU2RL6LAZQWA67U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Paul Bryan

On Wed, 2021-04-14 at 22:42 +0200, Baptiste Carvello wrote:

> That's assuming the syntax in the annotations doesn't diverge too
> much
> from the Python syntax as far as brackets etc are concerned. I must
> say
> I'm not too worried about typing. But the hypothetic "def foo(prec:
> --precision int):" is already less readable. Will finding the closing
> comma or colon always be obvious to the human reader?

To push the limit, let's add some default value:

def foo(prec: --precision int = 123):
    ...

vs.

def foo(prec: "--precision int" = 123):
    ...

And if a "type parameter" becomes numeric. For example:

def foo(prec: --max bar 3000 = 123):
    ...

vs.

def foo(prec: "--max bar 3000" = 123):
    ...

Now, the quotes seem even more readable as a delimiter.

Paul

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UITB2A657TAINAGWGRD6GCKWFC5PEBIZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Baptiste Carvello

Hi,

Le 14/04/2021 à 19:44, Guido van Rossum a écrit :
> 
> No, what I heard is that, since in *most* cases the string quotes are
> not needed, people are surprised and annoyed when they encounter cases
> where they are needed. And if you have a large code base it takes an
> expensive run of the static type checker to find out that you've
> forgotten the quotes.

Well, I had assumed quotes would be used in all cases for consistency.
Indeed, using them only if needed leads to surprises. Are there specific
annoyances associated with quoting always, apart from the 2 more characters?

>  
> [...]
> 
> They will treat it as anything else they don't quite understand -- they
> will ignore it unless it bites them. And the rule for finding the end of
> an annotation would be very simple -- just skip words until the next
> comma, close paren or colon, skipping matching brackets etc.

That's assuming the syntax in the annotations doesn't diverge too much
from the Python syntax as far as brackets etc are concerned. I must say
I'm not too worried about typing. But the hypothetic "def foo(prec:
--precision int):" is already less readable. Will finding the closing
comma or colon always be obvious to the human reader?

Cheers,
Baptiste
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/E654LKI36TFURBOSBMFUPEMLJQ32CZNN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Carl Meyer via Python-Dev

Hi Larry,

On 4/14/21, 1:56 PM, "Larry Hastings"  wrote:

>My plan was to post it here and see what the response was first.  Back in 
> January, when I posted the first draft, I got some very useful feedback that 
> resulted in some dramatic changes.  This time around, so far, nobody has 
> suggested even minor changes.  Folks have just expressed their opinions about 
> it (which is fine).

This is not true. I suggested yesterday (in 
https://mail.python.org/archives/list/python-dev@python.org/message/DSZFE7XTRK2ESRJDPQPZIDP2I67E76WH/
 ) that PEP 649 could avoid making life worse for users of type annotations 
(relative to PEP 563) if it replaced runtime-undefined names with forward 
reference markers, as implemented in 
https://github.com/larryhastings/co_annotations/pull/3

Perhaps you've chosen to ignore the suggestion, but that's not the same as 
nobody suggesting any changes ;)

Carl

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4LXOED3ABKDSNUDJ3JTNEGTXD3R7TEWT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2



My plan was to post it here and see what the response was first. Back in 
January, when I posted the first draft, I got some very useful feedback 
that resulted in some dramatic changes.  This time around, so far, 
nobody has suggested even minor changes.  Folks have just expressed 
their opinions about it (which is fine).


Still left to do: ping the project leads of some other static type 
analysis projects and see if they have any feedback to contribute.  Once 
the dust completely settles around the conversation here, I expect to 
formally submit the PEP, hopefully later this week.


Cheers,


//arry/

On 4/14/21 12:22 PM, Brett Cannon wrote:



On Wed, Apr 14, 2021 at 12:08 PM Guido van Rossum > wrote:


Let's just wait for the SC to join the discussion. I'm sure they
will, eventually.


FYI the PEP has not been sent to us via 
https://github.com/python/steering-council/issues 
 as ready for 
pronouncement, so we have not started officially discussing this PEP yet.


-Brett


On Wed, Apr 14, 2021 at 11:12 AM Larry Hastings
mailto:la...@hastings.org>> wrote:

On 4/14/21 10:44 AM, Guido van Rossum wrote:

besides the cost of closing the door to relaxed annotation
syntax, there's the engineering work of undoing the work that
was done to make `from __future__ import annotations` the
default (doing this was a significant effort spread over many
commits, and undoing will be just as hard).



I'm not sure either of those statements is true.

Accepting PEP 649 as written would deprecate stringized
annotations, it's true.  But the SC can make any decision it
wants here, including only accepting the new semantics of 649
without deprecating stringized annotations.  They could remain
in the language for another release (or two? or three?) while
we "kick the can down the road". This is not without its costs
too but it might be the best approach for now.

As for undoing the effort to make stringized annotations the
default, git should do most of the heavy lifting here. 
There's a technique where you check out the revision that made
the change, generate a reverse patch, apply it, and check that
in.  This creates a new head which you then merge. That's what
I did when I created my co_annotations branch, and at the time
it was literally the work of ten minutes.  I gather the list
of changes is more substantial now, so this would have to be
done multiple times, and it may be more involved.  Still, if
PEP 649 is accepted, I would happily volunteer to undertake
this part of the workload.


Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/LRVFVLH4AHF7SX5MOEUBPPII7UNINAMJ/


Code of Conduct: http://python.org/psf/codeofconduct/




-- 
--Guido van Rossum (python.org/~guido )

/Pronouns: he/him //(why is my pronoun here?)/


___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/V5ASSMVVAP4RZX3DOGJIDS52OEJ6LP7C/


Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MZNDSJ2Z5M6VXHBHWPOD2HYUQI72KGX2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Brett Cannon

On Wed, Apr 14, 2021 at 12:08 PM Guido van Rossum  wrote:

> Let's just wait for the SC to join the discussion. I'm sure they will,
> eventually.
>

FYI the PEP has not been sent to us via
https://github.com/python/steering-council/issues as ready for
pronouncement, so we have not started officially discussing this PEP yet.

-Brett


>
> On Wed, Apr 14, 2021 at 11:12 AM Larry Hastings 
> wrote:
>
>> On 4/14/21 10:44 AM, Guido van Rossum wrote:
>>
>> besides the cost of closing the door to relaxed annotation syntax,
>> there's the engineering work of undoing the work that was done to make
>> `from __future__ import annotations` the default (doing this was a
>> significant effort spread over many commits, and undoing will be just as
>> hard).
>>
>>
>> I'm not sure either of those statements is true.
>>
>> Accepting PEP 649 as written would deprecate stringized annotations, it's
>> true.  But the SC can make any decision it wants here, including only
>> accepting the new semantics of 649 without deprecating stringized
>> annotations.  They could remain in the language for another release (or
>> two? or three?) while we "kick the can down the road".  This is not without
>> its costs too but it might be the best approach for now.
>>
>> As for undoing the effort to make stringized annotations the default, git
>> should do most of the heavy lifting here.  There's a technique where you
>> check out the revision that made the change, generate a reverse patch,
>> apply it, and check that in.  This creates a new head which you then
>> merge.  That's what I did when I created my co_annotations branch, and at
>> the time it was literally the work of ten minutes.  I gather the list of
>> changes is more substantial now, so this would have to be done multiple
>> times, and it may be more involved.  Still, if PEP 649 is accepted, I would
>> happily volunteer to undertake this part of the workload.
>>
>>
>> Cheers,
>>
>>
>> */arry*
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/LRVFVLH4AHF7SX5MOEUBPPII7UNINAMJ/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/V5ASSMVVAP4RZX3DOGJIDS52OEJ6LP7C/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7LXQATCTABW7HCSAUGAGFRJFFRLW5DVU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

Let's just wait for the SC to join the discussion. I'm sure they will,
eventually.

On Wed, Apr 14, 2021 at 11:12 AM Larry Hastings  wrote:

> On 4/14/21 10:44 AM, Guido van Rossum wrote:
>
> besides the cost of closing the door to relaxed annotation syntax, there's
> the engineering work of undoing the work that was done to make `from
> __future__ import annotations` the default (doing this was a significant
> effort spread over many commits, and undoing will be just as hard).
>
>
> I'm not sure either of those statements is true.
>
> Accepting PEP 649 as written would deprecate stringized annotations, it's
> true.  But the SC can make any decision it wants here, including only
> accepting the new semantics of 649 without deprecating stringized
> annotations.  They could remain in the language for another release (or
> two? or three?) while we "kick the can down the road".  This is not without
> its costs too but it might be the best approach for now.
>
> As for undoing the effort to make stringized annotations the default, git
> should do most of the heavy lifting here.  There's a technique where you
> check out the revision that made the change, generate a reverse patch,
> apply it, and check that in.  This creates a new head which you then
> merge.  That's what I did when I created my co_annotations branch, and at
> the time it was literally the work of ten minutes.  I gather the list of
> changes is more substantial now, so this would have to be done multiple
> times, and it may be more involved.  Still, if PEP 649 is accepted, I would
> happily volunteer to undertake this part of the workload.
>
>
> Cheers,
>
>
> */arry*
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LRVFVLH4AHF7SX5MOEUBPPII7UNINAMJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V5ASSMVVAP4RZX3DOGJIDS52OEJ6LP7C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

On Wed, Apr 14, 2021 at 11:03 AM Paul Bryan  wrote:

> What would you expect get_type_hints(...) to return with relaxed syntax?
> Today, for type hint annotations, it returns a type, which I'd argue is an
> important feature to preserve (in it or some successor).
>

It would have to return some other representation. Presumably the (purely
hypothetical) new syntax would be syntactic sugar for something that can be
expressed as an object, just like (as of PEP 604) X | Y is syntactic sugar
for Union[X, Y].

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z4D4WUCLWUVO53EAKIY2WMCRL5MMZ6GD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2


On 4/14/21 10:44 AM, Guido van Rossum wrote:
besides the cost of closing the door to relaxed annotation syntax, 
there's the engineering work of undoing the work that was done to make 
`from __future__ import annotations` the default (doing this was a 
significant effort spread over many commits, and undoing will be just 
as hard).



I'm not sure either of those statements is true.

Accepting PEP 649 as written would deprecate stringized annotations, 
it's true.  But the SC can make any decision it wants here, including 
only accepting the new semantics of 649 without deprecating stringized 
annotations.  They could remain in the language for another release (or 
two? or three?) while we "kick the can down the road".  This is not 
without its costs too but it might be the best approach for now.


As for undoing the effort to make stringized annotations the default, 
git should do most of the heavy lifting here.  There's a technique where 
you check out the revision that made the change, generate a reverse 
patch, apply it, and check that in.  This creates a new head which you 
then merge.  That's what I did when I created my co_annotations branch, 
and at the time it was literally the work of ten minutes.  I gather the 
list of changes is more substantial now, so this would have to be done 
multiple times, and it may be more involved.  Still, if PEP 649 is 
accepted, I would happily volunteer to undertake this part of the workload.



Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LRVFVLH4AHF7SX5MOEUBPPII7UNINAMJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Paul Bryan

What would you expect get_type_hints(...) to return with relaxed
syntax? Today, for type hint annotations, it returns a type, which I'd
argue is an important feature to preserve (in it or some successor).

On Wed, 2021-04-14 at 10:54 -0700, Guido van Rossum wrote:
> On Wed, Apr 14, 2021 at 10:47 AM Paul Bryan  wrote:
> > I favour annotations for type hints; the writing's been on the wall
> > for some time. I think the necessary escape hatch for those using
> > it for other purposes should be Annotated[Any, ...] (or a similar,
> > nicer alternative).
> > 
> > Guido, one of the difficulties I'm having is understanding the
> > direction you're going with "relaxed syntax". PEP 649 is concrete;
> > it's hard to weigh its merits against the usability—even
> > feasibility—of incorporating an as yet undefined relaxed syntax.
> > 
> > At the end of the day, such syntax is going to have to be
> > represented in some structure. If one were to accept that
> > annotations are for type hints only, is the debate then the
> > difference between a Python type (which PEP 649 would yield) and
> > some other as yet undefined structure?  
> > 
> 
> 
> In `__annotations__` it would be a string, as currently implemented
> in the 3.10 alpha code. The string just might not be parsable as an
> expression.
> 
> In the AST, it will have to be a new node that just collects tokens
> and bracketed things; that could be an array of low-level tokens.
> 

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AP6TJNGH5SKFX3H5EL2QWACYG3F7WQF7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

On Wed, Apr 14, 2021 at 10:47 AM Paul Bryan  wrote:

> I favour annotations for type hints; the writing's been on the wall for
> some time. I think the necessary escape hatch for those using it for other
> purposes should be Annotated[Any, ...] (or a similar, nicer alternative).
>
> Guido, one of the difficulties I'm having is understanding the direction
> you're going with "relaxed syntax". PEP 649 is concrete; it's hard to weigh
> its merits against the usability—even feasibility—of incorporating an as
> yet undefined relaxed syntax.
>
> At the end of the day, such syntax is going to have to be represented in
> some structure. If one were to accept that annotations are for type hints
> only, is the debate then the difference between a Python type (which PEP
> 649 would yield) and some other as yet undefined structure?
>

In `__annotations__` it would be a string, as currently implemented in the
3.10 alpha code. The string just might not be parsable as an expression.

In the AST, it will have to be a new node that just collects tokens and
bracketed things; that could be an array of low-level tokens.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FKOKFK7CPMMQEIPNZYJS56J3U3KFCFJE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Paul Bryan

I favour annotations for type hints; the writing's been on the wall for
some time. I think the necessary escape hatch for those using it for
other purposes should be Annotated[Any, ...] (or a similar, nicer
alternative).

Guido, one of the difficulties I'm having is understanding the
direction you're going with "relaxed syntax". PEP 649 is concrete; it's
hard to weigh its merits against the usability—even feasibility—of
incorporating an as yet undefined relaxed syntax.

At the end of the day, such syntax is going to have to be represented
in some structure. If one were to accept that annotations are for type
hints only, is the debate then the difference between a Python type
(which PEP 649 would yield) and some other as yet undefined structure?
 

Paul


On Wed, 2021-04-14 at 10:24 -0700, Guido van Rossum wrote:
> On Tue, Apr 13, 2021 at 6:48 PM Larry Hastings 
> wrote:
> > 
> > On 4/13/21 1:52 PM, Guido van Rossum wrote:
> > 
> > > On Tue, Apr 13, 2021 at 12:32 PM Larry Hastings
> > >  wrote:
> > > 
> > > > 
> > > > On 4/12/21 7:24 PM, Guido van Rossum wrote:
> > > > 
> > > > > I've been thinking about this a bit, and I think that the way
> > > > > forward is for Python to ignore the text of annotations
> > > > > ("relaxed annotation syntax"), not to try and make it
> > > > > available as an expression.
> > > > > 
> > > > > To be honest, the most pressing issue with annotations is the
> > > > > clumsy way that type variables have to be introduced. The
> > > > > current convention, `T = TypeVar('T')`, is both verbose (why
> > > > > do I have to repeat the name?) and widely misunderstood (many
> > > > > help request for mypy and pyright follow from users making a
> > > > > mistaken association between two type variables that are
> > > > > unrelated but share the same TypeVar definition). And relaxed
> > > > > annotation syntax alone doesn't solve this.
> > > > > 
> > > > > Nevertheless I think that it's time to accept that
> > > > > annotations are for types -- the intention of PEP 3107 was to
> > > > > experiment with different syntax and semantics for types, and
> > > > > that experiment has resulted in the successful adoption of a
> > > > > specific syntax for types that is wildly successful.
> > > > 
> > > > I don't follow your reasoning.  I'm glad that type hints have
> > > > found success, but I don't see why that implies "and therefore
> > > > we should restrict the use of annotations solely for type
> > > > hints".  Annotations are a useful, general-purpose feature of
> > > > Python, with legitimate uses besides type hints.  Why would it
> > > > make Python better to restrict their use now?
> > > > 
> > > 
> > > Because typing is, to many folks, a Really Important Concept, and
> > > it's confusing to use the same syntax ("x: blah blah") for
> > > different purposes, in a way that makes it hard to tell whether a
> > > particular "blah blah" is meant as a type or as something else --
> > > because you have to know what's introspecting the annotations
> > > before you can tell. And that introspection could be signalled by
> > > a magical decorator, but it could also be implicit: maybe you
> > > have a driver that calls a function based on a CLI entry point
> > > name, and introspects that function even if it's not decorated.
> > 
> > I'm not sure I understand your point.  Are you saying that we need
> > to take away the general-purpose functionality of annotations,
> > that's been in the language since 3.0, and restrict annotations to
> > just type hints... because otherwise an annotation might not be
> > used for a type hint, and then the programmer would have to figure
> > out what it means?  We need to take away the functionality from all
> > other use cases in order to lend clarity to one use case?
> > 
> 
> 
> Yes, that's how I see it.
> 
> And before you get too dramatic about it, the stringification of
> annotations has been in the making a long time, with the community's
> and the SC's support. You came up with a last-minute attempt to
> change it, using the PEP process to propose to *revert* the decision
> already codified in PEP 563 and implemented in the master branch. But
> you've waited until the last minute (feature freeze is in three
> weeks) and IMO you're making things awkward for the SC (who can and
> will speak for themselves).
>  
> > Also, if you're stating that programmers get confused reading
> > source code because annotations get used for different things at
> > different places--surely that confirms that annotations are useful
> > for more than just type hints, in real-world code, today.
> > 
> 
> No, it doesn't, it's just a hypothetical that they *would* be
> confused if there *were* other uses. Personally I haven't used any
> libraries that use non-type-hint annotations, but I've been told they
> exist.
> 
> > I genuinely have no sense of how important static type analysis is
> > in Python--personally I have no need for it--but I find it hard to
> > believe that type hints

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

On Wed, Apr 14, 2021 at 9:42 AM Baptiste Carvello <
devel2...@baptiste-carvello.net> wrote:

> Hi,
>
> tl;dr: imho the like or dislike of PEP 563 is related to whether people
> intend to learn a second syntax for typing, or would rather ignore it;
> both groups should be taken into account.
>
> Le 13/04/2021 à 19:30, Guido van Rossum a écrit :
> > On Tue, Apr 13, 2021 at 9:39 AM Baptiste Carvello
> >  > > wrote:
> >
> > Then, what's wrong with quoting? It's just 2 characters, and prevent
> the
> > user (or their IDE) from trying to parse them as Python syntax.
> >
> >
> > Informal user research has shown high resistance to quoting.
>
> OK, but why? I'd bet it's due to an "aesthetic" concern: for typing
> users, type hints are code, not textual data. So it irks them to see
> them quoted and syntax-highlighted as text strings.
>

No, what I heard is that, since in *most* cases the string quotes are not
needed, people are surprised and annoyed when they encounter cases where
they are needed. And if you have a large code base it takes an expensive
run of the static type checker to find out that you've forgotten the quotes.

> > As a comparison: docstrings do get quoting, even though they also
> have
> > special semantics in the language.
> >
> >
> > Not the same thing. Docstrings use English, which has no formal (enough)
> > syntax. The idea for annotations is that they *do* have a formal syntax,
> > it just evolves separately from that of Python itself.
>
> If I may say it in my words: to both the parser and (more importantly)
> typing-savvy developers, type hints are code. I now see the point.
>
> But what about developers who won't learn this (future) incompatible
> typing syntax, and only encounter it in the wild? To them, those
> annotations are akin to docstrings: pieces of textual data that Python
> manages specially because of their role in the greater ecosystem, but
> that they can ignore because the program behavior is not modified.
>
> So it will irk them if annotations in this new syntax are not quoted or
> otherwise made distinguishable from code written in the normal Python
> syntax they understand. Again the "aesthetic" concern, and imho it
> explains in large part why some people dislike PEP 563.
>

They will treat it as anything else they don't quite understand -- they
will ignore it unless it bites them. And the rule for finding the end of an
annotation would be very simple -- just skip words until the next comma,
close paren or colon, skipping matching brackets etc.

Certainly I use this strategy all the time for quickly skimming code (in
any language) that I don't need to completely understand -- "oh, this is
where the parameters are processed, this is where the defaults are sorted
out, and this is where the work is being done; and since I'm investigating
why the default is weird, let me look at that part of the code in more
detail."

> Can the needs of both groups of developers be addressed? Could code in
> the new typing syntax be marked with a specific syntactic marker,
> distinguishing it from both normal Python syntax and text strings? Then
> this new marker could also be used outside of annotations, to mark
> analysis-time-only imports or statements?
>

There already is a special marker for annotations in function definitions
-- for arguments, it's the colon following the parameter name, and for
return types, it's the arrow after the parameter list. And for variable
declarations ("x: int") the same colon also suffices.

The idea to use the same marker for other analysis-time code is
interesting, but the syntactic requirements are somewhat different --
annotations live at the "expression" level (informally speaking) and are
already in a clearly indicated syntactic position -- analysis-time code
looks just like other code and can occur in any position where statements
can appear. So an appropriate marker would probably be an if-statement with
a special condition, like "if TYPE_CHECKING" (I am all for making that a
built-in constant, BTW).

If you were thinking of backticks, sorry, I'm not biting. (There are folks
who have other plans for those -- presumably because it's one of the few
ASCII characters that currently has no meaning.)

> Or is this all not worth the expense, and typing syntax can manage to
> stay compatible with normal Python syntax, in which case PEP 649 is the
> way to go?
>

I don't see much expense in the proposal to relax the syntax, and I see
benefits for new kinds of type annotations (e.g. PEP 647 would have
benefited).

And certainly PEP 649 has considerable cost as well -- besides the cost of
closing the door to relaxed annotation syntax, there's the engineering work
of undoing the work that was done to make `from __future__ import
annotations` the default (doing this was a significant effort spread over
many commits, and undoing will be just as hard). And we would still have to
support stringification

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Brett Cannon

On Tue, Apr 13, 2021 at 6:58 PM Inada Naoki  wrote:

> On Wed, Apr 14, 2021 at 10:44 AM Larry Hastings 
> wrote:
> >
> >
> > On 4/13/21 1:52 PM, Guido van Rossum wrote:
> >
> >
> > Because typing is, to many folks, a Really Important Concept, and it's
> confusing to use the same syntax ("x: blah blah") for different purposes,
> in a way that makes it hard to tell whether a particular "blah blah" is
> meant as a type or as something else -- because you have to know what's
> introspecting the annotations before you can tell. And that introspection
> could be signalled by a magical decorator, but it could also be implicit:
> maybe you have a driver that calls a function based on a CLI entry point
> name, and introspects that function even if it's not decorated.
> >
> >
> > I'm not sure I understand your point.  Are you saying that we need to
> take away the general-purpose functionality of annotations, that's been in
> the language since 3.0, and restrict annotations to just type hints...
> because otherwise an annotation might not be used for a type hint, and then
> the programmer would have to figure out what it means?  We need to take
> away the functionality from all other use cases in order to lend clarity to
> one use case?
> >
>
> I don't think we need to take away "general purpose functionality".
> But if we define type hinting is 1st class use case of annotations,
> annotations should be optimized for type hinting.  General purpose use
> case should accept some limitation and overhead.
>
> On the other hand, if we decide general purpose functionality is 1st
> class too, we shouldn't annotation syntax different from Python
> syntax.
>

Has anyone reached out to people like Pydantic, FastAPI, typer, etc. to see
what they think of this PEP? For instance, are they having issues with the
way things are today enough that this is a very clear win for them?

-Brett


>
> But annotations should be optimized for type hinting anyway. General
> purpose use case used only is a limited part of application. On the
> other hand, type hint can be used almost everywhere in application
> code base. It must cheap enough.
>
> Regards,
>
> --
> Inada Naoki  
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UGFWTZUGH6QZRHF3FKTQHZLYG2ZNX5EG/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YKVYJMLUWUVT4KMLUNEQYVBZWNAPR4GV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

On Tue, Apr 13, 2021 at 6:48 PM Larry Hastings  wrote:

>
> On 4/13/21 1:52 PM, Guido van Rossum wrote:
>
> On Tue, Apr 13, 2021 at 12:32 PM Larry Hastings 
> wrote:
>
>>
>> On 4/12/21 7:24 PM, Guido van Rossum wrote:
>>
>> I've been thinking about this a bit, and I think that the way forward is
>> for Python to ignore the text of annotations ("relaxed annotation syntax"),
>> not to try and make it available as an expression.
>>
>> To be honest, the most pressing issue with annotations is the clumsy way
>> that type variables have to be introduced. The current convention, `T =
>> TypeVar('T')`, is both verbose (why do I have to repeat the name?) and
>> widely misunderstood (many help request for mypy and pyright follow from
>> users making a mistaken association between two type variables that are
>> unrelated but share the same TypeVar definition). And relaxed annotation
>> syntax alone doesn't solve this.
>>
>> Nevertheless I think that it's time to accept that annotations are for
>> types -- the intention of PEP 3107 was to experiment with different syntax
>> and semantics for types, and that experiment has resulted in the successful
>> adoption of a specific syntax for types that is wildly successful.
>>
>>
>> I don't follow your reasoning.  I'm glad that type hints have found
>> success, but I don't see why that implies "and therefore we should restrict
>> the use of annotations solely for type hints".  Annotations are a useful,
>> general-purpose feature of Python, with legitimate uses besides type
>> hints.  Why would it make Python better to restrict their use now?
>>
>
> Because typing is, to many folks, a Really Important Concept, and it's
> confusing to use the same syntax ("x: blah blah") for different purposes,
> in a way that makes it hard to tell whether a particular "blah blah" is
> meant as a type or as something else -- because you have to know what's
> introspecting the annotations before you can tell. And that introspection
> could be signalled by a magical decorator, but it could also be implicit:
> maybe you have a driver that calls a function based on a CLI entry point
> name, and introspects that function even if it's not decorated.
>
>
> I'm not sure I understand your point.  Are you saying that we need to take
> away the general-purpose functionality of annotations, that's been in the
> language since 3.0, and restrict annotations to just type hints... because
> otherwise an annotation might not be used for a type hint, and then the
> programmer would have to figure out what it means?  We need to take away
> the functionality from all other use cases in order to lend *clarity* to
> one use case?
>

Yes, that's how I see it.

And before you get too dramatic about it, the stringification of
annotations has been in the making a long time, with the community's and
the SC's support. You came up with a last-minute attempt to change it,
using the PEP process to propose to *revert* the decision already codified
in PEP 563 and implemented in the master branch. But you've waited until
the last minute (feature freeze is in three weeks) and IMO you're making
things awkward for the SC (who can and will speak for themselves).


> Also, if you're stating that programmers get confused reading source code
> because annotations get used for different things at different
> places--surely that confirms that annotations are *useful* for more than
> just type hints, in real-world code, today.
>

No, it doesn't, it's just a hypothetical that they *would* be confused if
there *were* other uses. Personally I haven't used any libraries that use
non-type-hint annotations, but I've been told they exist.

I genuinely have no sense of how important static type analysis is in
> Python--personally I have no need for it--but I find it hard to believe
> that type hints are so overwhelmingly important that they should become the
> sole use case for annotations, and we need to take away this long-standing
> functionality, that you suggest is being successfully used side-by-side
> with type hints today, merely to make type hints clearer.
>

For projects and teams that use type hints, they are *very* important. For
example, they are so important to the Instagram team at Facebook that they
wrote their own static type checker when they found mypy wasn't fast enough
for their million-line codebase. And of course they were so important to
Dropbox that they sponsored a multi-year, multi-person effort to create
mypy in the first place. The amount of feedback we've received for mypy
indicates that it's not just those two companies that are using type hints.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

2021-04-14 Thread Baptiste Carvello

Hi,

tl;dr: imho the like or dislike of PEP 563 is related to whether people
intend to learn a second syntax for typing, or would rather ignore it;
both groups should be taken into account.

Le 13/04/2021 à 19:30, Guido van Rossum a écrit :
> On Tue, Apr 13, 2021 at 9:39 AM Baptiste Carvello
>  > wrote:
> 
> Then, what's wrong with quoting? It's just 2 characters, and prevent the
> user (or their IDE) from trying to parse them as Python syntax.
> 
> 
> Informal user research has shown high resistance to quoting.

OK, but why? I'd bet it's due to an "aesthetic" concern: for typing
users, type hints are code, not textual data. So it irks them to see
them quoted and syntax-highlighted as text strings.

> As a comparison: docstrings do get quoting, even though they also have
> special semantics in the language.
> 
> 
> Not the same thing. Docstrings use English, which has no formal (enough)
> syntax. The idea for annotations is that they *do* have a formal syntax,
> it just evolves separately from that of Python itself.

If I may say it in my words: to both the parser and (more importantly)
typing-savvy developers, type hints are code. I now see the point.

But what about developers who won't learn this (future) incompatible
typing syntax, and only encounter it in the wild? To them, those
annotations are akin to docstrings: pieces of textual data that Python
manages specially because of their role in the greater ecosystem, but
that they can ignore because the program behavior is not modified.

So it will irk them if annotations in this new syntax are not quoted or
otherwise made distinguishable from code written in the normal Python
syntax they understand. Again the "aesthetic" concern, and imho it
explains in large part why some people dislike PEP 563.

Can the needs of both groups of developers be addressed? Could code in
the new typing syntax be marked with a specific syntactic marker,
distinguishing it from both normal Python syntax and text strings? Then
this new marker could also be used outside of annotations, to mark
analysis-time-only imports or statements?

Or is this all not worth the expense, and typing syntax can manage to
stay compatible with normal Python syntax, in which case PEP 649 is the
way to go?

Cheers,
Baptiste
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YI3DPTJXKTKZRVVWBJ6UYTUH3PPAJM6Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2



On 4/12/21 7:24 PM, Guido van Rossum wrote:
To be honest, the most pressing issue with annotations is the clumsy 
way that type variables have to be introduced. The current convention, 
`T = TypeVar('T')`, is both verbose (why do I have to repeat the 
name?) and widely misunderstood (many help request for mypy and 
pyright follow from users making a mistaken association between two 
type variables that are unrelated but share the same TypeVar definition).



This repeat-the-name behavior has been in Python for a long time, e.g.

   Point = namedtuple('Point', ['x', 'y'])

namedtuple() shipped with Python 2.6 in 2008.  So if that's the most 
pressing issue with annotations, annotations must be going quite well, 
because we've known about this for at least 13 years without attempting 
to solve it.


I've always assumed that this repetition was worth the minor 
inconvenience.  You only have to retype the name once, and the resulting 
code is clear and readable, with predictable behavior. A small price to 
pay to preserve Python's famous readability.



For what it's worth--and forgive me for straying slightly into 
python-ideas territory--/if/ we wanted to eliminate the need to repeat 
the name, I'd prefer a general-purpose solution rather than something 
tailored specifically for type hints.  In a recent private email 
conversation on a different topic, I proposed this syntax:


   bind  

This statement would be equivalent to

   id = expression('id')


Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L5EBHWHUOQZDW33FH5VI2D6KFN75LLVW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2