date:20210112

On Tue, Jan 12, 2021 at 8:00 PM Brett Cannon  wrote:

>
>
>>- It turns a None annotation into type(None).  Which means now you
>>can't tell the difference between "None" and "type(None)".
>>
>> Huh, I wasn't aware of that.
>

This has tripped up many people. Maybe we should just bite the bullet and
change this?

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XMX4OKYQR6QWTIGB7DJBKJFB7G4QZMRO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Let's Fix Class Annotations -- And Maybe Annotations Generally

2021-01-12 Thread Brett Cannon

On Tue, Jan 12, 2021 at 6:31 PM Larry Hastings  wrote:

>
> On 1/12/21 5:28 PM, Brett Cannon wrote:
>
> The other thing to keep in mind is we are talking about every module,
> class, and function getting 64 bytes ... which I bet isn't that much.
>
> Actually it's only every module and class.  Functions don't have this
> problem because they've always stored __annotations__ internally--meaning,
> peeking in their __dict__ doesn't work, and they don't support inheritance
> anyway.  So the number is even smaller than that.
>
> If we can just make __annotations__ default to an empty dict on classes
> and modules, and not worry about the memory consumption, that goes a long
> way to cleaning up the semantics.
>

Great!


>
> And I know you were somewhat joking when you mentioned using
> sys.version_info, but since this would be behind a __future__ import
>
> Would it?
>

I thought you had proposed that initially, but it appears I mixed this with
your PEP email.  Sorry about that!


> My original proposal would make breaking changes to how you examine
> __annotations__.  Let's say we put those behind a from __future__ import.
> Now we're gonna write library code that examines annotations.  A user
> passes in a class and asks us to examine its annotations.  The old
> semantics might be active on it, or the new ones.  How do we know which set
> of semantics we need to use?
>
> It occurs to me that you could take kls.__module__, pull out the module
> from sys.modules, then look inside to see if it contains the correct
> "future" object imported from the __future__ module.  Is that an approach
> we would suggest to our users?
>
> Also, very little code ever examines annotations; most code with
> annotations merely defines them.  So I suspect most annotations users
> wouldn't care either way--which also means a "from __future__ import" that
> changes the semantics of examining or modifying annotations isn't going to
> see a lot of uptake, because it doesn't really affect them.  The change in
> semantics only affects people whose code examines annotations, which I
> suspect is very few.
>
> So I wasn't really joking when I proposed making these changes without a
> from __future__ import, and suggested users use a version check.  The
> library code would know based on the Python version number which semantics
> were active, no peeking in modules to find future object.  They could
> literally write what I suggested:
>
> if you know you're running python 3.10 or higher:
> examine using the new semantics
> else:
> examine using the old semantics
>
> I realize that's a pretty aggressive approach, which is why I prefaced it
> with "if I could wave my magic wand".  But if we're going to make breaking
> changes, then whatever we do, it's going to break some people's code until
> it gets updated to cope with the new semantics.  In that light this
> approach seemed reasonable.
>
> But really this is why I started this thread in the first place.  My idea
> of what's reasonable is probably all out of whack.  So I wanted to start
> the conversation, to get feedback on how much breakage is allowable and how
> best to mitigate it.  If it wasn't a controversial change, then we wouldn't
> need to talk about it!
>
>
> And finally: if we really do set a default of an empty dict on classes and
> modules, then my other in-theory breaking changes:
>
>- you can't delete __annotations__
>- you can only set __annotations__ to a dict or None (this is already
>true of functions, but not of classes or modules)
>
> will, I expect, in practice breaking exactly zero code.  Who deletes
> __annotations__?  Who ever sets __annotations__ to something besides a
> dict?  So if the practical breakage is zero, why bother gating it with
> "from __future__ import" at all?
>
>
> I think it really means people need to rely on typing.get_type_hints()
> more than they may be doing right now.
>
> What I find frustrating about that answer--and part of what motivated me
> to work on this in the first place--is that typing.get_type_hints()
> requires your annotations to be type hints.  All type hints are
> annotations, but not all annotations are type hints, and it's entirely
> plausible for users to have reasonable uses for non-type-hint annotations
> that typing.get_type_hints() wouldn't like.
>

You and I have talked about this extensively, so I'm aware. 


> The two things typing.get_type_hints() does, that I know of, that can
> impede such non-type-hint annotations are:
>
>- It turns a None annotation into type(None).  Which means now you
>can't tell the difference between "None" and "type(None)".
>
> Huh, I wasn't aware of that.

-Brett


>
>-
>- It regards all string annotations as "forward references", which
>means they get eval()'d and the result returned as the annotation.
>typing.get_type_hints() doesn't catch any exceptions here, so if the eval
>fails, typing.get_type_hints() fails and you can't use it

[Python-Dev] Re: Let's Fix Class Annotations -- And Maybe Annotations Generally

On Tue, Jan 12, 2021 at 6:35 PM Larry Hastings  wrote:

>
> On 1/12/21 5:28 PM, Brett Cannon wrote:
>
> The other thing to keep in mind is we are talking about every module,
> class, and function getting 64 bytes ... which I bet isn't that much.
>
> Actually it's only every module and class.  Functions don't have this
> problem because they've always stored __annotations__ internally--meaning,
> peeking in their __dict__ doesn't work, and they don't support inheritance
> anyway.  So the number is even smaller than that.
>
> If we can just make __annotations__ default to an empty dict on classes
> and modules, and not worry about the memory consumption, that goes a long
> way to cleaning up the semantics.
>

I would like that very much. And the exception for functions is especially
helpful.

> And I know you were somewhat joking when you mentioned using
> sys.version_info, but since this would be behind a __future__ import
>
> Would it?
>
> My original proposal would make breaking changes to how you examine
> __annotations__.  Let's say we put those behind a from __future__ import.
> Now we're gonna write library code that examines annotations.  A user
> passes in a class and asks us to examine its annotations.  The old
> semantics might be active on it, or the new ones.  How do we know which set
> of semantics we need to use?
>
> It occurs to me that you could take kls.__module__, pull out the module
> from sys.modules, then look inside to see if it contains the correct
> "future" object imported from the __future__ module.  Is that an approach
> we would suggest to our users?
>

You're kidding, right?

Also, very little code ever examines annotations; most code with
> annotations merely defines them.  So I suspect most annotations users
> wouldn't care either way--which also means a "from __future__ import" that
> changes the semantics of examining or modifying annotations isn't going to
> see a lot of uptake, because it doesn't really affect them.  The change in
> semantics only affects people whose code examines annotations, which I
> suspect is very few.
>

I agree, but they're pretty vocal -- the breakage in get_type_hints() due
to the scope issue in 3.10 (which isn't even in beta) has drawn plenty of
complaints.

Also, dataclasses (which I have to assume is fairly popular :-) introspects
`__annotations__`, and even mutates and sets it.

So I wasn't really joking when I proposed making these changes without a
> from __future__ import, and suggested users use a version check.  The
> library code would know based on the Python version number which semantics
> were active, no peeking in modules to find future object.  They could
> literally write what I suggested:
>
> if you know you're running python 3.10 or higher:
> examine using the new semantics
> else:
> examine using the old semantics
>
> I realize that's a pretty aggressive approach, which is why I prefaced it
> with "if I could wave my magic wand".  But if we're going to make breaking
> changes, then whatever we do, it's going to break some people's code until
> it gets updated to cope with the new semantics.  In that light this
> approach seemed reasonable.
>

Is there a way that such code could be written without a version check?
E.g. for modules we could recommend `getattr(m, "__attributes__", None) or
{}`, and that would work in earlier versions too.

I'm not sure what would work for classes, since most code will want to
combine the annotations for all classes in the MRO, and the way to do that
would change -- before 3.10, you *must* use
`cls.__dict__.get("__attributes__")` whereas for 3.10+ you *must* use
`cls.__attributes__`.

Note, for a moment I thought that for modules we don't need to evaluate
annotations lazily (I know that's your other PEP/thread, but still, it
seems related). But we do, because there's an idiom where people write
```
from __future__ import annotations
import typing
if typing.TYPE_CHECKING:
   from somewhere import Class
a: Class
```
Here introspecting the annotations would fail, but clearly the intention
was to use them purely for static type checking, so the user presumably
doesn't care.

(But does that mean that if a single annotation cannot be evaluated, the
entire annotations dict becomes inaccessible? That's a general weakness of
the PEP 649 scheme, right?)

But really this is why I started this thread in the first place.  My idea
> of what's reasonable is probably all out of whack.  So I wanted to start
> the conversation, to get feedback on how much breakage is allowable and how
> best to mitigate it.  If it wasn't a controversial change, then we wouldn't
> need to talk about it!
>
>
> And finally: if we really do set a default of an empty dict on classes and
> modules, then my other in-theory breaking changes:
>
>- you can't delete __annotations__
>- you can only set __annotations__ to a dict or None (this is already
>true of functions, but not of classes or modules)
>
> will, I expect,

[Python-Dev] Re: Let's Fix Class Annotations -- And Maybe Annotations Generally



On 1/12/21 5:28 PM, Brett Cannon wrote:
The other thing to keep in mind is we are talking about every module, 
class, and function getting 64 bytes ... which I bet isn't that much.


Actually it's only every module and class.  Functions don't have this 
problem because they've always stored __annotations__ 
internally--meaning, peeking in their __dict__ doesn't work, and they 
don't support inheritance anyway.  So the number is even smaller than that.


If we can just make __annotations__ default to an empty dict on classes 
and modules, and not worry about the memory consumption, that goes a 
long way to cleaning up the semantics.



And I know you were somewhat joking when you mentioned using 
sys.version_info, but since this would be behind a __future__ import


Would it?

My original proposal would make breaking changes to how you examine 
__annotations__.  Let's say we put those behind a from __future__ 
import.  Now we're gonna write library code that examines annotations.  
A user passes in a class and asks us to examine its annotations.  The 
old semantics might be active on it, or the new ones.  How do we know 
which set of semantics we need to use?


It occurs to me that you could take kls.__module__, pull out the module 
from sys.modules, then look inside to see if it contains the correct 
"future" object imported from the __future__ module. Is that an approach 
we would suggest to our users?


Also, very little code ever examines annotations; most code with 
annotations merely defines them.  So I suspect most annotations users 
wouldn't care either way--which also means a "from __future__ import" 
that changes the semantics of examining or modifying annotations isn't 
going to see a lot of uptake, because it doesn't really affect them.  
The change in semantics only affects people whose code examines 
annotations, which I suspect is very few.


So I wasn't really joking when I proposed making these changes without a 
from __future__ import, and suggested users use a version check.  The 
library code would know based on the Python version number which 
semantics were active, no peeking in modules to find future object.  
They could literally write what I suggested:


   if you know you're running python 3.10 or higher:
    examine using the new semantics
   else:
    examine using the old semantics

I realize that's a pretty aggressive approach, which is why I prefaced 
it with "if I could wave my magic wand".  But if we're going to make 
breaking changes, then whatever we do, it's going to break some people's 
code until it gets updated to cope with the new semantics.  In that 
light this approach seemed reasonable.


But really this is why I started this thread in the first place. My idea 
of what's reasonable is probably all out of whack.  So I wanted to start 
the conversation, to get feedback on how much breakage is allowable and 
how best to mitigate it.  If it wasn't a controversial change, then we 
wouldn't need to talk about it!



And finally: if we really do set a default of an empty dict on classes 
and modules, then my other in-theory breaking changes:


 * you can't delete __annotations__
 * you can only set __annotations__ to a dict or None (this is already
   true of functions, but not of classes or modules)

will, I expect, in practice breaking exactly zero code.  Who deletes 
__annotations__?  Who ever sets __annotations__ to something besides a 
dict?  So if the practical breakage is zero, why bother gating it with 
"from __future__ import" at all?



I think it really means people need to rely on typing.get_type_hints() 
more than they may be doing right now.


What I find frustrating about that answer--and part of what motivated me 
to work on this in the first place--is that typing.get_type_hints() 
requires your annotations to be type hints.  All type hints are 
annotations, but not all annotations are type hints, and it's entirely 
plausible for users to have reasonable uses for non-type-hint 
annotations that typing.get_type_hints() wouldn't like.


The two things typing.get_type_hints() does, that I know of, that can 
impede such non-type-hint annotations are:


 * It turns a None annotation into type(None).  Which means now you
   can't tell the difference between "None" and "type(None)".
 * It regards all string annotations as "forward references", which
   means they get eval()'d and the result returned as the annotation. 
   typing.get_type_hints() doesn't catch any exceptions here, so if the
   eval fails, typing.get_type_hints() fails and you can't use it to
   examine your annotations.

PEP 484 "explicitly does NOT prevent other uses of annotations". But if 
you force everyone to use typing.get_type_hints() to examine their 
annotations, then you have de facto prevented any use of annotations 
that isn't compatible with type hints.



Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors

2021-01-12 Thread Inada Naoki

On Wed, Jan 13, 2021 at 1:47 AM Larry Hastings  wrote:
>
> On 1/11/21 5:33 PM, Inada Naoki wrote:
>
> Note that PEP 563 semantics allows more efficient implementation.
> Annotation is just a single constant tuple, not a dict.
> We already have the efficient implementation for Python 3.10.
>
> The efficient implementation in 3.10 can share tuples. If there are
> hundreds of methods with the same signature, annotation is just a
> single tuple, not hundreds of tuples. This is very efficient for auto
> generated codebase. I think this PEP can share the code objects for
> same signature by removing co_firstlineno information too.
>
> That's very clever!  My co_annotations repo was branched from before this 
> feature was added, and I haven't pulled and merged recently.  So I hadn't 
> seen it.
>

Please see this pull request too. It merges co_code and co_consts. It
will save more RAM and importing time of your implementation.
https://github.com/python/cpython/pull/23056

>
> Additionally, we should include the cost for loading annotations from
> PYC files, because most annotations are "load once, set once".
> Loading "simple code object" from pyc files is not so cheap. It may
> affect importing time of large annotated codebase and memory
> footprints.
>
> I did some analysis in a separate message.  The summary is, the code object 
> for a single annotation costs us 232 bytes; that includes the code object 
> itself, the bytestring for the bytecode, and the bytestring for the lnotab.  
> This grows slowly as you add new parameters; the code object for ten 
> parameters is 360 bytes.
>
> It seems possible to create a hybrid of these two approaches!  Here's my 
> idea: instead of the compiler storing a code object as the annotations 
> argument to MAKE_FUNCTION, store a tuple containing the fields you'd need to 
> recreate the code object at runtime--bytecode, lnotab, names, consts, etc. 
> func_get_annotations would create the code object from that, bind it to a 
> function object, call it, and return the result.  These code-object-tuples 
> would then be automatically shared in the .pyc file and at runtime the same 
> way that 3.10 shares the tuples of stringized annotations today.

It may be good idea if we can strip most code object members, like
argcount, kwonlyargcount, nlocals, flags, freevars, cellvars,
filename, name, firstlineno, linetable.
It can be smaller than Python 3.9.

>
> That said, I suggest PEP 649's memory consumption isn't an urgent 
> consideration in choosing to accept or reject it.  PEP 649 is competitive in 
> terms of startup time and memory usage with PEP 563, and PEP 563 was accepted 
> and shipped with several versions of Python.
>

I still want a real-world application/library with heavy annotation.
My goal is to use annotations in the stdlib without caring about
resource usage or importtime.
But I agree with you if PEP 649 will be smaller than Python 3.9.

Regards,
-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CNXCFW2PXUCZ75OBFZTUS3TVKI3IEKZH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.10 change (?) for bool

2021-01-12 Thread Brett Cannon

On Tue, Jan 12, 2021 at 9:51 AM Chris Angelico  wrote:

> On Wed, Jan 13, 2021 at 4:24 AM Richard Damon 
> wrote:
> >
> > On 1/12/21 10:53 AM, Mark Shannon wrote:
> > > Hi everyone,
> > >
> > > Should the optimizer eliminate tests that it can prove have no effect
> > > on the control flow of the program, even if that may eliminate some
> > > side effects in __bool__()?
> > >
> > > For several years we have converted
> > >
> > > if a and b:
> > > ...
> > >
> > > to
> > >
> > > if a:
> > > if b:
> > > ...
> > >
> > > which are equivalent, unless bool(a) has side effects the second time
> > > it is called.
> > >
> > > In master we convert `if x: pass` to `pass` which is equivalent,
> > > unless bool(x) has side effects the first time it is called. This is a
> > > recent change.
> > >
> > > This is one of those "easy to fix, if we can decide on the semantics"
> > > bugs.
> > >
> > >
> > > Submit your thoughts to https://bugs.python.org/issue42899, please.
> > >
> > > Cheers,
> > > Mark.
> >
> > One key point about 'and' and 'or' is that those operators are defined
> > to be 'short circuiting', i.e. that  with a and b, that if a is false,
> > then b is not evaluated at all. This can be important if a is a
> > condition that test if the expression b is 'safe' to evaluate. In fact,
> > isn't it true that 'a and b' is defined to be the equivalent to:  'a if
> > not a else b' so b doesn't need to be evaluated unless a is truthy.
>

https://snarky.ca/unravelling-boolean-operations/ if you want the gory
details.


>
> Yes, the shortcircuiting behaviour isn't in question. But consider:
>
> class A:
> def __bool__(self):
> print("A().__bool__")
> return False
>
> def f():
> print("if A() and A()")
> if A() and A(): x = 1
> print("cond = A() and A()")
> cond = A() and A()
> if cond: x = 1
>
> f()
>
> And once you've run the code, disassemble the function for extra insight.
>
> There's a very definite difference here, and it's the same difference as:
>
> for x in thing:
>
> and
>
> for x in iter(thing):
>
> I'd be fine with documenting that __bool__ is (guaranteed to be)
> called only if the interpreter needs to know the result, leaving open
> the option for things like this to be optimized out. That'd leave open
> the option for "foo() if x else foo()" to be optimized down to just
> "foo()", although I don't think that particular one is needed.
>

I think that's the key question here: it is a language change, but is it
one we want (specifically in this case and in general)? It will require a
language reference change at least, and as it has been shown, some people
don't expect Python to take shortcuts in execution knowing full well that
CPython dutifully executes things. So saying we only call __bool__() *if
necessary* is definitely a change. But then this begs the question of
whether we want to do this more widely in the language in the name of
optimization, or not in the name of consistency that a user can easily
reason about.


>
> > I know that I would be very surpised if a statement like
> >
> >
> > if foo():
> >
> > pass
> >
> > ended up optimizing itself and not calling foo(), which is only one step
> > away from the quoted case of
> >
> > if b:
> >
> > pass
> >
> > which is the equivalent to
> >
> > if b.__bool__():
> >
> > pass
>
> Yes, they do look similar. The difference is that calling __bool__ is
> under the interpreter's control, and it's easier to document that it
> be assumed to be side-effect-free.
>
> > Yes, perhaps there is more of an expectation that __bool__() is
> > innocuous, but is that a an assumption that should be baked into the
> > language.
> >
>
> I think so. Consider that sometimes other dunders won't be called, if
> the interpreter believes it's not necessary:
>
> class A(float):
> def __index__(self):
> print("A().__index__")
> return 10
>
> class B(int):
> def __index__(self):
> print("B().__index__")
> return 10
>
> print(range(20)[A():B()])
>
> If it's a subclass of float, slicing will call __index__, but if it's
> a subclass of int, Python knows already that it can use the internal
> integer value.
>

https://snarky.ca/unravelling-not-in-python/  But basically,
https://docs.python.org/3.8/reference/datamodel.html#object.__index__ says
"called *if *conversion to an int is necessary" which isn't the case when
something is already an int.


>
> ChrisA
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GHKDF6YE3D43JPWS7GVG34FVPJNYE5SO/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To

[Python-Dev] Re: Let's Fix Class Annotations -- And Maybe Annotations Generally

2021-01-12 Thread Brett Cannon

On Mon, Jan 11, 2021 at 5:57 PM Larry Hastings  wrote:

>
> On 1/11/21 5:05 PM, Greg Ewing wrote:
>
> On 12/01/21 6:22 am, Larry Hastings wrote:
>
>  * The language will set __annotations__ to a dict if the object has
>
>annotations, or None if it has no annotations.
>
>
> That sounds inconvenient -- it means that any code referencing
> __annotations__ has to guard against the possibility of it being
> None.
>
> It was a balancing act.  Using an 64-byte empty dict per object with no
> defined annotations seems so wasteful.  And anything short of an empty
> dict, you'd have to guard against.  Current code already has to guard
> against "__annotations__ aren't set" anyway, so I figured the cost of
> migrating to checking a different condition would be small.  And None is so
> cheap, and the guard is so easy:
>
> if o.__annotations__:
>
>
But if things could fall through in the default case such that use in a
`for` loop, it is nice as Guido pointed out.

The other thing to keep in mind is we are talking about every module,
class, and function getting 64 bytes ... which I bet isn't that much. I bet
you save more memory running with -OO than what this will cost users in
memory.

And I know you were somewhat joking when you mentioned using
sys.version_info, but since this would be behind a __future__ import it
means the version check just means you then need to *potentially* worry
about the semantic shift (until the change becomes permanent). It seems the
changes are all still easy enough to have fallthrough and semantic checks
that it won't be much of a problem. I think it really means people need to
rely on typing.get_type_hints() more than they may be doing right now.

>
> If we're changing things, I'm wondering if the best thing would be
> to introduce an annotations() function as the new best practice for
> getting an object's annotations. It would know how to handle all
> the type-specific pecularities, and could take care of things such
> as manufacturing an empty dict if the object doesn't have any
> annotations.
>
> I guess I'm marginally against this, just because it seems like a needless
> change.  We don't need the flexibility of a function with optional
> parameters and such, and with a data descriptor we can already put code
> behind __annotations__ (as I have already done).  Plus, the function should
> probably cache its result--you wouldn't want code that called it ten times
> to generate ten fresh dicts, would you?--and already we're most of the way
> to what I proposed in PEP 649.
>

I also don't think introspection on annotations is common enough to warrant
a built-in function; this stuff is meant for tools, not for the average
developer to be dynamically playing with. As Guido pointed out,
typing.get_type_hints() already covers this.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WGDIGZSMGFDXBI5ZTVPDMYBPRAKBNIG4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors



On 1/12/21 3:53 PM, Greg Ewing wrote:

On 13/01/21 5:47 am, Larry Hastings wrote:


instead of the compiler storing a code object as the annotations 
argument to MAKE_FUNCTION, store a tuple containing the fields you'd 
need to /recreate/ the code object at runtime--bytecode, lnotab, 
names, consts, etc.


Would that tuple be significantly smaller than the corresponding
code object, though?


It would only be slightly smaller.  The point of doing it would be to 
boil out fields that change per-object (e.g. co_name) so that functions 
with identical signatures would share the same tuple both in the .pyc 
and at runtime.  This idea is predicated on Inada-san's assertion that 
this is an important memory optimization, that there are large 
heavily-annotated projects with lots of functions/methods with identical 
signatures where this memory savings is significant.



//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QCARB4FO3DYT46U2R5VX5AQYZ4OW7K3I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Bumping minimum Sphinx version to 3.2 for cpython 3.10?

2021-01-12 Thread Senthil Kumaran

On Wed, Jan 13, 2021 at 02:53:30AM +0300, Ivan Pozdeev via Python-Dev wrote:
> > I support keeping same Sphinx version across all the supported python 
> > versions.
> 
> This is not a sustainable route since this way, there's no way to change the 
> version at all.
> 

By supported, I mean the active versions that we backport patches to.
Same as what Victor said.

--
Senthil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YF7KWOG4L7AEJWMP3WY7YCXAICC63LDL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Bumping minimum Sphinx version to 3.2 for cpython 3.10?

2021-01-12 Thread Ivan Pozdeev via Python-Dev




On 13.01.2021 2:47, Senthil Kumaran wrote:

On Wed, Jan 13, 2021 at 12:28:42AM +0100, Victor Stinner wrote:


The alternative is to keep Sphinx 2 support, use
strip_signature_backslash and don't use :no-trim-doctest-flags: ?

+1. :no-trim-doctest-flags: was introduced to python docs only recently, so not
using that is a reasonable suggestion.

I support keeping same Sphinx version across all the supported python versions.


This is not a sustainable route since this way, there's no way to change the 
version at all.



--
Senthil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HS2OYF7QMSOEQDDYTXGEA7HRQDCYF6X6/
Code of Conduct: http://python.org/psf/codeofconduct/


--
Regards,
Ivan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/36DN32A233MURV5AECCPO2WUXLLJFGFA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors

2021-01-12 Thread Greg Ewing


On 13/01/21 5:47 am, Larry Hastings wrote:


instead of the compiler storing a code object as the annotations 
argument to MAKE_FUNCTION, store a tuple containing the fields you'd 
need to /recreate/ the code object at runtime--bytecode, lnotab, names, 
consts, etc.


Would that tuple be significantly smaller than the corresponding
code object, though?

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2NHFIGK55XN4U77V6V2KTBRGJ7TSGAAB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Bumping minimum Sphinx version to 3.2 for cpython 3.10?

2021-01-12 Thread Miro Hrončok


On 13. 01. 21 0:28, Victor Stinner wrote:

I looked at Sphinx and Python versions of Debian, Ubuntu and Fedora:
https://bugs.python.org/issue42843#msg384963

In my list, there is only Debian Buster (stable) which doesn't have
Sphinx 3 yet. It uses Python 3.7 and so would not be affected by
Python 3.8 changes.


Hello Victor.

In Fedora 32, we have Sphinx 2.2.2 and Python 3.8. Fedora 32 goes EOL in May 
2021. Until then, it would eb great if we could keep compatibility with Sphinx 2 
in Python 3.8 (but if we don't, no big deal, we just won't update the docs there).


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R574346ZSKEJMUBMOGO27RFCHHFR64MB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors

2021-01-12 Thread Greg Ewing


On 13/01/21 6:54 am, Larry Hastings wrote:
Note that this only works in the current version of the PEP / prototype, 
where annotations are strictly evaluated in module scope.  If we start 
supporting closures, those need "cell" objects, which IIUC can't be 
marshalled.


The cells belong to the enclosing scope, not the function that uses
them. I don't think they would interfere with sharing parts of code
objects between identical annotations.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QKMYWGBSQKNFUNBHXXWFJZ7WIRHGTEWI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Bumping minimum Sphinx version to 3.2 for cpython 3.10?

2021-01-12 Thread Senthil Kumaran

On Wed, Jan 13, 2021 at 12:28:42AM +0100, Victor Stinner wrote:

> The alternative is to keep Sphinx 2 support, use
> strip_signature_backslash and don't use :no-trim-doctest-flags: ?

+1. :no-trim-doctest-flags: was introduced to python docs only recently, so not
using that is a reasonable suggestion.

I support keeping same Sphinx version across all the supported python versions.

--
Senthil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HS2OYF7QMSOEQDDYTXGEA7HRQDCYF6X6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Bumping minimum Sphinx version to 3.2 for cpython 3.10?

2021-01-12 Thread Victor Stinner

Since documentation changes are backported to 3.8 and 3.9 stable
branches, if we increase the minimum required Sphinx version in
master, I would prefer to also increase it in 3.8 and 3.9 branches.

I would prefer to not have to check manually if a doc backport PR is
still compatible with Sphinx 2 or not. If we skip some doc backports,
conflicts will become more likely with following doc backports.

--

I looked at Sphinx and Python versions of Debian, Ubuntu and Fedora:
https://bugs.python.org/issue42843#msg384963

In my list, there is only Debian Buster (stable) which doesn't have
Sphinx 3 yet. It uses Python 3.7 and so would not be affected by
Python 3.8 changes.

--

The alternative is to keep Sphinx 2 support, use
strip_signature_backslash and don't use :no-trim-doctest-flags: ?

Victor

On Tue, Jan 12, 2021 at 9:44 PM Julien Palard via Python-Dev
 wrote:
>
> During the development of cpython 3.10, Sphinx was bumped to 3.2.1.
>
> Problem is Sphinx 3 have some incompatibilities with Sphinx 2, some that
> we could work around, some are bit harder, so we may need to bump
> `needs_sphinx = '3.2'` (currently it is 1.8).
>
> I found two incompatibilities:
>
> - We're using :no-trim-doctest-flags:, introduced in Sphinx 3.2.0, if we
> build with Sphinx < 3.2.0 the blocks using it are dropped by Sphinx (we
> use it in library/doctest).
>
> - double backslashes in domain directives are no longer replaced by
> single backslashes. But a configuration value
> (strip_signature_backslash) can be used to have the same behavior on
> Sphinx 2 and 3.
>
> So yes, it's still possible to build the docs with Sphinx 1.8 using
> `make html SPHINXERRORHANDLING=`, it "works" but:
>
> - Doctests code blocks using no-trim-doctest-flags dissapear.
> - Some functions declarations are lacking a backslash, like
>print(*objects, sep=' ', end='n', ...
>
> Which is bad.
>
> For the first one we could still workaround with an ugly `sed -i
> /no-trim-doctest-flags/d`, or simply stop using it (and reopen bpo-36675).
> For the 2nd one we could use `strip_signature_backslash` in cpython 3.10
> to have the same behavior.
>
> So the question is: Should we bump minimum version of Sphinx from 1.8 to
> 3.2.1 along with Python 3.10? For the moment this is where we go, but if
> you're having to maintain a release of Python along with Sphinx < 3,
> please make speak.
>
> Conversation has already started here [1] and here [2].
>
> [1]: https://bugs.python.org/issue42843
> [2]: https://github.com/python/cpython/pull/24142
>
> --
> [Julien Palard](https://mdk.fr)
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/E64YE3DQGOHLFQOJAJHS7VW3PK5KLB4W/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VABPDAAMCNMHLSG4N2FXK6S7NHYTPBRM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Bumping minimum Sphinx version to 3.2 for cpython 3.10?

2021-01-12 Thread Senthil Kumaran

On Tue, Jan 12, 2021 at 08:38:17PM +, Julien Palard via Python-Dev wrote:

> - Some functions declarations are lacking a backslash, like
>print(*objects, sep=' ', end='n', ...
> 
> Which is bad.

Wouldn't this a bug with Sphinx?
Why would that be special cased with a flag (strip_signature_backslash)?

> For the moment this is where we go, but if 

+1. Keeping the documentation dependency updated

I noticed that you suggested not to backport this PR
https://github.com/python/cpython/pull/24142

* Would that mean we have to careful not to use/merge sphinx features like
  `no-trim-doctest-flags` to older docs?

* What would it take to keep all active versions of Cpython docs build at the
  same min version (possibly 3.2)?

As Cpython developer, I see nothing bad, but platform maintainers seem to have
some concerns, and it will be good to see points against this.

Thank you,
Senthil

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KSMNZLNDTNOIKADXBCAGXFE5MTQTF42Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors

On Tue, Jan 12, 2021 at 11:31 AM Jim J. Jewett  wrote:

> Larry Hastings wrote:
> > The control-flow exclusion is for /module//attribute/ or /class
> > attribute/ annotations:
> > class C:
> >if random.random() > 0.5:
> >  my_attr:int=3
> >else:
> >  my_attr2:float=3.5
>
> That very example would be helpful in the FAQ, though I understand if
> you're concerned about making a minor sub-section seem too long.
>

This elucidates a crucial point to me: Larry's proposal looks at the source
code of the annotations.


> If I understand correctly, the problem is that you can't store multiple
> alternative annotations on my_attr.  Therefore:
>
> class C:
> my_attr:(int if random.random > 0.5 else float)
>
> should be OK, because there is only a single annotation.
>

Does that mean that the generated function would contain the entire
expression `(int if random.random > 0.5 else float)`? I guess that's what
it has to mean. But the PEP only uses such simple examples that it's easy
to miss this.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CQYPHXLTKOI3ZWUOFTX5VAOUIXLVA3OP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors



On 1/12/21 12:16 PM, Paul Bryan wrote:

On Tue, 2021-01-12 at 09:54 -0800, Larry Hastings wrote:


Note that this only works in the current version of the PEP / 
prototype, where annotations are strictly evaluated in module scope.  
If we start supporting closures, those need "cell" objects, which 
IIUC can't be marshalled.


Since the __co_annotations__ function will get globals from the 
function it annotates, doesn't it get more than just module scope?



I'm not sure what you're asking.  And, uh, what's the difference between 
"globals" and "module scope"?



//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OTIQH6QOZ7TTUPARANPSF3NNC275LGJS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Bumping minimum Sphinx version to 3.2 for cpython 3.10?

2021-01-12 Thread Julien Palard via Python-Dev

During the development of cpython 3.10, Sphinx was bumped to 3.2.1.

Problem is Sphinx 3 have some incompatibilities with Sphinx 2, some that
we could work around, some are bit harder, so we may need to bump
`needs_sphinx = '3.2'` (currently it is 1.8).

I found two incompatibilities:

- We're using :no-trim-doctest-flags:, introduced in Sphinx 3.2.0, if we
build with Sphinx < 3.2.0 the blocks using it are dropped by Sphinx (we
use it in library/doctest).

- double backslashes in domain directives are no longer replaced by
single backslashes. But a configuration value
(strip_signature_backslash) can be used to have the same behavior on
Sphinx 2 and 3.

So yes, it's still possible to build the docs with Sphinx 1.8 using
`make html SPHINXERRORHANDLING=`, it "works" but:

- Doctests code blocks using no-trim-doctest-flags dissapear.
- Some functions declarations are lacking a backslash, like
print(*objects, sep=' ', end='n', ...

Which is bad.

For the first one we could still workaround with an ugly `sed -i
/no-trim-doctest-flags/d`, or simply stop using it (and reopen bpo-36675).
For the 2nd one we could use `strip_signature_backslash` in cpython 3.10
to have the same behavior.

So the question is: Should we bump minimum version of Sphinx from 1.8 to
3.2.1 along with Python 3.10? For the moment this is where we go, but if
you're having to maintain a release of Python along with Sphinx < 3,
please make speak.

Conversation has already started here [1] and here [2].

[1]: https://bugs.python.org/issue42843
[2]: https://github.com/python/cpython/pull/24142

--
[Julien Palard](https://mdk.fr)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/E64YE3DQGOHLFQOJAJHS7VW3PK5KLB4W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors

2021-01-12 Thread Paul Sokolovsky

Hello,

On Mon, 11 Jan 2021 13:44:45 -0800
Larry Hastings  wrote:

> The control-flow exclusion is for /module//attribute/ or /class 
> attribute/ annotations:
> 
> class C:
>    if random.random() > 0.5:
>      my_attr:int=3
>    else:
>      my_attr2:float=3.5

Ok, let's take "module attribute" as an example. Why do you think
there's anything wrong with this code:

==
import config
from .types import *

if config.SUPPORT_BIGINT:
var: bigint = 1
else:
var: int64 = 1
==

> Your example doesn't define any module attributes or class attributes 
> inside flow control statements, so that code should work fine.  
> (Defining functions/methods inside flow control statements isn't a
> problem.)

PEP649 criticizes PEP563's approach with big words like "It requires
Python implementations to stringize their annotations. This is
surprising behavior — unprecedented for a language-level feature." But
itself devolves to clauses like:

>>>  It seems reasonable to declare that both are at the very least
>>> unsupported, and their use results in undefined behavior. It might
>>> be worth making a small effort to explicitly prohibit them with
>>> compile-time checks.

Isn't the fact that PEP563 doesn't have problems with annotations in
conditionals is a sign of PEP563 technical superiority? And its
"unprecedented" measure of storing annotations as strings is actually a
clever technical feat - it should have stored annotations as AST trees,
but such trees would take quite a lot of memory. So, PEP563 smartly
went to store those ASTs in a serialized format. So, those strings
aren't strings, but serialized ASTs.

Overall 2 comments/questions:

1. Was there an attempt to devise how to make PEP649 deal with existing
Python language features (like conditionals)?
2. As a general comment, PEP649, by placing arbitrary restrictions on
where annotations can appear, tries to dig under the foundations of
Python as a dynamic language. Which is absolutely great, it just the
PEP should be viewed as such - undermining Python's dynamic nature, as
if there's something wrong with it.

-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2GLVNLJ4WZMQ5FPMNAFZFU7MSVK5GOTJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors

2021-01-12 Thread Paul Bryan via Python-Dev

On Tue, 2021-01-12 at 09:54 -0800, Larry Hastings wrote:
> Note that this only works in the current version of the PEP /
> prototype, where annotations are strictly evaluated in module scope. 
> If we start supporting closures, those need "cell" objects, which
> IIUC can't be marshalled.
Since the __co_annotations__ function will get globals from the
function it annotates, doesn't it get more than just module scope?

Paul

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VWY42HWKA5LQ7BZ7VROCEDWVYHWELSBI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol





On 1/12/21 8:23 PM, Neil Schemenauer wrote:

On 2021-01-12, Pablo Galindo Salgado wrote:

One worry that I have in general with this move is the usage of
_PyType_GetModuleByDef to get the type object from the module
definition. This normally involves getting a TLS in every instance
creation, which can impact notably performance for some
perf-sensitive types or types that are created a lot.


I would say _PyType_GetModuleByDef is the problem.  Why do we need
to use such an ugly approach (walking the MRO) when Python defined
classes don't have the same performance issue?  E.g.

 class A:
 def b():
 pass
 A.b.__globals__

IMHO, we should be working to make types and functions defined in
extensions more like the pure Python versions.

Related, my "__namespace__" idea[1] might be helpful in reducing the
differences between pure Python modules and extension modules.
Rather than functions having a __globals__ property, which is a
dict, they would have a __namespace__, which is a module object.
Basically, functions and methods known which global namespace
(module) they have been defined in.  For extension modules, when you
call a function or method defined in the extension, it could be
passed the module instance, by using the __namespace__ property.

Maybe I'm missing some details on why this approach wouldn't work.
However, at a high level, I don't see why it shouldn't.  Maybe
performance would be an issue?  Reducing the number of branches in
code paths like CALL_FUNCTION should help.


The main difference between Python and C functions is that in C, you 
need type safety. You can't store C state in a mutable dict (or module) 
accessible from Python, because when users invalidate your C invariants, 
you get a segfault rather than a nice AttributeError.


Making methods "remember" their context does work though, and has 
already been implemented -- see PEP 573!
It uses the *defining class* instead of __namespace__, but you can get 
the module from that quite easily.


The only place it doesn't work are slot methods, which have a fixed C 
API. For example:


PyObject *tp_repr(PyObject *self);
int tp_init(PyObject *self, PyObject *args, PyObject *kwds);

There is no good way to pass the method, module object, globals() or the 
defining class to such functions.




1. https://github.com/nascheme/cpython/tree/frame_no_builtins

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TZVSCCCTUISV32U2OTE5LY7F3X5QAVCX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

2021-01-12 Thread Neil Schemenauer

On 2021-01-12, Petr Viktorin wrote:
> Unfortunately, it's not just the creation that needs to be changed.
> You also need to decref Foo_Type somewhere.

Add the type to the module dict?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7LSR4IASUNVVEAV6M6FRHZN7DABWHSBY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors



On 1/12/21 11:26 AM, Jim J. Jewett wrote:

If I understand correctly, the problem is that you can't store multiple 
alternative annotations on my_attr.  Therefore:

 class C:
 my_attr:(int if random.random > 0.5 else float)

should be OK, because there is only a single annotation.


Sure, that works fine.  Any expression (except "yield" and ":=") is okay 
in an annotation.




What about optional attributes, like:

  class C:
     if random.random() > 0.5:
       my_attr:int=3

Also, would (conditionally defined) function variable attributes become a 
problem if they were actually stored?  (Take Larry's class example, and make if 
a def instead of a class statement.)


You mean attributions on function locals?

   def foo():
  if random.random() > 0.5:
    x:int=3
  else:
    x:float=3.5

As I mentioned in my PEP, attributions on function locals have no effect 
at runtime.  If they did, this would cause the same problem that doing 
it in classes has.



Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RXNT6SUAIBOEUBGQLZ4TCW7CJVWAFY7I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

2021-01-12 Thread Neil Schemenauer

On 2021-01-12, Pablo Galindo Salgado wrote:
> One worry that I have in general with this move is the usage of
> _PyType_GetModuleByDef to get the type object from the module
> definition. This normally involves getting a TLS in every instance
> creation, which can impact notably performance for some
> perf-sensitive types or types that are created a lot.

I would say _PyType_GetModuleByDef is the problem.  Why do we need
to use such an ugly approach (walking the MRO) when Python defined
classes don't have the same performance issue?  E.g.

class A:
def b():
pass
A.b.__globals__

IMHO, we should be working to make types and functions defined in
extensions more like the pure Python versions.

Related, my "__namespace__" idea[1] might be helpful in reducing the
differences between pure Python modules and extension modules.
Rather than functions having a __globals__ property, which is a
dict, they would have a __namespace__, which is a module object.
Basically, functions and methods known which global namespace
(module) they have been defined in.  For extension modules, when you
call a function or method defined in the extension, it could be
passed the module instance, by using the __namespace__ property.

Maybe I'm missing some details on why this approach wouldn't work.
However, at a high level, I don't see why it shouldn't.  Maybe
performance would be an issue?  Reducing the number of branches in
code paths like CALL_FUNCTION should help.

1. https://github.com/nascheme/cpython/tree/frame_no_builtins
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ERZFFBSO6J4G4X3V5QFWH6CBEEECEIAG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors

2021-01-12 Thread Jim J. Jewett

Larry Hastings wrote:
> The control-flow exclusion is for /module//attribute/ or /class 
> attribute/ annotations:
> class C:
>    if random.random() > 0.5:
>      my_attr:int=3
>    else:
>      my_attr2:float=3.5

That very example would be helpful in the FAQ, though I understand if you're 
concerned about making a minor sub-section seem too long.

If I understand correctly, the problem is that you can't store multiple 
alternative annotations on my_attr.  Therefore:

class C:
my_attr:(int if random.random > 0.5 else float)

should be OK, because there is only a single annotation.

What about optional attributes, like:

 class C:
    if random.random() > 0.5:
      my_attr:int=3

Also, would (conditionally defined) function variable attributes become a 
problem if they were actually stored?  (Take Larry's class example, and make if 
a def instead of a class statement.)

My (weakly held, personal) opinion is that these restrictions would be 
reasonable, and a single release of deprecation would be enough, but it would 
be better if that code could trigger a deprecation warning during that release, 
even for code that hasn't done the future import.  It would also be OK to just 
say "implementation-defined behavior; CPython 3.x ignores the annotation" 
instead of banning them.

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TRX23XWZ3NGROVQRC6DXLIU7NPPNEZRB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

2021-01-12 Thread Nelson, Karl E. via Python-Dev

Having used the heap types extensively for JPype, I believe that converting all 
types too heap types would be a great benefit.  There are still minor rough 
spots in which a static type can do things that heap types cannot (such as you 
can derive a type which is marked final when it is static but not heap such as 
function).  But generally I found heap types to be much more flexible.   I 
found that heap types were better in concept than static but because the 
majority of the API (and the examples on using CAPI) were static the heap types 
paths were less exercised.   I eventually puzzled out most of the mysteries, 
but having the everything be the same (except for old static types that should 
be marked as immortal) likely has a lot of side benefits.   

Of course the other issue that I have with heap types is that they currently 
lack the concept of meta classes.   Thus there are things that you can do from 
the Python language that you can't do from the C API.  See...

https://bugs.python.org/issue42617

The downside of course is there are a lot of calls in the C API that infer that 
static type is fixed address.   Perhaps those call all be macros to the which 
equate to evaluating the address of the heap type.   

But that is just my 2 cents.

--Karl

-Original Message-
From: Neil Schemenauer  
Sent: Tuesday, January 12, 2021 10:17 AM
To: Victor Stinner 
Cc: Python Dev 
Subject: [Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the 
GC protocol

On 2021-01-12, Victor Stinner wrote:
> It seems like a safer approach is to continue the work on
> bpo-40077: "Convert static types to PyType_FromSpec()".

I agree that trying to convert static types is a good idea.  Another possible 
bonus might be that we can gain some performance by integrating garbage 
collection with the Python object memory allocator.  Static types frustrate 
that effort.

Could we have something easier to use than PyType_FromSpec(), for the purposes 
of coverting existing code?  I was thinking of something like:

static PyTypeObject Foo_TypeStatic = {
}
static PyTypeObject *Foo_Type;

PyInit_foo(void)
{
Foo_Type = PyType_FromStatic(_TypeStatic);
}

The PyType_FromStatic() would return a new heap type, created by copying the 
static type.  The static type could be marked as being unusable (e.g. with a 
type flag).
___
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email 
to python-dev-le...@python.org 
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RPG2TRQLONM2OCXKPVCIDKVLQOJR7EUU/
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DV4SPP2TTXGYMTMRMEO6TG5W7XPZKPXX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

2021-01-12 Thread Antoine Pitrou

On Tue, 12 Jan 2021 18:48:39 +
Pablo Galindo Salgado  wrote:
> One worry that I have in general with this move
> is the usage of _PyType_GetModuleByDef to get the type object
> from the module definition. This normally involves getting a TLS in every
> instance creation,
> which can impact notably performance for some perf-sensitive types or types
> that are created a lot.

If it's inlined C TLS it should be fast (*).  If it's Python's emulated
TLS then probably not :-)

(*) see https://godbolt.org/z/d7eKx7

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CKABGMLSJLKDGKUOMXA6MKO36MEWZIIS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

On 1/12/21 7:48 PM, Pablo Galindo Salgado wrote:

One worry that I have in general with this move
is the usage of _PyType_GetModuleByDef to get the type object
from the module definition. This normally involves getting a TLS in 
every instance creation,

Not TLS, it's walking the MRO.

which can impact notably performance for some perf-sensitive types or types
that are created a lot.

But yes, that's right. _PyType_GetModuleByDef should not be used in 
perf-sensitive spots, at least not without profiling.
There's often an alternative, though. Do you have any specific cases 
you're concerned about?

On Tue, 12 Jan 2021 at 18:21, Neil Schemenauer > wrote:

On 2021-01-12, Victor Stinner wrote:
 > It seems like a safer approach is to continue the work on
 > bpo-40077: "Convert static types to PyType_FromSpec()".

I agree that trying to convert static types is a good idea.  Another
possible bonus might be that we can gain some performance by
integrating garbage collection with the Python object memory
allocator.  Static types frustrate that effort.

Could we have something easier to use than PyType_FromSpec(), for
the purposes of coverting existing code?  I was thinking of
something like:

     static PyTypeObject Foo_TypeStatic = {
     }
     static PyTypeObject *Foo_Type;

     PyInit_foo(void)
     {
         Foo_Type = PyType_FromStatic(_TypeStatic);
     }

The PyType_FromStatic() would return a new heap type, created by
copying the static type.  The static type could be marked as being
unusable (e.g. with a type flag).
___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/RPG2TRQLONM2OCXKPVCIDKVLQOJR7EUU/

Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HOCGUW3S6AXBSQ5BWX5KYPFVXEGWQJ6H/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QYXDVMTI5CBKQOGYC557ER45IZZLJZGS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol


On 1/12/21 7:16 PM, Neil Schemenauer wrote:

On 2021-01-12, Victor Stinner wrote:

It seems like a safer approach is to continue the work on
bpo-40077: "Convert static types to PyType_FromSpec()".


I agree that trying to convert static types is a good idea.  Another
possible bonus might be that we can gain some performance by
integrating garbage collection with the Python object memory
allocator.  Static types frustrate that effort.

Could we have something easier to use than PyType_FromSpec(), for
the purposes of coverting existing code?  I was thinking of
something like:

 static PyTypeObject Foo_TypeStatic = {
 }
 static PyTypeObject *Foo_Type;

 PyInit_foo(void)
 {
 Foo_Type = PyType_FromStatic(_TypeStatic);
 }


The PyType_FromStatic() would return a new heap type, created by
copying the static type.  The static type could be marked as being
unusable (e.g. with a type flag).


Unfortunately, it's not just the creation that needs to be changed.
You also need to decref Foo_Type somewhere.

Your example is for "single-phase init" modules (pre-PEP 489). Those 
don't have a dealloc hook, so they will leak memory (e.g. in multiple 
Py_Initialize/Py_Finalize cycles).


Multi-phase init (PEP 489) allows multiple module instances of extension 
modules. Assigning PyType_FromStatic's result to a static pointer would 
mean that every instance of the module will create a new type, and 
overwrite any existing one. And the deallocation will either leave a 
dangling pointer or NULL the pointer for other module instances.


So, you need to make the type part of the module state, so that the 
module has proper ownership of the type. And that means you need to 
access the type from the module state any time you need to use it.


At that point, IMO, PyType_FromStatic saves you so little work that it's 
not worth supporting a third variation of type creation code.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WAKYSLYIUZN7NPCE6G6SRRCJK5RELJQ3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

2021-01-12 Thread Pablo Galindo Salgado

One worry that I have in general with this move
is the usage of _PyType_GetModuleByDef to get the type object
from the module definition. This normally involves getting a TLS in every
instance creation,
which can impact notably performance for some perf-sensitive types or types
that are created a lot.

On Tue, 12 Jan 2021 at 18:21, Neil Schemenauer 
wrote:

> On 2021-01-12, Victor Stinner wrote:
> > It seems like a safer approach is to continue the work on
> > bpo-40077: "Convert static types to PyType_FromSpec()".
>
> I agree that trying to convert static types is a good idea.  Another
> possible bonus might be that we can gain some performance by
> integrating garbage collection with the Python object memory
> allocator.  Static types frustrate that effort.
>
> Could we have something easier to use than PyType_FromSpec(), for
> the purposes of coverting existing code?  I was thinking of
> something like:
>
> static PyTypeObject Foo_TypeStatic = {
> }
> static PyTypeObject *Foo_Type;
>
> PyInit_foo(void)
> {
> Foo_Type = PyType_FromStatic(_TypeStatic);
> }
>
>
> The PyType_FromStatic() would return a new heap type, created by
> copying the static type.  The static type could be marked as being
> unusable (e.g. with a type flag).
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RPG2TRQLONM2OCXKPVCIDKVLQOJR7EUU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HOCGUW3S6AXBSQ5BWX5KYPFVXEGWQJ6H/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

2021-01-12 Thread Neil Schemenauer

On 2021-01-12, Victor Stinner wrote:
> It seems like a safer approach is to continue the work on
> bpo-40077: "Convert static types to PyType_FromSpec()".

I agree that trying to convert static types is a good idea.  Another
possible bonus might be that we can gain some performance by
integrating garbage collection with the Python object memory
allocator.  Static types frustrate that effort.

Could we have something easier to use than PyType_FromSpec(), for
the purposes of coverting existing code?  I was thinking of
something like:

static PyTypeObject Foo_TypeStatic = {
}
static PyTypeObject *Foo_Type;

PyInit_foo(void)
{
Foo_Type = PyType_FromStatic(_TypeStatic);
}

The PyType_FromStatic() would return a new heap type, created by
copying the static type.  The static type could be marked as being
unusable (e.g. with a type flag).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RPG2TRQLONM2OCXKPVCIDKVLQOJR7EUU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors

Given all the effort that get_type_hints() puts into evaluating those
strings it seems important to spell out explicitly that they're not
special. (IIRC they *are* special in PEP 563.)

On Tue, Jan 12, 2021 at 8:56 AM Larry Hastings  wrote:

>
> Yes, PEP 649 is completely agnostic about what values you put in as
> annotations.  You can put in strings, complex objects,
> expressions--whatever you put in, you get back out later.
>
> I'm happy to add some text to the PEP if this needs clarifying; I just
> thought it was obvious.
>
>
> Cheers,
>
>
> */arry*
> On 1/11/21 9:11 PM, Guido van Rossum wrote:
>
>  Another thought about this PEP (hopefully my last one tonight).
>
> The section on backwards compatibility doesn't mention what should happen
> with annotations that are stringified by the user (as is needed for forward
> references in code that hasn't been PEP-563-ified yet).
>
> That's a PEP 484 feature. Should we start deprecating that at the same
> time? Static checkers support it but don't need it (for example, stubs in
> typeshed don't use it since their code is never evaluated).
>
> At the very least I think your PEP should mention what happens for these
> -- presumably `__annotations__` will just contain the string literal, so
> get_type_hints() would be needed to evaluate these.
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>
>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MFPOC45A5IWY6HZU6FQQQV2ZYIRVRWUP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors



On 1/12/21 8:47 AM, Larry Hastings wrote:
It seems possible to create a hybrid of these two approaches! Here's 
my idea: instead of the compiler storing a code object as the 
annotations argument to MAKE_FUNCTION, store a tuple containing the 
fields you'd need to /recreate/ the code object at runtime--bytecode, 
lnotab, names, consts, etc. func_get_annotations would create the code 
object from that, bind it to a function object, call it, and return 
the result.  These code-object-tuples would then be automatically 
shared in the .pyc file and at runtime the same way that 3.10 shares 
the tuples of stringized annotations today.


Note that this only works in the current version of the PEP / prototype, 
where annotations are strictly evaluated in module scope.  If we start 
supporting closures, those need "cell" objects, which IIUC can't be 
marshalled.



//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I5TKGDL4KIRQIL5G56ZTCVG7O2PKAPBI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.10 change (?) for bool

2021-01-12 Thread Chris Angelico

On Wed, Jan 13, 2021 at 4:24 AM Richard Damon  wrote:
>
> On 1/12/21 10:53 AM, Mark Shannon wrote:
> > Hi everyone,
> >
> > Should the optimizer eliminate tests that it can prove have no effect
> > on the control flow of the program, even if that may eliminate some
> > side effects in __bool__()?
> >
> > For several years we have converted
> >
> > if a and b:
> > ...
> >
> > to
> >
> > if a:
> > if b:
> > ...
> >
> > which are equivalent, unless bool(a) has side effects the second time
> > it is called.
> >
> > In master we convert `if x: pass` to `pass` which is equivalent,
> > unless bool(x) has side effects the first time it is called. This is a
> > recent change.
> >
> > This is one of those "easy to fix, if we can decide on the semantics"
> > bugs.
> >
> >
> > Submit your thoughts to https://bugs.python.org/issue42899, please.
> >
> > Cheers,
> > Mark.
>
> One key point about 'and' and 'or' is that those operators are defined
> to be 'short circuiting', i.e. that  with a and b, that if a is false,
> then b is not evaluated at all. This can be important if a is a
> condition that test if the expression b is 'safe' to evaluate. In fact,
> isn't it true that 'a and b' is defined to be the equivalent to:  'a if
> not a else b' so b doesn't need to be evaluated unless a is truthy.

Yes, the shortcircuiting behaviour isn't in question. But consider:

class A:
def __bool__(self):
print("A().__bool__")
return False

def f():
print("if A() and A()")
if A() and A(): x = 1
print("cond = A() and A()")
cond = A() and A()
if cond: x = 1

f()

And once you've run the code, disassemble the function for extra insight.

There's a very definite difference here, and it's the same difference as:

for x in thing:

and

for x in iter(thing):

I'd be fine with documenting that __bool__ is (guaranteed to be)
called only if the interpreter needs to know the result, leaving open
the option for things like this to be optimized out. That'd leave open
the option for "foo() if x else foo()" to be optimized down to just
"foo()", although I don't think that particular one is needed.

> I know that I would be very surpised if a statement like
>
>
> if foo():
>
> pass
>
> ended up optimizing itself and not calling foo(), which is only one step
> away from the quoted case of
>
> if b:
>
> pass
>
> which is the equivalent to
>
> if b.__bool__():
>
> pass

Yes, they do look similar. The difference is that calling __bool__ is
under the interpreter's control, and it's easier to document that it
be assumed to be side-effect-free.

> Yes, perhaps there is more of an expectation that __bool__() is
> innocuous, but is that a an assumption that should be baked into the
> language.
>

I think so. Consider that sometimes other dunders won't be called, if
the interpreter believes it's not necessary:

class A(float):
def __index__(self):
print("A().__index__")
return 10

class B(int):
def __index__(self):
print("B().__index__")
return 10

print(range(20)[A():B()])

If it's a subclass of float, slicing will call __index__, but if it's
a subclass of int, Python knows already that it can use the internal
integer value.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GHKDF6YE3D43JPWS7GVG34FVPJNYE5SO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.10 change (?) for bool

2021-01-12 Thread Richard Damon

On 1/12/21 10:53 AM, Mark Shannon wrote:
> Hi everyone,
>
> Should the optimizer eliminate tests that it can prove have no effect
> on the control flow of the program, even if that may eliminate some
> side effects in __bool__()?
>
> For several years we have converted
>
>     if a and b:
>     ...
>
> to
>
>     if a:
>     if b:
>     ...
>
> which are equivalent, unless bool(a) has side effects the second time
> it is called.
>
> In master we convert `if x: pass` to `pass` which is equivalent,
> unless bool(x) has side effects the first time it is called. This is a
> recent change.
>
> This is one of those "easy to fix, if we can decide on the semantics"
> bugs.
>
>
> Submit your thoughts to https://bugs.python.org/issue42899, please.
>
> Cheers,
> Mark. 

One key point about 'and' and 'or' is that those operators are defined
to be 'short circuiting', i.e. that  with a and b, that if a is false,
then b is not evaluated at all. This can be important if a is a
condition that test if the expression b is 'safe' to evaluate. In fact,
isn't it true that 'a and b' is defined to be the equivalent to:  'a if
not a else b' so b doesn't need to be evaluated unless a is truthy.

I know that I would be very surpised if a statement like


if foo():

    pass

ended up optimizing itself and not calling foo(), which is only one step
away from the quoted case of

if b:

    pass

which is the equivalent to

if b.__bool__():

    pass


Yes, perhaps there is more of an expectation that __bool__() is
innocuous, but is that a an assumption that should be baked into the
language.

-- 
Richard Damon
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HODNO42SFFRFX4IJY5K562YHT2MTIHIQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors



Yes, PEP 649 is completely agnostic about what values you put in as 
annotations.  You can put in strings, complex objects, 
expressions--whatever you put in, you get back out later.


I'm happy to add some text to the PEP if this needs clarifying; I just 
thought it was obvious.



Cheers,


//arry/

On 1/11/21 9:11 PM, Guido van Rossum wrote:

 Another thought about this PEP (hopefully my last one tonight).

The section on backwards compatibility doesn't mention what should 
happen with annotations that are stringified by the user (as is needed 
for forward references in code that hasn't been PEP-563-ified yet).


That's a PEP 484 feature. Should we start deprecating that at the same 
time? Static checkers support it but don't need it (for example, stubs 
in typeshed don't use it since their code is never evaluated).


At the very least I think your PEP should mention what happens for 
these -- presumably `__annotations__` will just contain the string 
literal, so get_type_hints() would be needed to evaluate these.


--
--Guido van Rossum (python.org/~guido )
/Pronouns: he/him //(why is my pronoun here?)/ 

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7Q3SHNMRKLROBESEB2U4NNMZAFS2XFRU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors

On 1/11/21 5:33 PM, Inada Naoki wrote:

Note that PEP 563 semantics allows more efficient implementation.
Annotation is just a single constant tuple, not a dict.
We already have the efficient implementation for Python 3.10.

The efficient implementation in 3.10 can share tuples. If there are
hundreds of methods with the same signature, annotation is just a
single tuple, not hundreds of tuples. This is very efficient for auto
generated codebase. I think this PEP can share the code objects for
same signature by removing co_firstlineno information too.

That's very clever! My co_annotations repo was branched from before
this feature was added, and I haven't pulled and merged recently. So I
hadn't seen it.

Additionally, we should include the cost for loading annotations from
PYC files, because most annotations are "load once, set once".
Loading "simple code object" from pyc files is not so cheap. It may
affect importing time of large annotated codebase and memory
footprints.

I did some analysis in a separate message. The summary is, the code
object for a single annotation costs us 232 bytes; that includes the
code object itself, the bytestring for the bytecode, and the bytestring
for the lnotab. This grows slowly as you add new parameters; the code
object for ten parameters is 360 bytes.

It seems possible to create a hybrid of these two approaches! Here's my
idea: instead of the compiler storing a code object as the annotations
argument to MAKE_FUNCTION, store a tuple containing the fields you'd
need to /recreate/ the code object at runtime--bytecode, lnotab, names,
consts, etc. func_get_annotations would create the code object from
that, bind it to a function object, call it, and return the result.
These code-object-tuples would then be automatically shared in the .pyc
file and at runtime the same way that 3.10 shares the tuples of
stringized annotations today.

That said, I suggest PEP 649's memory consumption isn't an urgent
consideration in choosing to accept or reject it. PEP 649 is
competitive in terms of startup time and memory usage with PEP 563, and
PEP 563 was accepted and shipped with several versions of Python.

Cheers,

//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/375OTKI42EUKMB5VQAAIBXMQWQTXPYHH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors



On 1/11/21 5:37 PM, Larry Hastings wrote:
I'll have to let people with large code bases speak up about this, but 
my understanding is that most people would prefer Python to use less 
memory.  On my 64-bit Linux machine, a code object is 136 bytes, its 
empty __dict__ is 64 bytes, [...]



Oops: where I said "a code object is 136 bytes", I meant, a /function/ 
object is 136 bytes.  We regret the error.



//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XPEY2HJLVBW23YWQ7SK4YBNKMNW53GFT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

On 1/12/21 4:34 PM, Antoine Pitrou wrote:

On Tue, 12 Jan 2021 15:22:36 +0100
Petr Viktorin wrote:

On 1/11/21 5:26 PM, Victor Stinner wrote:

Hi,

There are multiple PEPs covering heap types. The latest one refers to
other PEPs: PEP 630 "Isolating Extension Modules" by Petr Viktorin.
https://www.python.org/dev/peps/pep-0630/#motivation

The use case is to embed multiple Python instances (interpreters) in
the same application process, or to embed Python with multiple calls
to Py_Initialize/Py_Finalize (sequentially, not in parallel). Static
types are causing different issues for these use cases.

If a type is immutable and has no references to heap-allocated objects,
it could stay as a static type.
The issue is that very many types don't fit that. For example, if some
method needs to raise a module-specific exception, that's a reference to
a heap-allocated type, because custom exceptions generally aren't static.

Aren't we confusing two different things here?

- a mutable *type*, i.e. a type with mutable state attached to itself
(not to instances)

- a mutable *instance*, where the mutable state is per-instance

While it's very common for custom exceptions to have mutable instance
state (e.g. a backend-specific error number), I can't think of any
custom exception that has mutable state attached to the exception
*type*.

You're right, exception types *could* generally be static. However, the
most common API for creating them, PyErr_NewException[WithDoc], creates
heap types.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/ZSQ4HGBIMFBOVXBUKB7C7UPIJPW76OLA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol




On 1/12/21 4:09 PM, Victor Stinner wrote:

On Tue, Jan 12, 2021 at 3:28 PM Petr Viktorin  wrote:

If a type is immutable and has no references to heap-allocated objects,
it could stay as a static type.
The issue is that very many types don't fit that. For example, if some
method needs to raise a module-specific exception, that's a reference to
a heap-allocated type, because custom exceptions generally aren't static.
(...)
I don't see why we would need to destroy immutable static objects. They
don't need to be freed.


I'm not sure of your definition of "immutable" here. At the C level,
many immutable Python objects are mutable. For example, a str instance
*can* be modified with the C level, and computing hash()
modifies the object as well (the internal cached hash value).

Any type contains at least one Python object: the __mro__ tuple. Most
types also contain a __subclass__ dictionary (by default, it's NULL).
These objects are created at Python startup, but not destroyed at
Python exit. See also tp_bases (tuple) and tp_dict (dict).


Ah, right. __subclasses__ is the reason these need to be heap types (if 
they allow subclassing, which – isn't).
If __mro__ is a tuple of static types, it could probably be made static 
as well; hashes could be protected by a lock.




I tried once to "finalize" static types, but it didn't go well:

* https://github.com/python/cpython/pull/20763
* https://bugs.python.org/issue1635741#msg371119

It doesn't look to be safe to clear static types. Many functions rely
on the fact that static types are "always there" and are never
finalized. Also, only a few static types are cleared by my PR: many
static types are left unchanged. For example, static types of the _io
module. It seems like a safer approach is to continue the work on
bpo-40077: "Convert static types to PyType_FromSpec()".


Yes, seems so. And perhaps this has enough subtle details to want a PEP?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3TLWPQT76RZ2Q6HEUKFURB3J45AXYWFE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.10 change (?) for bool

2021-01-12 Thread Mark Shannon

Hi everyone,

Should the optimizer eliminate tests that it can prove have no effect on 
the control flow of the program, even if that may eliminate some side 
effects in __bool__()?

For several years we have converted

if a and b:
...

to

if a:
if b:
...

which are equivalent, unless bool(a) has side effects the second time it 
is called.

In master we convert `if x: pass` to `pass` which is equivalent, unless 
bool(x) has side effects the first time it is called. This is a recent 
change.

This is one of those "easy to fix, if we can decide on the semantics" bugs.

Submit your thoughts to https://bugs.python.org/issue42899, please.

Cheers,
Mark.

On 12/01/2021 12:45 am, Guido van Rossum wrote:

Ah never mind. Seems to be a real bug -- thanks for reporting!

On Mon, Jan 11, 2021 at 2:57 PM Mats Wichmann > wrote:

On 1/11/21 1:00 PM, Guido van Rossum wrote:
 > All that said (I agree it's surprising that 3.10 seems backwards
 > incompatible here) I would personally not raise AttributeError but
 > TypeError in the `__bool__()` method.

eh, that was just me picking a cheap something to demo it.  the program
raises an application-specific error that I didn't feel like
defining to
keep the repro as short as possible.
___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/SVGFN4DCDN462QVVMHY45IKH2XL4GVRD/
Code of Conduct: http://python.org/psf/codeofconduct/

--
--Guido van Rossum (python.org/~guido )
/Pronouns: he/him //(why is my pronoun here?)/ 

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BK4IUDXCZDDQCRSX3QGY7XUHOKMIDPG4/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DFX7HMZ7RFUQJMJI7MABHKEK4EOYHR4A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

2021-01-12 Thread Antoine Pitrou

On Tue, 12 Jan 2021 15:22:36 +0100
Petr Viktorin  wrote:
> On 1/11/21 5:26 PM, Victor Stinner wrote:
> > Hi,
> > 
> > There are multiple PEPs covering heap types. The latest one refers to
> > other PEPs: PEP 630 "Isolating Extension Modules" by Petr Viktorin.
> > https://www.python.org/dev/peps/pep-0630/#motivation
> > 
> > The use case is to embed multiple Python instances (interpreters) in
> > the same application process, or to embed Python with multiple calls
> > to Py_Initialize/Py_Finalize (sequentially, not in parallel). Static
> > types are causing different issues for these use cases.  
> 
> If a type is immutable and has no references to heap-allocated objects, 
> it could stay as a static type.
> The issue is that very many types don't fit that. For example, if some 
> method needs to raise a module-specific exception, that's a reference to 
> a heap-allocated type, because custom exceptions generally aren't static.

Aren't we confusing two different things here?

- a mutable *type*, i.e. a type with mutable state attached to itself
  (not to instances)

- a mutable *instance*, where the mutable state is per-instance

While it's very common for custom exceptions to have mutable instance
state (e.g. a backend-specific error number), I can't think of any
custom exception that has mutable state attached to the exception
*type*.

> > Also, it's not possible to destroy static types at Python exit, which
> > goes against the on-going effort to destroy all Python objects at exit
> > (bpo-1635741).  
> 
> I don't see why we would need to destroy immutable static objects. They 
> don't need to be freed.

Right.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FNSTYH7AQQUJFCD354AWDXVZWCLNV462/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

2021-01-12 Thread Victor Stinner

On Tue, Jan 12, 2021 at 3:28 PM Petr Viktorin  wrote:
> If a type is immutable and has no references to heap-allocated objects,
> it could stay as a static type.
> The issue is that very many types don't fit that. For example, if some
> method needs to raise a module-specific exception, that's a reference to
> a heap-allocated type, because custom exceptions generally aren't static.
> (...)
> I don't see why we would need to destroy immutable static objects. They
> don't need to be freed.

I'm not sure of your definition of "immutable" here. At the C level,
many immutable Python objects are mutable. For example, a str instance
*can* be modified with the C level, and computing hash()
modifies the object as well (the internal cached hash value).

Any type contains at least one Python object: the __mro__ tuple. Most
types also contain a __subclass__ dictionary (by default, it's NULL).
These objects are created at Python startup, but not destroyed at
Python exit. See also tp_bases (tuple) and tp_dict (dict).

I tried once to "finalize" static types, but it didn't go well:

* https://github.com/python/cpython/pull/20763
* https://bugs.python.org/issue1635741#msg371119

It doesn't look to be safe to clear static types. Many functions rely
on the fact that static types are "always there" and are never
finalized. Also, only a few static types are cleared by my PR: many
static types are left unchanged. For example, static types of the _io
module. It seems like a safer approach is to continue the work on
bpo-40077: "Convert static types to PyType_FromSpec()".

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OMDNS54ZVOAMJALDCFB7IVD26WF47WB4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

2021-01-12 Thread encukou

Simon Cross wrote:
> We've run into some minor issues with the limitations in PyType_Slot
> (https://docs.python.org/3/c-api/type.html#c.PyType_Slot.PyType_Slot.slot)
> but we are working around them for the moment.
> It would be useful to have some sense of where PyType_FromSpec is
> headed -- e.g. is it a goal to have it support all of the features of
> static types in the future -- so that we can perhaps help suggest /
> implement small changes that head in the right direction and also
> ensure that HPy is aligned with the immediate future of the C API.

Yes, the goal is to have it support all the features of static types.
If you see something that's not in PEP 630 open issues 
(https://www.python.org/dev/peps/pep-0630/#open-issues),
I'd like to know.
I'm using https://github.com/encukou/abi3/issues to collect issues related to 
the stable ABI. Maybe have some of HPy's issues there already.

And fixes are always welcome, of course :)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BFLYTI2WZZY5W3LKF5WBT4MZBTB6O7N6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol