[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Chris Angelico]

On Sat, 23 Apr 2022 at 15:32, Larry Hastings  wrote:
>
>
> On 4/22/22 22:03, Chris Angelico wrote:
>
> Anyhow, [a forward-defined class object is] a class, with some special
> features (notably that you can't instantiate it).
>
> Yes.  Specifically, here's my intention for "forward-defined class objects": 
> you can examine some generic dunder values (__name__, __mro__), and you can 
> take references to it.  You can't instantiate it or meaningfully examine its 
> contents, because it hasn't been fully initialized yet.
>
>
> It seems odd that you define a blessed way of monkeypatching a class,
> but then demand that it can only be done once unless you mess with
> dunders. Why not just allow multiple continuations?
>
> I think monkeypatching is bad, and I'm trying to avoid Python condoning it.
>
> On that note, the intent of my proposal is that "continue class" is not 
> viewed as "monkeypatching" the class, it's the second step in defining the 
> class.
>
> I considered attempting to prevent the user modifying the "forward-declared 
> class object".  But a) that just seemed like an arms race with the user--"oh 
> yeah? well watch THIS!" and b) I thought the Consenting Adults rule applied.
>
> Still, it's not the intent of my PEP to condone or facilitate monkeypatching.
>

I guess perception is everything. This really *is* monkeypatching; you
have an object, and that object will be mutated, and those mutations
are defined by adding functionality to the class. It's not
fundamentally different from X.__dict__.update() with a pile of new
methods etc. The only difference is that you call it something
different.

Is that sufficient justification? I'm unsure.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DIEGQM5ARFGIYPWG34CQODAENKNBVUET/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Larry Hastings]

On 4/22/22 22:03, Chris Angelico wrote:

Anyhow, [a forward-defined class object is] a class, with some special
features (notably that you can't instantiate it).


Yes.  Specifically, here's my intention for "forward-defined class 
objects": you can examine some generic dunder values (__name__, 
__mro__), and you can take references to it.  You can't instantiate it 
or meaningfully examine its contents, because it hasn't been fully 
initialized yet.




It seems odd that you define a blessed way of monkeypatching a class,
but then demand that it can only be done once unless you mess with
dunders. Why not just allow multiple continuations?


I think monkeypatching is bad, and I'm trying to avoid Python condoning it.

On that note, the intent of my proposal is that "continue class" is not 
viewed as "monkeypatching" the class, it's the second step in defining 
the class.


I considered attempting to prevent the user modifying the 
"forward-declared class object".  But a) that just seemed like an arms 
race with the user--"oh yeah? well watch THIS!" and b) I thought the 
Consenting Adults rule applied.


Still, it's not the intent of my PEP to condone or facilitate 
monkeypatching.



//arry/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CXRMKKMKIHGGPAW3F4W4QCNRDIBHFLTB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Larry Hastings]

On 4/22/22 20:58, Steven D'Aprano wrote:

On Fri, Apr 22, 2022 at 06:13:57PM -0700, Larry Hastings wrote:


This PEP proposes an additional syntax for declaring a class which splits
this work across two statements:
* The first statement is `forward class`, which declares the class and binds
   the class object.
* The second statement is `continue class`, which defines the contents
   of the class in the "class body".

To be clear: `forward class` creates the official, actual class object.
Code that wants to take a reference to the class object may take references
to the `forward class` declared class, and interact with it as normal.
However, a class created by `forward class` can't be *instantiated*
until after the matching `continue class` statement finishes.

Since the "forward class" is a real class,


It's a "forward-declared class object".  It's the real class object, but 
it hasn't been fully initialized yet, and won't be until the "continue 
class" statement.




it doesn't need any new
syntax to create it. Just use plain old regular class syntax.

 class X(object):
 """Doc string"""
 attribute = 42

And now we have our X class, ready to use in annotations.

To add to it, or "continue class" in your terms, we can already do this:

 X.value = "Hello World"


But if X has a metaclass that defines __new__ , "value" won't be defined 
yet, so metaclass.__new__ won't be able to react to and possibly modify 
it.  Similarly for metaclass.__init__ and BaseClass.__init_subclass__.


So, while your suggested technique doesn't "break" class creation per 
se, it prevents the user from benefiting from metaclasses and base 
classes using these advanced techniques.




Counter proposal:

`continue class expression:` evaluates expression to an existing class
object (or raises an exception) and introduces a block. The block is
executed inside that class' namespace, as the `class` keyword does,
except the class already exists.


If "continue class" is run on an already-created class, this breaks the 
functionality of __prepare__, which creates the namespace used during 
class body execution and is thrown away afterwards.  The "dict-like 
object" returned by __prepare__ will have been thrown away by the time 
"continue class" is executed.


Also, again, this means that the contents added to the class in the 
"continue class" block won't be visible to metaclass.__new__, 
metaclass.__init__, and BaseClass.__init_subclass__.


Also, we would want some way of preventing the user from running 
"continue class" multiple times on the same class--else we accidentally 
condone monkey-patching in Python, which we don't want to do.




Isn't this case solved by either forward references:

 class A:
 value: "B"

or by either of PEP 563 or PEP 649?


It is, but:

a) manual stringizing was rejected by the community as too tiresome and 
too error-prone (the syntax of the string isn't checked until you run 
your static type analysis tool).  Also, if you need the actual Python 
value at runtime, you need to eval() it, which causes a lot of headaches.


b) PEP 649 doesn't solve this only-slightly-more-advanced case:

   @dataclass
   class A:
  value: B

   @dataclass
   class B:
  value: A

as the dataclass decorator examines the contents of the class, including 
its annotations.


c) PEP 563 has the same "what if you need the actual Python value at 
runtime" problem as manual stringization, which I believe is why the SC 
has delayed its becoming default behavior.


Perhaps my example for b) would be a better example for the PEP.



That could become:

 class A:
 pass

 class B:
 value: A  # This is fine, A exists.

 A.value: B  # And here B exists, so this is fine too.

No new syntax is needed. This is already legal.


It's legal, but it doesn't set the annotation of "value" on A. Perhaps 
this is just a bug and could be fixed.  (TBH I'm not sure what the 
intended semantics of this statement are, or where that annotation ends 
up currently.  I couldn't find it in A.__annotations__ or the module's 
__annotations__.  Is it just forgotten?)


I assert this approach will be undesirable to Python programmers.  This 
makes for two very-different feeling approaches to defining the members 
of a class.  One of the goals of my PEP was to preserve the existing 
"feel" of Python as much as possible.


Also, as previously mentioned, your technique prevents "A.value", and 
all other attributes and methods set using this technique, from being 
visible to metaclass.__new__, metaclass.__init__, and 
BaseClass.__init_subclass__.




This proposed  `forward class` / `continue class` syntax should permit
solving *every* forward-reference and circular-reference problem faced
in Python,

I think that's overselling the concept.


Okay, perhaps I should have said "the forward-reference and 
circular-reference problems of class definitions" or something like 
that.  I'll adjust the 

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Chris Angelico]

On Sat, 23 Apr 2022 at 12:50, Larry Hastings  wrote:
>
>
> On 4/22/22 19:17, Chris Angelico wrote:
>
> I'm unsure about the forward class. How is it different from subclassing an 
> ABC?
>
> They're just different objects.  A subclass of an ABC is either itself 
> another abstract base class, which will never be instantiatable, or a 
> non-abstract class, which is immediately instantiatable.  A forward-declared 
> class object is not currently instantiatable, and is not fully defined, but 
> will become fully defined and instantiatable after the matching "continue 
> class" statement.
>

Ah okay, I think I had the idea that an ABC could demand certain
methods, but I think I'm mismatching semantics when it comes to
half-implementing something. Anyhow, it's a class, with some special
features (notably that you can't instantiate it).

>
> What happens if you try to continue a non-forward class?
>
> From the proto-PEP:
>
> Executing a `continue class` statement with a class defined by the `class` 
> statement raises a `ValueError` exception.
>
> And also:
>
> It's expected that knowledgeable users will be able to trick Python into 
> executing `continue class` on the same class multiple times by interfering 
> with "dunder" attributes.  The same tricks may also permit users to trick 
> Python into executing `continue class` on a class defined by the `class` 
> statement.  This is undefined and unsupported behavior, but Python will not 
> prevent it.
>

Huh. I skimmed back and forth looking, and did a search, and didn't
find that. My bad.

It seems odd that you define a blessed way of monkeypatching a class,
but then demand that it can only be done once unless you mess with
dunders. Why not just allow multiple continuations?

And if multiple continuations are permitted, there's really no
difference between a forward class and a regular class. It'd be a
"non-instantiable class", which is an aspect which can be removed by
reopening the class, but otherwise it's just like any other class.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AELFL53RTT2GXOZ6NA7BFEJUE2BH67UR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Steven D'Aprano]

On Fri, Apr 22, 2022 at 06:13:57PM -0700, Larry Hastings wrote:

> This PEP proposes an additional syntax for declaring a class which splits
> this work across two statements:
> * The first statement is `forward class`, which declares the class and binds
>   the class object.
> * The second statement is `continue class`, which defines the contents
>   of the class in the "class body".
> 
> To be clear: `forward class` creates the official, actual class object.
> Code that wants to take a reference to the class object may take references
> to the `forward class` declared class, and interact with it as normal.
> However, a class created by `forward class` can't be *instantiated*
> until after the matching `continue class` statement finishes.

Since the "forward class" is a real class, it doesn't need any new 
syntax to create it. Just use plain old regular class syntax.

class X(object):
"""Doc string"""
attribute = 42


And now we have our X class, ready to use in annotations.

To add to it, or "continue class" in your terms, we can already do this:

X.value = "Hello World"

Methods are a bit trickier. The status quo pollutes the global 
namespace, and repeats the method name three times:

def method(self):
pass

X.method = method

but we can do better with just a small amount of new syntax:

def X.method(self):
pass

This has been proposed before, and rejected for lack of any real need. 
But your "forward class/continue class" suggests that the need has now 
appeared.

So where you would write this:


    forward class Node()

    continue class Node:
"""Node in a linked list."""
    def __init__(self, value, next:Optional[Node]):
    self.value = value
self.next = next


we could instead do this:

    class Node():
"""Node in a linked list."""

    def Node.__init__(self, value, next:Optional[Node]):
    self.value = value
self.next = next


The only downside -- although perhaps that's an upside -- is that we 
save an indent level, which may make it marginally harder to recognise 
which function defs are being added into a class, and which are not.

If that really is an issue, then we could keep the "continue class ..." 
syntax solely to give us that block structure.

Counter proposal:

`continue class expression:` evaluates expression to an existing class 
object (or raises an exception) and introduces a block. The block is 
executed inside that class' namespace, as the `class` keyword does, 
except the class already exists.

But we don't need the "forward class". Just create a class!

Any methods, docstrings, attributes etc which aren't part of a forward 
reference cycle can go in the initial class creation, and only those 
which are part of a cycle need to be added afterwards.


Here is your circular class example:


> ```Python
>     class A:
>     value: B
> 
>     class B:
>     value: A
> ```

Isn't this case solved by either forward references:

class A:
value: "B"

or by either of PEP 563 or PEP 649? I don't think this is a compelling 
example. But let's continue anyway.

That could become:

class A:
pass

class B:
value: A  # This is fine, A exists.

A.value: B  # And here B exists, so this is fine too.


No new syntax is needed. This is already legal. We only need minimal new 
syntax to allow methods to be defined outside of their class body:

def A.method(self):
...


with no need to overload `continue` for a second meaning. Or we could 
use the `continue A` syntax, but either way, the `forward A` syntax 
seems to be unneeded.


> But nothing so far has been both satisfying and complete; either it
> is wordy and clumsy to use (manually stringizing annotations), or it
> added restrictions and caused massive code breakage for runtime use of
> annotations (PEP 563), or simply didn't solve every problem (PEP 649).

But doesn't PEP 649 solve problems which this proposal does not? 
Delaying the evaluation of annotations is more general than merely the 
"forward reference" problem.


> This proposed  `forward class` / `continue class` syntax should permit
> solving *every* forward-reference and circular-reference problem faced
> in Python,

I think that's overselling the concept. How about forward references for 
function defaults?

def func(arg=func(0)):
if arg == 0:
return "something"
...


Obviously there are other ways to solve that, but the point is, your 
forward/continue class proposal is not one of them!



> using an elegant and Pythonic new syntax.

That's a matter of opinion. Personally, I think that *declarations* as 
in `forward MyClass` are not especially Pythonic, although at least 
your declaration also creates a real class object. But having two ways 
to create a class (three if we count type()) is not especially Pythonic 
either.


> As a side benefit, `forward class` and `continue class` 

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Larry Hastings]

On 4/22/22 19:36, Terry Reedy wrote:

On 4/22/2022 9:13 PM, Larry Hastings wrote:

 forward class X()


New keywords are a nuisance.  And the proposed implementation seems 
too complex.


My proposed implementation seemed necessary to handle the complexity of 
the problem.  I would welcome a simpler solution that also worked for 
all the same use cases.



How about a 'regular' class statement with a special marker of some 
sort.  Example: 'body=None'.


It's plausible.  I take it "body=None" would mean the declaration would 
not be permitted to have a colon and a class body.  So now we have two 
forms of the "class" statement, and which syntax you're using is 
controlled by a named parameter argument.


I think this "body=None" argument changing the syntax of the "class" 
statement is clumsy.  It lacks the visibility and clarity of "forward 
class"; the keyword here makes it pretty obvious that this is not a 
conventional class declaration.  So I still prefer "forward class".


In my PEP I proposed an alternate syntax for "forward class": "def 
class", which has the feature that it doesn't require adding a new 
keyword.  But again, I don't think it's as clear as "forward class", and 
I think clarity is vital here.



Either __new__ or __init__ could raise XError("Cannot instantiate 
until this is continued.", so no special instantiation code would 
needed and X could be a real class, with a special limitation.


Yes, my proposal already suggests that __new__ raise an exception.  
That's not the hard part of the problem.


The problem with X being a "real class" is that creating a "real class" 
means running all the class creation code, and a lot of the class 
creation code is designed with the assumption that the namespace has 
already been filled by executing the class body.


For example, Enum in enum.py relies on EnumMeta, which defines __new__, 
which examines the already-initialized namespace of the class you want 
to create.  If you propose a "body=None" class be a "real class object", 
then how do you declare a class that inherits from Enum using 
"body=None"?  Creating the class object will call EnumMeta.__new__, 
which needs to examine the namespace, which hasn't been initialized yet.


Changing the class object creation code so we can construct a class 
object in two steps, with the execution of the class body being part of 
the second step, was the major sticking point--and the source of most of 
the complexity of my proposal.




 continue class X:
 # class body goes here
 def __init__(self, key):
 self.key = key


'continue' is already a keyword. 


I'm aware.  I'm not sure why you mentioned it.


Given that X is a real class, could implementation be 
X.__dict__.update(new-body-dict)


That's what the proof-of-concept does.  But the proof-of-concept fails 
with a lot of common use cases:


 * metaclasses that override __new__ or __init__
 * base classes that implement __init_subclass__

because these methods assume the namespace of the class has already been 
filled in, but it doesn't get filled in until the @continue_() class 
decorator.  Handling these cases is why my proposal is sadly as complex 
as it is, and why in practice the proof-of-concept doesn't work a lot of 
the time.



//arry/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KVDT632CHPORAA6KWJG6ZZVY4FFSZI27/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Larry Hastings]

On 4/22/22 19:17, Chris Angelico wrote:

I'm unsure about the forward class. How is it different from subclassing an ABC?


They're just different objects.  A subclass of an ABC is either itself 
another abstract base class, which will never be instantiatable, or a 
non-abstract class, which is immediately instantiatable.  A 
forward-declared class object is not currently instantiatable, and is 
not fully defined, but will become fully defined and instantiatable 
after the matching "continue class" statement.




What happens if you try to continue a non-forward class?


From the proto-PEP:

   Executing a `continue class` statement with a class defined by the
   `class` statement raises a `ValueError` exception.

And also:

   It's expected that knowledgeable users will be able to trick Python
   into executing `continue class` on the same class multiple times by
   interfering with "dunder" attributes.  The same tricks may also
   permit users to trick Python into executing `continue class` on a
   class defined by the `class` statement. This is undefined and
   unsupported behavior, but Python will not prevent it.


//arry/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QU2HE5AAMLHU33Z5Y4EPDF46CXO7F3NF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Terry Reedy]

On 4/22/2022 9:13 PM, Larry Hastings wrote:


     forward class X()


New keywords are a nuisance.  And the proposed implementation seems too 
complex.  How about a 'regular' class statement with a special marker of 
some sort.

Example: 'body=None'.

Either __new__ or __init__ could raise XError("Cannot instantiate until 
this is continued.", so no special instantiation code would needed and X 
could be a real class, with a special limitation.



     continue class X:
     # class body goes here
     def __init__(self, key):
     self.key = key


'continue' is already a keyword.  Given that X is a real class, could 
implementation be

X.__dict__.update(new-body-dict)


--
Terry Jan Reedy
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UAZ5P7N5S4OWFSVZSKK6DUEPZTQPV3GP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Chris Angelico]

On Sat, 23 Apr 2022 at 11:16, Larry Hastings  wrote:
> This PEP proposes an additional syntax for declaring a class which splits
> this work across two statements:
> * The first statement is `forward class`, which declares the class and binds
>the class object.
> * The second statement is `continue class`, which defines the contents
>of the class in the "class body".
>
> [chomp all the details]

Hmm. The "continue class" part looks great IMO (and I have in fact
implemented the decorator version, which is clunky - your version,
where "continue class EXPR:" will reference any class - is way more
flexible than anything a decorator can do easily), but I'm unsure
about the forward class. How is it different from subclassing an ABC?
You're claiming a keyword (and one which is definitely going to
conflict - if this goes forward in this form, I would certainly be
bikeshedding the exact choice of keyword) for something that's
extremely similar to simply defining a class, and then marking it as
abstract.

What happens if you try to continue a non-forward class? For example:

class Demo: pass
continue class Demo: pass

If that works fine, then I'd be inclined to downgrade "forward class"
to a keyword argument (like metaclass is), eg "class
Demo(abstract=True)" or "class Demo(forward=True)".

But, I definitely like the "continue class" part of the idea, and
would love to see that happen. Of course, it's going to be horribly
abused for monkeypatching, but we could do that already anyway :)

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LDYKP7NSVUECEZPW7SP5FMZLASMGJ6TT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Proto-PEP part 3: Closing thoughts on "forward class", etc.

[Larry Hastings]

Just a quick note from me on the proto-PEP and the two proposed 
implementations.  When I started exploring this approach, I didn't 
suspect it'd require such sweeping changes to be feasible. Specifically, 
I didn't think I was going to propose changing the fundamental mechanism 
used to create class objects.  That's an enormous change, and it makes 
me uncomfortable; I suspect I won't be alone in having that reaction.


The alternate implementation with proxy objects was borne of my 
reaction, but it's worrisome too.  It's a hack--though whether it's a 
"big" hack or a "small" hack is debatable.  Anyway, I'm specifically 
worried about the underlying class object escaping the proxy and 
becoming visible inside Python somehow.  If that happened, we'd have two 
objects representing the same "type" at runtime, a situation that could 
quickly become confusing.


Also, as I hopefully made clear in the "alternate implementation" 
approach using a class proxy object, I'm not 100% certain that the proxy 
will work in all cases.  I ran out of time to investigate it more--I 
wanted to post this idea with some lead time before the 2022 Language 
Summit, so that folks had time to read and digest it and discuss it 
before the Summit.  I have some implementation ideas--the "class proxy" 
class may need its own exotic metaclass.


Ultimately I'm posting this proto-PEP to foster discussion.  I'm 
confident that "forward class" / "continue class" could solve all our 
forward-reference and circular-reference problems; the questions we need 
to collectively answer are:


 * how should the implementation work, and
 * is the cost of the implementation worth it?


Best wishes,


//arry/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HD7YPONSPL5ZFZISKCOUWVUXMIJTQG2M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object

[Larry Hastings]

Here's one alternate idea for how to implement the "forward class" syntax.

The entire point of the "forward class" statement is that it creates
the real actual class object.  But what if it wasn't actually the
"real" class object?  What if it was only a proxy for the real object?

In this scenario, the syntax of "forward object" remains the same.
You define the class's bases and metaclass.  But all "forward class"
does is create a simple, lightweight class proxy object.  This object
has a few built-in dunder values, __name__ etc.  It also allows you
to set attributes, so let's assume (for now) it calls
metaclass.__prepare__ and uses the returned "dict-like object" as
the class proxy object __dict__.

"continue class" internally performs all the rest of the
class-creation machinery.  (Everything except __prepare__, as we
already called that.)  The first step is metaclass.__new__, which
returns the real class object.  "continue class" takes that
object and calls a method on the class proxy object that says
"here's your real class object".  From that moment on, the proxy
becomes a pass-through for the "real" class object, and nobody
ever sees a reference to the "real" class object ever again.
Every interaction with the class proxy object is passed through
to the underlying class object.  __getattribute__ calls on the
proxy look up the attribute in the underlying class object.  If
the object returned is a bound method object, it rebinds that
callable with the class proxy instead, so that the "self" passed
in to methods is the proxy object.  Both base_cls.__init_subclass__
and cls.__init__ see the proxy object during class creation.  As far
as Python user code is concerned, the class proxy *is* the class,
in every way, important or not.

The upside: this moves all class object creation code into "continue
class" call.  We don't have to replace __new__ with two new calls.

The downside: a dinky overhead to every interaction with a "forward
class" class object and with instances of a "forward class" class
object.


A huge concern: how does this interact with metaclasses implemented
in C?  If you make a method call on a proxy class object, and that
calls a C function from the metaclass, we'd presumably have to pass
in the "real class object", not the proxy class object.  Which means
references to the real class object could leak out somewhere, and
now we have a real-class-object vs proxy-class-object identity crisis.
Is this a real concern?


A possible concern: what if metaclass.__new__ keeps a reference to
the object it created?  Now we have two objects with an identity
crisis.  I don't know if people ever do that.  Fingers crossed that
they don't.  Or maybe we add a new dunder method:

    @special_cased_staticmethod
    metaclass.__bind_proxy__(metaclass, proxy, cls)

This tells the metaclass "bind cls to this proxy object", so
metaclasses that care can update their database or whatever.
The default implementation uses the appropriate mechanism,
whatever it is.

One additional probably-bad idea: in the case where it's just a
normal "class" statement, and we're not binding it to a proxy,
should we call this?

    metaclass.__bind_proxy__(metaclass, None, cls)

The idea there being "if you register the class objects you create,
do the registration in __bind_proxy__, it's always called, and you'll
always know the canonical object in there".  I'm guessing probably not,
in which case we tell metaclasses that track the class objects we
create "go ahead and track the object you return from __new__, but
be prepared to update your tracking info in case we call __bind_proxy__
on you".


A small but awfully complicated wrinkle here: what do we do if the
metaclass implements __del__?  Obviously, we have to call __del__
with the "real" class object, so it can be destroyed properly.
But __del__ might resurrect that object, which means someone took a 
reference to it.




One final note.  Given that, in this scenario, all real class creation
happens in "continue class", we could move the bases and metaclass
declaration down to the "continue class" statement.  The resulting
syntax would look like:

    forward class X

    ...

    continue class X(base1, base2, metaclass=AmazingMeta, rocket="booster")

Is that better? worse? doesn't matter?  I don't have an intuition about
it right now--I can see advantages to both sides, and no obvious
deciding factor.  Certainly this syntax prevents us from calling
__prepare__ so early, so we'd have to use a real dict in the "forward
class" proxy object until we reached continue, then copy the values from
that dict into the "dict-like object", etc.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OJRA7F7EMRJCIXQPRRKZZ7YMFD2ZKQV2/

[Python-Dev] Proto-PEP part 1: Forward declaration of classes

[Larry Hastings]

This document is a loose proto-PEP for a new "forward class" / "continue 
class" syntax.  Keep in mind, the formatting is a mess. If I wind up 
submitting it as a real PEP I'll be sure to clean it up first.



/arry

--


PEP : Forward declaration of classes

Overview


Python currently has one statement to define a class, the `class` statement:

```Python
    class X():
    # class body goes here
    def __init__(self, key):
    self.key = key
```

This single statement declares the class, including its bases and metaclass,
and also defines the contents of the class in the "class body".

This PEP proposes an additional syntax for declaring a class which splits
this work across two statements:
* The first statement is `forward class`, which declares the class and binds
  the class object.
* The second statement is `continue class`, which defines the contents
  of the class in the "class body".

To be clear: `forward class` creates the official, actual class object.
Code that wants to take a reference to the class object may take references
to the `forward class` declared class, and interact with it as normal.
However, a class created by `forward class` can't be *instantiated*
until after the matching `continue class` statement finishes.

Defining class `X` from the previous example using this new syntax would 
read

as follows:

```
    forward class X()

    continue class X:
    # class body goes here
    def __init__(self, key):
    self.key = key
```

This PEP does not propose altering or removing the traditional `class` 
statement;

it would continue to work as before.


Rationale
-

Python programmers have had a minor problem with classes for years: there's
no way to have early-bound circular dependencies between objects. If A
depends on B, and B depends on A, there's no linear order that allows
you to cleanly declare both.

Most of the time, the dependencies were in late-binding code, e.g. A refers
to B inside a method.  So this was rarely an actual problem at runtime.  
When

this problem did arise, in code run at definition-time, it was usually only
a minor headache and could be easily worked around.

But the explosion of static type analysis in Python, particularly with
the `typing` module and the `mypy` tool, has made circular definition-time
dependencies between classes commonplace--and much harder to solve.  Here's
one simple example:

```Python
    class A:
    value: B

    class B:
    value: A
```

An attribute of `B` is defined using a type annotation of `A`, and an
attribute of `A` is defined using a type annotation of `B`. There's
no order to these two definitions that works; either `A` isn't defined
yet, or `B` isn't defined yet.

Various workarounds and solutions have been proposed to solve this problem,
including two PEPs: PEP 563 (automatic stringized annotations) and PEP 649
(delayed evaluation of annotations using functions).
But nothing so far has been both satisfying and complete; either it
is wordy and clumsy to use (manually stringizing annotations), or it
added restrictions and caused massive code breakage for runtime use of
annotations (PEP 563), or simply didn't solve every problem (PEP 649).
This proposed  `forward class` / `continue class` syntax should permit
solving *every* forward-reference and circular-reference problem faced
in Python, using an elegant and Pythonic new syntax.

As a side benefit, `forward class` and `continue class` syntax enables
rudimentary separation of "interface" from "implementation", at least for
classes.  A user seeking to "hide" the implementation details of their
code could put their class definitions in one module, and the
implementations of those classes in a different module.

This new syntax is not intended to replace the traditional `class`
declaration syntax in Python.  If this PEP were accepted, the `class`
statement would still be the preferred mechanism for creating classes
in Python; `forward class` should only be used when it confers some
specific benefit.


Syntax
--

The `forward class` statement is the same as the `class` statement,
except it doesn't end with a colon and is not followed by an indented block.
Without any base classes or metaclass, the `forward class` statement is
as follows:

```
    forward class X
```

This would declare class `X`.

If `X` needs base classes or metaclass, the corresponding `forward 
class` statement

would be as follows, rendered in a sort of "function prototype" manner:

```
    forward class X(*bases, metaclass=object, **kwargs)
```

The `continue class` statement is similar to a `class` statement
without any bases or metaclass.  It ends with a colon,
and is followed by the "class body":

    continue class X:
    # class body goes here
    pass

One important difference: the `X` in `continue class X:` is not a *name*,
it's an *expression*.  This code is valid:

```
    forward 

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2022-04-15 - 2022-04-22)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open7146 ( +0)
  closed 51841 ( +0)
  total  58987 ( +0)

Open issues with patches: 2890 


Most recent 15 issues with no replies (15)
==

#47258: Python 3.10 hang at exit in drop_gil() (due to resource warnin
https://bugs.python.org/issue47258

#47256: re: limit the maximum capturing group to 1,073,741,823, reduce
https://bugs.python.org/issue47256

#47253: LOAD_GLOBAL instruction with wrong source position
https://bugs.python.org/issue47253

#47252: socket.makefile documentation is missing data regarding the 'b
https://bugs.python.org/issue47252

#47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE
https://bugs.python.org/issue47251

#47244: email.utils.formataddr does not respect double spaces
https://bugs.python.org/issue47244

#47242: Annoying white bar in IDLE (line 457 in sidebar.py)
https://bugs.python.org/issue47242

#47241: [C API] Move the PyCodeObject structure to the internal C API 
https://bugs.python.org/issue47241

#47238: Python threading.Event().wait() depends on the system time
https://bugs.python.org/issue47238

#47236: Document types.CodeType.replace() changes about co_exceptionta
https://bugs.python.org/issue47236

#47228: Document that na??ve datetime objects represent local time
https://bugs.python.org/issue47228

#47222: subprocess.Popen() should allow capturing output and sending i
https://bugs.python.org/issue47222

#47219: asyncio with two interpreter instances
https://bugs.python.org/issue47219

#47218: adding name to lzmafile
https://bugs.python.org/issue47218

#47217: adding name to BZ2File
https://bugs.python.org/issue47217



Most recent 15 issues waiting for review (15)
=

#47256: re: limit the maximum capturing group to 1,073,741,823, reduce
https://bugs.python.org/issue47256

#47255: Many broken :meth: roles in the docs
https://bugs.python.org/issue47255

#47254: enhanced dir?
https://bugs.python.org/issue47254

#47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE
https://bugs.python.org/issue47251

#47243: Duplicate entry in 'Objects/unicodetype_db.h'
https://bugs.python.org/issue47243

#47233: show_caches option affects code positions reported by dis.get_
https://bugs.python.org/issue47233

#47222: subprocess.Popen() should allow capturing output and sending i
https://bugs.python.org/issue47222

#47218: adding name to lzmafile
https://bugs.python.org/issue47218

#47217: adding name to BZ2File
https://bugs.python.org/issue47217

#47216: adding mtime option to gzip open()
https://bugs.python.org/issue47216

#47215: Add "unstable" frame stack api
https://bugs.python.org/issue47215

#47208: Support libffi implementations that cannot support invocations
https://bugs.python.org/issue47208

#47205: posix.sched_{get|set}affinity(-1) no longer returns ProcessLoo
https://bugs.python.org/issue47205

#47200: Add ZipInfo.mode property
https://bugs.python.org/issue47200

#47199: multiprocessing: micro-optimize Connection.send_bytes() method
https://bugs.python.org/issue47199
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T4FH4WLLKDRAS2YSQI75RHXVVZE3R4JZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

On 22. 04. 22 14:47, Fabio Zadrozny wrote:



Em sex., 22 de abr. de 2022 às 09:02, Petr Viktorin > escreveu:


Hello Fabio,
Let's talk a bit about which API should, exactly, be guaranteed to not
change across minor releases.
So far it looks like:
- PyEval_RequestCodeExtraIndex
- PyCode_GetExtra
- PyCode_SetExtra
- PyFrameEvalFunction
- PyInterpreterState_GetEvalFrameFunc
- PyInterpreterState_SetEvalFrameFunc

Do any more come to mind?

The issue with this set is that in 3.11, _PyFrameEvalFunction changes
its signature to take _PyInterpreterFrame rather than PyFrameObject.
Exposing _PyInterpreterFrame would be quite problematic. For example,
since it's not a PyObject, it has its own lifetime management that's
controlled by the interpreter itself,. And it includes several
pointers whose lifetime and semantics also isn't guaranteed (they
might be borrowed, cached or filled on demand). I don't think we can
make any guarantees on these, so the info needs to be accessed using
getter functions.

There is the function _PyFrame_GetFrameObject, which returns a
PyFrameObject.
I think it would be best to only expose _PyInterpreterFrame as an
opaque structure, and expose PyFrame_GetFrameObject so debuggers can
get a PyFrameObject from it.
Does that sound reasonable?



Humm, now I'm a bit worried... the approach the debugger is using gets 
the PyFrameObject that's about to be executed and changes the 
PyFrameObject.f_code just before the execution so that the new code is 
executed instead.


 From what you're saying the PyFrameObject isn't really used anymore 
(apparently it's substituted by a _PyInterpreterFrame?)... in this case, 
will this approach still let the debugger patch the code object in the 
frame before it's actually executed?


PyFrameObject is a fairly thin wrapper around _PyInterpreterFrame -- it 
adds PyObject metadata (type & refcount), and not much else. It's 
allocated at most once for each _PyInterpreterFrame -- once it's created 
it stays attached to the frame.
So, for the most heavily optimized code paths a PyFrameObject is not 
allocated, but it's trivial to get it whenever it's needed.


-- i.e.: the debugger changes the state.interp.eval_frame to its own 
custom evaluation function, but _PyEval_EvalFrameDefault is still what 
ends up being called afterwards (it works more as a hook to change the 
PyFrameObject.f_code prior to execution than as an alternate interpreter).


Ah, you also need PyEval_EvalFrameDefault exposed. The public version 
would take PyFrameObject and pass its _PyInterpreterFrame to the 
internal _PyEval_EvalFrameDefault.




On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny mailto:fabi...@gmail.com>> wrote:
 >
 >
 > Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny
mailto:fabi...@gmail.com>> escreveu:
 >>>
 >>> PEP 523 API added more private functions for code objects:
 >>>
 >>> * _PyEval_RequestCodeExtraIndex()
 >>> * _PyCode_GetExtra()
 >>> * _PyCode_SetExtra()
 >>>
 >>> The _PyEval_RequestCodeExtraIndex() function seems to be used
by the
 >>> pydevd debugger. The two others seem to be unused in the wild.
I'm not
 >>> sure if these ones should be moved to the internal C API. They
can be
 >>> left unchanged, since they don't use a type only defined by the
 >>> internal C API.
 >>
 >> Just to note, the pydevd/debugpy debuggers actually uses all of
those APIs.
 >>
 >> i.e.:
 >>
 >>

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187


 >>

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232


 >>

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311


 >>
 >> The debugger already has workarounds because of changes to
evaluation api over time (see:

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491

)
and I know 3.11 won't be different.
 >>
 >> I'm ok with changes as I understand that this is a special API
-- as long as there's still a way to use it and get the information
needed (the debugger already goes through many hops because it needs
to use many internals of CPython -- in every new 

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Fabio Zadrozny]

Em sex., 22 de abr. de 2022 às 09:02, Petr Viktorin 
escreveu:

> Hello Fabio,
> Let's talk a bit about which API should, exactly, be guaranteed to not
> change across minor releases.
> So far it looks like:
> - PyEval_RequestCodeExtraIndex
> - PyCode_GetExtra
> - PyCode_SetExtra
> - PyFrameEvalFunction
> - PyInterpreterState_GetEvalFrameFunc
> - PyInterpreterState_SetEvalFrameFunc
>
> Do any more come to mind?
>
> The issue with this set is that in 3.11, _PyFrameEvalFunction changes
> its signature to take _PyInterpreterFrame rather than PyFrameObject.
> Exposing _PyInterpreterFrame would be quite problematic. For example,
> since it's not a PyObject, it has its own lifetime management that's
> controlled by the interpreter itself,. And it includes several
> pointers whose lifetime and semantics also isn't guaranteed (they
> might be borrowed, cached or filled on demand). I don't think we can
> make any guarantees on these, so the info needs to be accessed using
> getter functions.
>
> There is the function _PyFrame_GetFrameObject, which returns a
> PyFrameObject.
> I think it would be best to only expose _PyInterpreterFrame as an
> opaque structure, and expose PyFrame_GetFrameObject so debuggers can
> get a PyFrameObject from it.
> Does that sound reasonable?
>


Humm, now I'm a bit worried... the approach the debugger is using gets the
PyFrameObject that's about to be executed and changes the
PyFrameObject.f_code just before the execution so that the new code is
executed instead.

>From what you're saying the PyFrameObject isn't really used anymore
(apparently it's substituted by a _PyInterpreterFrame?)... in this case,
will this approach still let the debugger patch the code object in the
frame before it's actually executed?

-- i.e.: the debugger changes the state.interp.eval_frame to its own custom
evaluation function, but _PyEval_EvalFrameDefault is still what ends up
being called afterwards (it works more as a hook to change the
PyFrameObject.f_code prior to execution than as an alternate interpreter).



On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny  wrote:
> >
> >
> > Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny 
> escreveu:
> >>>
> >>> PEP 523 API added more private functions for code objects:
> >>>
> >>> * _PyEval_RequestCodeExtraIndex()
> >>> * _PyCode_GetExtra()
> >>> * _PyCode_SetExtra()
> >>>
> >>> The _PyEval_RequestCodeExtraIndex() function seems to be used by the
> >>> pydevd debugger. The two others seem to be unused in the wild. I'm not
> >>> sure if these ones should be moved to the internal C API. They can be
> >>> left unchanged, since they don't use a type only defined by the
> >>> internal C API.
> >>
> >> Just to note, the pydevd/debugpy debuggers actually uses all of those
> APIs.
> >>
> >> i.e.:
> >>
> >>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187
> >>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232
> >>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311
> >>
> >> The debugger already has workarounds because of changes to evaluation
> api over time (see:
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491)
> and I know 3.11 won't be different.
> >>
> >> I'm ok with changes as I understand that this is a special API -- as
> long as there's still a way to use it and get the information needed (the
> debugger already goes through many hops because it needs to use many
> internals of CPython -- in every new release it's a **really** big task to
> update to the latest version as almost everything that the debugger relies
> to make debugging fast changes across versions and I never really know if
> it'll be possible to support it until I really try to do the port -- I
> appreciate having less things in a public API so it's easier to have
> extensions work in other interpreters/not recompiling on newer versions,
> but please keep it possible to use private APIs which provides the same
> access that CPython has to access things internally for special cases such
> as the debugger).
> >>
> >> Maybe later on that PEP from mark which allows a better debugger API
> could alleviate that (but until then, if possible I appreciate it if
> there's some effort not to break things unless really needed -- ideally
> with instructions on how to port).
> >>
> >> Anyways, to wrap up, the debugger already needs to be built with
> `Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API
> (as long as it's still accessible in that case) is probably not a big issue
> for the debugger and having setters/getters to set it instead of relying on
> `state.interp.eval_frame` seems good to me.
> >>
> >> Cheers,
> >>
> >> Fabio
> >>
> >
> >
> > I think the main issue here is the compatibility across the same version
> though... is it 

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

Hello Fabio,
Let's talk a bit about which API should, exactly, be guaranteed to not
change across minor releases.
So far it looks like:
- PyEval_RequestCodeExtraIndex
- PyCode_GetExtra
- PyCode_SetExtra
- PyFrameEvalFunction
- PyInterpreterState_GetEvalFrameFunc
- PyInterpreterState_SetEvalFrameFunc

Do any more come to mind?

The issue with this set is that in 3.11, _PyFrameEvalFunction changes
its signature to take _PyInterpreterFrame rather than PyFrameObject.
Exposing _PyInterpreterFrame would be quite problematic. For example,
since it's not a PyObject, it has its own lifetime management that's
controlled by the interpreter itself,. And it includes several
pointers whose lifetime and semantics also isn't guaranteed (they
might be borrowed, cached or filled on demand). I don't think we can
make any guarantees on these, so the info needs to be accessed using
getter functions.

There is the function _PyFrame_GetFrameObject, which returns a PyFrameObject.
I think it would be best to only expose _PyInterpreterFrame as an
opaque structure, and expose PyFrame_GetFrameObject so debuggers can
get a PyFrameObject from it.
Does that sound reasonable?


On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny  wrote:
>
>
> Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny  
> escreveu:
>>>
>>> PEP 523 API added more private functions for code objects:
>>>
>>> * _PyEval_RequestCodeExtraIndex()
>>> * _PyCode_GetExtra()
>>> * _PyCode_SetExtra()
>>>
>>> The _PyEval_RequestCodeExtraIndex() function seems to be used by the
>>> pydevd debugger. The two others seem to be unused in the wild. I'm not
>>> sure if these ones should be moved to the internal C API. They can be
>>> left unchanged, since they don't use a type only defined by the
>>> internal C API.
>>
>> Just to note, the pydevd/debugpy debuggers actually uses all of those APIs.
>>
>> i.e.:
>>
>> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187
>> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232
>> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311
>>
>> The debugger already has workarounds because of changes to evaluation api 
>> over time (see: 
>> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491)
>>  and I know 3.11 won't be different.
>>
>> I'm ok with changes as I understand that this is a special API -- as long as 
>> there's still a way to use it and get the information needed (the debugger 
>> already goes through many hops because it needs to use many internals of 
>> CPython -- in every new release it's a **really** big task to update to the 
>> latest version as almost everything that the debugger relies to make 
>> debugging fast changes across versions and I never really know if it'll be 
>> possible to support it until I really try to do the port -- I appreciate 
>> having less things in a public API so it's easier to have extensions work in 
>> other interpreters/not recompiling on newer versions, but please keep it 
>> possible to use private APIs which provides the same access that CPython has 
>> to access things internally for special cases such as the debugger).
>>
>> Maybe later on that PEP from mark which allows a better debugger API could 
>> alleviate that (but until then, if possible I appreciate it if there's some 
>> effort not to break things unless really needed -- ideally with instructions 
>> on how to port).
>>
>> Anyways, to wrap up, the debugger already needs to be built with 
>> `Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API (as 
>> long as it's still accessible in that case) is probably not a big issue for 
>> the debugger and having setters/getters to set it instead of relying on 
>> `state.interp.eval_frame` seems good to me.
>>
>> Cheers,
>>
>> Fabio
>>
>
>
> I think the main issue here is the compatibility across the same version 
> though... is it possible to have some kind of guarantee on private APIs that 
> something won't change across micro-releases?
>
> I.e.: having the frame evaluation function change across major releases and 
> having them be reworked seems reasonable, but then having the frame 
> evaluation be changed across micro-releases wouldn't be.
>
> So, I'm ok in pushing things to the internal API, but then I still would like 
> guarantees about the compatibility of that API in the same major release 
> (otherwise those setters/getters/frame evaluation should probably remain on 
> the public API if the related structure was moved to the internal API).
>
> Cheers,
>
> Fabio
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
>