[Python-Dev] Re: __init_subclass__ and metaclasses

[Ethan Furman]

On 1/5/21 10:25 AM, Guido van Rossum wrote:
> On Tue, Jan 5, 2021 at 8:50 AM Ethan Furman wrote:

>> [...]
>> Perhaps `ABCMeta` can be easily fixed -- it is calling the function that
>> records all abstract methods, etc., after the `type.__new__` call; can
>> it be called before?
>
> Why do you keep insisting that this is "broken"?

Because `__init_subclass__` is getting a new class object that is not ready to 
be gotten.

> Yes, you have identified a way that an abstract class can be instantiated
> *by its own constructor* despite being abstract.

It doesn't feel that way to me.  Referring to the code below [1], the constructor for `MyPlugin` is `abc.ABCMeta`, but 
`PluginBase.__init_subclass__` is getting `MyPlugin` before `abc.ABCMeta` has finished doing its job.


> I think that's not something to get worked up about. As the doctor in the
> old joke said, "then don't do that." Or perhaps this is one of those
> situations where we're all still considered to be consenting adults --
> surely abstract classes and methods are meant as a way to help users do
> the right thing, not as an absolute guarantee.

According to the documentation:
> This module provides abstract base classes that can be used to test whether
> a class provides a particular interface; for example, whether it is hashable
> or whether it is a mapping.

Do we need to add a warning to that?

> except inside `__init_subclass__` as the class has not been fully initialized.

For that matter, if we are not going to address this issue with code, we should definitely update the 
`__init_subclass__` and `__set_name__` documentation:


> Note: custom metaclasses must have all changes needed already in the class
> namespace before calling `type.__new__`, as `type.__new__` will invoke
> `__init_subclass__` and `__set_name__` before returning.  Any modifications
> made to the class after that point will not be available to those two methods.
>
> Note: because of the above, any `__init_subclass__` used with `Enum` classes 
will
> not see any members as members are added after the call to `type.__new__`.
>
> Note: because of the above, any `__init_subclass__` used with `ABC` classes 
will
> have to determine for itself if the class is abstract or concrete as that is
> determined after the call to `type.__new__`.

As I said before, perhaps `_abc_init` can be called before `type.__new__`, 
which would make that last note unnecessary.

That would not solve the problem for `Enum`, though, nor for any other metaclass that cannot complete its customization 
before calling `type.__new__`.


--
~Ethan~


[1]
class PluginBase(abc.ABC):

# base class for plugins
# must override `do_something`

def __init_subclass__(cls, **kwargs):

instance = cls()
print(f"Created instance of {cls} easily: {instance}")


@abc.abstractmethod
def do_something(self):
pass


class MyPlugin(PluginBase):
pass



[2] https://docs.python.org/3/library/collections.abc.html
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KE3ZJEDFMZT4TXTTHXMVGEZSLX2HDUSZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: __init_subclass__ and metaclasses

[Guido van Rossum]

On Tue, Jan 5, 2021 at 8:50 AM Ethan Furman  wrote:

> [...]
> Perhaps `ABCMeta` can be easily fixed -- it is calling the function that
> records all abstract methods, etc., after the
> `type.__new__` call; can it be called before?
>

Why do you keep insisting that this is "broken"? Yes, you have identified a
way that an abstract class can be instantiated *by its own constructor*
despite being abstract. I think that's not something to get worked up
about. As the doctor in the old joke said, "then don't do that." Or perhaps
this is one of those situations where we're all still considered to be
consenting adults -- surely abstract classes and methods are meant as a way
to help users do the right thing, not as an absolute guarantee.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CSL6XXC53JFOB3RXJ5L3KVI4SDRAX5SR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: __init_subclass__ and metaclasses

[Ethan Furman]

On 1/5/21 4:22 AM, Nick Coghlan wrote:
> On Wed, 30 Dec 2020 at 04:38, Ethan Furman wrote:

>> ... there will be a few custom metaclasses that need to move some code
>> from their `__new__` to `__init__` instead, and a few that need to add
>> an `__init__` to consume any keyword arguments that don't need to get
>> passed to `__init_subclass__`.  That seems like a small price to pay to
>> be able to write custom metaclasses that are able to fully
>> participate in the `__set_name__` and `__init_subclass__` protocols.
>
> If a metaclass wants to modify the attributes on the type in a way
> that is visible to __init_subclass__ and __set_name__, it needs to
> modify the namespace dict passed to type().__new__, not modify the
> type after it has already been created.

And if that's not possible?  (More on that later.)

> If you need to access the class being defined from elsewhere, then the
> `__class__` cell used by zero-arg super is already populated before
> any of those hooks get called.

If the custom metaclass hasn't finished its work, I don't see how that is 
helpful.

> As a CPython implementation detail, that cell is actually passed
> through to type().__new__ as `__classcell__` in the namespace. If that
> was elevated to a language feature rather than a CPython
> implementation detail, then metaclasses could write:
>
>  cls_cell = namespace["__classcell__"]
>  @property
>  def defining_class(obj):
>  return cls_cell.get()
>  namespace[defining_class.fget.__name__] = defining_class

I have two concrete examples of custom metaclasses currently broken by calling 
`__init_subclass__` from `type.__new__`:

- `ABCMeta`

- `Enum`

Perhaps `ABCMeta` can be easily fixed -- it is calling the function that records all abstract methods, etc., after the 
`type.__new__` call; can it be called before?


`Enum` cannot be fixed: it must call `type.__new__` to get the new class object, and it must have the new class object 
in order to create the members as instances of that `Enum` class -- which means that anyone trying to use 
`__init_subclass__` to do anything with an `Enum`'s members is doomed to failure because those members don't exist yet.


Possible work-arounds for `Enum`:

- call `__init_subclass__` twice - once from `type.__new__`, and then again
  by `EnumMeta` after the members have been added

  * problem: authors of `__init_subclass__` have to be aware that they will
get called twice

- do horribly brittle mro rewrites in `EnumMeta` so the `type.__new__` call
  is a no-op, then `EnumMeta` calls `__init_subclass__` when the members
  have been created

  * problem: horribly brittle

--

While I would prefer to see the calls to `__init_subclass__` and `__set_name__` (and any other such enhancements) moved 
to `type.__init__`, another possibility would be to add a class attribute/keyword, `_pep487_to_init`, which if present 
causes `type.__new__` to skip those calls and `type.__init__` to make them.  Then the only headaches would be when 
metaclasses with different desires were combined.


I am also open to other solutions that would allow `EnumMeta` and its users to be able to use `__init_subclass__` with a 
fully created class.


--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NIIQPQO3DLDTMV6PJQNGM3ODUQHJBYSQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Dusting off PEP 533?

[Stestagg]

For what it's worth, this recent issue: https://bugs.python.org/issue42762
is somewhat related to the non-async aspect of 533. (generator func getting
closed during __del__ and causing odd side-effects at seemingly random
points in the execution flow)

The example code provided in that issue is contrived, and involves catching
unconditional exceptions and recursing, but following the code flow in that
situation feels unreasonably hard, and deciding if it's a python bug or not
is even harder.

I'm not sure how easy an implementation of 533 would be that doesn't break
a lot of existing code, but if one could be made, it might help make
generator lifecycles more predictable.




On Tue, Jan 5, 2021 at 1:08 PM Senthil Kumaran  wrote:

> Hi Paul,
>
> > Per PEP 525, I can call aclose coroutine method to cleanup the
> generator, but
> > it requires the code iterating to be aware that that closing the
> generator is
> > necessary.
>
> How about treating this as a bug for the specific use case that you
> mentioned,
> rather than a complete addition of " PEP 533 -- Deterministic cleanup for
> iterators". I am not certain why PEP 533 was deffered.
>
> I'd also suggest a documentation update if the details of aclose
> requirement is
> mentioned only in a PEP and not in the documentation.
>
> Thank you,
> Senthil
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/EPKMYH7DH5G6YR3AQHZTQKHBQF46YXLC/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VB64QJAYU2CCLOHRCBUIQOHMS2DIQQCC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Dusting off PEP 533?

[Senthil Kumaran]

Hi Paul,

> Per PEP 525, I can call aclose coroutine method to cleanup the generator, but
> it requires the code iterating to be aware that that closing the generator is
> necessary.

How about treating this as a bug for the specific use case that you mentioned,
rather than a complete addition of " PEP 533 -- Deterministic cleanup for
iterators". I am not certain why PEP 533 was deffered.

I'd also suggest a documentation update if the details of aclose requirement is
mentioned only in a PEP and not in the documentation.

Thank you,
Senthil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EPKMYH7DH5G6YR3AQHZTQKHBQF46YXLC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: __init_subclass__ and metaclasses

[Nick Coghlan]

On Wed, 30 Dec 2020 at 04:38, Ethan Furman  wrote:
> > No, we can't. There is a window where the subclass is initialized after 
> > `typing_new()` returned before `__init__`
> > starts, and you propose to move the subclass after that window. There may 
> > be code that depends on the class being
> > initialized at that point, and you will break that code.
>
> True, there will be a few custom metaclasses that need to move some code from 
> their `__new__` to `__init__` instead, and
> a few that need to add an `__init__` to consume any keyword arguments that 
> don't need to get passed to
> `__init_subclass__`.  That seems like a small price to pay to be able to 
> write custom metaclasses that are able to fully
> participate in the `__set_name__` and `__init_subclass__` protocols.

>From https://www.python.org/dev/peps/pep-0487/#implementation-details:

"""As a third option, all the work could have been done in
type.__init__. Most metaclasses do their work in __new__, as this is
recommended by the documentation. Many metaclasses modify their
arguments before they pass them over to super().__new__. For
compatibility with those kind of classes, the hooks should be called
from __new__."""

That isn't the clearest way of saying it, but the intent was to make
sure that metaclass __new__ implementations continued to see fully
initialised class objects as soon as they returned from
type.__new__(), with type.__init__() continuing to be essentially a
no-op.

If a metaclass wants to modify the attributes on the type in a way
that is visible to __init_subclass__ and __set_name__, it needs to
modify the namespace dict passed to type().__new__, not modify the
type after it has already been created:

```
class Meta(type):
 #
 def __new__(mcls, name, bases, namespace, **kwds):
 # create new class, which will call __init_subclass__ and __set_name__
new_namespace = namespace.copy()
# Customize class contents
new_namespace["some_attr"] = 9
return type.__new__(mcls, name, bases, new_namespace, **kwds)
```

If you need to access the class being defined from __init_subclass__,
it gets passed in, and if you need to access it from __set_name__,
then it gets passed in there as well.

If you need to access the class being defined from elsewhere, then the
`__class__` cell used by zero-arg super is already populated before
any of those hooks get called.

As a CPython implementation detail, that cell is actually passed
through to type().__new__ as `__classcell__` in the namespace. If that
was elevated to a language feature rather than a CPython
implementation detail, then metaclasses could write:

cls_cell = namespace["__classcell__"]
@property
def defining_class(obj):
return cls_cell.get()
namespace[defining_class.fget.__name__] = defining_class

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QWVL2UPV2MCJJXTO3TOANH5YSR2GEDBP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 642 v3: Explicit patterns for structured pattern matching

[Paul Sokolovsky]

Hello,

On Tue, 5 Jan 2021 20:37:27 +1000
Nick Coghlan  wrote:

> > > object(host=as host, port=as port}:", but that couldn't ever be

I'd like to point out the weirdness of the "as" syntax when applied to
positional arguments, e.g.:

case [as x, as y]:
case Cls(as x, as y):

That feels unnatural, and the fact behind that intuitive feeling is
that "as" *never* used like that in Python so far. Where it's used
now, there's explicit "what" is available before "as":

import org_name as alias_name
with expr as var:

So again, standalone "as" feels weird. To feel it natural, it would
need to be written as "case [_ as x, _ as y]:", but that's exactly what
PEP634 already allows, and nothing needs to be done! And I'd like to
remind that one of the original alternatives raised for the syntax was
something like:

case [>x, >y]:

Which is both concise and intuitive (for as long as you accept that
">" here represent an arrow, which should be one-time-in-life learning
event).


The argument for "lone as" can be "just let that syntax keep you
hostages for some time (say, one week of regular usage) and you'll
acquire Stockholm Syndrome for it." That may very well be true. But
1-or-2-character punctuation prefix sigil is still more concise and more
intuitive.

[]

> For the other point, consider the following examples (using "match
> SUBJECT as case PTRN" as a strawman one-shot pattern matching syntax):
> 
># One shot PEP 634 pattern
>match addr as case object(host=host, port=port)
> 
># One shot PEP 642 pattern
>match addr as case object{.host as host, .port as port}
> 
> # Attribute tuple (PEP 642 based deferred idea)
> host, port = addr{.host, .port}
> 
> One shot pattern matching turns out to have an inherent verbosity
> problem,

My bet that it has "inherent verbosity problem" only because there was
no such proposal. And any such proposal would start with basics:

What's syntax for sequence destructuring aka pattern matching against
catch-all variables? Assignment to a tuple/list:

(a, b) = tuply

Ok, what happens if you assign to a dict syntax now:

{a: b} = dicty

Syntax error? Ok, that's new one-time dict matching. 

What happens if you assign to a function call syntax now (also object
constructor syntax):

Addr(ip, port) = addr_obj

Syntax error? That's new one-time object matching.

Not verbose at all:

(a, b) = tuply
{a: b} = dicty
Addr(ip, port) = addr_obj


Now, those throw ValueError, what if you want to get a boolean? Just
wrap it in the "if" context:

if (a, b) = tuply:

if {a: b} = dicty:

if Addr(ip, port) = addr_obj:


>From there, bikeshedding can start. E.g., there will be strong cry that
"= and == are easy to mix up", so it'll become:

if match (a, b) = tuply:

if match {a: b} = dicty:

if match Addr(ip, port) = addr_obj:

That's a whole 1 token more verbose, but 6 chars more, so someone will
come up with following alternatives:

if_m (a, b) = tuply:

if (a, b) m= tuply:


Etc, the usual bikeshedding whirlwind.


[]

-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OAKCB4JI7K7CN4A64EDYL6GDM6KX2M7U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 642 v3: Explicit patterns for structured pattern matching

[Nick Coghlan]

On Mon, 4 Jan 2021 at 18:38, Paul Moore  wrote:
>
> On Sun, 3 Jan 2021 at 23:38, Nick Coghlan  wrote:
> >
> > The instance attribute syntax arose from trying to deal with two problems 
> > from class patterns in PEP 634:
> >
> > * "ATTR=TARGET" using "=" to bind to the right instead of to the left
> > * no subsequent path to ever offering a syntax for *retrieving* multiple 
> > attributes in ordinary expressions outside pattern matching
> >
> > The minimal fix for the first point would have been just "case 
> > object(host=as host, port=as port}:", but that couldn't ever be used to 
> > retrieve multiple attributes, as "object(host, port)" is already a function 
> > call.
>
> OK, so there's our dispute. Neither of those seem to me to be problems
> with PEP 634.
>
> 1. I view ATTR=PLACEHOLDER as *equality* with a placeholder that gets
> filled in, not a binding that goes left to right. (And no, I don't
> have a problem with the rule that the placeholder must be on the
> right).

But why would you assume `=` means that in Python code? The only
places we currently use `=` are in assignment statements and in
keyword arguments, and in both of those cases, the target is on the
left:

TARGET = EXPR
func(TARGET_PARAM=EXPR)

However, you're right, if "SOURCE = DESTINATION" doesn't read
strangely to you, then the switch to mapping-pattern inspired syntax
for attribute patterns in PEP 642 isn't going to feel compelling.

For the other point, consider the following examples (using "match
SUBJECT as case PTRN" as a strawman one-shot pattern matching syntax):

   # One shot PEP 634 pattern
   match addr as case object(host=host, port=port)

   # One shot PEP 642 pattern
   match addr as case object{.host as host, .port as port}

# Attribute tuple (PEP 642 based deferred idea)
host, port = addr{.host, .port}

One shot pattern matching turns out to have an inherent verbosity
problem, as it forces the specification of the destructuring to be
separated from the specification of the subject. That separation is
desirable in match statements, where one subject is potentially
destructured in different ways. It's far from clear that the
separation would be desirable in one-shot matching - we might instead
be better off with a model that only allows simple destructuring
operations that produce a tuple, and then uses ordinary iterable
unpacking for name binding.

While that's a relatively minor point in the grand scheme of things, I
do like having more future design options available to us.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XJZQDW44LVCHCUSH4BSO3BWADRBORC4K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Where is the SQLite module maintainer

[Serhiy Storchaka]

27.12.20 22:20, Erlend Aasland пише:
> Who can help me review code that touches the sqlite3 module code base?
> 
> 
> I've received a lot of constructive reviews from Victor Stinner, Dong-he
> Na and Pablo Galindo on my past sqlite3 PR's; thank you so much! I still
> feel uncomfortable requesting their review, as none of them are sqlite3
> module maintainers.

I fixed many bugs in sqlite3 and did reviews, so I have some experience,
although I am not an expert. I payed less attention to it since Berker
Peksag took its maintenance, so my attention was not needed. But seems
he is not very active now (I hope it is temporary). You can request my
review. My reaction may be not instant because last weeks I am very busy
with curses and tkinter.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DYA4XLJMSAEHDU4YQEG5QEU5SGRT6HBC/
Code of Conduct: http://python.org/psf/codeofconduct/