[Python-Dev] checking what atexit handlers are registered in Python 3
Hi All, I'm finally getting around to porting some of the packages I maintain over to Python 3. One rough edge I've hit: I see the atexit module has moved to be C-based and, as far as I can tell, no longer allows you to introspect what atexit functions have been registered. If I'm writing tests for code that registers atexit handlers, how can I check that they've been correctly registered? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
I wrote PEP 422 (simple class initialization hook) last year after PJE pointed out that Python 3 style metaclasses made some Python 2 code impossible to migrate (since the class body could no longer request modifications to be made to the class after initialization was complete). It then languished, as I never found the time to actually implement it. Fortunately, Daniel Urban has now agreed to be a co-author on the PEP, and has fixed a couple of lingering technical errors in the PEP, as well as providing a reference implementation at http://bugs.python.org/issue17044 For those that don't recall the original discussion, the proposal is to add a new __init_class__ hook, invoked after the class object is created, but before the class decorators are applied. This provides a simple approach to inherited post-creation modification of classes, without the need for a custom metaclass. There's currently no BDFL delegate assigned, so this is a request for a pronouncement from Guido. Regards, Nick. === PEP: 422 Title: Simple class initialisation hook Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Daniel Urban Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 5-Jun-2012 Python-Version: 3.4 Post-History: 5-Jun-2012, 10-Feb-2012 Abstract In Python 2, the body of a class definition could modify the way a class was created (or simply arrange to run other code after the class was created) by setting the ``__metaclass__`` attribute in the class body. While doing this implicitly from called code required the use of an implementation detail (specifically, ``sys._getframes()``), it could also be done explicitly in a fully supported fashion (for example, by passing ``locals()`` to a function that calculated a suitable ``__metaclass__`` value) There is currently no corresponding mechanism in Python 3 that allows the code executed in the class body to directly influence how the class object is created. Instead, the class creation process is fully defined by the class header, before the class body even begins executing. This PEP proposes a mechanism that will once again allow the body of a class definition to more directly influence the way a class is created (albeit in a more constrained fashion), as well as replacing some current uses of metaclasses with a simpler, easier to understand alternative. Background == For an already created class ``cls``, the term "metaclass" has a clear meaning: it is the value of ``type(cls)``. *During* class creation, it has another meaning: it is also used to refer to the metaclass hint that may be provided as part of the class definition. While in many cases these two meanings end up referring to one and the same object, there are two situations where that is not the case: * If the metaclass hint refers to an instance of ``type``, then it is considered as a candidate metaclass along with the metaclasses of all of the parents of the class being defined. If a more appropriate metaclass is found amongst the candidates, then it will be used instead of the one given in the metaclass hint. * Otherwise, an explicit metaclass hint is assumed to be a factory function and is called directly to create the class object. In this case, the final metaclass will be determined by the factory function definition. In the typical case (where the factory functions just calls ``type``, or, in Python 3.3 or later, ``types.new_class``) the actual metaclass is then determined based on the parent classes. It is notable that only the actual metaclass is inherited - a factory function used as a metaclass hook sees only the class currently being defined, and is not invoked for any subclasses. In Python 3, the metaclass hint is provided using the ``metaclass=Meta`` keyword syntax in the class header. This allows the ``__prepare__`` method on the metaclass to be used to create the ``locals()`` namespace used during execution of the class body (for example, specifying the use of ``collections.OrderedDict`` instead of a regular ``dict``). In Python 2, there was no ``__prepare__`` method (that API was added for Python 3 by PEP 3115). Instead, a class body could set the ``__metaclass__`` attribute, and the class creation process would extract that value from the class namespace to use as the metaclass hint. There is `published code`_ that makes use of this feature. Another new feature in Python 3 is the zero-argument form of the ``super()`` builtin, introduced by PEP 3135. This feature uses an implicit ``__class__`` reference to the class being defined to replace the "by name" references required in Python 2. Just as code invoked during execution of a Python 2 metaclass could not call methods that referenced the class by name (as the name had not yet been bound in the containing scope), similarly, Python 3 metaclasses cannot call methods that rely on the implicit ``__class__`` reference (as it is
Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
On Sun, 10 Feb 2013 22:32:50 +1000 Nick Coghlan wrote: > > Replaces many use cases for dynamic setting of ``__metaclass__`` > - > > For use cases that don't involve completely replacing the defined class, > Python 2 code that dynamically set ``__metaclass__`` can now dynamically > set ``__init_class__`` instead. For more advanced use cases, introduction of > an explicit metaclass (possibly made available as a required base class) will > still be necessary in order to support Python 3. So, what are the use cases? I probably write metaclasses once a year, I wonder how much incentive there is to bring an additional complication to the already complicated class construction process. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
On Sun, Feb 10, 2013 at 10:47 PM, Antoine Pitrou wrote: > On Sun, 10 Feb 2013 22:32:50 +1000 > Nick Coghlan wrote: >> >> Replaces many use cases for dynamic setting of ``__metaclass__`` >> - >> >> For use cases that don't involve completely replacing the defined class, >> Python 2 code that dynamically set ``__metaclass__`` can now dynamically >> set ``__init_class__`` instead. For more advanced use cases, introduction of >> an explicit metaclass (possibly made available as a required base class) will >> still be necessary in order to support Python 3. At least PEAK uses it for interface declarations, I believe Zope do something similar. However, just as I championed PEP 414 to make porting Unicode-aware projects easier, I'm championing this one because I don't consider it acceptable for Python 3 to be a less capable language than Python 2 in this regard. Having come up with what I consider to be an elegant design to restore that capability, I'm not especially interested in passing judgment on the worthiness of what PJE is doing with Python 2 metaclasses that cannot currently be replicated in Python 3. > So, what are the use cases? > I probably write metaclasses once a year, I wonder how much incentive > there is to bring an additional complication to the already > complicated class construction process. One of the main goals of the PEP is actually to give people even *less* reason to write custom metaclasses, as you won't need to write one any more if all you want to do is adjust the class attributes after execution of the class body completes, and automatically do the same for any subclasses. (The latter part is key - if you don't want inheritance, then class decorators fill the role just fine. That's why the design in the PEP is the way it us - __init_class__ effectively becomes the innermost class decorator, which is called even for subclasses) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
On Sun, Feb 10, 2013 at 2:32 PM, Nick Coghlan wrote: > For those that don't recall the original discussion, the proposal is > to add a new __init_class__ hook, invoked after the class object is > created, but before the class decorators are applied. This provides a > simple approach to inherited post-creation modification of classes, > without the need for a custom metaclass. Couldn't one just write a metaclass that calls __init_class__ for you? Schiavo Simon ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
On Sun, 10 Feb 2013 23:17:00 +1000 Nick Coghlan wrote: > On Sun, Feb 10, 2013 at 10:47 PM, Antoine Pitrou wrote: > > On Sun, 10 Feb 2013 22:32:50 +1000 > > Nick Coghlan wrote: > >> > >> Replaces many use cases for dynamic setting of ``__metaclass__`` > >> - > >> > >> For use cases that don't involve completely replacing the defined class, > >> Python 2 code that dynamically set ``__metaclass__`` can now dynamically > >> set ``__init_class__`` instead. For more advanced use cases, introduction > >> of > >> an explicit metaclass (possibly made available as a required base class) > >> will > >> still be necessary in order to support Python 3. > > At least PEAK uses it for interface declarations, I believe Zope do > something similar. zope.interface has been ported to Python 3, so the annoyance can't be very blocking. > However, just as I championed PEP 414 to make porting Unicode-aware > projects easier, I'm championing this one because I don't consider it > acceptable for Python 3 to be a less capable language than Python 2 in > this regard. This sounds like a theoretical concern rather than a practical one. > Having come up with what I consider to be an elegant > design to restore that capability, I'm not especially interested in > passing judgment on the worthiness of what PJE is doing with Python 2 > metaclasses that cannot currently be replicated in Python 3. It may be locally elegant, but it still brings additional complication to the global model. Reading your proposal reminds me of the various ways of influencing the pickle protocol: http://docs.python.org/dev/library/pickle.html#pickling-class-instances Nobody can claim this is simple and easy to wrap your head around. It is a maintenance burden, and it discourages understanding of the underlying model by anyone but language experts. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
On Sun, Feb 10, 2013 at 11:33 PM, Simon Cross wrote: > On Sun, Feb 10, 2013 at 2:32 PM, Nick Coghlan wrote: >> For those that don't recall the original discussion, the proposal is >> to add a new __init_class__ hook, invoked after the class object is >> created, but before the class decorators are applied. This provides a >> simple approach to inherited post-creation modification of classes, >> without the need for a custom metaclass. > > Couldn't one just write a metaclass that calls __init_class__ for you? And, indeed, that's what Zope did more than 10 years ago. The problem is that you're now locked in to that metaclass - metaclass conflicts become a permanent risk. You also can't *add* a metaclass to public classes, the risk of introducing metaclass conflicts means that such a change is always backwards incompatible. Adding __init_class__ is fine though, because that's no higher risk than adding a __new__ or __init__ method (it's actually lower risk, since the signature is always consistent). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
On Sun, Feb 10, 2013 at 11:34 PM, Antoine Pitrou wrote: > Nobody can claim this is simple and easy to wrap your head around. It > is a maintenance burden, and it discourages understanding of the > underlying model by anyone but language experts. You think anyone but language experts fully understands the metaclass mechanism now? *I* got the description of some details wrong in the PEP, and I'm probably in the top half dozen people on the planet when it comes to understanding how Python 3's class model works. Plenty of people understand decorators though, and that's all __init_class__ is: a special class decorator that is applied before any other decorators, and is automatically inherited by subclasses. It massively lowers the barrier to entry for inherited post-modification of classes. Will it mean people will only bother to understand metaclasses if they actually *need* metaclass.__prepare__ or to control the identity of the object returned by metaclass.__call__? Absolutely. However, I don't see that as any different from the fact that vastly more people understand how to use instance __init__ methods correctly than understand the ins and outs of using __new__ to control the *creation* of the object, rather than merely initialising it after it is already created. In some respects, cleanly separating out the two steps of controlling initialisation and controlling creation can actually make the two easier to grasp (since you can tackle them individually as distinct concepts, rather than having to wrap your head around both of them at the same time). As far as the maintenance burden goes, the patch to enable PEP 422 for types.new_class() is trivial: -return meta(name, bases, ns, **kwds) +cls = meta(name, bases, ns, **kwds) +try: +initcl = cls.__init_class__ +except AttributeError: +pass +else: +initcl() +return cls The change in the builtin.__build_class__ implementation is similarly trivial (and has the same semantics), it's just more verbose due to its being written in C. The documentation changes are quite straightforward (moving __init_class__ and decorator invocation out to their own subsection), and the tests Daniel has provided are extensive. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
On Mon, 11 Feb 2013 00:09:55 +1000 Nick Coghlan wrote: > > As far as the maintenance burden goes, the patch to enable PEP 422 for > types.new_class() is trivial: > > -return meta(name, bases, ns, **kwds) > +cls = meta(name, bases, ns, **kwds) > +try: > +initcl = cls.__init_class__ > +except AttributeError: > +pass > +else: > +initcl() > +return cls I didn't know types.new_class(). I suppose that's because I'm not part of the top half dozen people on the planet :-) I've stopped trying to understand when I saw about the __build_class__ additions and whatnot. I now consider that part of the language totally black magic. I'm not sure it's a good thing when even some maintainers of the language don't understand some of its (important) corners. > The change in the builtin.__build_class__ implementation is similarly > trivial (and has the same semantics), it's just more verbose due to > its being written in C. Sure, every little addition is "trivial". At the end you have a scary monster made of many little trivial additions along the years, and everyone has to take care not to break it. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
Antoine Pitrou, 10.02.2013 15:28: > On Mon, 11 Feb 2013 00:09:55 +1000 Nick Coghlan wrote: >> As far as the maintenance burden goes, the patch to enable PEP 422 for >> types.new_class() is trivial: >> >> -return meta(name, bases, ns, **kwds) >> +cls = meta(name, bases, ns, **kwds) >> +try: >> +initcl = cls.__init_class__ >> +except AttributeError: >> +pass >> +else: >> +initcl() >> +return cls > > I didn't know types.new_class(). I suppose that's because I'm not part > of the top half dozen people on the planet :-) I've stopped trying to > understand when I saw about the __build_class__ additions and whatnot. > I now consider that part of the language totally black magic. > > I'm not sure it's a good thing when even some maintainers of the > language don't understand some of its (important) corners. I, for one, only learned most of this stuff when we implemented it for Cython and I'm sure there are lots of little details that I'm not aware of. Use cases are too rare to fully learn it and the matter is certainly complex. >> The change in the builtin.__build_class__ implementation is similarly >> trivial (and has the same semantics), it's just more verbose due to >> its being written in C. > > Sure, every little addition is "trivial". At the end you have a scary > monster made of many little trivial additions along the years, and > everyone has to take care not to break it. FWIW, I agree with Nick that this actually kills some use cases of metaclasses and could thus lower the barrier for "doing stuff to classes" for those who don't want to dig into metaclasses for one reason or another. However, it's hard to say if this "new way of doing it" doesn't come with its own can of worms. For example, would cooperative calls to "__init_class__" work if a superclass already defines it? Do implementors have to remember that? And is it clear how this should be done, e.g. what should normally go first, my own code or the superclass call? Supporting cooperative __init_class__() calls properly might actually be a good thing. Currently, there's only one metaclass, plus a sequence of decorators, which makes the setup rather static and sometimes tedious for subclasses that need to do certain things by themselves, but in addition to what happens already. So, the way to explain it to users would be 1) don't use it, 2) if you really need to do something to a class, use a decorator, 3) if you need to decide dynamically what to do, define __init_class__() and 4) don't forget to call super's __init_class__() in that case, and 5) only if you need to do something substantially more involved and know what you're doing, use a metaclass. Sounds a bit smoother than the current metaclass hole that we throw people into directly when a decorator won't catch it for them. It's usually not a good idea to add complexity for something that's essentially a rare use case. Only the assumption that it might actually make it easier for users to handle certain other use cases where they'd currently end up with their own home-grown metaclass is a point that makes it worth considering for me. But then again, maybe these use cases are really rare enough to just let people figure out how to do it with metaclasses, or even a combination of metaclasses and decorators. Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
On Sun, Feb 10, 2013 at 5:48 PM, Stefan Behnel wrote: > However, it's hard to say if this "new way of doing it" doesn't come with > its own can of worms. For example, would cooperative calls to > "__init_class__" work if a superclass already defines it? Do implementors > have to remember that? And is it clear how this should be done, e.g. what > should normally go first, my own code or the superclass call? Supporting > cooperative __init_class__() calls properly might actually be a good thing. > Currently, there's only one metaclass, plus a sequence of decorators, which > makes the setup rather static and sometimes tedious for subclasses that > need to do certain things by themselves, but in addition to what happens > already. Cooperative multiple inheritance of __init_class__ would work exactly the same way as it works e.g., for __init__ of any other method (actually it is easier, since as Nick mentioned, the signature is always the same): __init_class__ can simply use the zero argument form of super. There is a simple example in the tests at http://bugs.python.org/issue17044. Daniel ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
On Sun, Feb 10, 2013 at 7:32 AM, Nick Coghlan wrote:
>class Example:
>@classmethod
>def __init_class__(cls):
Is the @classmethod required? What happens if it's not present?
Second, will type have a default __init_class__? (IMO, it should,
otherwise it will be impossible to write co-operative __init_class__
functions.)
Only other comment is that the PEP could use a more concrete use case, e.g.:
class Record:
__fields = {}
@classmethod
def __init_class__(cls):
cls.__fields = dict(cls.__fields) # inherited fields
cls.__fields.update({attr:val for attr, val in
cls.__dict__.iteritems() if isinstance(val, Field)})
super().__init_class__() # be co-operative
# ...other methods that use the __fields attribute
class SomeRecord(Record):
foo = Field(int)
bar = Field(str)
Putting something like this early on might help to demonstrate the
usefulness of the feature on its own merits, independent of the
porting issue, etc. ;-)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
On Sun, Feb 10, 2013 at 11:48 AM, Stefan Behnel wrote: > So, the way to explain it to users would be 1) don't use it, 2) if you > really need to do something to a class, use a decorator, 3) if you need to > decide dynamically what to do, define __init_class__() and 4) don't forget > to call super's __init_class__() in that case, and 5) only if you need to > do something substantially more involved and know what you're doing, use a > metaclass. I'd revise that to: 1) if there's no harm in forgetting to decorate a subclass, use a class decorator 2) if you want to ensure that a modification is applied to every subclass of a single common base class, define __init_class__ (and always call its super) 3) If you need to make the class object *act* differently (not just initialize it or trigger some other side-effect at creation time), or if you want the class suite to return some other kind of object, you'll need a metaclass. Essentially, this change fixes a hole in class decorators that doesn't exist with function decorators: if you need the decoration applied to subclasses, you can end up with silent failures right now. Conversely, if you try prevent such failures using a metaclass, you not only have a big hill to climb, but the resulting code will be vulnerable to metaclass conflicts. The proposed solution neatly fixes both of these problems, providing One Obvious Way to do subclass initialization. (An alternative, I suppose, would be to let you do something like @@someClassDecorator to have inheritable class decorators, but as Nick has pointed out, implementing inheritable decorators is a lot more complicated.) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does Signature.from_function() have to check the type of its argument?
On Fri, Feb 8, 2013 at 5:44 PM, Stefan Behnel wrote: > Argh - sorry, got it all wrong. "__instancecheck__()" works exactly the > other way round. In the type check above, it's the FunctionType type that > gets asked for an instance check, which doesn't help at all in this case > because it simply doesn't know about the desired subtype relation. It would > work if type(func).__instancecheck__() was used, but that doesn't happen. > > So, no help from that side, sadly. How about you return FunctionType as your __class__ attribute? ;-) Your type() will still be different, but isinstance() also considers the __class__ if it's different from the type(). (At least it does in 2.x, I've not tried it in any 3.x versions yet...) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does Signature.from_function() have to check the type of its argument?
On 02/10/2013 01:46 PM, PJ Eby wrote: How about you return FunctionType as your __class__ attribute? ;-) Your type() will still be different, but isinstance() also considers the __class__ if it's different from the type(). (At least it does in 2.x, I've not tried it in any 3.x versions yet...) Python 3.2.3 (default, Oct 19 2012, 19:53:16) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. --> class Test(): ... pass ... --> t = Test() --> t.__class__ --> import types --> class Test(): ... __class__ = types.FunctionType ... --> t = Test() --> t.__class__ --> isinstance(t, types.FunctionType) True -- ~Ethan~ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.3): Reject float as uid or gid.
On 2/10/2013 4:29 PM, serhiy.storchaka wrote:
> http://hg.python.org/cpython/rev/4ef048f4834e
> changeset: 82147:4ef048f4834e
> branch: 3.3
> parent: 82145:b322655a4a88
> user:Serhiy Storchaka
> date:Sun Feb 10 23:28:02 2013 +0200
> summary:
> Reject float as uid or gid.
> A regression was introduced in the commit for issue issue #4591.
>
> files:
> Modules/posixmodule.c | 16 ++--
> 1 files changed, 14 insertions(+), 2 deletions(-)
>
>
> diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
> --- a/Modules/posixmodule.c
> +++ b/Modules/posixmodule.c
> @@ -437,7 +437,13 @@
> _Py_Uid_Converter(PyObject *obj, void *p)
> {
> int overflow;
> -long result = PyLong_AsLongAndOverflow(obj, &overflow);
> +long result;
> +if (PyFloat_Check(obj)) {
> +PyErr_SetString(PyExc_TypeError,
> +"integer argument expected, got float");
> +return 0;
> +}
> +result = PyLong_AsLongAndOverflow(obj, &overflow);
> if (overflow < 0)
> goto OverflowDown;
> if (!overflow && result == -1) {
Instead of special-casing float, isn't using __index__ the preferred way
to do this?
--
Eric.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
