Re: getting special from type, not instance (was Re: [Python-Dev] copy confusion)
Hi Guido, On Wed, Jan 12, 2005 at 09:59:13AM -0800, Guido van Rossum wrote: The descriptor for __getattr__ and other special attributes could claim to be a data descriptor This has the nice effect that x[y] and x.__getitem__(y) would again be equivalent, which looks good. On the other hand, I fear that if there is a standard metamethod decorator (named after Phillip's one), it will be misused. Reading the documentation will probably leave most programmers with the feeling it's something magical to put on methods with __ in their names, and it won't be long before someone notices that you can put this decorator everywhere in your classes (because it won't break most programs) and gain a tiny performance improvement. I guess that a name-based hack in type_new() to turn all __*__() methods into data descriptors would be even more obscure? Finally, I wonder if turning all methods whatsoever into data descriptors (ouch! don't hit!) would be justifiable by the feeling that it's often bad style and confusing to override a method in an instance (as opposed to defining a method in an instance when there is none on the class). (Supporting this claim: Psyco does this simplifying hypothesis for performance reasons and I didn't see yet a bug report for this.) In all cases, I'm +1 on seeing built-in method objects (PyMethodDescr_Type) become data descriptors (classy descriptors? :-). Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
getting special from type, not instance (was Re: [Python-Dev] copy confusion)
Since this bug isn't the cause of Fredrik's problem I'm changing the subject (and keep discussing the specific problem that Fredrik uncovered under the original subject). On 2005 Jan 12, at 05:11, Guido van Rossum wrote: ... I had exactly the same metabug in the pep 246 reference implementation, Armin Rigo showed how to fix it in his only recent post. Don't recall seeing that, but if you or he can fix this without breaking other stuff, it's clear you should go ahead. (This worked in 2.2, FWIW; it broke in 2.3.) Armin's fix was to change: conform = getattr(type(obj), '__conform__', None) into: for basecls in type(obj).__mro__: if '__conform__' in basecls.__dict__: conform = basecls.__dict__['__conform__'] break else: # not found I have only cursorily examined the rest of the standard library, but it seems to me there may be a few other places where getattr is being used on a type for this purpose, such as pprint.py which has a couple of occurrences of r = getattr(typ, __repr__, None) Since this very same replacement is needed in more than one place for get the following special attribute from the type of the object', it seems that a function to do it should be introduced in one place and used from where it's needed: def get_from_first_dict(dicts, name, default=None): for adict in dicts: try: return adict[name] except KeyError: pass return default to be called, e.g. in the above example with '__conform__', as: conform = get_from_first_dict( (basecls.__dict__ for basecls in type(obj).__mro__), '__conform__' ) The needed function could of course be made less general, by giving more work to the function and less work to the caller, all the way down to: def getspecial(obj, name, default=None): for basecls in type(obj).__mro__: try: return basecls.__dict__[name] except KeyError: pass return default to be called, e.g. in the above example with '__conform__', as: conform = getspecial(obj, '__conform__') This has the advantage of not needing the genexp, so it's usable to implement the fix in 2.3.5 as well as in 2.4.1. Moreover, it can specialcase old-style class instances to provide the backwards compatible behavior, if desired -- that doesn't matter (but doesn't hurt) to fix the bug in copy.py, because in that case old-style instances have been already specialcases previously, and might help to avoid breaking anything in other similar bugfixes, so that's what I would suggest: def getspecial(obj, name, default=None): if isinstance(obj, types.InstanceType): return getattr(obj, name, default) for basecls in type(obj).__mro__: try: return basecls.__dict__[name] except KeyError: pass return default The tradeoff between using type(obj) and obj.__class__ isn't crystal-clear to me, but since the latter might apparently be faked by some proxy to survive isinstance calls type(obj) appears to me to be right. Where in the standard library to place this function is not clear to me either. Since it's going into bugfix-only releases, I assume it shouldn't be published. Maybe having it as copy._getspecial (i.e. with a private name) is best, as long as it's OK to introduce some coupling by having (e.g.) pprint call copy._getspecial too. Performance might be a problem, but the few bugfix locations where a getattr would be replaced by this getspecial don't seem to be hotspots, so maybe we don't need to worry about it for 2.3 and 2.4 (it might be nice to have this functionality published in 2.5, of course, and then it should probably be made fast). Feedback welcome -- the actual patch will doubtless be tiny, but it would be nice to have it right the first time (as it needs to go into both the 2.3 and 2.4 bugfix branches and the 2.5 head). Alex ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] copy confusion
On 2005 Jan 11, at 23:20, Fredrik Lundh wrote: back in Python 2.1 (and before), an object could define how copy.copy should work simply by definining a __copy__ method. here's the relevant portion: ... try: copierfunction = _copy_dispatch[type(x)] except KeyError: try: copier = x.__copy__ except AttributeError: raise error, \ un(shallow)copyable object of type %s % type(x) y = copier() ... I recently discovered that this feature has disappeared in 2.3 and 2.4. in- stead of looking for an instance method, the code now looks at the object's type: Hmmm, yes, we were discussing this general issue as part of the huge recent thread about pep 246. In the new-style object model, special methods are supposed to be looked up on the type, not on the object; otherwise, having a class with special methods would be a problem -- are the methods meant to apply to the class object itself, or to its instances? However, apparently, the code you quote is doing it wrong: cls = type(x) copier = _copy_dispatch.get(cls) if copier: return copier(x) copier = getattr(cls, __copy__, None) if copier: return copier(x) ...because getattr is apparently the wrong way to go about it (e.g., it could get the '__copy__' from type(cls), which would be mistaken). Please see Armin Rigo's only recent post to Python-Dev for the way it should apparently be done instead -- assuming Armin is right (he generally is), there should be plenty of bugs in copy.py (ones that emerge when you're using custom metaclasses c -- are you doing that?). Still, if you're using an instance of an old-style class, the lookup in _copy_dispatch should be on types.InstanceType -- is that what you're trying to copy, an instance of an old-style class? (copy.deepcopy still seems to be able to use __deepcopy__ hooks, though) It starts with a peek into a dispatch dictionary for the type of the object, too, just like shallow copy does. What's the type of what you're trying to copy? is this a bug, or a feature of the revised copy/pickle design? (the code in copy_reg/copy/pickle might be among the more convoluted pieces of python coding that I ever seen... and what's that smiley doing in copy.py?) and if it's a bug, does the fact that nobody reported this for 2.3 indicate that I'm the only one using this feature? is there a better way to control copying that I should use instead? When I can, I use __getstate__ and __setstate__, simply because they seem clear and flexible to be (usable for copying, deep copying, pickling). But that doesn't mean __copy__ or __deepcopy__ should be left broken, of course. Although there are features of design intent here, it does appear to me there may be bugs too (if the getattr on the type is wrong); in this case it's worrisome, not just that nobody else reported problems, but also that the unit tests didn't catch them...:-( Alex ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] copy confusion
At 11:56 PM 1/11/05 +0100, Alex Martelli wrote: What both issues? There's only one issue, it seems to me -- one of metaconfusion. I was relying on Fredrik's report of a problem with the code; that is the other issue I referred to. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] copy confusion
At 02:58 PM 1/11/05 -0800, Guido van Rossum wrote: [Phillip] Looks like a bug to me; it breaks the behavior of classic classes, since type(classicInstance) returns InstanceType. I'm not so sure. I can't seem to break this for classic classes. Sorry; I was extrapolating from what I thought was Fredrik's description of this behavior as a bug, and examining of the history of the code that he referenced. I saw that the current version of that code had evolved directly from a version that was retrieving instance.__copy__; I therefore assumed that the loss-of-feature Fredrik was reporting was that. That is, I thought that the problem he was experiencing was that classic classes no longer supported __copy__ because this code had changed. I guess I should have looked at other lines of code besides the ones he pointed out; sorry about that. :( The only thing this intends to break, and then only for new-style classes, is the ability to have __copy__ be an instance variable (whose value should be a callable without arguments) -- it must be a method on the class. This is the same thing that I've done for all built-in operations (__add__, __getitem__ etc.). Presumably, this is the actual feature loss that Fredrik's describing; i.e. lack of per-instance __copy__ on new-style classes. That would make more sense. However, it also looks like it might have been introduced to fix the possibility that calling '__copy__' on a new-style class with a custom metaclass would result in ending up with an unbound method. (Similar to the metaconfusion issue being recently discussed for PEP 246.) Sorry, my head just exploded. :-( The issue is that for special attributes (like __copy__, __conform__, etc.) that do not have a corresponding type slot, using getattr() is not sufficient to obtain slot-like behavior. This is because 'aType.__special__' may refer to a __special__ intended for *instances* of 'aType', instead of the __special__ for aType. As Armin points out, the only way to fully emulate type slot behavior for unslotted special attributes is to perform a search of the __dict__ of each type in the MRO of the type of the object for which you wish to obtain the special attribute. So, in this specific case, __copy__ does not have a type slot, so it is impossible using getattr (or simple attribute access) to guarantee that you are retrieving the correct version of __copy__ in the presence of metaclasses. This is what Alex and I dubbed metaconfusion in discussion of the same issue for PEP 246's __adapt__ and __conform__ methods; until they have tp_adapt and tp_conform slots, they can have this same problem. Alex and I also just speculated that perhaps the stdlib should include a function that can do this, so that stdlib modules that define unslotted special attributes (such as __copy__) can ensure they work correctly in the presence of metaclasses. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
