On Mon, 3 Nov 2014 08:48:07 -0800 Guido van Rossum <gu...@python.org> wrote: > Gotta be brief, but NotImplemented is for all binary ops.
Even in-place ops? Regards Antoine. > Power may be an > exception because it's ternary? > On Nov 3, 2014 8:08 AM, "Brett Cannon" <br...@python.org> wrote: > > > > > > > On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman <et...@stoneleaf.us> wrote: > > > >> Just to be clear, this is about NotImplemented, not NotImplementedError. > >> > >> tl;dr When a binary operation fails, should an exception be raised or > >> NotImplemented returned? > >> > > > > The docs for NotImplemented suggest it's only for rich comparison methods > > and not all binary operators: > > https://docs.python.org/3/library/constants.html#NotImplemented . But > > then had I not read that I would have said all binary operator methods > > should return NotImplemented when the types are incompatible. > > > > -Brett > > > > > >> > >> > >> When a binary operation in Python is attempted, there are two > >> possibilities: > >> > >> - it can work > >> - it can't work > >> > >> The main reason [1] that it can't work is that the two operands are of > >> different types, and the first type does not know > >> how to deal with the second type. > >> > >> The question then becomes: how does the first type tell Python that it > >> cannot perform the requested operation? The most > >> obvious answer is to raise an exception, and TypeError is a good > >> candidate. The problem with the exception raising > >> approach is that once an exception is raised, Python doesn't try anything > >> else to make the operation work. > >> > >> What's wrong with that? Well, the second type might know how to perform > >> the operation, and in fact that is why we have > >> the reflected special methods, such as __radd__ and __rmod__ -- but if > >> the first type raises an exception the __rxxx__ > >> methods will not be tried. > >> > >> Okay, how can the first type tell Python that it cannot do what is > >> requested, but to go ahead and check with the second > >> type to see if it does? That is where NotImplemented comes in -- if a > >> special method (and only a special method) > >> returns NotImplemented then Python will check to see if there is anything > >> else it can do to make the operation succeed; > >> if all attempts return NotImplemented, then Python itself will raise an > >> appropriate exception [2]. > >> > >> In an effort to see how often NotImplemented is currently being returned > >> I crafted a test script [3] to test the types > >> bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, > >> deque, and OrderedDict with the operations for > >> __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__, > >> __ilshift__, __imod__, __imul__, __ior__, __ipow__, > >> __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__, > >> __mul__, __or__, __pow__, __rshift__, __sub__, > >> __truediv__, and __xor__. > >> > >> Here are the results of the 275 tests: > >> ------------------------------------------------------------ > >> -------------------- > >> testing control... > >> > >> ipow -- Exception <unsupported operand type(s) for ** or pow(): 'Control' > >> and 'subtype'> raised > >> errors in Control -- misunderstanding or bug? > >> > >> testing types against a foreign class > >> > >> iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute > >> 'items'> raised instead of TypeError > >> iand(Counter()) -- NotImplemented not returned, TypeError not raised > >> ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute > >> 'items'> raised instead of TypeError > >> isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute > >> 'items'> raised instead of TypeError > >> > >> > >> testing types against a subclass > >> > >> mod(str()) -- NotImplemented not returned, TypeError not raised > >> > >> iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'> > >> raised (should have worked) > >> iand(Counter()) -- NotImplemented not returned, TypeError not raised > >> ior(Counter()) -- Exception <'subtype' object has no attribute 'items'> > >> raised (should have worked) > >> isub(Counter()) -- Exception <'subtype' object has no attribute 'items'> > >> raised (should have worked) > >> ------------------------------------------------------------ > >> -------------------- > >> > >> Two observations: > >> > >> - __ipow__ doesn't seem to behave properly in the 3.x line (that error > >> doesn't show up when testing against 2.7) > >> > >> - Counter should be returning NotImplemented instead of raising an > >> AttributeError, for three reasons [4]: > >> - a TypeError is more appropriate > >> - subclasses /cannot/ work with the current implementation > >> - __iand__ is currently a silent failure if the Counter is empty, > >> and the other operand should trigger a failure > >> > >> Back to the main point... > >> > >> So, if my understanding is correct: > >> > >> - NotImplemented is used to signal Python that the requested operation > >> could not be performed > >> - it should be used by the binary special methods to signal type > >> mismatch failure, so any subclass gets a chance to work. > >> > >> Is my understanding correct? Is this already in the docs somewhere, and > >> I just missed it? > >> > >> -- > >> ~Ethan~ > >> > >> [1] at least, it's the main reason in my code > >> [2] usually a TypeError, stating either that the operation is not > >> supported, or the types are unorderable > >> [3] test script at the end > >> [4] https://bugs.python.org/issue22766 [returning NotImplemented was > >> rejected] > >> > >> -- 8< ------------------------------------------------------------ > >> ---------------- > >> from collections import Counter, defaultdict, deque, OrderedDict > >> from fractions import Fraction > >> from decimal import Decimal > >> from enum import Enum > >> import operator > >> import sys > >> > >> py_ver = sys.version_info[:2] > >> > >> types = ( > >> bytes, bytearray, str, dict, list, tuple, > >> Enum, Counter, defaultdict, deque, OrderedDict, > >> ) > >> numeric_types = int, float, Decimal, Fraction > >> > >> operators = ( > >> '__add__', '__and__', '__floordiv__', > >> '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__', > >> '__imod__', '__imul__', '__ior__', '__ipow__', > >> '__irshift__', '__isub__', '__itruediv__', '__ixor__', > >> '__lshift__', '__mod__', '__mul__', > >> '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__', > >> '__xor__', > >> ) > >> > >> if py_ver >= (3, 0): > >> operators += ('__gt__', '__ge__', '__le__','__lt__') > >> > >> ordered_reflections = { > >> '__le__': '__ge__', > >> '__lt__': '__gt__', > >> '__ge__': '__le__', > >> '__gt__': '__lt__', > >> } > >> > >> > >> # helpers > >> > >> class SomeOtherClass: > >> """" > >> used to test behavior when a different type is passed in to the > >> special methods > >> """ > >> def __repr__(self): > >> return 'SomeOtherClass' > >> some_other_class = SomeOtherClass() > >> > >> class MainClassHandled(Exception): > >> """ > >> called by base class if both operands are of type base class > >> """ > >> > >> class SubClassCalled(Exception): > >> """ > >> called by reflected operations for testing > >> """ > >> > >> def create_control(test_op): > >> def _any(self, other): > >> if not type(other) is self.__class__: > >> return NotImplemented > >> raise MainClassHandled > >> class Control: > >> "returns NotImplemented when other object is not supported" > >> _any.__name__ = op > >> setattr(Control, test_op, _any) > >> return Control() > >> > >> def create_subtype(test_op, base_class=object): > >> def _any(*a): > >> global subclass_called > >> subclass_called = True > >> raise SubClassCalled > >> class subtype(base_class): > >> __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = _any > >> __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = _any > >> __and__ = __xor__ = __or__ = _any > >> __radd__ = __rsub__ = __rmul__ = __rtruediv__ = __rfloordiv__ = > >> _any > >> __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = __rrshift__ = > >> _any > >> __rand__ = __rxor__ = __ror__ = _any > >> __le__ = __lt__ = __gt__ = __ge__ = _any > >> if issubclass(subtype, (bytes, bytearray)): > >> value = b'hello' > >> elif issubclass(subtype, str): > >> value = 'goodbye' > >> elif issubclass(subtype, (list, tuple)): > >> value = (1, 2, 3) > >> elif issubclass(subtype, (int, float, Decimal, Fraction)): > >> value = 42 > >> else: > >> # ignore value > >> return subtype() > >> return subtype(value) > >> > >> > >> # test exceptions > >> > >> # control against some other class > >> print('testing control...\n') > >> errors = False > >> for op in operators: > >> control = create_control(op) > >> op = getattr(operator, op) > >> try: > >> op(control, some_other_class) > >> except TypeError: > >> # the end result of no method existing, or each method called > >> returning > >> # NotImplemented because it does not know how to perform the > >> requested > >> # operation between the two types > >> pass > >> except Exception as exc: > >> errors = True > >> print('%s(%s()) -- Exception <%s> raised instead of TypeError' % > >> (op.__name__, test_type.__name__, exc)) > >> else: > >> errors = True > >> print('Control -- TypeError not raised for op %r' % op) > >> if errors: > >> print('errors in Control -- misunderstanding or bug?\n') > >> > >> # control against a subclass > >> errors = False > >> for op in operators: > >> subclass_called = False > >> control = create_control(op) > >> subtype = create_subtype(op, control.__class__) > >> op = getattr(operator, op) > >> try: > >> op(control, subtype) > >> except SubClassCalled: > >> # if the control class properly signals that it doesn't know how > >> to > >> # perform the operation, of if Python notices that a reflected > >> # operation exists, we get here (which is good) > >> pass > >> except MainClassHandled: > >> errors = True > >> print('Control did not yield to subclass for op %r' % op) > >> except Exception as exc: > >> if subclass_called: > >> # exception was subverted to something more appropriate (like > >> # unorderable types) > >> pass > >> errors = True > >> print('%s -- Exception <%s> raised' % > >> (op.__name__, exc)) > >> else: > >> errors = True > >> print('Control -- op %r appears to have succeeded (it should not > >> have)' % op) > >> if errors: > >> print('errors in Control -- misunderstanding or bug?\n') > >> > >> > >> # tests > >> print('testing types against a foreign class\n') > >> for test_type in types + numeric_types: > >> errors = False > >> for op in operators: > >> op = getattr(operator, op) > >> try: > >> op(test_type(), some_other_class) > >> except TypeError: > >> pass > >> except Exception as exc: > >> errors = True > >> print('%s(%s()) -- Exception <%s> raised instead of > >> TypeError' % > >> (op.__name__, test_type.__name__, exc)) > >> else: > >> print('%s(%s()) -- NotImplemented not returned, TypeError > >> not raised' % > >> (op.__name__, test_type.__name__)) > >> if errors: > >> print() > >> > >> print() > >> > >> # test subclasses > >> print('testing types against a subclass\n') > >> for test_type in types: > >> errors = False > >> for op in operators: > >> subclass_called = False > >> if not test_type.__dict__.get(op): > >> continue > >> subclass = create_subtype(op, test_type) > >> op = getattr(operator, op) > >> try: > >> if test_type is str: > >> op('%s', subtype) > >> else: > >> op(test_type(), subtype) > >> except SubClassCalled: > >> # expected, ignore > >> pass > >> except Exception as exc: > >> if subclass_called: > >> # exception raised by subclass was changed > >> pass > >> errors = True > >> print('%s(%s()) -- Exception <%s> raised (should have > >> worked)' % > >> (op.__name__, test_type.__name__, exc)) > >> else: > >> errors = True > >> print('%s(%s()) -- NotImplemented not returned, TypeError > >> not raised' % > >> (op.__name__, test_type.__name__)) > >> if errors: > >> print() > >> for test_type in numeric_types: > >> errors = False > >> for op in operators: > >> subclass_called = False > >> if not test_type.__dict__.get(op): > >> continue > >> subtype = create_subtype(op, test_type) > >> op = getattr(operator, op) > >> try: > >> op(test_type(), subtype) > >> except SubClassCalled: > >> # expected, ignore > >> pass > >> except Exception as exc: > >> if subclass_called: > >> # exception raised by subclass was changed > >> pass > >> errors = True > >> print('%s(%s()) -- Exception <%s> raised (should have > >> worked)' % > >> (op.__name__, test_type.__name__, exc)) > >> else: > >> errors = True > >> print('%s(%s)) -- NotImplemented not returned' % > >> (op.__name__, test_type.__name__)) > >> if errors: > >> print() > >> -- 8< ------------------------------------------------------------ > >> ---------------- > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev@python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > >> brett%40python.org > >> > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev@python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
