Just to be clear, this is about NotImplemented, not NotImplementedError.
tl;dr When a binary operation fails, should an exception be raised or
NotImplemented returned?
When a binary operation in Python is attempted, there are two possibilities:
- it can work
- it can't work
The main reason [1] that it can't work is that the two operands are of different types, and the first type does not know
how to deal with the second type.
The question then becomes: how does the first type tell Python that it cannot perform the requested operation? The most
obvious answer is to raise an exception, and TypeError is a good candidate. The problem with the exception raising
approach is that once an exception is raised, Python doesn't try anything else to make the operation work.
What's wrong with that? Well, the second type might know how to perform the operation, and in fact that is why we have
the reflected special methods, such as __radd__ and __rmod__ -- but if the first type raises an exception the __rxxx__
methods will not be tried.
Okay, how can the first type tell Python that it cannot do what is requested, but to go ahead and check with the second
type to see if it does? That is where NotImplemented comes in -- if a special method (and only a special method)
returns NotImplemented then Python will check to see if there is anything else it can do to make the operation succeed;
if all attempts return NotImplemented, then Python itself will raise an appropriate exception [2].
In an effort to see how often NotImplemented is currently being returned I crafted a test script [3] to test the types
bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, deque, and OrderedDict with the operations for
__add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__, __ilshift__, __imod__, __imul__, __ior__, __ipow__,
__irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__, __mul__, __or__, __pow__, __rshift__, __sub__,
__truediv__, and __xor__.
Here are the results of the 275 tests:
--------------------------------------------------------------------------------
testing control...
ipow -- Exception <unsupported operand type(s) for ** or pow(): 'Control' and
'subtype'> raised
errors in Control -- misunderstanding or bug?
testing types against a foreign class
iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute 'items'>
raised instead of TypeError
iand(Counter()) -- NotImplemented not returned, TypeError not raised
ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute 'items'>
raised instead of TypeError
isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute 'items'>
raised instead of TypeError
testing types against a subclass
mod(str()) -- NotImplemented not returned, TypeError not raised
iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'> raised
(should have worked)
iand(Counter()) -- NotImplemented not returned, TypeError not raised
ior(Counter()) -- Exception <'subtype' object has no attribute 'items'> raised
(should have worked)
isub(Counter()) -- Exception <'subtype' object has no attribute 'items'> raised
(should have worked)
--------------------------------------------------------------------------------
Two observations:
- __ipow__ doesn't seem to behave properly in the 3.x line (that error
doesn't show up when testing against 2.7)
- Counter should be returning NotImplemented instead of raising an
AttributeError, for three reasons [4]:
- a TypeError is more appropriate
- subclasses /cannot/ work with the current implementation
- __iand__ is currently a silent failure if the Counter is empty, and the
other operand should trigger a failure
Back to the main point...
So, if my understanding is correct:
- NotImplemented is used to signal Python that the requested operation could
not be performed
- it should be used by the binary special methods to signal type mismatch
failure, so any subclass gets a chance to work.
Is my understanding correct? Is this already in the docs somewhere, and I just
missed it?
--
~Ethan~
[1] at least, it's the main reason in my code
[2] usually a TypeError, stating either that the operation is not supported, or
the types are unorderable
[3] test script at the end
[4] https://bugs.python.org/issue22766 [returning NotImplemented was rejected]
-- 8<
----------------------------------------------------------------------------
from collections import Counter, defaultdict, deque, OrderedDict
from fractions import Fraction
from decimal import Decimal
from enum import Enum
import operator
import sys
py_ver = sys.version_info[:2]
types = (
bytes, bytearray, str, dict, list, tuple,
Enum, Counter, defaultdict, deque, OrderedDict,
)
numeric_types = int, float, Decimal, Fraction
operators = (
'__add__', '__and__', '__floordiv__',
'__iadd__', '__iand__', '__ifloordiv__', '__ilshift__',
'__imod__', '__imul__', '__ior__', '__ipow__',
'__irshift__', '__isub__', '__itruediv__', '__ixor__',
'__lshift__', '__mod__', '__mul__',
'__or__', '__pow__', '__rshift__', '__sub__', '__truediv__',
'__xor__',
)
if py_ver >= (3, 0):
operators += ('__gt__', '__ge__', '__le__','__lt__')
ordered_reflections = {
'__le__': '__ge__',
'__lt__': '__gt__',
'__ge__': '__le__',
'__gt__': '__lt__',
}
# helpers
class SomeOtherClass:
""""
used to test behavior when a different type is passed in to the
special methods
"""
def __repr__(self):
return 'SomeOtherClass'
some_other_class = SomeOtherClass()
class MainClassHandled(Exception):
"""
called by base class if both operands are of type base class
"""
class SubClassCalled(Exception):
"""
called by reflected operations for testing
"""
def create_control(test_op):
def _any(self, other):
if not type(other) is self.__class__:
return NotImplemented
raise MainClassHandled
class Control:
"returns NotImplemented when other object is not supported"
_any.__name__ = op
setattr(Control, test_op, _any)
return Control()
def create_subtype(test_op, base_class=object):
def _any(*a):
global subclass_called
subclass_called = True
raise SubClassCalled
class subtype(base_class):
__add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = _any
__mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = _any
__and__ = __xor__ = __or__ = _any
__radd__ = __rsub__ = __rmul__ = __rtruediv__ = __rfloordiv__ = _any
__rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = __rrshift__ = _any
__rand__ = __rxor__ = __ror__ = _any
__le__ = __lt__ = __gt__ = __ge__ = _any
if issubclass(subtype, (bytes, bytearray)):
value = b'hello'
elif issubclass(subtype, str):
value = 'goodbye'
elif issubclass(subtype, (list, tuple)):
value = (1, 2, 3)
elif issubclass(subtype, (int, float, Decimal, Fraction)):
value = 42
else:
# ignore value
return subtype()
return subtype(value)
# test exceptions
# control against some other class
print('testing control...\n')
errors = False
for op in operators:
control = create_control(op)
op = getattr(operator, op)
try:
op(control, some_other_class)
except TypeError:
# the end result of no method existing, or each method called returning
# NotImplemented because it does not know how to perform the requested
# operation between the two types
pass
except Exception as exc:
errors = True
print('%s(%s()) -- Exception <%s> raised instead of TypeError' %
(op.__name__, test_type.__name__, exc))
else:
errors = True
print('Control -- TypeError not raised for op %r' % op)
if errors:
print('errors in Control -- misunderstanding or bug?\n')
# control against a subclass
errors = False
for op in operators:
subclass_called = False
control = create_control(op)
subtype = create_subtype(op, control.__class__)
op = getattr(operator, op)
try:
op(control, subtype)
except SubClassCalled:
# if the control class properly signals that it doesn't know how to
# perform the operation, of if Python notices that a reflected
# operation exists, we get here (which is good)
pass
except MainClassHandled:
errors = True
print('Control did not yield to subclass for op %r' % op)
except Exception as exc:
if subclass_called:
# exception was subverted to something more appropriate (like
# unorderable types)
pass
errors = True
print('%s -- Exception <%s> raised' %
(op.__name__, exc))
else:
errors = True
print('Control -- op %r appears to have succeeded (it should not have)'
% op)
if errors:
print('errors in Control -- misunderstanding or bug?\n')
# tests
print('testing types against a foreign class\n')
for test_type in types + numeric_types:
errors = False
for op in operators:
op = getattr(operator, op)
try:
op(test_type(), some_other_class)
except TypeError:
pass
except Exception as exc:
errors = True
print('%s(%s()) -- Exception <%s> raised instead of TypeError' %
(op.__name__, test_type.__name__, exc))
else:
print('%s(%s()) -- NotImplemented not returned, TypeError not
raised' %
(op.__name__, test_type.__name__))
if errors:
print()
print()
# test subclasses
print('testing types against a subclass\n')
for test_type in types:
errors = False
for op in operators:
subclass_called = False
if not test_type.__dict__.get(op):
continue
subclass = create_subtype(op, test_type)
op = getattr(operator, op)
try:
if test_type is str:
op('%s', subtype)
else:
op(test_type(), subtype)
except SubClassCalled:
# expected, ignore
pass
except Exception as exc:
if subclass_called:
# exception raised by subclass was changed
pass
errors = True
print('%s(%s()) -- Exception <%s> raised (should have worked)' %
(op.__name__, test_type.__name__, exc))
else:
errors = True
print('%s(%s()) -- NotImplemented not returned, TypeError not
raised' %
(op.__name__, test_type.__name__))
if errors:
print()
for test_type in numeric_types:
errors = False
for op in operators:
subclass_called = False
if not test_type.__dict__.get(op):
continue
subtype = create_subtype(op, test_type)
op = getattr(operator, op)
try:
op(test_type(), subtype)
except SubClassCalled:
# expected, ignore
pass
except Exception as exc:
if subclass_called:
# exception raised by subclass was changed
pass
errors = True
print('%s(%s()) -- Exception <%s> raised (should have worked)' %
(op.__name__, test_type.__name__, exc))
else:
errors = True
print('%s(%s)) -- NotImplemented not returned' %
(op.__name__, test_type.__name__))
if errors:
print()
-- 8<
----------------------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
