On Thu, May 14, 2015 at 9:29 PM, Guido van Rossum <gu...@python.org> wrote:
> I expect you can make something that behaves like list by defining __mul__
> and __rmul__ and returning NotImplemented.
Hmm, it's fairly tricky, and part of the trick is that you can never
return NotImplemented (because you have to pretty much take over and
entirely replace the normal dispatch rules inside __mul__ and
__rmul__), but see attached for something I think should work.
So I guess this is just how Python's list, tuple, etc. work, and PyPy
and friends need to match...
-n
> On Thursday, May 14, 2015, Stefan Richthofer <stefan.richtho...@gmx.de>
> wrote:
>>
>> >>Should I be worried?
>>
>> You mean should *I* be worried ;)
>>
>> Stuff like this is highly relevant for JyNI, so thanks very much for
>> clarifying this
>> subtle behavior. It went onto my todo-list right now to ensure that JyNI
>> will emulate
>> this behavior as soon as I am done with gc-support. (Hopefully it will be
>> feasible,
>> but I can only tell in half a year or so since there are currently other
>> priorities.)
>> Still, this "essay" potentially will save me a lot of time.
>>
>> So, everybody please feel encouraged to post things like this as they come
>> up. Maybe
>> there could be kind of a pitfalls-page somewhere in the docs collecting
>> these things.
>>
>> Best
>>
>> Stefan
>>
>>
>> > Gesendet: Freitag, 15. Mai 2015 um 02:45 Uhr
>> > Von: "Nathaniel Smith" <n...@pobox.com>
>> > An: "Python Dev" <python-dev@python.org>
>> > Betreff: [Python-Dev] Python-versus-CPython question for __mul__
>> > dispatch
>> >
>> > Hi all,
>> >
>> > While attempting to clean up some of the more squamous aspects of
>> > numpy's operator dispatch code [1][2], I've encountered a situation
>> > where the semantics we want and are using are possible according to
>> > CPython-the-interpreter, but AFAICT ought not to be possible according
>> > to Python-the-language, i.e., it's not clear to me whether it's
>> > possible even in principle to implement an object that works the way
>> > numpy.ndarray does in any other interpreter. Which makes me a bit
>> > nervous, so I wanted to check if there was any ruling on this.
>> >
>> > Specifically, the quirk we are relying on is this: in CPython, if you do
>> >
>> > [1, 2] * my_object
>> >
>> > then my_object's __rmul__ gets called *before* list.__mul__,
>> > *regardless* of the inheritance relationship between list and
>> > type(my_object). This occurs as a side-effect of the weirdness
>> > involved in having both tp_as_number->nb_multiply and
>> > tp_as_sequence->sq_repeat in the C API -- when evaluating "a * b",
>> > CPython tries a's nb_multiply, then b's nb_multiply, then a's
>> > sq_repeat, then b's sq_repeat. Since list has an sq_repeat but not an
>> > nb_multiply, this means that my_object's nb_multiply gets called
>> > before any list method.
>> >
>> > Here's an example demonstrating how weird this is. list.__mul__ wants
>> > an integer, and by "integer" it means "any object with an __index__
>> > method". So here's a class that list is happy to be multiplied by --
>> > according to the ordinary rules for operator dispatch, in the example
>> > below Indexable.__mul__ and __rmul__ shouldn't even get a look-in:
>> >
>> > In [3]: class Indexable(object):
>> > ...: def __index__(self):
>> > ...: return 2
>> > ...:
>> >
>> > In [4]: [1, 2] * Indexable()
>> > Out[4]: [1, 2, 1, 2]
>> >
>> > But, if I add an __rmul__ method, then this actually wins:
>> >
>> > In [6]: class IndexableWithMul(object):
>> > ...: def __index__(self):
>> > ...: return 2
>> > ...: def __mul__(self, other):
>> > ...: return "indexable forward mul"
>> > ...: def __rmul__(self, other):
>> > ...: return "indexable reverse mul"
>> >
>> > In [7]: [1, 2] * IndexableWithMul()
>> > Out[7]: 'indexable reverse mul'
>> >
>> > In [8]: IndexableWithMul() * [1, 2]
>> > Out[8]: 'indexable forward mul'
>> >
>> > NumPy arrays, of course, correctly define both __index__ method (which
>> > raises an array on general arrays but coerces to int for arrays that
>> > contain exactly 1 integer), and also defines an nb_multiply slot which
>> > accepts lists and performs elementwise multiplication:
>> >
>> > In [9]: [1, 2] * np.array(2)
>> > Out[9]: array([2, 4])
>> >
>> > And that's all great! Just what we want. But the only reason this is
>> > possible, AFAICT, is that CPython 'list' is a weird type with
>> > undocumented behaviour that you can't actually define using pure
>> > Python code.
>> >
>> > Should I be worried?
>> >
>> > -n
>> >
>> > [1] https://github.com/numpy/numpy/pull/5864
>> > [2] https://github.com/numpy/numpy/issues/5844
>> >
>> > --
>> > Nathaniel J. Smith -- http://vorpus.org
>> > _______________________________________________
>> > Python-Dev mailing list
>> > Python-Dev@python.org
>> > https://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> > https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.de
>> >
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (on iPad)
--
Nathaniel J. Smith -- http://vorpus.org
# Attempt to implement in pure Python a Sequence class which behaves the same
# way as a CPython object that provides tp_as_sequence but not
# tp_as_number (like, for example, the built-in types list, tuple, ...).
def _proper_subinstance(a, b):
ta = type(a)
tb = type(b)
return ta is not tb and issubclass(ta, tb)
class Sequence(object):
### -- Overload these --
### They should NOT return NotImplemented
def _sq_repeat(self, count):
raise NotImplementedError
def _sq_concat(self, other):
raise NotImplementedError
### -- Leave the rest alone --
# Rules for 'a * b':
# a is a sequence and b is not:
# call b.__rmul__
# then fall back on a.__mul__
# a is not a sequence but b is:
# call a.__mul__
# then fall back on b.__rmul__
# a and b are both sequences:
# call a.__mul__
# The class hierarchy is irrelevant to all of these cases. Of course,
# Python thinks that the class hierarchy is relevant, so we have to pay
# attention to it and counteract its attempts to use its idea of the
# proper dispatch order.
def __do_sq_repeat(self, other):
if hasattr(other, "__index__"):
return self._sq_repeat(other.__index__())
else:
# Failed, and we *don't* want to unwind and return NotImplemented
raise TypeError("can't multiply sequence by non-int of type %s"
% (type(other).__name__))
def __mul__(self, other):
if (hasattr(other, "__rmul__")
and not isinstance(other, Sequence)
and not _proper_subinstance(other, self)):
result = other.__rmul__(self):
if result is not NotImplemented:
return result
return self.__do_sq_repeat(other)
def __rmul__(self, other):
if hasattr(other, "__mul__") and _proper_subinstance(self, other):
result = other.__mul__(self)
if result is not NotImplemented:
return result
return self.__do_sq_repeat(other)
# Rules for 'a + b':
# a is a sequence and b is not:
# call b.__radd__
# then fall back on a.__add__
# a is not a sequence and b is:
# call a.__add__
# a and b are both sequences:
# call a.__add__
# Again, we don't care about the class hierarchy, but Python does.
def __add__(self, other):
if (hasattr(other, "__radd__")
and not isinstance(other, Sequence)
and not _proper_subinstance(other, self)):
result = other.__radd__(self)
if result is not NotImplemented:
return result
return self._sq_concat(result)
# __radd__ intentionally omitted
class list(Sequence):
# ...
pass
class tuple(Sequence):
# ...
pass
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
