Re: Tuples and immutability
On Tue, 11 Mar 2014 23:25:19 -0400, Terry Reedy wrote: On 3/11/2014 10:01 PM, Rick Johnson wrote: On Thursday, February 27, 2014 4:18:01 PM UTC-6, Ian wrote: x += y is meant to be equivalent, except possibly in-place and more efficient, than x = x + y. The manual actually says An augmented assignment expression like x += 1 can be rewritten as x = x + 1 to achieve a similar, but not exactly equal effect. In the augmented version, x is only evaluated once. Also, when possible, the actual operation is performed in-place, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead. In an ideal world, the speed of these two codes should be the same, Nope, 'similar' is not 'equivalent'. Evaluating x twice instead of once and possibly allocating a new object versus not take extra time. In a statement like x.y.z[3*n+m] += 1, calculating the target dominates the time to increment, so this form should be nearly twice as fast. Excellent point Terry! I always forget that the target of an augmented assignment may not be a simple name like x but can be an arbitrary complex reference, anything that is a legal assignment target. Because += is documented as only evaluating the expression once it can behave quite differently to the `spam = spam + 1` case. Evaluating the right hand side may have side- effects that change what the left hand side evaluates to. This is not the case with the augmented assignment. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On 03/11/2014 08:25 PM, Terry Reedy wrote: On 3/11/2014 10:01 PM, Rick Johnson wrote: On Thursday, February 27, 2014 4:18:01 PM UTC-6, Ian wrote: x += y is meant to be equivalent, except possibly in-place and more efficient, than x = x + y. The manual actually says An augmented assignment expression like x += 1 can be rewritten as x = x + 1 to achieve a similar, but not exactly equal effect. In the augmented version, x is only evaluated once. Also, when possible, the actual operation is performed in-place, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead. ... and reassigned to the target. :) (If it doesn't say that last part, it should.) -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Op 12-03-14 07:28, Steven D'Aprano schreef:
On Tue, 11 Mar 2014 23:25:19 -0400, Terry Reedy wrote:
Nope, 'similar' is not 'equivalent'. Evaluating x twice instead of once
and possibly allocating a new object versus not take extra time. In a
statement like x.y.z[3*n+m] += 1, calculating the target dominates the
time to increment, so this form should be nearly twice as fast.
Excellent point Terry!
I always forget that the target of an augmented assignment may not be a
simple name like x but can be an arbitrary complex reference, anything
that is a legal assignment target. Because += is documented as only
evaluating the expression once it can behave quite differently to the
`spam = spam + 1` case. Evaluating the right hand side may have side-
effects that change what the left hand side evaluates to. This is not the
case with the augmented assignment.
The documentation is wrong at that point as the following code illustrates.
| import sys
| write = sys.stdout.write
|
| class logdict:
| def __init__(self):
| self.lst = {}
|
| def __setitem__(self, key, value):
| write('[%s] = %s\n' % (key, value))
| self.lst[key] = value
|
| def __getitem__(self, key):
| value = self.lst[key]
| write('[%s] = %s\n' % (key, value))
| return value
|
| tab = logdict()
| tab['key'] = 'value'
| tab['key'] += ' with extra tail'
| write('\n')
| tab = logdict()
| tab['key'] = 'value'
| tab['key'] = tab['key'] + ' with extra tail'
If you run this code, you get the following result:
| [key] = value
| [key] = value
| [key] = value with extra tail
|
| [key] = value
| [key] = value
| [key] = value with extra tail
As you can see there is no difference here in the evaluations done
between using
| tab['key'] += ' with extra tail'
or
| tab['key'] = tab['key'] + ' with extra tail'
--
https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Wed, Mar 12, 2014 at 12:28 AM, Steven D'Aprano st...@pearwood.info wrote: On Tue, 11 Mar 2014 23:25:19 -0400, Terry Reedy wrote: Nope, 'similar' is not 'equivalent'. Evaluating x twice instead of once and possibly allocating a new object versus not take extra time. In a statement like x.y.z[3*n+m] += 1, calculating the target dominates the time to increment, so this form should be nearly twice as fast. Excellent point Terry! I always forget that the target of an augmented assignment may not be a simple name like x but can be an arbitrary complex reference, anything that is a legal assignment target. Because += is documented as only evaluating the expression once it can behave quite differently to the `spam = spam + 1` case. Evaluating the right hand side may have side- effects that change what the left hand side evaluates to. This is not the case with the augmented assignment. Of course one could also do: z = x.y.z i = 3*n+m z[i] = z[i] + 1 which reduces the duplicated work down to storing and loading a couple of locals, and also prevents side-effects from affecting the LHS evaluation. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Wed, Mar 12, 2014 at 3:39 AM, Antoon Pardon antoon.par...@rece.vub.ac.be wrote: The documentation is wrong at that point as the following code illustrates. Either way it still has to do a getitem and a setitem, but if you have a more nested structure then the extra getitems are not repeated. For example, using your logdict class: tab = logdict() tab[1] = logdict() [1] = __main__.logdict object at 0x02A2E430 tab[1][2] = logdict() [1] = __main__.logdict object at 0x02A2E430 [2] = __main__.logdict object at 0x02A2EB10 tab[1][2][3] = ['value'] [1] = __main__.logdict object at 0x02A2E430 [2] = __main__.logdict object at 0x02A2EB10 [3] = ['value'] tab[1][2][3] += [' with extra tail'] [1] = __main__.logdict object at 0x02A2E430 [2] = __main__.logdict object at 0x02A2EB10 [3] = ['value'] [3] = ['value', ' with extra tail'] versus: tab[1][2][3] = ['value'] [1] = __main__.logdict object at 0x02A2E430 [2] = __main__.logdict object at 0x02A2EB10 [3] = ['value'] tab[1][2][3] = tab[1][2][3] + [' with extra tail'] [1] = __main__.logdict object at 0x02A2E430 [2] = __main__.logdict object at 0x02A2EB10 [3] = ['value'] [1] = __main__.logdict object at 0x02A2E430 [2] = __main__.logdict object at 0x02A2EB10 [3] = ['value', ' with extra tail'] As you can see the += version does two fewer getitem calls in this case. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On 12 March 2014 03:25, Terry Reedy tjre...@udel.edu wrote:
On 3/11/2014 10:01 PM, Rick Johnson wrote:
On Thursday, February 27, 2014 4:18:01 PM UTC-6, Ian wrote:
x += y is meant to be equivalent, except possibly in-place and
more efficient, than x = x + y.
The manual actually says An augmented assignment expression like x += 1 can
be rewritten as x = x + 1 to achieve a similar, but not exactly equal
effect. In the augmented version, x is only evaluated once. Also, when
possible, the actual operation is performed in-place, meaning that rather
than creating a new object and assigning that to the target, the old object
is modified instead.
In an ideal world, the speed of these two codes should be the same,
Nope, 'similar' is not 'equivalent'. Evaluating x twice instead of once and
possibly allocating a new object versus not take extra time. In a statement
like x.y.z[3*n+m] += 1, calculating the target dominates the time to
increment, so this form should be nearly twice as fast.
An example where the result is semantically different:
from numpy import array
a = array([1, 2, 3], dtype=int)
a
array([1, 2, 3])
a + 1.1
array([ 2.1, 3.1, 4.1])
a += 1.1
a
array([2, 3, 4])
The reason is that numpy arrays are both mutable and have statically
determined elementary data type:
(a + 1.1).dtype
dtype('float64')
a.dtype
dtype('int64')
Oscar
--
https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Op 12-03-14 10:51, Ian Kelly schreef: On Wed, Mar 12, 2014 at 3:39 AM, Antoon Pardon antoon.par...@rece.vub.ac.be wrote: The documentation is wrong at that point as the following code illustrates. Either way it still has to do a getitem and a setitem, but if you have a more nested structure then the extra getitems are not repeated. For example, using your logdict class: Sure, but the documentation doesn't say for sufficiently nested structures some evaluations are not repeated. Take the following example. | tab['key'] = [1] | tab['key'] += [2] Now let us rewrite that second statment in two different ways. | tab['key'] = tab['key'] + [2] or | tmp = tab['key'] | tmp += [2] Now which of these two rewritings is closer to the augmented concatenation? A natural reading of the documentation would conclude the second one, because in that case we only need to evaluate tab['key'] once as righthand sided. However it turns out the augmented concantenation is closer to the first rewriting here, evaluating tab['key'] twice once a lefthand sided and once as right hand sided, which IMO will surprise people that rely on the documentation. -- Antoon Pardon -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Tue, 11 Mar 2014 17:06:43 -0600, Ian Kelly wrote: On Tue, Mar 11, 2014 at 10:46 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: There are a number of possible solutions. One possibility would be to copy the Circle as an Ellipse and return the new object instead of mutating it. Then you have the situation where, given a mutable object x that satisfies isinstance(x, Ellipse), the stretch method *may* be able to update the object in-place, or it *may* not. That is a really lousy design. Of course we are free to create classes with ugly, inconsistent, stupid or unworkable APIs if we like. Python won't stop us: That's true but irrelevant to my point, which was to counter the assertion that mutable types can always be assumed to be able to perform operations in-place. Always? Not so fast. This is Python. We have freedom to abuse nearly everything, and if you want to shoot yourself in the foot, you can. With the exception of a handful of things which cannot be overridden (e.g. None, numeric literals, syntax) you cannot strictly assume anything about anything. Python does not enforce that iterators raise StopIteration when empty, or that indexing beyond the boundaries of a sequence raises IndexError, or that __setitem__ of a mapping sets the key and value, or that __len__ returns a length. Augmented assignment is no different. The docs describe the intention of the designers and the behaviour of the classes that they control, so with standard built-in classes like int, str, list, tuple etc. you can safely assume that mutable types will perform the operation in place and immutable types won't, but with arbitrary types from some arbitrarily eccentric or twisted programmer, who knows what it will do? -- Steven D'Aprano http://import-that.dreamwidth.org/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Wed, Mar 12, 2014 at 5:20 PM, Steven D'Aprano
steve+comp.lang.pyt...@pearwood.info wrote:
On Tue, 11 Mar 2014 17:06:43 -0600, Ian Kelly wrote:
That's true but irrelevant to my point, which was to counter the
assertion that mutable types can always be assumed to be able to perform
operations in-place.
Always? Not so fast.
This is Python. We have freedom to abuse nearly everything, and if you
want to shoot yourself in the foot, you can. With the exception of a
handful of things which cannot be overridden (e.g. None, numeric
literals, syntax) you cannot strictly assume anything about anything.
Python does not enforce that iterators raise StopIteration when empty, or
that indexing beyond the boundaries of a sequence raises IndexError, or
that __setitem__ of a mapping sets the key and value, or that __len__
returns a length.
Thank you; you've stated my point more succinctly than I did.
Augmented assignment is no different. The docs describe the intention of
the designers and the behaviour of the classes that they control, so with
standard built-in classes like int, str, list, tuple etc. you can safely
assume that mutable types will perform the operation in place and
immutable types won't, but with arbitrary types from some arbitrarily
eccentric or twisted programmer, who knows what it will do?
This got me curious about how consistent the standard library is about
this exactly, so I did some grepping. In the standard library there
are 5 mutable types that support concatenation that I was able to
find: list, deque, array, bytearray, and Counter. There are none that
support addition, which I find interesting in that the language
provides hooks for in-place addition but never uses them itself.
All of the classes above appear to follow the rule that if you can
concatenate an operand, you can in-place concatenate the same operand.
The converse however does not hold: list.__iadd__ and
Counter.__iadd__ are both more permissive in what types they will
accept than their __add__ counterparts, and especially interesting to
me is that deque implements __iadd__ but does not implement __add__ at
all. This last in particular seems to support the assertion that +=
should be viewed more as a shorthand for an in-place operation, less
as an equivalent for x = x + y.
l = [1,2,3]
l + (4,5,6)
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: can only concatenate list (not tuple) to list
l += (4,5,6)
l
[1, 2, 3, 4, 5, 6]
c = collections.Counter('mississippi')
c + collections.Counter('alabama')
Counter({'s': 4, 'a': 4, 'i': 4, 'p': 2, 'm': 2, 'b': 1, 'l': 1})
c + dict({'a': 4, 'l': 1, 'b': 1, 'm': 1})
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: unsupported operand type(s) for +: 'Counter' and 'dict'
c += dict({'a': 4, 'l': 1, 'b': 1, 'm': 1})
c
Counter({'s': 4, 'a': 4, 'i': 4, 'p': 2, 'm': 2, 'b': 1, 'l': 1})
d = collections.deque([1,2,3])
d += [4,5,6]
d
deque([1, 2, 3, 4, 5, 6])
d + [7,8,9]
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: unsupported operand type(s) for +: 'collections.deque' and 'list'
d.__add__
Traceback (most recent call last):
File stdin, line 1, in module
AttributeError: 'collections.deque' object has no attribute '__add__'
--
https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On 3/12/2014 9:35 PM, Ian Kelly wrote:
On Wed, Mar 12, 2014 at 5:20 PM, Steven D'Aprano
steve+comp.lang.pyt...@pearwood.info wrote:
On Tue, 11 Mar 2014 17:06:43 -0600, Ian Kelly wrote:
That's true but irrelevant to my point, which was to counter the
assertion that mutable types can always be assumed to be able to perform
operations in-place.
Always? Not so fast.
This is Python. We have freedom to abuse nearly everything, and if you
want to shoot yourself in the foot, you can. With the exception of a
handful of things which cannot be overridden (e.g. None, numeric
literals, syntax) you cannot strictly assume anything about anything.
Python does not enforce that iterators raise StopIteration when empty, or
that indexing beyond the boundaries of a sequence raises IndexError, or
that __setitem__ of a mapping sets the key and value, or that __len__
returns a length.
Thank you; you've stated my point more succinctly than I did.
Augmented assignment is no different. The docs describe the intention of
the designers and the behaviour of the classes that they control, so with
standard built-in classes like int, str, list, tuple etc. you can safely
assume that mutable types will perform the operation in place and
immutable types won't, but with arbitrary types from some arbitrarily
eccentric or twisted programmer, who knows what it will do?
This got me curious about how consistent the standard library is about
this exactly, so I did some grepping. In the standard library there
are 5 mutable types that support concatenation that I was able to
find: list, deque, array, bytearray, and Counter. There are none that
support addition, which I find interesting in that the language
provides hooks for in-place addition but never uses them itself.
All of the classes above appear to follow the rule that if you can
concatenate an operand, you can in-place concatenate the same operand.
The converse however does not hold: list.__iadd__ and
Counter.__iadd__ are both more permissive in what types they will
accept than their __add__ counterparts, and especially interesting to
me is that deque implements __iadd__ but does not implement __add__ at
all. This last in particular seems to support the assertion that +=
should be viewed more as a shorthand for an in-place operation, less
as an equivalent for x = x + y.
l = [1,2,3]
l + (4,5,6)
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: can only concatenate list (not tuple) to list
l += (4,5,6)
l
[1, 2, 3, 4, 5, 6]
Like it or not, one should actually think of 'somelist += iterable' as
equivalent to 'somelist.extend(iterable)'. Without looking at the C
code, I suspect that the latter is the internal implementation.
Collections.deque also has .extend. Collections.Counter has .update and
that is += seems to be doing.
c = collections.Counter('mississippi')
c + collections.Counter('alabama')
Counter({'s': 4, 'a': 4, 'i': 4, 'p': 2, 'm': 2, 'b': 1, 'l': 1})
c + dict({'a': 4, 'l': 1, 'b': 1, 'm': 1})
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: unsupported operand type(s) for +: 'Counter' and 'dict'
c += dict({'a': 4, 'l': 1, 'b': 1, 'm': 1})
c
Counter({'s': 4, 'a': 4, 'i': 4, 'p': 2, 'm': 2, 'b': 1, 'l': 1})
d = collections.deque([1,2,3])
d += [4,5,6]
d
deque([1, 2, 3, 4, 5, 6])
d + [7,8,9]
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: unsupported operand type(s) for +: 'collections.deque' and 'list'
d.__add__
Traceback (most recent call last):
File stdin, line 1, in module
AttributeError: 'collections.deque' object has no attribute '__add__'
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Mon, Mar 10, 2014 at 11:03 PM, Gregory Ewing greg.ew...@canterbury.ac.nz wrote: As far as observable effects are concerned, it's quite clear: mutable objects can *always* be updated in-place, and immutable objects can *never* be. Hm. Consider the circle-ellipse problem. Briefly, a circle is-an ellipse, so in an inheritance hierarchy it is natural to make Circle a subclass of Ellipse. Now suppose the Ellipse has a stretch method that mutates the ellipse by changing the length of one of its axes while preserving the other. To avoid violating LSP, the Circle class must support all the methods of its ancestor. However it cannot, because the stretch method would invalidate the invariant of the Circle class that both of its axes must always be equal. There are a number of possible solutions. One possibility would be to copy the Circle as an Ellipse and return the new object instead of mutating it. Then you have the situation where, given a mutable object x that satisfies isinstance(x, Ellipse), the stretch method *may* be able to update the object in-place, or it *may* not. I can't think of a reasonable example that would replace the stretch method here with an augmented assignment, but then it is rather late. It might be the obvious way for that particular operation on that particular type. But what about all the others? What's the obvious way to spell in-place set intersection, for example? (Quickly -- no peeking at the docs!) You mean set.intersection_update? The in-place set methods are not hard to remember, because they all end in _update. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Tue, 11 Mar 2014 04:39:39 -0600, Ian Kelly wrote:
On Mon, Mar 10, 2014 at 11:03 PM, Gregory Ewing
greg.ew...@canterbury.ac.nz wrote:
As far as observable effects are concerned, it's quite clear: mutable
objects can *always* be updated in-place, and immutable objects can
*never* be.
Hm. Consider the circle-ellipse problem.
Oh, not that old chestnut! The circle-ellipse problem demonstrates one
limitation of OOP (specifically subtype polymorphism), as well as a
general difficulty with hierarchical taxonomies. Mammals have hair --
what do you make of hairless[1] dolphins? Circle/Ellipse is not a good
place to start from in order to critic augmented assignment in Python.
You're starting from a place where inheritance itself is problematic, so
naturally you're going to find problems.
Briefly, a circle is-an ellipse, so in an inheritance hierarchy it is
natural to make Circle a subclass of Ellipse.
Natural and wrong. It's only natural if you don't think through the
consequences. As you go on to say:
Now suppose the Ellipse has a stretch method that
mutates the ellipse by changing the length of one of its axes while
preserving the other. To avoid violating LSP, the Circle class must
support all the methods of its ancestor. However it cannot, because the
stretch method would invalidate the invariant of the Circle class that
both of its axes must always be equal.
Right. So *Circles cannot be Ellipses*, not without violating the Liskov
Substitution Principle. If I think that they are, I haven't thought it
through. Nor can Ellipses be Circles. That's the problem of the Circle/
Ellipse relationship.
(Aside: the LSP is not a Law Of Physics that cannot be touched. There are
other OOP models that don't require LSP.)
Even in the case of immutable shapes, one might not wish to inherit
Circle from Ellipsis. Ellipses have five degrees of freedom:
2 x position
size (scale)
orientation
shape
while circles only have three:
2 x position
size
Orientation and shape are meaningless for circles! So they should not
inherit from a class where they are meaningful: given the LSP, a subclass
cannot be more restrictive than a superclass.
There are a number of possible solutions. One possibility would be to
copy the Circle as an Ellipse and return the new object instead of
mutating it. Then you have the situation where, given a mutable object
x that satisfies isinstance(x, Ellipse), the stretch method *may* be
able to update the object in-place, or it *may* not.
That is a really lousy design. Of course we are free to create classes
with ugly, inconsistent, stupid or unworkable APIs if we like. Python
won't stop us:
class MyList(list):
def append(self, obj):
if len(self) % 17 == 3:
return self + [obj]
super(MyList, self).append(obj)
I can't think of a reasonable example that would replace the stretch
method here with an augmented assignment, but then it is rather late.
It might be the obvious way for that particular operation on that
particular type.
Um, yes? Nobody suggests that a method of type X has to be the most
obvious way for *any operation* on *any type*. What's your point?
But what about all the others? What's the obvious way
to spell in-place set intersection, for example?
I would expect it to be =, let's find out if I'm right:
py a = set(abcde)
py b = a # save a reference to it
py c = set(cdefg)
py a = c
py a, b
({'c', 'd', 'e'}, {'c', 'd', 'e'})
py a is b
True
(Quickly -- no peeking at the docs!)
The only reason I'd need to look at the docs is because I always forget
whether is intersection and | is union, or the other way around. But
having remembered which is which, going from to = was easy.
You mean set.intersection_update? The in-place set methods are not hard
to remember, because they all end in _update.
And hard to spell.
[1] Technically they aren't *entirely* hairless. They may have a few
hairs around the blowhole, and baby dolphins are born with whiskers which
they soon lose. But from a functional perspective, they are hairless.
--
Steven D'Aprano
--
https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Steven D'Aprano wrote: On Tue, 11 Mar 2014 04:39:39 -0600, Ian Kelly wrote: On Mon, Mar 10, 2014 at 11:03 PM, Gregory Ewing What's the obvious way to spell in-place set intersection, for example? I would expect it to be =, That's my point -- once you know the binary operator for an operation, the corresponding in-place operator is obvious. But there's no established standard set of method names for in-place operations -- each type does its own thing. You mean set.intersection_update? The in-place set methods are not hard to remember, because they all end in _update. For that particular type, maybe, but I wouldn't trust that rule to extend to other types. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Wed, Mar 12, 2014 at 1:01 PM, Rick Johnson rantingrickjohn...@gmail.com wrote: On Thursday, February 27, 2014 4:18:01 PM UTC-6, Ian wrote: x += y is meant to be equivalent, except possibly in-place and more efficient, than x = x + y. In an ideal world, the speed of these two codes should be the same, of course i'm assuming that most competent language designers would optimise the slower version. Except that they don't mean the same thing, so it's not an optimization target. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Tue, Mar 11, 2014 at 10:46 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: There are a number of possible solutions. One possibility would be to copy the Circle as an Ellipse and return the new object instead of mutating it. Then you have the situation where, given a mutable object x that satisfies isinstance(x, Ellipse), the stretch method *may* be able to update the object in-place, or it *may* not. That is a really lousy design. Of course we are free to create classes with ugly, inconsistent, stupid or unworkable APIs if we like. Python won't stop us: That's true but irrelevant to my point, which was to counter the assertion that mutable types can always be assumed to be able to perform operations in-place. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Thursday, February 27, 2014 4:18:01 PM UTC-6, Ian wrote: x += y is meant to be equivalent, except possibly in-place and more efficient, than x = x + y. In an ideal world, the speed of these two codes should be the same, of course i'm assuming that most competent language designers would optimise the slower version. But Rick, Python is an interpreted language and does not benefit from a compile stage. Even if the bytecode can't be optimized on the current run, it CAN be optimized by updating the .pyo file for future runs without degrading current (or future) runtime performance. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On 3/11/2014 10:01 PM, Rick Johnson wrote: On Thursday, February 27, 2014 4:18:01 PM UTC-6, Ian wrote: x += y is meant to be equivalent, except possibly in-place and more efficient, than x = x + y. The manual actually says An augmented assignment expression like x += 1 can be rewritten as x = x + 1 to achieve a similar, but not exactly equal effect. In the augmented version, x is only evaluated once. Also, when possible, the actual operation is performed in-place, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead. In an ideal world, the speed of these two codes should be the same, Nope, 'similar' is not 'equivalent'. Evaluating x twice instead of once and possibly allocating a new object versus not take extra time. In a statement like x.y.z[3*n+m] += 1, calculating the target dominates the time to increment, so this form should be nearly twice as fast. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Sun, Mar 9, 2014 at 8:37 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On Sun, 09 Mar 2014 17:42:42 -0600, Ian Kelly wrote: On Sun, Mar 9, 2014 at 4:03 PM, Gregory Ewing greg.ew...@canterbury.ac.nz wrote: Note that it says when possible, not if the implementation feels like it. That's quite vague, and not much stronger a guarantee than maybe. It's technically possible for this augmented assignment to be performed in place: x = 12 x += 4 But it's not done in-place, because ints are meant to be immutable. That's incorrect. Ints aren't merely meant to be immutable, which implies that's it's optional, they are defined by the language specification and the reference implementation as immutable. Any interpreter where ints are mutable *is not Python*. That's true, but is beside the point, which is that when possible is not very meaningful. In any case, this means that whether the operation is actually performed in-place is an implementation detail -- if not of the Python implementation then at least of the class -- and not something the user should take for granted. Whether += operates in place or not is part of the interface of the class, not the implementation. Would you say that whether list.append operates in place or creates a new list is an implementation detail? Whether str.upper() creates a new string or modifies the existing one in place? Of course not. list.append is documented as modifying the list. str.upper is documented as returning a copy of the string. Mutability versus immutability is part of the interface, not implementation, not withstanding that somebody could create an alternative class with the opposite behaviour: a MutableStr, or ImmutableList. If the in-place behavior of += is held to be part of the interface, then we must accept that += is not polymorphic across mutable and immutable types, which in my mind largely* defeats the purpose of having it. After all, there should be one -- and preferably only one -- obvious way to do it. If you want in-place concatenation, the obvious way to do it is by calling extend. If you want copy concatenation, the obvious way to do it is with the + operator. Why then should not just mutable sequences but immutable sequences as well even offer the += operator? * The one exception I can think of is numpy, where there is no more obvious way to do in-place addition, and in that context I would consider the in-place behavior to be part of the interface. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Mon, 10 Mar 2014 02:35:36 -0600, Ian Kelly wrote: On Sun, Mar 9, 2014 at 8:37 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On Sun, 09 Mar 2014 17:42:42 -0600, Ian Kelly wrote: On Sun, Mar 9, 2014 at 4:03 PM, Gregory Ewing greg.ew...@canterbury.ac.nz wrote: Note that it says when possible, not if the implementation feels like it. That's quite vague, and not much stronger a guarantee than maybe. It's technically possible for this augmented assignment to be performed in place: x = 12 x += 4 But it's not done in-place, because ints are meant to be immutable. That's incorrect. Ints aren't merely meant to be immutable, which implies that's it's optional, they are defined by the language specification and the reference implementation as immutable. Any interpreter where ints are mutable *is not Python*. That's true, but is beside the point, which is that when possible is not very meaningful. It's meaningful. It refers not to ints, but the infinite number of possible classes which might include augmented assignment. Some of them will be immutable, in which case it is not possible for += etc. to be performed in-place. Some of them will be mutable, but there won't be any reasonable (or even unreasonable) way to perform += in-place. But for some mutable classes, it will be possible to perform += in place, in which case the docs say that they have to do so. In any case, this means that whether the operation is actually performed in-place is an implementation detail -- if not of the Python implementation then at least of the class -- and not something the user should take for granted. Whether += operates in place or not is part of the interface of the class, not the implementation. Would you say that whether list.append operates in place or creates a new list is an implementation detail? Whether str.upper() creates a new string or modifies the existing one in place? Of course not. list.append is documented as modifying the list. str.upper is documented as returning a copy of the string. Right. And += is documented as modifying the list too. Mutability versus immutability is part of the interface, not implementation, not withstanding that somebody could create an alternative class with the opposite behaviour: a MutableStr, or ImmutableList. If the in-place behavior of += is held to be part of the interface, then we must accept that += is not polymorphic across mutable and immutable types, I'm fine with that. which in my mind largely* defeats the purpose of having it. Not to my mind. I think that having augmented assignment is worthwhile even if it behaves differently and incompatibly for different classes. After all, so does * (multiplication): py x = 24 py x*x 576 but: py x = [] py x*x Traceback (most recent call last): File stdin, line 1, in module TypeError: can't multiply sequence by non-int of type 'list' I don't think that we ought to throw away * and I don't think we ought to throw away *= either. After all, there should be one -- and preferably only one -- obvious way to do it. If you want in-place concatenation, the obvious way to do it is by calling extend. If you want copy concatenation, the obvious way to do it is with the + operator. Why then should not just mutable sequences but immutable sequences as well even offer the += operator? *shrug* I can see that each individual operation: list.extend list + list += makes good sense in isolation, but I can also see that the combination don't quite gel together as smoothly as we might hope. -- Steven D'Aprano http://import-that.dreamwidth.org/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Ian Kelly wrote: It's technically possible for this augmented assignment to be performed in place: x = 12 x += 4 But it's not done in-place, because ints are meant to be immutable. Which means it's *not* possible, because doing so would violate the documented properties of the int type. In any case, this means that whether the operation is actually performed in-place is an implementation detail The implementation could theoretically perform this optimisation if there are no other references to the object. But this will be completely invisible, because to even find out whether it's the same object, you need to keep another reference to the original object, preventing the optimisation from being performed. As far as observable effects are concerned, it's quite clear: mutable objects can *always* be updated in-place, and immutable objects can *never* be. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Ian Kelly wrote: If the in-place behavior of += is held to be part of the interface, then we must accept that += is not polymorphic across mutable and immutable types, That's quite correct, it's not. As I said, it's one notation doing double duty. Usually there isn't any confusion, because you know whether any particular instance of it is intended to operate on a mutable or immutable type. If that's not the case -- if you're writing a function intended to operate on a variety of types, some mutable and some not -- then using in-place operators would not be appropriate. If you want in-place concatenation, the obvious way to do it is by calling extend. It might be the obvious way for that particular operation on that particular type. But what about all the others? What's the obvious way to spell in-place set intersection, for example? (Quickly -- no peeking at the docs!) The in-place operators provide a standardised spelling for in-place versions of all the binary operations. That's a useful thing to have, I think. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Ian Kelly ian.g.ke...@gmail.com: In my view the second one is wrong. a += b should be understood as being equivalent to a = a + b, but with the *possible* and by no means guaranteed optimization that the operation may be performed in-place. Some call it an optimization, others call it a side effect. Anyway, that's how it has been explicitly defined in the language specification so that's the reality whether one likes it or not. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On 28 February 2014 14:43, Chris Angelico ros...@gmail.com wrote: On Sat, Mar 1, 2014 at 1:41 AM, Joshua Landau jos...@landau.ws wrote: Would it be better to add a check here, such that if this gets raised to the top-level it includes a warning (Addition was inplace; variable probably mutated despite assignment failure)? That'd require figuring out whether or not the variable was actually mutated, and that's pretty hard to work out. It does not. First, the warning is actually an attachment to the exception so is only shown if the exception is uncaught. This should basically never happen in working code. The warning exists only to remove likely misunderstanding in these odd cases. Even if x = (1,); x[0] += 1 warned addition was inplace; possible mutation occurred or whatever phrasing you wish, this would only cause a quick check of the results. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Mon, Mar 10, 2014 at 4:54 AM, Joshua Landau jos...@landau.ws wrote: On 28 February 2014 14:43, Chris Angelico ros...@gmail.com wrote: On Sat, Mar 1, 2014 at 1:41 AM, Joshua Landau jos...@landau.ws wrote: Would it be better to add a check here, such that if this gets raised to the top-level it includes a warning (Addition was inplace; variable probably mutated despite assignment failure)? That'd require figuring out whether or not the variable was actually mutated, and that's pretty hard to work out. It does not. First, the warning is actually an attachment to the exception so is only shown if the exception is uncaught. This should basically never happen in working code. The warning exists only to remove likely misunderstanding in these odd cases. Even if x = (1,); x[0] += 1 warned addition was inplace; possible mutation occurred or whatever phrasing you wish, this would only cause a quick check of the results. I think I see what you're saying here. But ignore top-level; this should just be a part of the exception message, no matter what. Otherwise things that recreate the REPL (like IDLE) or that isolate two sections of the code (like a web server framework) wouldn't get the benefit, because the exception's not caught at the true top-level. x=1, x[0]+=0 Traceback (most recent call last): File pyshell#1, line 1, in module x[0]+=0 TypeError: 'tuple' object does not support item assignment What you're saying is that this should notice that it's doing an augmented assignment and give some more text. This can be done; all you need to do is catch the error and reraise it with more info: try: x[0]+=0 except TypeError as e: if 'does not support item assignment' in e.args[0]: raise TypeError(e.args[0]+\nAugmented assignment used; mutation may have occurred.) from None raise Traceback (most recent call last): File pyshell#16, line 5, in module raise TypeError(e.args[0]+\nAugmented assignment used; mutation may have occurred.) from None TypeError: 'tuple' object does not support item assignment Augmented assignment used; mutation may have occurred. Now you can look at writing an import hook that does an AST transform, locating every instance of item assignment and wrapping it like that. It's certainly possible. I'm not sure how much benefit you'd get, but it could be done. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On 9 March 2014 18:13, Chris Angelico ros...@gmail.com wrote: I think I see what you're saying here. But ignore top-level; this should just be a part of the exception message, no matter what. I don't think I was clear, but yes. That. What you're saying is that this should notice that it's doing an augmented assignment and give some more text. This can be done; all you need to do is catch the error and reraise it with more info: [...] Now you can look at writing an import hook that does an AST transform, locating every instance of item assignment and wrapping it like that. It's certainly possible. I'm not sure how much benefit you'd get, but it could be done. I would probably implement it closer to home. Inside tuple.__getitem__, there would be something like if context_is_augmented_assignment(): raise TypeError(message+warning) else: raise TypeError(message) which would have much lower technical costs. It does depend on how costly context_is_augmented_assignment is, though. A speed argument is totally valid, but I'd hope it's possible to work around. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Mon, Mar 10, 2014 at 6:57 AM, Joshua Landau jos...@landau.ws wrote: I would probably implement it closer to home. Inside tuple.__getitem__, there would be something like if context_is_augmented_assignment(): raise TypeError(message+warning) else: raise TypeError(message) which would have much lower technical costs. It does depend on how costly context_is_augmented_assignment is, though. A speed argument is totally valid, but I'd hope it's possible to work around. Yeah, I'm not sure there's an easy way to tell the tuple that it's an augmented assignment. You might be able to look at the backtrace, but that's tricky. In theory, you could catch TypeError after any augmented assignment, and check several things: 1) The target object does not have __setitem__ 2) The object being manipulated does have __iadd__ (or corresponding for others) 3) The error is that item assignment is not possible and if all three are the case, then add a tag to the message. But this is best done by catching the exception. Otherwise you'd be limiting this to tuples and lists; not to mention that this is really only an issue of error reporting, and correct code shouldn't be tripping this at all. So put a check like that at the place where you display the error, if you can. The AST transform that I described would also work. But I really think it's not worth the effort. If you get this error, understand that it may have consequences. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Ian Kelly wrote: In my view the second one is wrong. a += b should be understood as being equivalent to a = a + b, but with the *possible* and by no means guaranteed optimization that the operation may be performed in-place. This interpretation is at odds with the Language Reference, section 6.2.1, Augmented Assignment Statements: An augmented assignment expression like x += 1 can be rewritten as x = x + 1 to achieve a similar, but not exactly equal effect... when possible, the actual operation is performed in-place, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead. Note that it says when possible, not if the implementation feels like it. In fact, if you read the documentation for lists, you may notice that while they clearly cover the + operator and the extend method, they do not explicitly document the list class's += operator. The when possible clause, together with the fact that lists are mutable, implies that it *will* be done in-place. There is no need to document all the in-place operators explicitly for every type. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On 3/9/2014 6:03 PM, Gregory Ewing wrote: Ian Kelly wrote: In my view the second one is wrong. a += b should be understood as being equivalent to a = a + b, but with the *possible* and by no means guaranteed optimization that the operation may be performed in-place. This interpretation is at odds with the Language Reference, section 6.2.1, Augmented Assignment Statements: An augmented assignment expression like x += 1 can be rewritten as x = x + 1 to achieve a similar, but not exactly equal effect... when possible, the actual operation is performed in-place, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead. Note that it says when possible, not if the implementation feels like it. In fact, if you read the documentation for lists, you may notice that while they clearly cover the + operator and the extend method, they do not explicitly document the list class's += operator. The when possible clause, together with the fact that lists are mutable, implies that it *will* be done in-place. There is no need to document all the in-place operators explicitly for every type. The discussion of .__iop__ in the datamodel chapter was just revised slightly. http://bugs.python.org/issue19953 -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Sun, Mar 9, 2014 at 4:03 PM, Gregory Ewing greg.ew...@canterbury.ac.nz wrote: Ian Kelly wrote: In my view the second one is wrong. a += b should be understood as being equivalent to a = a + b, but with the *possible* and by no means guaranteed optimization that the operation may be performed in-place. This interpretation is at odds with the Language Reference, section 6.2.1, Augmented Assignment Statements: An augmented assignment expression like x += 1 can be rewritten as x = x + 1 to achieve a similar, but not exactly equal effect... when possible, the actual operation is performed in-place, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead. Note that it says when possible, not if the implementation feels like it. That's quite vague, and not much stronger a guarantee than maybe. It's technically possible for this augmented assignment to be performed in place: x = 12 x += 4 But it's not done in-place, because ints are meant to be immutable. In any case, this means that whether the operation is actually performed in-place is an implementation detail -- if not of the Python implementation then at least of the class -- and not something the user should take for granted. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Sun, 09 Mar 2014 17:42:42 -0600, Ian Kelly wrote: On Sun, Mar 9, 2014 at 4:03 PM, Gregory Ewing greg.ew...@canterbury.ac.nz wrote: Note that it says when possible, not if the implementation feels like it. That's quite vague, and not much stronger a guarantee than maybe. It's technically possible for this augmented assignment to be performed in place: x = 12 x += 4 But it's not done in-place, because ints are meant to be immutable. That's incorrect. Ints aren't merely meant to be immutable, which implies that's it's optional, they are defined by the language specification and the reference implementation as immutable. Any interpreter where ints are mutable *is not Python*. In any case, this means that whether the operation is actually performed in-place is an implementation detail -- if not of the Python implementation then at least of the class -- and not something the user should take for granted. Whether += operates in place or not is part of the interface of the class, not the implementation. Would you say that whether list.append operates in place or creates a new list is an implementation detail? Whether str.upper() creates a new string or modifies the existing one in place? Mutability versus immutability is part of the interface, not implementation, not withstanding that somebody could create an alternative class with the opposite behaviour: a MutableStr, or ImmutableList. -- Steven D'Aprano http://import-that.dreamwidth.org/ -- https://mail.python.org/mailman/listinfo/python-list
Balanced trees (was: Re: Tuples and immutability)
Ian Kelly ian.g.ke...@gmail.com: I already mentioned this earlier in the thread, but a balanced binary tree might implement += as node insertion and then return a different object if the balancing causes the root node to change. True. Speaking of which, are there plans to add a balanced tree to the batteries of Python? Timers, cache aging and the like need it. I'm using my own AVL tree implementation, but I'm wondering why Python still doesn't have one. In fact, since asyncio has timers but Python doesn't have balanced trees, I'm led to wonder how good the asyncio implementation can be. Note that Java batteries include TreeMap. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Balanced trees (was: Re: Tuples and immutability)
On Sat, Mar 8, 2014 at 1:34 AM, Marko Rauhamaa ma...@pacujo.net wrote: Speaking of which, are there plans to add a balanced tree to the batteries of Python? Timers, cache aging and the like need it. I'm using my own AVL tree implementation, but I'm wondering why Python still doesn't have one. None currently that I'm aware of. If you want to propose adding one, I suggest reading: http://docs.python.org/devguide/stdlibchanges.html In fact, since asyncio has timers but Python doesn't have balanced trees, I'm led to wonder how good the asyncio implementation can be. Peeking at the code, it appears to use a heapq-based priority queue. Why would a balanced binary tree be better? -- https://mail.python.org/mailman/listinfo/python-list
Re: Balanced trees (was: Re: Tuples and immutability)
On Sat, Mar 8, 2014 at 12:34 AM, Marko Rauhamaa ma...@pacujo.net wrote: Ian Kelly ian.g.ke...@gmail.com: I already mentioned this earlier in the thread, but a balanced binary tree might implement += as node insertion and then return a different object if the balancing causes the root node to change. True. Speaking of which, are there plans to add a balanced tree to the batteries of Python? Timers, cache aging and the like need it. I'm using my own AVL tree implementation, but I'm wondering why Python still doesn't have one. I think it'd probably be a good idea to add one or more balanced binary trees to the standard library. But I suspect it's been tried before, and didn't happen. It might be good to add an _un_balanced tree too, since they do quite well with random keys. Here's a performance comparison I did of a bunch of tree types in Python: http://stromberg.dnsalias.org/~strombrg/python-tree-and-heap-comparison/2014-01/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Ian Kelly wrote: class LessThanFilter: def __init__(self, the_list): self._the_list = the_list def __getitem__(self, bound): return [x for x in self._the_list if x bound] filter = LessThanFilter([10, 20, 30, 40, 50]) filter[25] += [15, 17, 23] Should that last line not raise an exception? In this case it will fail to catch what is probably an error, but you can't expect the language to find all your bugs for you. If you wrote the same bug this way: filter[25].extend([15, 17, 23]) it wouldn't be caught either. What's happening is that we're trying to use the syntax a += b to mean two different things: 1) Shorthand for a = a + b 2) A way of expressing an in-place modification, such as a.extend(b) Case (2) is not really an assignment at all, so arguably it shouldn't require the LHS to support assignment. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Ian Kelly wrote: I already mentioned this earlier in the thread, but a balanced binary tree might implement += as node insertion and then return a different object if the balancing causes the root node to change. That would be a really bad way to design a binary tree implementation. What if there is another reference to the tree somewhere? It's still going to be referring to the old root object, and will have an incoherent view of the data -- partly old and partly new. If you're going to have a mutable tree, it needs to be encapsulated in a stable top-level object. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Sat, Mar 8, 2014 at 5:40 PM, Gregory Ewing greg.ew...@canterbury.ac.nz wrote: Ian Kelly wrote: class LessThanFilter: def __init__(self, the_list): self._the_list = the_list def __getitem__(self, bound): return [x for x in self._the_list if x bound] filter = LessThanFilter([10, 20, 30, 40, 50]) filter[25] += [15, 17, 23] Should that last line not raise an exception? In this case it will fail to catch what is probably an error, but you can't expect the language to find all your bugs for you. If you wrote the same bug this way: filter[25].extend([15, 17, 23]) it wouldn't be caught either. What's happening is that we're trying to use the syntax a += b to mean two different things: 1) Shorthand for a = a + b 2) A way of expressing an in-place modification, such as a.extend(b) Case (2) is not really an assignment at all, so arguably it shouldn't require the LHS to support assignment. In my view the second one is wrong. a += b should be understood as being equivalent to a = a + b, but with the *possible* and by no means guaranteed optimization that the operation may be performed in-place. In fact, if you read the documentation for lists, you may notice that while they clearly cover the + operator and the extend method, they do not explicitly document the list class's += operator. So although I'm not entirely sure whether it is intentional or not, and I would be quite surprised if some implementation were actually to differ on this point, the language does *not* from what I can see guarantee that the += operator on lists is equivalent to calling .extend. That having been said, code that uses += and relies on the operation to be performed in-place should be considered buggy. If you need the operation to be performed in-place, then use in-place methods like list.extend. If you need the operation not to be performed in-place, then use a = a + b. If you're ambivalent on the in-place issue and just want to write polymorphic code, that's when you should consider using +=. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Sat, Mar 8, 2014 at 5:45 PM, Gregory Ewing greg.ew...@canterbury.ac.nz wrote: Ian Kelly wrote: I already mentioned this earlier in the thread, but a balanced binary tree might implement += as node insertion and then return a different object if the balancing causes the root node to change. That would be a really bad way to design a binary tree implementation. What if there is another reference to the tree somewhere? It's still going to be referring to the old root object, and will have an incoherent view of the data -- partly old and partly new. If you're going to have a mutable tree, it needs to be encapsulated in a stable top-level object. Well, as I parenthetically noted the first time I brought it up, whether this is good design is tangential; it's a possible design. The language shouldn't be written such that only those designs deemed good by some external committee can be implemented. What you dismiss as really bad may be exactly what somebody else needs, and maybe they intend that there won't be other references. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Chris Angelico ros...@gmail.com wrote: On Sat, Mar 1, 2014 at 1:41 AM, Joshua Landau jos...@landau.ws wrote: Would it be better to add a check here, such that if this gets raised to the top-level it includes a warning (Addition was inplace; variable probably mutated despite assignment failure)? That'd require figuring out whether or not the variable was actually mutated, and that's pretty hard to work out. So there's a FAQ entry, which Zachary already posted: http://docs.python.org/3/faq/programming.html#why-does-a-tuple-i-item-r aise-an-exception-when-the-addition-works Also, we just answer this question every now and then :) Presumably more often on -tutor than here. ChrisA Another take on this that I haven't seen discussed in this thread: Is there any reason why tuples need to throw an exception on assigning to the element if the old value and new value are the same object? If I say: a = (spam, [10, 30], eggs) then a[0] = a[0] won't actually mutate the object. So tuples could let that silently pass. Then you would be able to safely do: a[1] += [50] but this would still throw an exception: a[0] += x -- Duncan Booth http://kupuguy.blogspot.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Duncan Booth duncan.booth@invalid.invalid writes: Is there any reason why tuples need to throw an exception on assigning to the element if the old value and new value are the same object? Special cases aren't special enough to break the rules. -- \ “I do not believe in forgiveness as it is preached by the | `\church. We do not need the forgiveness of God, but of each | _o__)other and of ourselves.” —Robert G. Ingersoll | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Fri, Mar 7, 2014 at 8:33 PM, Duncan Booth duncan.booth@invalid.invalid wrote: Is there any reason why tuples need to throw an exception on assigning to the element if the old value and new value are the same object? It'd be easy enough to implement your own tuple subclass that behaves that way. Try it! See how many situations it actually helps. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Chris Angelico wrote:
On Fri, Mar 7, 2014 at 8:33 PM, Duncan Booth
duncan.booth@invalid.invalid wrote:
Is there any reason why tuples need to throw an exception on assigning to
the element if the old value and new value are the same object?
It'd be easy enough to implement your own tuple subclass that behaves
that way. Try it! See how many situations it actually helps.
class T(tuple):
... def __setitem__(self, index, value):
... if value is not self[index]:
... raise TypeError({} is not {}.format(value,
self[index]))
...
for i, k in zip(range(250, 260), range(250, 260)):
... T([i])[0] = k
...
Traceback (most recent call last):
File stdin, line 2, in module
File stdin, line 4, in __setitem__
TypeError: 257 is not 257
I'm not sure help is the right word here ;)
--
https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Fri, Mar 7, 2014 at 10:38 PM, Peter Otten __pete...@web.de wrote: TypeError: 257 is not 257 I'm not sure help is the right word here ;) It doesn't help with non-small integers, yes, but the original case was a list. Personally, I don't think there are many situations that would benefit from it, plus it'd be confusing (I can use += with a list but not a number, why not?!). ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Fri, 07 Mar 2014 09:33:49 +, Duncan Booth wrote: Chris Angelico ros...@gmail.com wrote: On Sat, Mar 1, 2014 at 1:41 AM, Joshua Landau jos...@landau.ws wrote: Would it be better to add a check here, such that if this gets raised to the top-level it includes a warning (Addition was inplace; variable probably mutated despite assignment failure)? That'd require figuring out whether or not the variable was actually mutated, and that's pretty hard to work out. So there's a FAQ entry, which Zachary already posted: http://docs.python.org/3/faq/programming.html#why-does-a-tuple-i-item-r aise-an-exception-when-the-addition-works Also, we just answer this question every now and then :) Presumably more often on -tutor than here. ChrisA Another take on this that I haven't seen discussed in this thread: Is there any reason why tuples need to throw an exception on assigning to the element if the old value and new value are the same object? If I say: a = (spam, [10, 30], eggs) then a[0] = a[0] won't actually mutate the object. So tuples could let that silently pass. Then you would be able to safely do: a[1] += [50] but this would still throw an exception: a[0] += x I would think it would be better if the exception was thrown before the assignment to the list took place simply seeing that a modification action was being applied to a tupple should be enough. this would alert the programmer to the fact that he was trying something that may have undesired consequences -- Old age is the harbor of all ills. -- Bion -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
In article XnsA2E95FA1E1EB6duncanbooth@127.0.0.1, Duncan Booth duncan.booth@invalid.invalid wrote: Is there any reason why tuples need to throw an exception on assigning to the element if the old value and new value are the same object? If I say: a = (spam, [10, 30], eggs) then a[0] = a[0] won't actually mutate the object. So tuples could let that silently pass. But, why would you want them to? What a way to introduce bugs which are difficult to test for. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Fri, Mar 7, 2014 at 4:51 AM, Alister alister.w...@ntlworld.com wrote: I would think it would be better if the exception was thrown before the assignment to the list took place simply seeing that a modification action was being applied to a tupple should be enough. this would alert the programmer to the fact that he was trying something that may have undesired consequences Then the behavior of tuples would be inconsistent with other immutable types. This can't be applied generally, because the Python interpreter doesn't generally know whether a given type is supposed to be immutable or not. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Duncan Booth wrote: Is there any reason why tuples need to throw an exception on assigning to the element if the old value and new value are the same object? It would make introspection misleading, because tuples would have a __setitem__ method event though they don't actually support item assignment. Also, it would solve the problem for tuples in particular, but not for any other immutable type -- they would all have to implement the same behaviour independently to enjoy the benefit. Here's another idea: If the __iadd__ method returns the same object, *and* the LHS doesn't have a __setitem__ method, then do nothing instead of raising an exception. Peter Otten wrote: Traceback (most recent call last): File stdin, line 2, in module File stdin, line 4, in __setitem__ TypeError: 257 is not 257 I'm not sure help is the right word here ;) I don't think that's a problem, because the use case being addressed is where the object performs in-place modification and always returns itself. Any object that doesn't return itself is not modifying in-place, even if the returned object happens to be equal to the original one. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Fri, Mar 7, 2014 at 7:17 PM, Gregory Ewing greg.ew...@canterbury.ac.nz wrote: Here's another idea: If the __iadd__ method returns the same object, *and* the LHS doesn't have a __setitem__ method, then do nothing instead of raising an exception. Maybe it doesn't have a __setitem__ because the object that was retrieved is computed rather than stored, and the result of the __iadd__ will simply be discarded. Somewhat contrived example: class LessThanFilter: def __init__(self, the_list): self._the_list = the_list def __getitem__(self, bound): return [x for x in self._the_list if x bound] filter = LessThanFilter([10, 20, 30, 40, 50]) filter[25] += [15, 17, 23] Should that last line not raise an exception? The __iadd__ call will return the same object, and the LHS doesn't have a __setitem__ method. I don't think that's a problem, because the use case being addressed is where the object performs in-place modification and always returns itself. Any object that doesn't return itself is not modifying in-place, even if the returned object happens to be equal to the original one. I already mentioned this earlier in the thread, but a balanced binary tree might implement += as node insertion and then return a different object if the balancing causes the root node to change. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Sat, Mar 1, 2014 at 7:04 PM, Eric Jacoboni eric.jacob...@gmail.com wrote: In fact, i think i'm gonna forget += on lists :) Well, do what you want, but I think you're taking the wrong lesson from this. Don't forget about using += on lists. Instead, forget about using assignments, augmented or otherwise, on tuple elements. Would you expect this to work? tup = (1, 2, 3) tup[1] += 42 I'm guessing probably not. So I have a hard time understanding why it should be expected to work any differently when the tuple element happens to be a list. The fact that the list is actually modified I think is a distraction from the main point: the error indicates that something illegal happened, not that nothing has changed. Assignments just are not allowed on tuple elements, and it doesn't matter what type they happen to be. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Le 02/03/2014 13:32, Ian Kelly a écrit : On Sat, Mar 1, 2014 at 7:04 PM, Eric Jacoboni eric.jacob...@gmail.com wrote: In fact, i think i'm gonna forget += on lists :) Well, do what you want, but I think you're taking the wrong lesson from this. Don't forget about using += on lists. Instead, forget about using assignments, augmented or otherwise, on tuple elements. Would you expect this to work? Well, the thing about += on lists that makes me forget it, like i said in my previous post, is that its behaviour is not consistent with +. Don't get me wrong: i don't expect that modifying a tuple element works. That's exactly my point: my initial question was, why it half works : it should not work at all. I was thinking that += on lists worked like update() or extend() : modifying lists in place... It was my mistake So, yes, i still don't get the point using a += operation, which is not even consistent with the + operation (see my exemple on spam in my previous post). The + operator to return the modified list and the update() or extend() methods to do in place replacements are well enough for my present needs. Maybe, in the future, i will find a use case of += for lists which is not covered by others methods, though... Who knows? I don't doubt that Python designers have made this choice and this behavior for a good reason: i've just not get it yet. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On 02/03/2014 13:38, Eric Jacoboni wrote: Le 02/03/2014 13:32, Ian Kelly a écrit : On Sat, Mar 1, 2014 at 7:04 PM, Eric Jacoboni eric.jacob...@gmail.com wrote: In fact, i think i'm gonna forget += on lists :) Well, do what you want, but I think you're taking the wrong lesson from this. Don't forget about using += on lists. Instead, forget about using assignments, augmented or otherwise, on tuple elements. Would you expect this to work? Well, the thing about += on lists that makes me forget it, like i said in my previous post, is that its behaviour is not consistent with +. The behaviour is consistent except when you try to modify a tuple. Don't get me wrong: i don't expect that modifying a tuple element works. That's exactly my point: my initial question was, why it half works : it should not work at all. I was thinking that += on lists worked like update() or extend() : modifying lists in place... It was my mistake I'd like to see you get update to work on a list. This shows how confused you are over this issue. So, yes, i still don't get the point using a += operation, which is not even consistent with the + operation (see my exemple on spam in my previous post). The + operator to return the modified list and the update() or extend() methods to do in place replacements are well enough for my present needs. Maybe, in the future, i will find a use case of += for lists which is not covered by others methods, though... Who knows? I don't doubt that Python designers have made this choice and this behavior for a good reason: i've just not get it yet. All of this comes about because of the known issue with tuples. Take them out of the equation and the behaviour of =, +=, extend and append is consistent. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Le 02/03/2014 15:05, Mark Lawrence a écrit : The behaviour is consistent except when you try to modify a tuple. Not in my opinion... li = [10, 30] li = li + spam -- TypeError: can only concatenate list (not str) li += spam -- Ok So, not, that's not what i call consistent. And it's not related to tuples... I'd like to see you get update to work on a list. This shows how confused you are over this issue. Well, sorry, i wanted to write .append() -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Sun, 02 Mar 2014 15:17:11 +0100, Eric Jacoboni eric.jacob...@gmail.com wrote: Le 02/03/2014 15:05, Mark Lawrence a écrit : The behaviour is consistent except when you try to modify a tuple. Not in my opinion... li = [10, 30] li = li + spam -- TypeError: can only concatenate list (not str) li += spam -- Ok possibly because you expect += to take spam as a string, but have you looked at the result? In [1]: mylist = ['1', '2'] In [2]: mylist += 'spam' In [3]: mylist Out[3]: ['1', '2', 's', 'p', 'a', 'm'] consequently, try adding something that can not be interpreted as a sequence: In [4]: mylist += 3 --- TypeError Traceback (most recent call last) ipython-input-4-782b544a29d1 in module() 1 mylist += 3 TypeError: 'int' object is not iterable -- Vriendelijke groeten / Kind regards, Albert Visser Using Opera's mail client: http://www.opera.com/mail/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On 01/03/2014 06:16, Mark H. Harris wrote: On Friday, February 28, 2014 11:16:18 PM UTC-6, Ian wrote: How would you propose doing that? Bear in mind that while Python knows that tuples specifically are immutable, it doesn't generally know whether a type is immutable. hi Ian, consider the problem... its a cake and eat it too scenario... you can't have your cake and eat it too This is what I mean... the error message is telling the user that it cannot do what he has requested, and yet IT DID. You have one of two scenarios: 1) the message is arbitrarily lying and it really can assign an immutable's item... (and it did!) or 2) It really should NOT assign an immutables item (the message is truth) but the action was allowed anyway despite the protocol and accepted policy... in which case the two scenarios add up to a giant logical inconsistency... QED a bug. There really is no way to get around this from the average user's perspective. And therefore, this is going to come up again and again and again... because coders don't like unexpected logical inconsistencies. It is not helpful either to explain it away by knowing how the policy works under the covers... the average user of the python language should not have to understand the policy of the underlying substructure... 1) either they can't assign the list item of an immutable tuple type (and the action is a flaw, say bug of 2) they really CAN set the immutable's list item type (which it did!) and the message is bogus. Its one way or the other we can't have our cake and eat it too... this is (one way or the other) a bug. IMHO would you please be kind enough to use plain English... your use of sentences like this... is starting to annoy me... you're also using google groups... it doesn't wrap paragraphs correctly... please read and action this https://wiki.python.org/moin/GoogleGroupsPython... it does wrap paragraphs correctly... it also prevents us seeing double line spacing... do you get my drift??? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On 3/1/14 12:50 AM, Mark H. Harris wrote: On Friday, February 28, 2014 11:34:56 PM UTC-6, Ian wrote: One very common example of tuples containing lists is when lists are passed to any function that accepts *args, because the extra arguments are passed in a tuple. A similarly common example is when returning multiple objects from a function, and one of them happens to be a list, because again they are returned in a tuple. def f(*args): print(args) return (args[1:] result = f(1, 2, 3, [4, 5]) (1, 2, 3, [4, 5]) print(result) (2, 3, [4, 5]) I agree Ian... good points all. ... again, I'm not arguing with anyone... just saying that an error (whatever we mean by that) should not half-way-fail we are only pointing out the problem... we have not idea what the solution is yet. Intuitively everyone can see that there is a problem here... the debate cannot be answered either because of the inherent design of python (almost all of which we love). So, as they say, what is a mother to do? ... I mean, some people's kids... I don't know how I propose to handle the problem... I think the first step is getting everyone to agree that there IS a problem... then debate how to tackle the solution proposals. marcus Everyone can agree that it is not great to raise an exception and also leave the list modified. But I very much doubt that anything will be done to change the situation. All of the solutions are too extreme, and bring their own infelicities, and the actual problems caused by this scenario are vanishingly small. BTW: I also am mystified why you uses ellipses to end your sentences, surely one period would be enough? :) -- Ned Batchelder, http://nedbatchelder.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On 27 February 2014 21:47, Nick Timkovich prometheus...@gmail.com wrote: On Thu, Feb 27, 2014 at 10:33 AM, Chris Angelico ros...@gmail.com wrote: It's unintuitive, but it's a consequence of the way += is defined. If you don't want assignment, don't use assignment :) Where is `.__iadd__()` called outside of `list += X`? Numpy uses it for in-place operations with numpy arrays: import numpy a = numpy.arange(10) a array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) a[::2] += 10 a array([10, 1, 12, 3, 14, 5, 16, 7, 18, 9]) It makes sense for any mutable type that supports +. The stdlib doesn't have many of these. The obvious one is list but there's also MutableString (in 2.x): from UserString import MutableString a = MutableString(qwe) a 'qwe' b = a b += asd a 'qweasd' If the only difference from `.extend()` is that it returns `self`, but the list was already modified anyway, why bother with reassignment? I don't know of any situation where __iadd__ is defined but doesn't return self and I can't think of a use case apart from operator abuse. I had thought that the other difference was that .extend would accept any iterable but it seems += does also. Perhaps that was changed at some point. l = [1, 2, 3] l + (1, 2, 3) Traceback (most recent call last): File stdin, line 1, in module TypeError: can only concatenate list (not tuple) to list l += (1, 2, 3) l [1, 2, 3, 1, 2, 3] Oscar -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Fri, Feb 28, 2014 at 11:25 PM, Mark H. Harris harrismh...@gmail.com wrote: On Friday, February 28, 2014 11:16:18 PM UTC-6, Ian wrote: How would you propose doing that? Bear in mind that while Python knows that tuples specifically are immutable, it doesn't generally know whether a type is immutable. I was going to bed, and then thought of the solution. Allow the action. Hold the error pending until the action either succeeds or fails for the item, in this case a list, and then post the error for the tuple depending... this would make the whole mess consistent, and even compatible with the idea of dynamic name and type binding... So, in other words, the action should be permitted, and actually was allowed, and the message is incorrectly reporting something that for this particular item is NOT true. The error message (and timing of same) is the bug (philosophically). Under the covers it may be (and probably is) more complicated. Indeed. For one thing, __iadd__ is not required to return the object it was called on; it could return some other object. Imagine a balanced binary tree that is stored by referencing the root node object (whether this is good design is tangential; it's a possible design). If the __iadd__ operator is implemented to insert a node, and it causes the root node to change, then it might return that new root node, expecting it to be assigned. If Python simply ignores the exception from the assignment, then the data structure has been modified in-place, but the tuple does not contain the correct object, and worst of all no exception has been raised to alert anybody of this failure condition. So at best you could only ignore these exceptions when the object returned by __iadd__ is self, which to be fair is probably most of the time. Additionally, recall my statement up-thread that If you skip the assignment, and that assignment is meaningful to whatever the left side may be (e.g. assigning to a descriptor or something that invokes __setitem__ or __setattr__), then the operation is not equivalent. If you quash the exception from the assignment because the __iadd__ call succeeded, then you have effectively skipped the assignment and then lied to the programmer about it. For example, imagine a list subclass that contains other lists but only permits itself to be nested 1-deep; the contained lists are required to be flat. The __setitem__ method for the class might enforce the constraint and perhaps also update a count of the total combined length of the nested lists. Then suppose we have this code: lol = ListOfLists([1,2,3], [4], [6, 7, 8, 9]) lol[1] += [5, [5.25, 5.5, 5.75]] The effect of this is that the middle sublist has been extended to include a doubly-nested list. __setitem__ was then called, which noticed that the list it was passed includes another nested list, so it raised an exception instead of performing the assignment and updating the count. The ListOfLists is now in an invalid state, but if Python swallows the exception raised by __setitem__, then there is nothing to warn the programmer of this (until something else mysteriously fails to process the ListOfLists later on). The result is code that is hard to debug. Now, we might notice that the above code would probably not raise a TypeError, which is the exception we would expect if trying to mutate an immutable object, so we might revise the rule again: we only silence the exception if __iadd__ returns self and the exception is a TypeError. But even then, we have no guarantee that this is what the TypeError was meant to convey. A TypeError may have simply propagated up from other buggy code that was indirectly called by __setitem__, a legitimate error that should be brought to the programmer's attention. tldr: Swallowing exceptions is a risky practice unless you're quite sure that you know where the exception is coming from and that it's safe to ignore, and the language runtime should practically never do this. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Saturday, March 1, 2014 8:54:43 AM UTC-6, Mark Lawrence wrote: you're also using google groups... it doesn't wrap paragraphs correctly... please read and action this https://wiki.python.org/moin/GoogleGroupsPython... it does wrap paragraphs correctly... it also prevents us seeing double line spacing... do you get my drift??? hi Mark, yes, I understand. I am using google groups (for the moment) and it does not wrap correctly, at least not on my end. It also adds additional carrots that make replying difficult. The elipses thing is a bad habit which I 'picked up' in academia lately. We use elipses frequently in papers (also this---which also annoys people) and there is no problem. In a writing context the elipses seem to function for some people (intentionally) as 'um'. When humans speak they use the word 'um' to pause between though context and its usually unintentional at the surface. Often when I write the elipses (...) show up for the same reason. I really don't know that I'm doing it... it just happens. So, I am working on it. No, you're not the first to mention it. Its one of my many flaws. Thanks for keeping me accountable. Enjoy the day! marcus -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Friday, February 28, 2014 11:16:18 PM UTC-6, Ian wrote: How would you propose doing that? Bear in mind that while Python knows that tuples specifically are immutable, it doesn't generally know whether a type is immutable. hi Ian, I thought of something else after I slept on it, so to speak. Consider this: s=(2,3,[42,43,44],7) s[2] [42, 43, 44] s[2] is a list. We should be able to change s[2] even though its within a tuple, like the: [42, 43, 44] s[2].append(45) s[2] [42, 43, 44, 45] = no error or like this: s[2] [42, 43, 44, 45] s[2]=s[2]+[46] Traceback (most recent call last): File pyshell#8, line 1, in module s[2]=s[2]+[46]==error, but look below== TypeError: 'tuple' object does not support item assignment s[2] [42, 43, 44, 45]=also no change to list !! The point I'm trying to make with this post is that s[2]+=[46] and s[2]=s[2]+[46] are handled inconsistently. In each case (whether the list is part of a tuple or not) s[2] is a valid list and should be mutable. In every case, in fact, it is. In each case, in fact, the change might occur. But, the change occurs in the first case (append) without an error. The change occurs in the next case too (+=) but with an error not related to the tuple. The change does not occur in the third case (s=s+ with an error) but the list is not changed ! We all know this is not good. We also all know that its because of implementation details the user/coder should not need to know. QED: a bug. Having slept on it, I am remembering my Zen of Python: Errors should never pass silently, unless explicitly silenced. It looks to me like (+=) in this case is not being handled correctly under the covers. At any rate the three scenarios of trying to change a mutable list as an immutable tuple item should be handled consistently. Either we can change the tuple item (if its a list) or we can't. Also, if we cannot then the expected error should be consistent. Wow. I think I got that whole thing out without one elipses. Cheers -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Le 01/03/2014 22:21, Mark H. Harris a écrit : The point I'm trying to make with this post is that s[2]+=[46] and s[2]=s[2]+[46] are handled inconsistently. For my own, the fact that, in Python, a_liste += e_elt gives a different result than a_list = a_list + e_elt is a big source of trouble... I don't really find a good reason to justify that : a_list += spam gives a valid result, when a_list = a_list + spam is not allowed. Why the former is like a_list.extend() in the first place ? (well, it's not exactly an extend() as extend returns None, not +=). And if both operations were equivalent, i think my initial question about tuples will be clear and the behavio(u)r would me more consistent for me : tu = (10, [10, 30]) tu[1] = tu[1] + [20]-- Big mistake! TypeError: 'tuple' object does not support item assignment -- ok tu (10, [10, 30]) -- ok too... In fact, i think i'm gonna forget += on lists :) -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Saturday, March 1, 2014 3:21:44 PM UTC-6, Mark H. Harris wrote: On Friday, February 28, 2014 11:16:18 PM UTC-6, Ian wrote: hi Ian, I thought of something else after I slept on it, so to speak. Consider this: s=(2,3,[42,43,44],7) s[2] [42, 43, 44] s[2] is a list. We should be able to change s[2] even though its within a tuple, like this: [42, 43, 44] s[2].append(45) s[2] [42, 43, 44, 45] = no error or like this: s[2] [42, 43, 44, 45] s[2]=s[2]+[46] Traceback (most recent call last): File pyshell#8, line 1, in module s[2]=s[2]+[46]==error, but look below== TypeError: 'tuple' object does not support item assignment s[2] [42, 43, 44, 45]=also no change to list !! The point I'm trying to make with this post is that s[2]+=[46] and s[2]=s[2]+[46] are handled inconsistently. In each case (whether the list is part of a tuple or not) s[2] is a valid list and should be mutable. In every case, in fact, it is. In each case, in fact, the change might occur. But, the change occurs in the first case (append) without an error. The change occurs in the next case too (+=) but with an error not related to the tuple. The change does not occur in the third case (s=s+ with an error) but the list is not changed ! We all know this is not good. We also all know that its because of implementation details the user/coder should not need to know. QED: a bug. Having slept on it, I am remembering my Zen of Python: Errors should never pass silently, unless explicitly silenced. It looks to me like (+=) in this case is not being handled correctly under the covers. At any rate the three scenarios of trying to change a mutable list as an immutable tuple item should be handled consistently. Either we can change the tuple item (if its a list) or we can't. Also, if we cannot then the expected error should be consistent. Cheers Hi , I'm experimenting a bit with TextWrangler and gg to see if I can at least solve my double spacing and wrap problems with cut a paste for the moment. Looks ok? Thanks -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Saturday, March 1, 2014 8:04:32 PM UTC-6, Eric Jacoboni wrote: In fact, i think i'm gonna forget += on lists :) hi Eric, well, that might be extreme, but you certainly want to avoid trying to change an immutable. Something you might want to consider is trying something like creating a new immutable. In the example we've been looking at I think I would change the tuple list item with an append(); however, you might try building a new tuple instead of trying to change an existing tuple item. I must admit that I would never have found this bug myself, because I don't try to change tuples (at all). Anyway, I'm glad you found it -- we all learn and that's the good news. Cheers -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On 27 February 2014 16:14, Chris Angelico ros...@gmail.com wrote: On Fri, Feb 28, 2014 at 3:01 AM, Eric Jacoboni eric.jacob...@gmail.com wrote: a_tuple = (spam, [10, 30], eggs) a_tuple[1] += [20] Traceback (most recent call last): File stdin, line 1, in module TypeError: 'tuple' object does not support item assignment Ok... I accept this message as += is a reassignment of a_tuple[1] and a tuple is immutable... But, then, why a_tuple is still modified? This is a common confusion. The += operator does two things. First, it asks the target to please do an in-place addition of the other operand. Then, it takes whatever result the target gives back, and assigns it back into the target. So with a list, it goes like this: foo = [10, 30] foo.__iadd__([20]) [10, 30, 20] foo = _ Would it be better to add a check here, such that if this gets raised to the top-level it includes a warning (Addition was inplace; variable probably mutated despite assignment failure)? -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Sat, Mar 1, 2014 at 1:41 AM, Joshua Landau jos...@landau.ws wrote: Would it be better to add a check here, such that if this gets raised to the top-level it includes a warning (Addition was inplace; variable probably mutated despite assignment failure)? That'd require figuring out whether or not the variable was actually mutated, and that's pretty hard to work out. So there's a FAQ entry, which Zachary already posted: http://docs.python.org/3/faq/programming.html#why-does-a-tuple-i-item-raise-an-exception-when-the-addition-works Also, we just answer this question every now and then :) Presumably more often on -tutor than here. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Thursday, February 27, 2014 10:01:39 AM UTC-6, Eric Jacoboni wrote:
('spam', [10, 30, 20], 'eggs')
I get a TypeError for an illegal operation, but this operation is still
completed?
hi Eric, others have answered your question specifically, but the issue (which
is recurring) begs a couple of questions, which will also be recurring... ehem.
This little snag (if it can be called that) is a side effect from two python
inherent design considerations:
1) not to be a nudge, but dynamic typing is the first...
2) and the concept of immutability is the second.
I'll address the second first by asking a question... should an immutable type
(object) be able to hold (contain) mutable objects ... should tuples be allowed
to hold lists?
lists within a tuple should be converted to tuples.If you want a tuple to
hold a list, make it a list in the first place. Tuples should not be
changed... and as you point out... half changing a tuple is not a good
condition if there is an exception...
Which brings me to my second point... dynamic binding... (and everything
associated with that) is at play here too please, don't get me wrong,
this is not flame bait and I'm not advocating static binding, nor do I want
static binding, nor do I want to open that can of worms again... just saying.
I really think this is a bug; honestly. IMHO it should be an error to use +=
with an immutable type and that means not at all. In other words, the list
should not even be considered, because we're talking about changing a tuple...
which should not be changed (nor should its members be changed).
S the type does not support item assignment, yet the item got
assigned. This is at least inconsistent... and to my small mind means... bug
report.
:)
--
https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Le 01/03/2014 01:22, Mark H. Harris a écrit : I'll address the second first by asking a question... should an immutable type (object) be able to hold (contain) mutable objects ... should tuples be allowed to hold lists? lists within a tuple should be converted to tuples.If you want a tuple to hold a list, make it a list in the first place. You're perfectly right and that why i've corrected my code to use a list of lists instead of tuple of list. I was hoping Python would prevent me for such a mistake (because such a situation can't be cleary intentional, isn't ?). Now, i will use tuple only with immutable items. IMHO it should be an error to use += with an immutable type and that means not at all. In other words, the list should not even be considered, because we're talking about changing a tuple... which should not be changed (nor should its members be changed). I agree with that too... My error was to first consider the list, then the tuple... I should have considered the tuple first... Anyway, the TypeError should rollback, not commit the mistake. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Sat, Mar 1, 2014 at 11:22 AM, Mark H. Harris harrismh...@gmail.com wrote: lists within a tuple should be converted to tuples.If you want a tuple to hold a list, make it a list in the first place. Tuples should not be changed... and as you point out... half changing a tuple is not a good condition if there is an exception... A tuple is perfectly fine containing a list. If you want a tuple to be recursively immutable, then you're talking about hashability, and *then* yes, you need to convert everything into tuples - but a tuple is not just an immutable list. The two are quite different in pupose. I really think this is a bug; honestly. IMHO it should be an error to use += with an immutable type and that means not at all. In other words, the list should not even be considered, because we're talking about changing a tuple... which should not be changed (nor should its members be changed). Definitely not! Incrementing an integer with += is a perfectly normal thing to do: x = 5 x += 1 It's just a matter of knowing the language and understanding what's going on. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Friday, February 28, 2014 7:27:17 PM UTC-6, Eric Jacoboni wrote: I agree with that too... My error was to first consider the list, then the tuple... I should have considered the tuple first... Anyway, the TypeError should rollback, not commit the mistake. I believe so too, but I'm not one of the core devs. And they do not agree. Ever since day one with me and python I have questioned whether a tuple even makes sense. Well, certainly it does, because it has so much less overhead and yet it acts like a list (which for so many situations thats really what we want anyway... a list, that never changes). Tuples are great, for what they are designed to do. But now consider, why would I purposely want to place a mutable object within an immutable list? A valid question of high importance. Why indeed? I really believe IMHO that the error should have come when you made the list an item of a tuple. An immutable object should have NO REASON to contain a mutable object like list... I mean the whole point is to eliminate the overhead of a list ... why would the python interpreter allow you to place a mutable object within an immutable list in the first place. This is just philosophical, and yes, the core dev's are not going to agree with me on this either. I think the situation is interesting for sure... and it will surface again, you can count on that. Cheers -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Fri, Feb 28, 2014 at 5:22 PM, Mark H. Harris harrismh...@gmail.com wrote: I really think this is a bug; honestly. IMHO it should be an error to use += with an immutable type and that means not at all. In other words, the list should not even be considered, because we're talking about changing a tuple... which should not be changed (nor should its members be changed). How would you propose doing that? Bear in mind that while Python knows that tuples specifically are immutable, it doesn't generally know whether a type is immutable. The best it can do is try the operation and raise a TypeError if it fails. So you can special-case the code for += to fail earlier if the left side of the assignment is an tuple index, but it's not currently possible to do the same for arbitrary immutable user types. Even if that is solved, how you do propose preventing the simple workaround: tup = (0, [1, 2, 3], 4) thelist = tup[1] thelist += [5, 6] which currently works just fine. I don't think that entirely disallowing mutable objects inside tuples is an acceptable solution. For one, it would mean never allowing instances of custom classes. They could then basically only contain strings, numbers, frozensets, and other tuples. For another, my experience suggests that if I'm putting a list inside a tuple in the first place, I am generally not relying on the immutability of that tuple anyway, so I don't really see this as fixing an actual problem. The usual reason for wanting an immutable object is to hash it, and a tuple containing a list already can't be hashed. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Fri, Feb 28, 2014 at 6:27 PM, Eric Jacoboni eric.jacob...@gmail.com wrote: Anyway, the TypeError should rollback, not commit the mistake. The Python interpreter isn't a database. It can't rollback the object because the operation that was performed may not be reversible. Consider for example a socket class where the += operator is overloaded to send a message on the socket. The message can't be unsent. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Fri, Feb 28, 2014 at 9:45 PM, Mark H. Harris harrismh...@gmail.com wrote: I really believe IMHO that the error should have come when you made the list an item of a tuple. An immutable object should have NO REASON to contain a mutable object like list... I mean the whole point is to eliminate the overhead of a list ... why would the python interpreter allow you to place a mutable object within an immutable list in the first place. This is just philosophical, and yes, the core dev's are not going to agree with me on this either. One very common example of tuples containing lists is when lists are passed to any function that accepts *args, because the extra arguments are passed in a tuple. A similarly common example is when returning multiple objects from a function, and one of them happens to be a list, because again they are returned in a tuple. def f(*args): print(args) return (args[1:]) result = f(1, 2, 3, [4, 5]) (1, 2, 3, [4, 5]) print(result) (2, 3, [4, 5]) Both of these things are quite handy, and if you restrict tuples from containing lists, then you lose both of them (or you switch to lists and lose the optimization for no good reason). -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Friday, February 28, 2014 11:34:56 PM UTC-6, Ian wrote: One very common example of tuples containing lists is when lists are passed to any function that accepts *args, because the extra arguments are passed in a tuple. A similarly common example is when returning multiple objects from a function, and one of them happens to be a list, because again they are returned in a tuple. def f(*args): print(args) return (args[1:] result = f(1, 2, 3, [4, 5]) (1, 2, 3, [4, 5]) print(result) (2, 3, [4, 5]) I agree Ian... good points all. ... again, I'm not arguing with anyone... just saying that an error (whatever we mean by that) should not half-way-fail we are only pointing out the problem... we have not idea what the solution is yet. Intuitively everyone can see that there is a problem here... the debate cannot be answered either because of the inherent design of python (almost all of which we love). So, as they say, what is a mother to do? ... I mean, some people's kids... I don't know how I propose to handle the problem... I think the first step is getting everyone to agree that there IS a problem... then debate how to tackle the solution proposals. marcus -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Friday, February 28, 2014 11:16:18 PM UTC-6, Ian wrote: How would you propose doing that? Bear in mind that while Python knows that tuples specifically are immutable, it doesn't generally know whether a type is immutable. hi Ian, consider the problem... its a cake and eat it too scenario... you can't have your cake and eat it too This is what I mean... the error message is telling the user that it cannot do what he has requested, and yet IT DID. You have one of two scenarios: 1) the message is arbitrarily lying and it really can assign an immutable's item... (and it did!) or 2) It really should NOT assign an immutables item (the message is truth) but the action was allowed anyway despite the protocol and accepted policy... in which case the two scenarios add up to a giant logical inconsistency... QED a bug. There really is no way to get around this from the average user's perspective. And therefore, this is going to come up again and again and again... because coders don't like unexpected logical inconsistencies. It is not helpful either to explain it away by knowing how the policy works under the covers... the average user of the python language should not have to understand the policy of the underlying substructure... 1) either they can't assign the list item of an immutable tuple type (and the action is a flaw, say bug of 2) they really CAN set the immutable's list item type (which it did!) and the message is bogus. Its one way or the other we can't have our cake and eat it too... this is (one way or the other) a bug. IMHO -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Sat, Mar 1, 2014 at 5:16 PM, Mark H. Harris harrismh...@gmail.com wrote: This is what I mean... the error message is telling the user that it cannot do what he has requested, and yet IT DID. You have one of two scenarios: 1) the message is arbitrarily lying and it really can assign an immutable's item... (and it did!) or 2) It really should NOT assign an immutables item (the message is truth) but the action was allowed anyway despite the protocol and accepted policy... in which case the two scenarios add up to a giant logical inconsistency... QED a bug. It did not assign anything. The error was raised because the tuple rejects item assignment. It's that simple. The object was mutated before the assignment was attempted. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Friday, February 28, 2014 11:16:18 PM UTC-6, Ian wrote: How would you propose doing that? Bear in mind that while Python knows that tuples specifically are immutable, it doesn't generally know whether a type is immutable. I was going to bed, and then thought of the solution. Allow the action. Hold the error pending until the action either succeeds or fails for the item, in this case a list, and then post the error for the tuple depending... this would make the whole mess consistent, and even compatible with the idea of dynamic name and type binding... So, in other words, the action should be permitted, and actually was allowed, and the message is incorrectly reporting something that for this particular item is NOT true. The error message (and timing of same) is the bug (philosophically). Under the covers it may be (and probably is) more complicated. Good night everyone. Cheers -- https://mail.python.org/mailman/listinfo/python-list
Tuples and immutability
Hi,
I'm using Python 3.3 and i have a problem for which i've still not found
any reasonable explanation...
a_tuple = (spam, [10, 30], eggs)
a_tuple[1] += [20]
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: 'tuple' object does not support item assignment
Ok... I accept this message as += is a reassignment of a_tuple[1] and a
tuple is immutable...
But, then, why a_tuple is still modified?
a_tuple
('spam', [10, 30, 20], 'eggs')
I get a TypeError for an illegal operation, but this operation is still
completed?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Thu, Feb 27, 2014 at 10:01 AM, Eric Jacoboni eric.jacob...@gmail.com wrote:
Hi,
I'm using Python 3.3 and i have a problem for which i've still not found
any reasonable explanation...
a_tuple = (spam, [10, 30], eggs)
a_tuple[1] += [20]
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: 'tuple' object does not support item assignment
Ok... I accept this message as += is a reassignment of a_tuple[1] and a
tuple is immutable...
But, then, why a_tuple is still modified?
a_tuple
('spam', [10, 30, 20], 'eggs')
I get a TypeError for an illegal operation, but this operation is still
completed?
You're not the first person to have this question :)
http://docs.python.org/3/faq/programming.html#why-does-a-tuple-i-item-raise-an-exception-when-the-addition-works
--
Zach
--
https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Fri, Feb 28, 2014 at 3:01 AM, Eric Jacoboni eric.jacob...@gmail.com wrote:
I'm using Python 3.3 and i have a problem for which i've still not found
any reasonable explanation...
a_tuple = (spam, [10, 30], eggs)
a_tuple[1] += [20]
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: 'tuple' object does not support item assignment
Ok... I accept this message as += is a reassignment of a_tuple[1] and a
tuple is immutable...
But, then, why a_tuple is still modified?
a_tuple
('spam', [10, 30, 20], 'eggs')
I get a TypeError for an illegal operation, but this operation is still
completed?
This is a common confusion.
The += operator does two things. First, it asks the target to please
do an in-place addition of the other operand. Then, it takes whatever
result the target gives back, and assigns it back into the target. So
with a list, it goes like this:
foo = [10, 30]
foo.__iadd__([20])
[10, 30, 20]
foo = _
Note that the second step returned a three-element list. Then the +=
operator stuffs that back into the source. In the case of a list, it
does that addition by extending the list, and then returning itself.
The putting-back-into-the-target step fails with a tuple, because a
tuple's members can't be reassigned. But the list has already been
changed by that point. So, when you go to look at it, you see a
changed list.
You can call on this behaviour more safely by using the append() or
extend() methods.
a_tuple = (spam, [10, 30], eggs)
a_tuple[1].extend([20])
a_tuple
('spam', [10, 30, 20], 'eggs')
a_tuple = (spam, [10, 30], eggs)
a_tuple[1].append(20)
a_tuple
('spam', [10, 30, 20], 'eggs')
(append will add one more element, extend is roughly the same as you
had). Then you're not trying to assign to the tuple, but you're still
mutating the list.
Hope that helps!
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Eric Jacoboni eric.jacob...@gmail.com: a_tuple[1] += [20] Traceback (most recent call last): File stdin, line 1, in module TypeError: 'tuple' object does not support item assignment [...] But, then, why a_tuple is still modified? That's because the += operator 1. modifies the list object in place 2. tries to replace the tuple slot with the list (even though the list hasn't changed) It's Step 2 that raises the exception, but the damage has been done already. One may ask why the += operator modifies the list instead of creating a new object. That's just how it is with lists. BTW, try: a_tuple[1].append(20) a_tuple[1].extend([20]) Try also: a_tuple[1] = a_tuple[1] Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
Le 27/02/2014 17:13, Zachary Ware a écrit : You're not the first person to have this question :) http://docs.python.org/3/faq/programming.html#why-does-a-tuple-i-item-raise-an-exception-when-the-addition-works Oh yes, i was aware of this explanation (thanks to Chris for his answer, too)... and that's why i wrote reasonable :) I know i should use append() or extend() and i understand the tricky implications of += in this context. But, imho, it's far from being a intuitive result, to say the least. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Fri, Feb 28, 2014 at 3:27 AM, Eric Jacoboni eric.jacob...@gmail.com wrote: But, imho, it's far from being a intuitive result, to say the least. It's unintuitive, but it's a consequence of the way += is defined. If you don't want assignment, don't use assignment :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Thu, Feb 27, 2014 at 10:27 AM, Eric Jacoboni eric.jacob...@gmail.com wrote:
Le 27/02/2014 17:13, Zachary Ware a écrit :
You're not the first person to have this question :)
http://docs.python.org/3/faq/programming.html#why-does-a-tuple-i-item-raise-an-exception-when-the-addition-works
Oh yes, i was aware of this explanation (thanks to Chris for his answer,
too)... and that's why i wrote reasonable :)
I know i should use append() or extend() and i understand the tricky
implications of += in this context. But, imho, it's far from being a
intuitive result, to say the least.
Well, once you understand what's actually going on, it's the result
that you should expect. The FAQ entry I linked to exists to help
people get to that point.
To answer your specific questions:
But, then, why a_tuple is still modified?
It is not. a_tuple is still ('spam', list object at specific
address, 'eggs'), exactly the same before and after the attempted
a_tuple[1] += [20]. The change is internal to list object at
specific address.
I get a TypeError for an illegal operation, but this operation is still
completed?
Half completed. The extension of list object at specific address
happened as expected, but the assignment of list object at specific
address to a_tuple[1] didn't. It looks like it did, though, because
the assignment was just trying to assign the same object to the same
index.
--
Zach
--
https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Thu, Feb 27, 2014 at 10:33 AM, Chris Angelico ros...@gmail.com wrote: On Fri, Feb 28, 2014 at 3:27 AM, Eric Jacoboni eric.jacob...@gmail.com wrote: But, imho, it's far from being a intuitive result, to say the least. It's unintuitive, but it's a consequence of the way += is defined. If you don't want assignment, don't use assignment :) ChrisA Where is `.__iadd__()` called outside of `list += X`? If the only difference from `.extend()` is that it returns `self`, but the list was already modified anyway, why bother with reassignment? -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Fri, Feb 28, 2014 at 8:47 AM, Nick Timkovich prometheus...@gmail.com wrote: On Thu, Feb 27, 2014 at 10:33 AM, Chris Angelico ros...@gmail.com wrote: On Fri, Feb 28, 2014 at 3:27 AM, Eric Jacoboni eric.jacob...@gmail.com wrote: But, imho, it's far from being a intuitive result, to say the least. It's unintuitive, but it's a consequence of the way += is defined. If you don't want assignment, don't use assignment :) ChrisA Where is `.__iadd__()` called outside of `list += X`? If the only difference from `.extend()` is that it returns `self`, but the list was already modified anyway, why bother with reassignment? Not everything handles += that way. You can't mutate the integer 5 into 7 because someone had 5 in x and wrote x += 2. So it has to reassign. Actually, integers just don't define __iadd__, but the principle applies. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Thu, Feb 27, 2014 at 2:47 PM, Nick Timkovich prometheus...@gmail.com wrote: On Thu, Feb 27, 2014 at 10:33 AM, Chris Angelico ros...@gmail.com wrote: On Fri, Feb 28, 2014 at 3:27 AM, Eric Jacoboni eric.jacob...@gmail.com wrote: But, imho, it's far from being a intuitive result, to say the least. It's unintuitive, but it's a consequence of the way += is defined. If you don't want assignment, don't use assignment :) ChrisA Where is `.__iadd__()` called outside of `list += X`? If the only difference from `.extend()` is that it returns `self`, but the list was already modified anyway, why bother with reassignment? x += y is meant to be equivalent, except possibly in-place and more efficient, than x = x + y. If you skip the assignment, and that assignment is meaningful to whatever the left side may be (e.g. assigning to a descriptor or something that invokes __setitem__ or __setattr__), then the operation is not equivalent. -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
On Thu, 27 Feb 2014 18:19:09 +0200 Marko Rauhamaa ma...@pacujo.net wrote: Eric Jacoboni eric.jacob...@gmail.com: a_tuple[1] += [20] Traceback (most recent call last): File stdin, line 1, in module TypeError: 'tuple' object does not support item assignment [...] But, then, why a_tuple is still modified? That's because the += operator 1. modifies the list object in place 2. tries to replace the tuple slot with the list (even though the list hasn't changed) It's Step 2 that raises the exception, but the damage has been done already. One may ask why the += operator modifies the list instead of creating a new object. That's just how it is with lists. BTW, try: a_tuple[1].append(20) a_tuple[1].extend([20]) Try also: a_tuple[1] = a_tuple[1] [...] Also try: x = a_tuple[1] #What's in a name? x is a_tuple[1] #Obviously, but: x += [1] #No error a_tuple[1] += [1] #Error Same object, just a different name - but a different result. I get why, but still find that odd. -- John -- https://mail.python.org/mailman/listinfo/python-list
Re: Tuples and immutability
John O'Hagan resea...@johnohagan.com: Same object, just a different name - but a different result. I get why, but still find that odd. The general principle is stated in the language specification: URL: http://docs.python.org/3.2/reference/simple_stmts.html #augmented-assignment-statements: Also, when possible, the actual operation is performed in-place, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead. [...] with the exception of the possible in-place behavior, the binary operation performed by augmented assignment [x += y] is the same as the normal binary operations [x = x + y]. We should call += dual-use technology. Marko -- https://mail.python.org/mailman/listinfo/python-list
