Re: Copy-on-write friendly Python garbage collection (Posting On Python-List Prohibited)

[Niles Rogoff]

> Niles, if you want to claim wxjmfauth is right, you'll have to present
> some actual evidence.  He's claimed for years that Python's Unicode
> support is buggy (as he does here), without ever demonstrating a bug.
> We've long ago tired of trying to reason with him.
> 
> The tradeoffs of memory use for algorithmic complexity are well
> understood, and have been endlessly discussed. There is not an
> "obviously right" answer to how to make those tradeoffs.
> 
> --Ned.

Aah, I didn't know, I'm new here. I thought he was referring to UTF-16 
being inefficient, which is mostly what I was agreeing with.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write friendly Python garbage collection (Posting On Python-List Prohibited)

[Ned Batchelder]

On 1/1/18 1:49 PM, Niles Rogoff wrote:

On Mon, 01 Jan 2018 10:42:58 -0800, breamoreboy wrote:


On Monday, January 1, 2018 at 10:14:59 AM UTC, wxjm...@gmail.com wrote:

Le lundi 1 janvier 2018 08:35:53 UTC+1, Lawrence D’Oliveiro a écrit :

On Monday, January 1, 2018 at 7:52:48 AM UTC+13, Paul Rubin wrote:

I wonder if things would suffer if they tried a more aggressive
approach and ditched refcounting completely.

One thing that would suffer is Python’s well-behaved memory usage.
You would need to start imposing heap usage limits, like you do in
Java.

Memory:


sys.getsizeof('abcdefghij' + '$')

36

sys.getsizeof('abcdefghij' + '€')

60

sys.getsizeof(('abcdefghij' + '€').encode('utf-8'))

30

sys.getsizeof('abcdefghij' + '\U0001')

84

sys.getsizeof(('abcdefghij' + '\U0001').encode('utf-8'))

31



Performance:
"anti - utf-32"

Buggyness:
Better to not comment.

Python is the single language, which is presenting the opposite of what
Unicode.org offers on the side of memory *and* on the side of
performance, utf-8 *and* utf-32 !

Happy new year.

He's right though. I would encourage anyone interested to check out
http://utf8everywhere.org/

Niles, if you want to claim wxjmfauth is right, you'll have to present 
some actual evidence.  He's claimed for years that Python's Unicode 
support is buggy (as he does here), without ever demonstrating a bug.  
We've long ago tired of trying to reason with him.


The tradeoffs of memory use for algorithmic complexity are well 
understood, and have been endlessly discussed. There is not an 
"obviously right" answer to how to make those tradeoffs.


--Ned.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write friendly Python garbage collection

[breamoreboy]

On Monday, January 1, 2018 at 12:53:03 PM UTC, Wu Xi wrote:
> breamoreboy:
> > On Sunday, December 31, 2017 at 6:19:13 PM UTC, Wu Xi wrote:
> >> breamoreboy:
> >>> An interesting write up on something that is incorporated into Python 3.7 
> >>> https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf
> >>
> >> Appearantly, Erlang is the way to go, when it comes to web frameworks.
> > 
> > What has that got to do with the subject of this thread?
> 
> Well, a little implicitly. Pointing out Erlang superiority in a pythonic 
> newsgroup is, of course, heresy.

Python is fourth in the latest TIOBE index, Erlang doesn't even make the top 
20, so in what way is it superior?

Python doesn't need Pinky and the Brain in its quest to take over the world.

--
Kindest regards.

Mark Lawrence.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write friendly Python garbage collection (Posting On Python-List Prohibited)

[Niles Rogoff]

On Mon, 01 Jan 2018 10:42:58 -0800, breamoreboy wrote:

> On Monday, January 1, 2018 at 10:14:59 AM UTC, wxjm...@gmail.com wrote:
>> Le lundi 1 janvier 2018 08:35:53 UTC+1, Lawrence D’Oliveiro a écrit :
>> > On Monday, January 1, 2018 at 7:52:48 AM UTC+13, Paul Rubin wrote:
>> > > I wonder if things would suffer if they tried a more aggressive
>> > > approach and ditched refcounting completely.
>> > 
>> > One thing that would suffer is Python’s well-behaved memory usage.
>> > You would need to start imposing heap usage limits, like you do in
>> > Java.
>> 
>> Memory:
>> 
>> >>> sys.getsizeof('abcdefghij' + '$')
>> 36
>> >>> sys.getsizeof('abcdefghij' + '€')
>> 60
>> >>> sys.getsizeof(('abcdefghij' + '€').encode('utf-8'))
>> 30
>> >>> sys.getsizeof('abcdefghij' + '\U0001')
>> 84
>> >>> sys.getsizeof(('abcdefghij' + '\U0001').encode('utf-8'))
>> 31
>> >>>
>> >>>
>> Performance:
>> "anti - utf-32"
>> 
>> Buggyness:
>> Better to not comment.
>> 
>> Python is the single language, which is presenting the opposite of what
>> Unicode.org offers on the side of memory *and* on the side of
>> performance, utf-8 *and* utf-32 !
>> 
>> Happy new year.
> 

He's right though. I would encourage anyone interested to check out 
http://utf8everywhere.org/
> Your usual drivel.  When are you going to stop banging this drum, you've
> done nothing else for the past five years?

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write friendly Python garbage collection

[Wu Xi]

breamore...@gmail.com:
> On Sunday, December 31, 2017 at 6:19:13 PM UTC, Wu Xi wrote:
>> breamoreboy:
>>> An interesting write up on something that is incorporated into Python 3.7 
>>> https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf
>>
>> Appearantly, Erlang is the way to go, when it comes to web frameworks.
> 
> What has that got to do with the subject of this thread?

Well, a little implicitly. Pointing out Erlang superiority in a pythonic 
newsgroup is, of course, heresy.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write friendly Python garbage collection

[breamoreboy]

On Sunday, December 31, 2017 at 6:19:13 PM UTC, Wu Xi wrote:
> breamoreboy:
> > An interesting write up on something that is incorporated into Python 3.7 
> > https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf
> 
> Appearantly, Erlang is the way to go, when it comes to web frameworks.

What has that got to do with the subject of this thread?

--
Kindest regards.

Mark Lawrence.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write friendly Python garbage collection

[INADA Naoki]

FYI: https://bugs.python.org/issue31558
INADA Naoki  


On Mon, Jan 1, 2018 at 12:39 AM,   wrote:
> An interesting write up on something that is incorporated into Python 3.7 
> https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf
>
> --
> Kindest regards.
>
> Mark Lawrence.
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write friendly Python garbage collection

[Wu Xi]

breamore...@gmail.com:
> An interesting write up on something that is incorporated into Python 3.7 
> https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf

Appearantly, Erlang is the way to go, when it comes to web frameworks.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write friendly Python garbage collection

[Wu Xi]

they said they run the largest deployment of Django world-wide.

be it as it may...

many still consider the web guys to be the "funny people".

Why did they not switch over to Erlang ?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write friendly Python garbage collection

[Wu Xi]

breamore...@gmail.com:
> An interesting write up on something that is incorporated into Python 3.7 
> https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf

kewl

py 3.7 does not fully make install here, but it built and works, as far as I 
can tell
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Rick Johnson]

On Feb 2, 11:28 pm, Chris Angelico ros...@gmail.com wrote:
 On Fri, Feb 3, 2012 at 4:04 PM, Steven D'Aprano

 steve+comp.lang.pyt...@pearwood.info wrote:
  No matter what Python did, somebody would complain.

 +1

 This is, I think, the ultimate truth of the matter.

People would not complain if they did not care. The only useless
complaint is people complaining about other people complaining. And
the only thing worse than that is rabid fanboi brown-nosing!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Wolfram Hinderer]

On 3 Feb., 11:47, John O'Hagan resea...@johnohagan.com wrote:

 But isn't it equally true if we say that z = t[1], then t[1] += x is 
 syntactic sugar for z = z.__iadd__(x)? Why should that fail, if z can handle 
 it?

It's more like syntactic sugar for
y = t; z = y.__getitem__(1); z.__iadd__(x); y.__setitem__(1, z)

It's clear that only the last expression fails, after the mutation has
taken place.

Just in case you wonder about the y: you need it for more complicated
cases.
t[1][1] += [4] is syntactic sugar for
y = t.__getitem__(1); z = y.__getitem__(1); z.__iadd__([4]);
y.__setitem__(1, z)

That makes clear why there's no exception in this code:
 t = (0, [1, [2, 3]])
 t[1][1] += [4]
 t
(0, [1, [2, 3, 4]])

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Antoon Pardon]

On 02/03/2012 06:04 AM, Steven D'Aprano wrote:

Ultimately, there is no right answer, because the multitude of
requirements are contradictory. No matter what Python did, somebody would
complain.
Which makes me wonder why it was introduced at all, or at least so fast 
If you see the difference in speed in introducing augmented assignment 
vs how long it took to get conditional expression I start thinking of a 
bikeshed. In the first case we have something that raises semantic 
questions that are difficult to resolve, in the second case the 
semantics were clear, the big problem that delayed introduction was what 
syntax to use.


But the second took a lot longer to become part of the language than the 
first, which seems very odd to me.


--
Antoon Pardon

--
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[John O'Hagan]

On 03 Feb 2012 05:04:39 GMT
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote:

 On Fri, 03 Feb 2012 14:08:06 +1100, John O'Hagan wrote:
 
  I think we're 12 years late on this one. It's PEP 203 from 2000 and
  the key phrase was:
  
  The in-place function should always return a new reference, either
  to the old `x' object if the operation was indeed performed
  in-place, or to a new object.
  
  If this had read:
  
  The in-place function should return a reference to a new object if
  the operation was not performed in-place.
  
  or something like that, we wouldn't be discussing this.
 
 And what should it return if the operation *is* performed in-place?


Not knowing anything about the inner workings of the interpreter, I'm agnostic 
on that as long as it's not a new reference. Perhaps the old reference? 

[...snip undoubted reasons why returning None wouldn't work...]

I don't know what would work. Maybe it is insoluble. But didn't Hrvoje Niksic's 
post in this thread suggest it could have been implemented to work the way I'm 
saying, even supplying code to demonstrate it?

All I'm saying is that however it's implemented, x[i] += y should simply mutate 
x[i] in-place if x[i] implements that, otherwise it should do x[i] = x[i] + y. 
If I can say it under 25 words, surely it's implementable? (Whether it's 
practical to do so is another question.) 

The x[i] in x[i] += y can be seen as a reference to an object to be incremented 
rather than an assignment (despite the name). In that view, whether the name 
x[i] needs to be rebound to a new object, resulting in an assignment, depends 
on the capabilities of x[i], not x.

 
 Yes, the current behaviour is a Gotcha, but it's a Gotcha that makes
 good sense compared to the alternatives.

I think it's worse than a Gotcha. IMHO a Gothcha is, for example, the mutable 
default arguments thing, which makes sense once you get it. This one has the 
bizarre consequence that what happens when you operate on an object depends on 
which name you use for the object. Not to mention that it succeeds after 
raising an exception.

 Ultimately, augmented assignment is *assignment*, just like it says
 on the tin. t[1] += x is syntactic sugar for t[1] = t[1].__iadd__(x).
 It can't and shouldn't fail to raise an exception if t is a tuple,
 because tuple item assignment *must* fail.

That makes sense if we view it strictly as assignment (but in that case the 
mutation of t[1] should not occur either).
But isn't it equally true if we say that z = t[1], then t[1] += x is syntactic 
sugar for z = z.__iadd__(x)? Why should that fail, if z can handle it?

[...]
 
 Ultimately, there is no right answer, because the multitude of 
 requirements are contradictory. No matter what Python did, somebody
 would complain.
 

Not complaining, just trying to contribute to the best of my ability. :)

John
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[OKB (not okblacke)]

Steven D'Aprano wrote:

 Perhaps lists shouldn't define += at all, but then people will
 complain that mylist += another_list is slow. Telling them to use
 mylist.extend instead just makes them cranky. After all, mylist +
 another_list works, so why shouldn't += work?

It would work, it just wouldn't work in-place.

-- 
--OKB (not okblacke)
Brendan Barnwell
Do not follow where the path may lead.  Go, instead, where there is
no path, and leave a trail.
--author unknown
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[88888 Dihedral]

在 2012年1月14日星期六UTC+8上午6时48分29秒，Evan Driscoll写道：
 On 01/13/2012 03:20 PM, Neil Cerutti wrote:
  They perform the same action, but their semantics are different.
  operator+ will always return a new object, thanks to its
  signature, and operator+= shall never do so. That's the main
  difference I was getting at.

Well, in any associative operation with an identity implemented in a computer 
language  personally I believe the operator overloading part in C++ is   
another teasing trick.

 
Do we have to work out the algebra here?

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[John O'Hagan]

On Thu, 2 Feb 2012 01:34:48 -0500
Devin Jeanpierre jeanpierr...@gmail.com wrote:

 On Wed, Feb 1, 2012 at 10:18 PM, John O'Hagan
 resea...@johnohagan.com wrote:
  On Fri, 13 Jan 2012 10:40:47 -0800
  Ethan Furman et...@stoneleaf.us wrote:
 
  Steven D'Aprano wrote:
   Normally this is harmless, but there is one interesting little
   glitch you can get:
  
   t = ('a', [23])
   t[1] += [42]
   Traceback (most recent call last):
     File stdin, line 1, in module
   TypeError: 'tuple' object does not support item assignment
   t
   ('a', [23, 42])
 
  IMHO, this is worthy of bug-hood: shouldn't we be able to conclude
  from the TypeError that the assignment failed?
 
 It did fail. The mutation did not.

You're right, in fact, for me the surprise is that t[1] += is interpreted as 
an assignment at all, given that for lists (and other mutable objects which use 
+=) it is a mutation. Although as Steven says elsewhere, it actually is an 
assignment, but one which ends up reassigning to the same object. 
 
But it shouldn't be both. I can't think of another example of (what appears to 
be but is not) a single operation failing with an exception, but still doing 
exactly what you intended.

 
 I can't think of any way out of this misleadingness, although if you
 can that would be pretty awesome.

In the case above, the failure of the assignment is of no consequence. I think 
it would make more sense if applying += to a tuple element were treated (by 
the interpreter I suppose) only on the merits of the element, and not as an 
assignment to the tuple. 

John



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Steven D'Aprano]

On Thu, 02 Feb 2012 19:11:53 +1100, John O'Hagan wrote:

 You're right, in fact, for me the surprise is that t[1] += is
 interpreted as an assignment at all, given that for lists (and other
 mutable objects which use +=) it is a mutation. Although as Steven
 says elsewhere, it actually is an assignment, but one which ends up
 reassigning to the same object.
  
 But it shouldn't be both.

Do you expect that x += 1 should succeed? After all, increment and 
decrement numbers is practically THE use-case for the augmented 
assignment operators.

How can you expect x += 1 to succeed without an assignment? Numbers in 
Python are immutable, and they have to stay immutable. It would cause 
chaos and much gnashing of teeth if you did this:

x = 2
y = 7 - 5
x += 1
print y * 100
= prints 300

So if you want x += 1 to succeed, += must do an assignment.

Perhaps you are thinking that Python could determine ahead of time 
whether x[1] += y involved a list or a tuple, and not perform the finally 
assignment if x was a tuple. Well, maybe, but such an approach (if 
possible!) is fraught with danger and mysterious errors even harder to 
debug than the current situation. And besides, what should Python do 
about non-built-in types? There is no way in general to predict whether 
x[1] = something will succeed except to actually try it.

 I can't think of another example of (what
 appears to be but is not) a single operation failing with an exception,
 but still doing exactly what you intended.

Neither can I, but that doesn't mean that the current situation is not 
the least-worst alternative.


 I can't think of any way out of this misleadingness, although if you
 can that would be pretty awesome.
 
 In the case above, the failure of the assignment is of no consequence. I
 think it would make more sense if applying += to a tuple element were
 treated (by the interpreter I suppose) only on the merits of the
 element, and not as an assignment to the tuple.

How should the interpreter deal with other objects which happen to raise 
TypeError? By always ignoring it?

x = [1, None, 3]
x[1] += 2  # apparently succeeds

Or perhaps by hard-coding tuples and only ignoring errors for tuples? So 
now you disguise one error but not others?



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Thomas Rachel]

Am 13.01.2012 13:30 schrieb Chris Angelico:


It seems there's a distinct difference between a+=b (in-place
addition/concatenation) and a=a+b (always rebinding),


There is indeed.

a = a + b is a = a.__add__(b), while

a += b is a = a.__iadd__(b).

__add__() is supposed to leave the original object intact and return a 
new one, while __iadd__() is free to modify (preference, to be done if 
possible) or return a new one.


A immutable object can only return a new one, and its __iadd__() 
behaviour is the same as __add__().


A mutable object, however, is free to and supposed to modify itself and 
then return self.



Thomas
--
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Hrvoje Niksic]

Steven D'Aprano steve+comp.lang.pyt...@pearwood.info writes:

 Perhaps you are thinking that Python could determine ahead of time
 whether x[1] += y involved a list or a tuple, and not perform the
 finally assignment if x was a tuple. Well, maybe, but such an approach
 (if possible!) is fraught with danger and mysterious errors even
 harder to debug than the current situation. And besides, what should
 Python do about non-built-in types? There is no way in general to
 predict whether x[1] = something will succeed except to actually try
 it.

An alternative approach is to simply not perform the final assignment if
the in-place method is available on the contained object.  No prediction
is needed to do it, because the contained object has to be examined
anyway.  No prediction is needed, just don't.  Currently,
lhs[ind] += rhs is implemented like this:

item = lhs[ind]
if hasattr(item, '__iadd__'):
lhs.__setitem__(ind, item.__iadd__(rhs))
else:
lhs.__setitem__(ind, item + rhs)
# (Note item assignment in both if branches.)

It could, however, be implemented like this:

item = lhs[ind]
if hasattr(item, '__iadd__'):
item += rhs  # no assignment, item supports in-place change
else:
lhs.__setitem__(ind, lhs[ind] + rhs)

This would raise the exact same exception in the tuple case, but without
executing the in-place assignment.  On the other hand, some_list[ind] += 1
would continue working exactly the same as it does now.

In the same vein, in-place methods should not have a return value
(i.e. they should return None), as per Python convention that functions
called for side effect don't return values.

The alternative behavior is unfortunately not backward-compatible (it
ignores the return value of augmented methods), so I'm not seriously
proposing it, but I believe it would have been a better implementation
of augmented assignments than the current one.  The present interface
doesn't just bite those who try to use augmented assignment on tuples
holding mutable objects, but also those who do the same with read-only
properties, which is even more reasonable.  For example, obj.list_attr
being a list, one would expect that obj.list_attr += [1, 2, 3] does the
same thing as obj.list_attr.extend([1, 2, 3]).  And it almost does,
except it also follows up with an assignment after the list has already
been changed, and the assignment to a read-only property raises an
exception.  Refusing to modify the list would have been fine, modifying
it without raising an exception (as described above) would have been
better, but modifying it and *then* raising an exception is a surprise
that takes some getting used to.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[88888 Dihedral]

在 2012年1月14日星期六UTC+8上午6时48分29秒，Evan Driscoll写道：
 On 01/13/2012 03:20 PM, Neil Cerutti wrote:
  They perform the same action, but their semantics are different.
  operator+ will always return a new object, thanks to its
  signature, and operator+= shall never do so. That's the main
  difference I was getting at.
 
 I was talking about the combination of + and =, since the discussion is 
 about 'a = a + b' vs 'a += b', not 'a + b' vs 'a += b' (where the 
 differences are obvious).
 
 And I stand by my statement. In 'a = a + b', operator+ obviously returns 
 a new object, but operator= should then go and assign the result to and 
 return a reference to 'a', just like how 'a += b' will return a 
 reference to 'a'.


The operation a+b means add(a,b) and returns a result instance, furthermore a 
and b can't be modified.

The expression a = a+b are two operations not one. But in C or C++ the  problem 
is mixing operations and expressions in a free style allowed.
 
The operation a+=b means a modified by b and b can't be changed.
Note that no new instance is necessary in a+=b.
 
 

 If you're working in C++ and overload your operators so that 'a += b' 
 and 'a = a + b' have different observable behaviors (besides perhaps 
 time), then either your implementation is buggy or your design is very 
 bad-mannered.
 
 Evan

Do you mean the result instances after 'a+=b' and 'a=a+b' or 
the actions of behaviors of instances involved  in performing 'a+=b' and 
'a=a+b'?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[John O'Hagan]

On 02 Feb 2012 09:16:40 GMT
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote:

 On Thu, 02 Feb 2012 19:11:53 +1100, John O'Hagan wrote:
 
  You're right, in fact, for me the surprise is that t[1] += is
  interpreted as an assignment at all, given that for lists (and other
  mutable objects which use +=) it is a mutation. Although as Steven
  says elsewhere, it actually is an assignment, but one which ends up
  reassigning to the same object.
   
  But it shouldn't be both.
 
 Do you expect that x += 1 should succeed? After all, increment and 
 decrement numbers is practically THE use-case for the augmented 
 assignment operators.
 
 How can you expect x += 1 to succeed without an assignment? 

I don't; obviously, for immutable objects assignment is the only possibility.

[...]
 
 Perhaps you are thinking that Python could determine ahead of time 
 whether x[1] += y involved a list or a tuple, and not perform the
 finally assignment if x was a tuple. Well, maybe, but such an
 approach (if possible!) is fraught with danger and mysterious errors
 even harder to debug than the current situation. And besides, what
 should Python do about non-built-in types? There is no way in general
 to predict whether x[1] = something will succeed except to actually
 try it.

It's not so much about the type of x but that of x[1]. Wouldn't it be possible 
to omit the assignment simply if the object referred to by x[1] uses += 
without creating a new object? That way, some_tuple[i] += y will succeed if 
some_tuple[i] is a list but not with, say, an int. That seems reasonable to me.

[...]

  
  In the case above, the failure of the assignment is of no
  consequence. I think it would make more sense if applying += to a
  tuple element were treated (by the interpreter I suppose) only on
  the merits of the element, and not as an assignment to the tuple.
 
 How should the interpreter deal with other objects which happen to
 raise TypeError? By always ignoring it?
 
 x = [1, None, 3]
 x[1] += 2  # apparently succeeds
 
 Or perhaps by hard-coding tuples and only ignoring errors for tuples?
 So now you disguise one error but not others?

I'm not suggesting either of those. None can't be modified in place. But for 
objects which can, wouldn't omitting the final assignment prevent the TypeError 
in the first place?

John

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[MRAB]

On 02/02/2012 10:53, Hrvoje Niksic wrote:

Steven D'Apranosteve+comp.lang.pyt...@pearwood.info  writes:


 Perhaps you are thinking that Python could determine ahead of time
 whether x[1] += y involved a list or a tuple, and not perform the
 finally assignment if x was a tuple. Well, maybe, but such an approach
 (if possible!) is fraught with danger and mysterious errors even
 harder to debug than the current situation. And besides, what should
 Python do about non-built-in types? There is no way in general to
 predict whether x[1] = something will succeed except to actually try
 it.


An alternative approach is to simply not perform the final assignment if
the in-place method is available on the contained object.  No prediction
is needed to do it, because the contained object has to be examined
anyway.  No prediction is needed, just don't.  Currently,
lhs[ind] += rhs is implemented like this:

item = lhs[ind]
if hasattr(item, '__iadd__'):
 lhs.__setitem__(ind, item.__iadd__(rhs))
else:
 lhs.__setitem__(ind, item + rhs)
# (Note item assignment in both if branches.)

It could, however, be implemented like this:

item = lhs[ind]
if hasattr(item, '__iadd__'):
 item += rhs  # no assignment, item supports in-place change
else:
 lhs.__setitem__(ind, lhs[ind] + rhs)

This would raise the exact same exception in the tuple case, but without
executing the in-place assignment.  On the other hand, some_list[ind] += 1
would continue working exactly the same as it does now.

In the same vein, in-place methods should not have a return value
(i.e. they should return None), as per Python convention that functions
called for side effect don't return values.

The alternative behavior is unfortunately not backward-compatible (it
ignores the return value of augmented methods), so I'm not seriously
proposing it, but I believe it would have been a better implementation
of augmented assignments than the current one.


[snip]
Could it not perform the assignment if the reference returned by
__iadd__ is the same as the current reference?

For example:

t[0] += x

would do:

r = t[0].__iadd__(x)
if t[0] is not r:
t[0] = r

Should failed assignment be raising TypeError? Is it really a type
error?
--
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Devin Jeanpierre]

On Thu, Feb 2, 2012 at 11:28 AM, MRAB pyt...@mrabarnett.plus.com wrote:
 Should failed assignment be raising TypeError? Is it really a type
 error?

A failed setitem should be a TypeError as much as a failed getitem
should. Should 1[0] be a TypeError?

-- Devin
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Terry Reedy]

On 2/2/2012 9:17 AM, John O'Hagan wrote:


It's not so much about the type of x but that of x[1]. Wouldn't it be
possible to omit the assignment simply if the object referred to by
x[1] uses += without creating a new object? That way, some_tuple[i]
+= y will succeed if some_tuple[i] is a list but not with, say, an
int. That seems reasonable to me.


There was considerable discussion of the exact semantics of augmented 
operations when they were introduced. I do not remember if that 
particular idea was suggested (and rejected) or not. You could try to 
look at the PEP, if there is one, or the dicussion ( probably on pydev 
list).


--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Evan Driscoll]

On 01/-10/-28163 01:59 PM, 8 Dihedral wrote:

If you're working in C++ and overload your operators so that 'a +='
and 'a = + b' have different observable behaviors (besides perhaps
time), then either your implementation is buggy or your design is very
bad-mannered.

Evan


Do you mean the result instances after 'a+= and 'a=a+b' or
the actions of behaviors of instances involved  in performing 'a+= and 'a=a+b'?



I mean if which operation you called is distinguishable in any way 
besides the time it takes to run or by tracing it through in a debugger


That means:

1. The value of 'a' should be the same after executing 'a+=b' and
   'a=a+b'
2. The actual result of the expression should be the same in both cases
   (in both cases it should be a reference to a)
3. Any additional side effects performed (ew!) should be the same in
   both cases

Evan
--
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[John O'Hagan]

On Thu, 02 Feb 2012 12:25:00 -0500
Terry Reedy tjre...@udel.edu wrote:

 On 2/2/2012 9:17 AM, John O'Hagan wrote:
 
  It's not so much about the type of x but that of x[1]. Wouldn't it
  be possible to omit the assignment simply if the object referred to
  by x[1] uses += without creating a new object? That way,
  some_tuple[i] += y will succeed if some_tuple[i] is a list but not
  with, say, an int. That seems reasonable to me.
 
 There was considerable discussion of the exact semantics of augmented 
 operations when they were introduced. I do not remember if that 
 particular idea was suggested (and rejected) or not. You could try to 
 look at the PEP, if there is one, or the dicussion ( probably on
 pydev list).
 

I think we're 12 years late on this one. It's PEP 203 from 2000 and the key 
phrase was:

The in-place function should always return a new reference, either
to the old `x' object if the operation was indeed performed
in-place, or to a new object.

If this had read:

The in-place function should return a reference to a new object
if the operation was not performed in-place.

or something like that, we wouldn't be discussing this.

The discussion on py-dev at the time was quite limited but there was some 
lively debate on this list the following year (in the context of widespread 
controversy over new-fangled features which also included list comprehensions 
and generators), to which the BDFL's response was:

You shouldn't think += is confusing because sometimes it modifies an
object and sometimes it does.  Gee, there are lots of places where
something that's *spelled* the same has a different effect depending
on the object types involved.


That's true, but I don't think there should be a different effect depending on 
what _name_ we use for an operand:

 t=([],)
 l=t[0]
 l is t[0]
True
 l+=[1]
 t
([1],)
 t[0]+=[1]
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: 'tuple' object does not support item assignment
 t
([1, 1],)
 l is t[0]
True

Same object, same operator, different name, different outcome. Maybe that was 
obvious from the foregoing discussion, but it shocked me when put that way.

John
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Steven D'Aprano]

On Fri, 03 Feb 2012 14:08:06 +1100, John O'Hagan wrote:

 I think we're 12 years late on this one. It's PEP 203 from 2000 and the
 key phrase was:
 
 The in-place function should always return a new reference, either to
 the old `x' object if the operation was indeed performed in-place, or to
 a new object.
 
 If this had read:
 
 The in-place function should return a reference to a new object if the
 operation was not performed in-place.
 
 or something like that, we wouldn't be discussing this.

And what should it return if the operation *is* performed in-place? 
Don't return anything is not an option, Python doesn't have procedures. 
That implies that __iadd__ etc. should return None. But two problems come 
to mind:

1) Using None as an out-of-band signal to the interpreter to say don't 
perform the assignment makes it impossible for the augmented assignment 
method to return None as the result. If we only think about numeric 
operations like x += 1 then we might not care, but once you consider the 
situation more widely the problem is clear:

x = Fact(foo)
y = Fact(bar)
x  y  # Returns a composite Fact, or None if they are contradictory

With your suggestion, x = y fails to work, but only sometimes. And when 
it fails, it doesn't fail with an explicit exception, but silently fails 
and then does the wrong thing. This makes debugging a horror.

2) And speaking of debugging, sometimes people forget to include the 
return statement in methods. Normally, the left hand side of the 
assignment then gets set to None, and the error is pretty obvious as soon 
as you try to do something with it. But with your suggestion, instead of 
getting an exception, it silently fails, and your code does the wrong 
thing.

I suppose that they could have invented a new sentinel, or a special 
exception to be raised as a signal, but that's piling complication on top 
of complication, and it isn't clear to me that it's worth it for an 
obscure corner case.

Yes, the current behaviour is a Gotcha, but it's a Gotcha that makes good 
sense compared to the alternatives.

Ultimately, augmented assignment is *assignment*, just like it says on 
the tin. t[1] += x is syntactic sugar for t[1] = t[1].__iadd__(x). It 
can't and shouldn't fail to raise an exception if t is a tuple, because 
tuple item assignment *must* fail.

The problem is that lists treat __iadd__ as an in-place optimization, and 
this clashes with tuple immutability. But if lists *didn't* treat 
__iadd__ as in-place, people would complain when they used it directly 
without a tuple wrapper. 

Perhaps lists shouldn't define += at all, but then people will complain 
that mylist += another_list is slow. Telling them to use mylist.extend 
instead just makes them cranky. After all, mylist + another_list works, 
so why shouldn't += work?

Ultimately, there is no right answer, because the multitude of 
requirements are contradictory. No matter what Python did, somebody would 
complain.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Chris Angelico]

On Fri, Feb 3, 2012 at 4:04 PM, Steven D'Aprano
steve+comp.lang.pyt...@pearwood.info wrote:
 No matter what Python did, somebody would complain.

+1

This is, I think, the ultimate truth of the matter.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[John O'Hagan]

On Fri, 13 Jan 2012 10:40:47 -0800
Ethan Furman et...@stoneleaf.us wrote:

 Steven D'Aprano wrote:
  Normally this is harmless, but there is one interesting little
  glitch you can get:
  
  t = ('a', [23])
  t[1] += [42]
  Traceback (most recent call last):
File stdin, line 1, in module
  TypeError: 'tuple' object does not support item assignment
  t
  ('a', [23, 42])

IMHO, this is worthy of bug-hood: shouldn't we be able to conclude from the 
TypeError that the assignment failed?

 There is one other glitch, and possibly my only complaint:
 
 -- a = [1, 2, 3]
 -- b = 'hello, world'
 -- a = a + b
 Traceback (most recent call last):
File stdin, line 1, in module
 TypeError: can only concatenate list (not str) to list
 -- a += b
 -- a
 [1, 2, 3, 'h', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd']
 
 IMO, either both + and += should succeed, or both should fail.
 
 ~Ethan~


This also happens for tuples, sets, generators and range objects (probably any 
iterable), AFAIK only when the left operand is a list. Do lists get special 
treatment in terms of implicitly converting the right-hand operand?

The behaviour of the in-place operator could be more consistent across types:

 a=[1,2]
 a+=(3,4)
 a
[1, 2, 3, 4]
 a=(1,2)
 a+=(3,4)
 a
(1, 2, 3, 4)
 a=(1,2)
 a+=[3,4]
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: can only concatenate tuple (not list) to tuple


John
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Rick Johnson]

On Jan 13, 10:48 am, Devin Jeanpierre jeanpierr...@gmail.com wrote:
 On Fri, Jan 13, 2012 at 10:13 AM, Grant Edwards inva...@invalid.invalid 
 wrote:
  On 2012-01-13, Devin Jeanpierre jeanpierr...@gmail.com wrote:
  On Fri, Jan 13, 2012 at 7:30 AM, Chris Angelico ros...@gmail.com wrote:
 There's a bit of a feeling
 that code should do what it looks like and be sort of understandable
 without exactly understanding everything.

Yeah there's a word for that; INTUITIVE, And I've been preaching its
virtues (sadly in vain it seems!) to these folks for some time now.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Steven D'Aprano]

On Wed, 01 Feb 2012 19:51:13 -0800, Rick Johnson wrote:

 Yeah there's a word for that; INTUITIVE, And I've been preaching its
 virtues (sadly in vain it seems!) to these folks for some time now.

Intuitive to whom?

Expert Python programmers?

VB coders?

Perl hackers?

School children who have never programmed before?

Mathematicians?

Babies?

Rocket scientists?

Hunter-gatherers from the Kalahari desert?


My intuition tells me you have never even considered that intuition 
depends on who is doing the intuiting. 


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Devin Jeanpierre]

On Wed, Feb 1, 2012 at 10:18 PM, John O'Hagan resea...@johnohagan.com wrote:
 On Fri, 13 Jan 2012 10:40:47 -0800
 Ethan Furman et...@stoneleaf.us wrote:

 Steven D'Aprano wrote:
  Normally this is harmless, but there is one interesting little
  glitch you can get:
 
  t = ('a', [23])
  t[1] += [42]
  Traceback (most recent call last):
    File stdin, line 1, in module
  TypeError: 'tuple' object does not support item assignment
  t
  ('a', [23, 42])

 IMHO, this is worthy of bug-hood: shouldn't we be able to conclude from the 
 TypeError that the assignment failed?

It did fail. The mutation did not.

I can't think of any way out of this misleadingness, although if you
can that would be pretty awesome.

-- Devin
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Eduardo Suarez-Santana]

El 13/01/12 11:33, Eduardo Suarez-Santana escribió:

I wonder whether this is normal behaviour.


Even simpler:

$ python
Python 2.7.2 (default, Oct 31 2011, 11:54:55)
[GCC 4.5.3] on linux2
Type help, copyright, credits or license for more information.
 r={'a':1};
 d={};
 d['x']=r;
 d['y']=r;
 d['x']['a']=3
 d['y']
{'a': 3}


--
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Jean-Michel Pichavant]

Eduardo Suarez-Santana wrote:

El 13/01/12 11:33, Eduardo Suarez-Santana escribió:

I wonder whether this is normal behaviour.


Even simpler:

$ python
Python 2.7.2 (default, Oct 31 2011, 11:54:55)
[GCC 4.5.3] on linux2
Type help, copyright, credits or license for more information.
 r={'a':1};
 d={};
 d['x']=r;
 d['y']=r;
 d['x']['a']=3
 d['y']
{'a': 3}



yes it is.

 d['x']=r;
 d['y']=r;

means that both d['x'] and d['y'] name the same object r. If you change 
r, you'll see these changes wheter using d['x'] or d['y'].


The operator '=' does not copy objects, it binds an object to a name, 
and an object can have multiple names.

Use the dictionary copy method to copy a dictionary:

d['x'] = r.copy()

JM
--
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Steven D'Aprano]

On Fri, 13 Jan 2012 11:33:24 +, Eduardo Suarez-Santana wrote:

 I wonder whether this is normal behaviour.
 
 I would expect equal sign to copy values from right to left. 

Assignment in Python never copies values.

 However, it
 seems there is a copy-on-write mechanism that is not working.

There is no copy-on-write.

Assignment in Python is name binding: the name on the left hand side is 
bound to the object on the right. An object can have zero, one or many 
names. If the object is mutable, changes to the object will be visible 
via any name:

 x = []  # lists are mutable objects
 y = x  # not a copy of x, but x and y point to the same object
 x.append(42)  # mutates the object in place
 print y
[42]

The same rules apply not just to names, but also to list items and dict 
items, as well as attributes, and any other reference:

 z = [x, y]  # z is a list containing the same sublist twice
 z[0].append(23)
 print z
[[42, 23], [42, 23]]

When you work with floats, ints or strings, you don't notice this because 
those types are immutable: you can't modify those objects in place. So 
for example:

 a = 42  # binds the name 'a' to the object 42
 b = a  # a and b point to the same object
 a += 1  # creates a new object, and binds it to a
 print b  # leaving b still pointing to the old object
42


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Chris Angelico]

On Fri, Jan 13, 2012 at 11:10 PM, Steven D'Aprano
steve+comp.lang.pyt...@pearwood.info wrote:
 z = [x, y]  # z is a list containing the same sublist twice
 z[0].append(23)
 print z
 [[42, 23], [42, 23]]

 When you work with floats, ints or strings, you don't notice this because
 those types are immutable: you can't modify those objects in place. So
 for example:

 a = 42  # binds the name 'a' to the object 42
 b = a  # a and b point to the same object
 a += 1  # creates a new object, and binds it to a
 print b  # leaving b still pointing to the old object
 42

I was about to say that it's a difference between .append() which is
a method on the object, and += which is normally a rebinding, but
unfortunately:

 a=[]
 b=a
 a+=[1]
 a
[1]
 b
[1]
 b+=[2]
 a
[1, 2]
 a
[1, 2]
 a=a+[3]
 a
[1, 2, 3]
 b
[1, 2]

(tested in Python 3.2 on Windows)

It seems there's a distinct difference between a+=b (in-place
addition/concatenation) and a=a+b (always rebinding), which is sorely
confusing to C programmers. But then, there's a lot about Python
that's sorely confusing to C programmers.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Steven D'Aprano]

On Fri, 13 Jan 2012 23:30:56 +1100, Chris Angelico wrote:

 It seems there's a distinct difference between a+=b (in-place
 addition/concatenation) and a=a+b (always rebinding), 

Actually, both are always rebinding. It just happens that sometimes a+=b 
rebinds to the same object that it was originally bound to.

In the case of ints, a+=b creates a new object (a+b) and rebinds a to it. 
In the case of lists, a+=b nominally creates a list a+b, but in fact it 
implements that as an in-place operation a.extend(b), and then rebinds 
the name a to the list already bound to a. 

It does that because the Python VM doesn't know at compile time whether 
a+=b will be in-place or not, and so it has to do the rebinding in order 
to support the fall-back case of a+=b = a=a+b. Or something -- go read 
the PEP if you really care :)

Normally this is harmless, but there is one interesting little glitch you 
can get:

 t = ('a', [23])
 t[1] += [42]
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: 'tuple' object does not support item assignment
 t
('a', [23, 42])




 which is sorely
 confusing to C programmers. But then, there's a lot about Python
 that's sorely confusing to C programmers.

I prefer to think of it as there's a lot about C that is sorely 
confusing to anyone who isn't a C programmer wink



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Devin Jeanpierre]

On Fri, Jan 13, 2012 at 7:30 AM, Chris Angelico ros...@gmail.com wrote:
 It seems there's a distinct difference between a+=b (in-place
 addition/concatenation) and a=a+b (always rebinding), which is sorely
 confusing to C programmers. But then, there's a lot about Python
 that's sorely confusing to C programmers.

I think this is confusing to just about everyone, when they first encounter it.

-- Devin
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Grant Edwards]

On 2012-01-13, Devin Jeanpierre jeanpierr...@gmail.com wrote:
 On Fri, Jan 13, 2012 at 7:30 AM, Chris Angelico ros...@gmail.com wrote:
 It seems there's a distinct difference between a+=b (in-place
 addition/concatenation) and a=a+b (always rebinding), which is sorely
 confusing to C programmers. But then, there's a lot about Python
 that's sorely confusing to C programmers.

 I think this is confusing to just about everyone, when they first
 encounter it.

That depends on what languages they've used in the past and whether
they skip reading any documentation and just assume that all languages
work the same way.

I would agree that for the majority of new users, they previously used
only languages where an assignment operator does a copy value, and
that 90+ percent of the time those new users they assume all languages
work that way.

I'm not sure what we can do about that -- Python's semantics are well
documented.

-- 
Grant Edwards   grant.b.edwardsYow! If our behavior is
  at   strict, we do not need fun!
  gmail.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Devin Jeanpierre]

On Fri, Jan 13, 2012 at 10:13 AM, Grant Edwards invalid@invalid.invalid wrote:
 On 2012-01-13, Devin Jeanpierre jeanpierr...@gmail.com wrote:
 On Fri, Jan 13, 2012 at 7:30 AM, Chris Angelico ros...@gmail.com wrote:
 It seems there's a distinct difference between a+=b (in-place
 addition/concatenation) and a=a+b (always rebinding), which is sorely
 confusing to C programmers. But then, there's a lot about Python
 that's sorely confusing to C programmers.

 I think this is confusing to just about everyone, when they first
 encounter it.

 That depends on what languages they've used in the past and whether
 they skip reading any documentation and just assume that all languages
 work the same way.

 I would agree that for the majority of new users, they previously used
 only languages where an assignment operator does a copy value, and
 that 90+ percent of the time those new users they assume all languages
 work that way.

That isn't what I was referring to. Specifically, it confuses almost
everyone the first time they encounter it that a += b is not the
same as a = a + b.

And sure, it's documented. That's a bit of a cop-out though... it
isn't in the tutorial, and even if it were, it's not as if people
remember everything they read. It's not about whether you _can_ know
it as much as whether it is obvious. There's a bit of a feeling
that code should do what it looks like and be sort of understandable
without exactly understanding everything. Maybe this idea is wrong if
taken to an extreme (since it's really impossible to do completely),
but the feeling of it is probably decent. It's why we use + for
addition and - for subtraction, and not the other way around. You
don't need to know the details of operator overloading and
NotImplemented and so on to get what X + Y means for numbers, or even
for lists.

I feel like a += b is sort of implicitly understood by most
programmers to be the same as a = a + b. If you asked someone what
it meant, their first answer would be Oh, it means a = a + b[*].
That is why it's confusing -- even to people that weren't already
exposed to that idea that these are equivalent, they get infected
fast. And then expectations get broken, because they're only *usually*
equivalent.

[*] Before posting this, I actually tried this on a Python IRC channel
-- and it happened exactly as so.

-- Devin
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Neil Cerutti]

On 2012-01-13, Devin Jeanpierre jeanpierr...@gmail.com wrote:
 On Fri, Jan 13, 2012 at 10:13 AM, Grant Edwards invalid@invalid.invalid 
 wrote:
 On 2012-01-13, Devin Jeanpierre jeanpierr...@gmail.com wrote:
 On Fri, Jan 13, 2012 at 7:30 AM, Chris Angelico ros...@gmail.com wrote:
 It seems there's a distinct difference between a+=b (in-place
 addition/concatenation) and a=a+b (always rebinding), which is sorely
 confusing to C programmers. But then, there's a lot about Python
 that's sorely confusing to C programmers.

 I think this is confusing to just about everyone, when they first
 encounter it.

 That depends on what languages they've used in the past and whether
 they skip reading any documentation and just assume that all languages
 work the same way.

 I would agree that for the majority of new users, they previously used
 only languages where an assignment operator does a copy value, and
 that 90+ percent of the time those new users they assume all languages
 work that way.

 That isn't what I was referring to. Specifically, it confuses
 almost everyone the first time they encounter it that a += b
 is not the same as a = a + b.

If you've ever implemented operator=, operator+, and operator+=
in C++ you'll know how and why they are different. A C++
programmer would be wondering how either can work on immutable
objects, and that's where Python's magical rebinding semantics
come into play.

-- 
Neil Cerutti
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Grant Edwards]

On 2012-01-13, Neil Cerutti ne...@norwich.edu wrote:

 If you've ever implemented operator=, operator+, and operator+=
 in C++ you'll know how and why they are different.

That assumes that C++ programmers understand C++.

;)

 A C++ programmer would be wondering how either can work on immutable
 objects, and that's where Python's magical rebinding semantics come
 into play.

-- 
Grant Edwards   grant.b.edwardsYow! Thousands of days of
  at   civilians ... have produced
  gmail.coma ... feeling for the
   aesthetic modules --
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Chris Angelico]

On Sat, Jan 14, 2012 at 5:15 AM, Grant Edwards invalid@invalid.invalid wrote:
 That assumes that C++ programmers understand C++.

I understand C++ very well. That's why I use Python or Pike.

(With apologies to Larry Wall)

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Grant Edwards]

On 2012-01-13, Chris Angelico ros...@gmail.com wrote:
 On Sat, Jan 14, 2012 at 5:15 AM, Grant Edwards invalid@invalid.invalid 
 wrote:
 That assumes that C++ programmers understand C++.

 I understand C++ very well. That's why I use Python or Pike.

 (With apologies to Larry Wall)

Were one inclined to troll a bit, one might be tempted to claim that
using C++ is prima facie evidence of not understanding C++.

Not that I would ever claim something inflamitory like that...

-- 
Grant Edwards   grant.b.edwardsYow! Thousands of days of
  at   civilians ... have produced
  gmail.coma ... feeling for the
   aesthetic modules --
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Ethan Furman]

Steven D'Aprano wrote:
Normally this is harmless, but there is one interesting little glitch you 
can get:



t = ('a', [23])
t[1] += [42]

Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: 'tuple' object does not support item assignment

t

('a', [23, 42])



There is one other glitch, and possibly my only complaint:

-- a = [1, 2, 3]
-- b = 'hello, world'
-- a = a + b
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: can only concatenate list (not str) to list
-- a += b
-- a
[1, 2, 3, 'h', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd']

IMO, either both + and += should succeed, or both should fail.

~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Neil Cerutti]

On 2012-01-13, Grant Edwards invalid@invalid.invalid wrote:
 On 2012-01-13, Chris Angelico ros...@gmail.com wrote:
 On Sat, Jan 14, 2012 at 5:15 AM, Grant Edwards
 invalid@invalid.invalid wrote:
 That assumes that C++ programmers understand C++.

 I understand C++ very well. That's why I use Python or Pike.

 (With apologies to Larry Wall)

 Were one inclined to troll a bit, one might be tempted to claim
 that using C++ is prima facie evidence of not understanding
 C++.

 Not that I would ever claim something inflamitory like that...

On the Python newsgroup, it's funny. ;)

-- 
Neil Cerutti
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Evan Driscoll]

On 01/13/2012 10:54 AM, Neil Cerutti wrote:

If you've ever implemented operator=, operator+, and operator+=
in C++ you'll know how and why they are different.


At the same time, you'd also know that that implementing them in such a 
way that 'a += b' does *not* perform the same action as 'a = a + b' is 
considered very bad-mannered.


In fact, it's often suggested (e.g. in More Effective C++'s Item 22, 
though this is not the main thrust of that section) to implement 
operator+ in terms of += to ensure that this is the case:

 MyType operator+ (MyType left, MyType right) {
 MyType copy = left; copy += right; return copy;
 }


A C++
programmer would be wondering how either can work on immutable
objects, and that's where Python's magical rebinding semantics
come into play.


IMO a C++ programmer wouldn't be likely to wonder that much at all 
because he or she wouldn't view the objects as immutable to begin with. 
:-) 'x = 5; x += 1;' makes perfect sense in C++, just for a somewhat 
different reason.


Evan
--
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Neil Cerutti]

On 2012-01-13, Evan Driscoll edrisc...@wisc.edu wrote:
 On 01/13/2012 10:54 AM, Neil Cerutti wrote:
 If you've ever implemented operator=, operator+, and operator+=
 in C++ you'll know how and why they are different.

 At the same time, you'd also know that that implementing them
 in such a way that 'a += b' does *not* perform the same action
 as 'a = a + b' is considered very bad-mannered.

 In fact, it's often suggested (e.g. in More Effective C++'s Item 22, 
 though this is not the main thrust of that section) to implement 
 operator+ in terms of += to ensure that this is the case:
   MyType operator+ (MyType left, MyType right) {
   MyType copy = left; copy += right; return copy;
   }

They perform the same action, but their semantics are different.
operator+ will always return a new object, thanks to its
signature, and operator+= shall never do so. That's the main
difference I was getting at.

 A C++ programmer would be wondering how either can work on
 immutable objects, and that's where Python's magical rebinding
 semantics come into play.

 IMO a C++ programmer wouldn't be likely to wonder that much at
 all because he or she wouldn't view the objects as immutable to
 begin with. :-) 'x = 5; x += 1;' makes perfect sense in C++,
 just for a somewhat different reason.

I was thinking of const objects, but you are correct that
immutable isn't really a C++ concept.

-- 
Neil Cerutti
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[88888 Dihedral]

Ethan Furman於 2012年1月14日星期六UTC+8上午2時40分47秒寫道：
 Steven D'Aprano wrote:
  Normally this is harmless, but there is one interesting little glitch you 
  can get:
  
  t = ('a', [23])
  t[1] += [42]
  Traceback (most recent call last):
File stdin, line 1, in module
  TypeError: 'tuple' object does not support item assignment
  t
  ('a', [23, 42])
 
 
 There is one other glitch, and possibly my only complaint:
 
 -- a = [1, 2, 3]
 -- b = 'hello, world'
 -- a = a + b
 Traceback (most recent call last):
File stdin, line 1, in module
 TypeError: can only concatenate list (not str) to list
 -- a += b
 -- a
 [1, 2, 3, 'h', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd']
 
 IMO, either both + and += should succeed, or both should fail.
 
 ~Ethan~

The += operator is not  only for value types in the above example. 
 
An operator of two operands and an operator of three operands  of 
general object types are two different operators. 

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: copy on write

[Evan Driscoll]

On 01/13/2012 03:20 PM, Neil Cerutti wrote:

They perform the same action, but their semantics are different.
operator+ will always return a new object, thanks to its
signature, and operator+= shall never do so. That's the main
difference I was getting at.


I was talking about the combination of + and =, since the discussion is 
about 'a = a + b' vs 'a += b', not 'a + b' vs 'a += b' (where the 
differences are obvious).


And I stand by my statement. In 'a = a + b', operator+ obviously returns 
a new object, but operator= should then go and assign the result to and 
return a reference to 'a', just like how 'a += b' will return a 
reference to 'a'.


If you're working in C++ and overload your operators so that 'a += b' 
and 'a = a + b' have different observable behaviors (besides perhaps 
time), then either your implementation is buggy or your design is very 
bad-mannered.


Evan
--
http://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write when forking a python process

[jac]

Heiko,
Thank you for pointing out POSH.  I have used some of python's other
shared memory facilities, but was completely unaware of POSH, it seems
nice.
Also, I agree that shared memory would solve the use-case I outlined
above, but it is not hard to imagine a slightly different case where
the child processes do want to mutate the dictionary, but do not want
the changes to show up in the parent process' dictionary. If shared
memory is used, then a more complicated algorithm would be needed to
get the desired behavior.
Long story short, shared-memory is great for some things.  Copy on
write is great for some other things. Sometimes they are easily
interchangeable, sometimes they are not.  Many, if not most languages,
allow the operating system to perform optimizations when a program
forks that python does not allow due to the way it counts references.
Many programs would be easier to write if python allowed the os to use
cow.
However, since I have gotten no other feedback in this list, I think
that I will post this in python-ideas as well.

Thanks,
--jac


On Apr 8, 4:29 pm, Heiko Wundram modeln...@modelnine.org wrote:
 Am 08.04.2011 20:34, schrieb jac:

  I disagree with your statement that COW is an optimization for a
  complete clone, it is an optimization that works at the memory page
  level, not at the memory image level.  In other words, if I write to a
  copy-on-write page, only that page is copied into my process' address
  space, not the entire parent image.  To the best of my knowledge by
  preventing the child process from altering an object's reference count
  you can prevent the object from being copied (assuming the object is
  not altered explicitly of course.)

 As I said before: COW for sharing a processes forked memory is simply
 an implementation-detail, and an _optimization_ (and of course a
 sensible one at that) for fork; there is no provision in the semantics
 of fork that an operating system should use COW memory-pages for
 implementing the copying (and early UNIXes didn't do that; they
 explicitly copied the complete process image for the child). The only
 semantic that is specified for fork is that the parent and the child
 have independent process images, that are equivalent copies (except for
 some details) immediately after the fork call has returned successfully
 (see SUSv4).

 What you're thinking of (and what's generally useful in the context
 you're describing) is shared memory; Python supports putting objects
 into shared memory using e.g. POSH (which is an extension that allows
 you to place Python objects in shared memory, using the SysV
 IPC-featureset that most UNIXes implement today).

 --
 --- Heiko.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write when forking a python process

[Heiko Wundram]

Am 08.04.2011 18:14, schrieb John Connor:
 Has anyone else looked into the COW problem?  Are there workarounds
 and/or other plans to fix it?  Does the solution I am proposing sound
 reasonable, or does it seem like overkill?  Does anyone foresee any
 problems with it?

Why'd you need a fix like this for something that isn't broken? COW
doesn't just refer to the object reference-count, but to the object
itself, too. _All_ memory of the parent (and, as such, all objects, too)
become unrelated to memory in the child once the fork is complete.

The initial object reference-count state of the child is guaranteed to
be sound for all objects (because the parent's final reference-count
state was, before the process image got cloned [remember, COW is just an
optimization for a complete clone, and it's up the operating-system to
make sure that you don't notice different semantics from a complete
copy]), and what you're proposing (opting in/out of reference counting)
breaks that.

-- 
--- Heiko.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write when forking a python process

[jac]

Hi Heiko,
I just realized I should probably have put a clearer use-case in my
previous message.  A example use-case would be if you have a parent
process which creates a large dictionary (say several gigabytes).  The
process then forks several worker processes which access this
dictionary.  The worker processes do not add or remove objects from
the dictionary, nor do they alter the individual elements of the
dictionary.  They simply perform lookups on the dictionary and perform
calculations which are then written to files.
If I wrote the above program in C, neither the dictionary nor its
contents would be copied into the memory of the child processes, but
in python as soon as you pass the dictionary itself or any of its
contents into a function as an argument, its reference count is
changed and the page of memory on which its reference count resides is
copied into the child process' memory.  What I am proposing is to
allow the parent process to disable reference counting for this
dictionary and its contents so that the child processes can access
them in a readonly fashion without them having to be copied.

I disagree with your statement that COW is an optimization for a
complete clone, it is an optimization that works at the memory page
level, not at the memory image level.  In other words, if I write to a
copy-on-write page, only that page is copied into my process' address
space, not the entire parent image.  To the best of my knowledge by
preventing the child process from altering an object's reference count
you can prevent the object from being copied (assuming the object is
not altered explicitly of course.)

Hopefully this clarifies my previous post,
--jac

On Apr 8, 12:26 pm, Heiko Wundram modeln...@modelnine.org wrote:
 Am 08.04.2011 18:14, schrieb John Connor:

  Has anyone else looked into the COW problem?  Are there workarounds
  and/or other plans to fix it?  Does the solution I am proposing sound
  reasonable, or does it seem like overkill?  Does anyone foresee any
  problems with it?

 Why'd you need a fix like this for something that isn't broken? COW
 doesn't just refer to the object reference-count, but to the object
 itself, too. _All_ memory of the parent (and, as such, all objects, too)
 become unrelated to memory in the child once the fork is complete.

 The initial object reference-count state of the child is guaranteed to
 be sound for all objects (because the parent's final reference-count
 state was, before the process image got cloned [remember, COW is just an
 optimization for a complete clone, and it's up the operating-system to
 make sure that you don't notice different semantics from a complete
 copy]), and what you're proposing (opting in/out of reference counting)
 breaks that.

 --
 --- Heiko.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Copy-on-write when forking a python process

[Heiko Wundram]

Am 08.04.2011 20:34, schrieb jac:
 I disagree with your statement that COW is an optimization for a
 complete clone, it is an optimization that works at the memory page
 level, not at the memory image level.  In other words, if I write to a
 copy-on-write page, only that page is copied into my process' address
 space, not the entire parent image.  To the best of my knowledge by
 preventing the child process from altering an object's reference count
 you can prevent the object from being copied (assuming the object is
 not altered explicitly of course.)

As I said before: COW for sharing a processes forked memory is simply
an implementation-detail, and an _optimization_ (and of course a
sensible one at that) for fork; there is no provision in the semantics
of fork that an operating system should use COW memory-pages for
implementing the copying (and early UNIXes didn't do that; they
explicitly copied the complete process image for the child). The only
semantic that is specified for fork is that the parent and the child
have independent process images, that are equivalent copies (except for
some details) immediately after the fork call has returned successfully
(see SUSv4).

What you're thinking of (and what's generally useful in the context
you're describing) is shared memory; Python supports putting objects
into shared memory using e.g. POSH (which is an extension that allows
you to place Python objects in shared memory, using the SysV
IPC-featureset that most UNIXes implement today).

-- 
--- Heiko.
-- 
http://mail.python.org/mailman/listinfo/python-list