Re: non-copy slices
On Thu, Nov 19, 2009 at 8:14 PM, Ethan Furman et...@stoneleaf.us wrote: No I'm well aware that there is no deep copy of the objects and the lists only keep references to the objects and in essence they have the same objects in there. But this doesn't mean they are the same list. Modifications to slices are not written back to the original list. x = range(5) y = x[1:3] y[0] = 13 x[1] == y[0] -- False Of course if I modify the object in the slice then the original list will see the change, but this is not what I was saying. Second and more importantly it's the performance penalty from allocating a large number of lists produced from the slices and the copy of the references. islice does not have this penalty, it should only instantiate a small object that iterates on the original list. Themis So shallow copy == new label created for existing object. So is your desired behavior to write back to the original list if your sub-list is modified? In other words, you are creating a window onto an existing list? If not, what would happen when a sublist element was modified (or deleted, or appended, or ...)? On a related note, GO encourages use of slices. http://golang.org/doc/effective_go.html#slices -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
No I'm well aware that there is no deep copy of the objects and the lists only keep references to the objects and in essence they have the same objects in there. But this doesn't mean they are the same list. Modifications to slices are not written back to the original list. x = range(5) y = x[1:3] y[0] = 13 x[1] == y[0] -- False Of course if I modify the object in the slice then the original list will see the change, but this is not what I was saying. Second and more importantly it's the performance penalty from allocating a large number of lists produced from the slices and the copy of the references. islice does not have this penalty, it should only instantiate a small object that iterates on the original list. Themis On Thu, Nov 19, 2009 at 3:00 AM, Rami Chowdhury rami.chowdh...@gmail.comwrote: I'm not sure you're understanding the point others have been making. A list item is merely another reference to an existing object -- it doesn't copy the object in any way. -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
On Wed, Nov 18, 2009 at 9:00 PM, Rami Chowdhury rami.chowdh...@gmail.comwrote: I'm not sure you're understanding the point others have been making. A list item is merely another reference to an existing object -- it doesn't copy the object in any way. It still has to copy the reference, though. That takes O(n) time if you're taking a big slice. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
Please don't top post. :) tbour...@doc.ic.ac.uk wrote: On Thu, Nov 19, 2009 at 3:00 AM, Rami Chowdhury rami.chowdh...@gmail.com mailto:rami.chowdh...@gmail.com wrote: I'm not sure you're understanding the point others have been making. A list item is merely another reference to an existing object -- it doesn't copy the object in any way. No I'm well aware that there is no deep copy of the objects and the lists only keep references to the objects and in essence they have the same objects in there. But this doesn't mean they are the same list. Modifications to slices are not written back to the original list. x = range(5) y = x[1:3] y[0] = 13 x[1] == y[0] -- False Of course if I modify the object in the slice then the original list will see the change, but this is not what I was saying. Second and more importantly it's the performance penalty from allocating a large number of lists produced from the slices and the copy of the references. islice does not have this penalty, it should only instantiate a small object that iterates on the original list. Themis So shallow copy == new label created for existing object. So is your desired behavior to write back to the original list if your sub-list is modified? In other words, you are creating a window onto an existing list? If not, what would happen when a sublist element was modified (or deleted, or appended, or ...)? ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
On Thu, 19 Nov 2009 02:39:42 -0800, tbour...@doc.ic.ac.uk wrote: Second and more importantly it's the performance penalty from allocating a large number of lists produced from the slices and the copy of the references. Ah, I see what you were getting at -- thanks for clarifying. On Thu, Nov 19, 2009 at 3:00 AM, Rami Chowdhury rami.chowdh...@gmail.comwrote: I'm not sure you're understanding the point others have been making. A list item is merely another reference to an existing object -- it doesn't copy the object in any way. -- Rami Chowdhury Never attribute to malice that which can be attributed to stupidity -- Hanlon's Razor 408-597-7068 (US) / 07875-841-046 (UK) / 0189-245544 (BD) -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
On Thu, Nov 19, 2009 at 2:44 PM, Ethan Furman et...@stoneleaf.us wrote: Please don't top post. :) So shallow copy == new label created for existing object. So is your desired behavior to write back to the original list if your sub-list is modified? In other words, you are creating a window onto an existing list? If not, what would happen when a sublist element was modified (or deleted, or appended, or ...)? ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list Yes a window / view on the existing list describes it best. So every modification you make in this view is actually modifying the original list accordingly. Blist that was suggested in a previous email in the thread seems lightweight but it does create a new list when a modification is made. In any case, I've already implemented the object myself and I can post it if you care to have a look, but I was just wondering if there was already something in the standard library. Themis -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
Themis Bourdenas wrote: On Thu, Nov 19, 2009 at 2:44 PM, Ethan Furman et...@stoneleaf.us mailto:et...@stoneleaf.us wrote: So shallow copy == new label created for existing object. So is your desired behavior to write back to the original list if your sub-list is modified? In other words, you are creating a window onto an existing list? If not, what would happen when a sublist element was modified (or deleted, or appended, or ...)? ~Ethan~ Yes a window / view on the existing list describes it best. So every modification you make in this view is actually modifying the original list accordingly. Blist that was suggested in a previous email in the thread seems lightweight but it does create a new list when a modification is made. In any case, I've already implemented the object myself and I can post it if you care to have a look, but I was just wondering if there was already something in the standard library. Themis Unfortunately, I am not very familiar with the stdlib yet (gotta buy that book!). I'm going to guess 'No' since nobody has chimed in with a 'Yes', though. I'd love to see what you have for that. Does in support a stepped window, or only contiguous sequences? The one I put together this afternoon only does contiguous sequences, as I had no use cases to decide how assignments of multiple items should be handled, and not a lot of time to implement something generic -- so, to answer John's question from a completely different thread, yes I do enjoy working on small projects even if IAGNI. :) Cheers! ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
non-copy slices
Hi, I was looking for a facility similar to slices in python library that would avoid the implicit creation of a new list and copy of elements that is the default behaviour. Instead I'd rather have a lazy iteratable object on the original sequence. Well, in the end I wrote it myself but I was wondering if I missed sth in the library. If I didn't is there a particular reason there isn't sth like that? I find it hard to believe that all slice needs have strictly copy semantics. Cheers, Themis -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
tbour...@doc.ic.ac.uk wrote: Hi, I was looking for a facility similar to slices in python library that would avoid the implicit creation of a new list and copy of elements that is the default behaviour. Instead I'd rather have a lazy iteratable object on the original sequence. Well, in the end I wrote it myself but I was wondering if I missed sth in the library. If I didn't is there a particular reason there isn't sth like that? I find it hard to believe that all slice needs have strictly copy semantics. I suspect that itertools is your friend, specifically itertools.islice TJG -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
Ahhh yes! that's exactly it. Thanks for pointing out! Themis On Wed, Nov 18, 2009 at 3:44 PM, Tim Golden m...@timgolden.me.uk wrote: tbour...@doc.ic.ac.uk wrote: Hi, I was looking for a facility similar to slices in python library that would avoid the implicit creation of a new list and copy of elements that is the default behaviour. Instead I'd rather have a lazy iteratable object on the original sequence. Well, in the end I wrote it myself but I was wondering if I missed sth in the library. If I didn't is there a particular reason there isn't sth like that? I find it hard to believe that all slice needs have strictly copy semantics. I suspect that itertools is your friend, specifically itertools.islice TJG -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
tbour...@doc.ic.ac.uk wrote: Hi, I was looking for a facility similar to slices in python library that would avoid the implicit creation of a new list and copy of elements that is the default behaviour. Instead I'd rather have a lazy iteratable object on the original sequence. Well, in the end I wrote it myself but I was wondering if I missed sth in the library If I didn't is there a particular reason there isn't sth like that? I find it hard to believe that all slice needs have strictly copy semantics. It is a strict *shallow* copy. There is no copying of contents. That aside, you are right, hence itertools.islice as already mentioned. In the design of 3.0, I believe the idea was raised of making slices iterables in 3.0, just as was done for map, filter, and range. However, it would have been highly disruptive, and not save much space. Map and range create an unbounded number of new objects, rather than just a sequence of references to existing objects (or bytes or words for bytes and str slices). There is also the problem of virtual slices preventing garbage collection of the source sequence when it is not otherwise needed. Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
Sth else that I noticed as I started using islice. The name is somewhat misleading. Having the slice part in the name I would expect it to imitate the functionality of normal slices. Random access, sub-slicing etc. However, it is only iteratable. Any particular reasons for that? My guess is that it's inside the itertools so it's meant only for iteration and not random access. However, as I told before the name implies the same functionality while the only thing they share is iteration. It's nothing in the library that completely imitates the slice without the copies, right? Cheers, Themis On Wed, Nov 18, 2009 at 6:49 PM, Terry Reedy tjre...@udel.edu wrote: tbour...@doc.ic.ac.uk wrote: Hi, I was looking for a facility similar to slices in python library that would avoid the implicit creation of a new list and copy of elements that is the default behaviour. Instead I'd rather have a lazy iteratable object on the original sequence. Well, in the end I wrote it myself but I was wondering if I missed sth in the library If I didn't is there a particular reason there isn't sth like that? I find it hard to believe that all slice needs have strictly copy semantics. It is a strict *shallow* copy. There is no copying of contents. That aside, you are right, hence itertools.islice as already mentioned. In the design of 3.0, I believe the idea was raised of making slices iterables in 3.0, just as was done for map, filter, and range. However, it would have been highly disruptive, and not save much space. Map and range create an unbounded number of new objects, rather than just a sequence of references to existing objects (or bytes or words for bytes and str slices). There is also the problem of virtual slices preventing garbage collection of the source sequence when it is not otherwise needed. Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
tbour...@doc.ic.ac.uk wrote: Hi, I was looking for a facility similar to slices in python library that would avoid the implicit creation of a new list and copy of elements that is the default behaviour. Instead I'd rather have a lazy iteratable object on the original sequence. Well, in the end I wrote it myself but I was wondering if I missed sth in the library. If I didn't is there a particular reason there isn't sth like that? I find it hard to believe that all slice needs have strictly copy semantics. Cheers, Themis Two questions: 1) What is sth? and 2), What copy? Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] In [1]: class dummy(object): ...: pass ...: In [2]: a = dummy() In [3]: b = dummy() In [4]: c = dummy() In [5]: d = dummy() In [6]: e = dummy() In [7]: list1 = [a, b, c, d, e] In [8]: list1 Out[8]: [__main__.dummy object at 0x0130C510, __main__.dummy object at 0x013F1A50, __main__.dummy object at 0x00A854F0, __main__.dummy object at 0x00A7EF50, __main__.dummy object at 0x00A7E650] In [9]: list2 = list1[1:3] In [10]: list2 Out[10]: [__main__.dummy object at 0x013F1A50, __main__.dummy object at 0x00A854F0] In [11]: list2[0] is list1[1] Out[11]: *True* In [12]: list2[1] is list1[2] Out[12]: *True* No copying of items going on here. What do you get? ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
Hi, sth == something :) sorry for the abbreviation. I'm talking about the shallow copy, still it's a copy. Unnecessary in my case and the worst part in my scenario is the creation (allocation) and deletion of a very large number of lists of moderate size (a few hundred objects) generated due to slices, while I only need to have a restricted view on the original list. The islice class partially solves the problem as I mentioned in the previous emails. Cheers, Themis On Wed, Nov 18, 2009 at 3:44 PM, Ethan Furman et...@stoneleaf.us wrote: tbour...@doc.ic.ac.uk wrote: Hi, I was looking for a facility similar to slices in python library that would avoid the implicit creation of a new list and copy of elements that is the default behaviour. Instead I'd rather have a lazy iteratable object on the original sequence. Well, in the end I wrote it myself but I was wondering if I missed sth in the library. If I didn't is there a particular reason there isn't sth like that? I find it hard to believe that all slice needs have strictly copy semantics. Cheers, Themis Two questions: 1) What is sth? and 2), What copy? Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] In [1]: class dummy(object): ...: pass ...: In [2]: a = dummy() In [3]: b = dummy() In [4]: c = dummy() In [5]: d = dummy() In [6]: e = dummy() In [7]: list1 = [a, b, c, d, e] In [8]: list1 Out[8]: [__main__.dummy object at 0x0130C510, __main__.dummy object at 0x013F1A50, __main__.dummy object at 0x00A854F0, __main__.dummy object at 0x00A7EF50, __main__.dummy object at 0x00A7E650] In [9]: list2 = list1[1:3] In [10]: list2 Out[10]: [__main__.dummy object at 0x013F1A50, __main__.dummy object at 0x00A854F0] In [11]: list2[0] is list1[1] Out[11]: *True* In [12]: list2[1] is list1[2] Out[12]: *True* No copying of items going on here. What do you get? ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
On Wed, Nov 18, 2009 at 2:22 PM, Themis Bourdenas t.bourdena...@imperial.ac.uk wrote: It's nothing in the library that completely imitates the slice without the copies, right? You might be interested in my blist extension type (link below). Syntactically, using a blist is just like using a list, but it has different performance characteristics. In particular for your needs, taking a slice is cheap. The blist implements copy-on-write behind the scenes, so taking a slice takes O(log n) time, where n is the size of the initial blist. http://pypi.python.org/pypi/blist/ Here is a simple example, which creates a blist with over 500 million elements and takes a slice of over 500 million elements. In under 22 microseconds. :-) from blist import blist small_list = blist([0]) BIG_list = small_list * 2**29 BIG_slice = BIG_list[4:-5] Cashew:~$ python2.5 -m timeit -s 'from blist import blist' 'small_list=blist([0])' ''BIG_list=small_list*2**29' 'BIG_slice=BIG_list[4:-5]' 1 loops, best of 3: 21.5 usec per loop -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com -- http://mail.python.org/mailman/listinfo/python-list
Re: non-copy slices
On Wednesday 18 November 2009 17:47:09 tbour...@doc.ic.ac.uk wrote: Hi, sth == something :) sorry for the abbreviation. I'm talking about the shallow copy, still it's a copy. I'm not sure you're understanding the point others have been making. A list item is merely another reference to an existing object -- it doesn't copy the object in any way. Unnecessary in my case and the worst part in my scenario is the creation (allocation) and deletion of a very large number of lists of moderate size (a few hundred objects) generated due to slices, while I only need to have a restricted view on the original list. The islice class partially solves the problem as I mentioned in the previous emails. Cheers, Themis On Wed, Nov 18, 2009 at 3:44 PM, Ethan Furman et...@stoneleaf.us wrote: tbour...@doc.ic.ac.uk wrote: Hi, I was looking for a facility similar to slices in python library that would avoid the implicit creation of a new list and copy of elements that is the default behaviour. Instead I'd rather have a lazy iteratable object on the original sequence. Well, in the end I wrote it myself but I was wondering if I missed sth in the library. If I didn't is there a particular reason there isn't sth like that? I find it hard to believe that all slice needs have strictly copy semantics. Cheers, Themis Two questions: 1) What is sth? and 2), What copy? Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] In [1]: class dummy(object): ...: pass ...: In [2]: a = dummy() In [3]: b = dummy() In [4]: c = dummy() In [5]: d = dummy() In [6]: e = dummy() In [7]: list1 = [a, b, c, d, e] In [8]: list1 Out[8]: [__main__.dummy object at 0x0130C510, __main__.dummy object at 0x013F1A50, __main__.dummy object at 0x00A854F0, __main__.dummy object at 0x00A7EF50, __main__.dummy object at 0x00A7E650] In [9]: list2 = list1[1:3] In [10]: list2 Out[10]: [__main__.dummy object at 0x013F1A50, __main__.dummy object at 0x00A854F0] In [11]: list2[0] is list1[1] Out[11]: *True* In [12]: list2[1] is list1[2] Out[12]: *True* No copying of items going on here. What do you get? ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list Rami Chowdhury As an online discussion grows longer, the probability of a comparison involving Nazis or Hitler approaches one. -- Godwin's Law 408-597-7068 (US) / 07875-841-046 (UK) / 0189-245544 (BD) -- http://mail.python.org/mailman/listinfo/python-list
