Re: Efficient way to break up a list into two pieces
On 21 Feb., 04:40, Steven D'Aprano st...@remove-this- cybersource.com.au wrote: Additionally, Python lists are over-allocated so that appends are fast. A list of (say) 1000 items might be over-allocated to (say) 1024 items, so that you can do 24 appends before the array is full and the array needs to be resized. This means that, on average, Python list appending is O(1) instead of O(N). You can't just change the length blindly, you need to worry about the over-allocation. Ok, I see your point. However, other data representation might still be able to optimize such a multi-element pop. I'm thinking of deques, for example. I'm sympathetic to your concern: I've often felt offended that doing something like this: x = SomeReallyBigListOrString for item in x[1:]: process(item) has to copy the entire list or string (less the first item). But honestly, I've never found a situation where it actually mattered. Good grief, it copies that, too? I assumed that the copying is at least delayed until the assignment and that x[1:] is represented by some wrapper that just shuffles the indices around (much like the .indices method that MRAB suggested). Maybe copying chunks of data really isn't that much of an issue as it used to be (and maybe I'm getting old). The application I have in mind has to do with symbolic computation, and expressions would be represented by python lists. It's too early to do any profiling (in fact, it's at the deciding if python is the right language to implement it stage), but it will have to handle expressions with many terms (i.e. long lists), and in the end taking these lists apart and putting them back together in different order is the only thing the code will do. That to explain my interest in performance issues related to pyhton lists. Anyway, thanks for your help. Martin. -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficient way to break up a list into two pieces
On 21 Feb., 07:38, Carl Banks pavlovevide...@gmail.com wrote: Numpy arrays can share underlying data like that when you take slices. For instance, this probably works the way you want: a = numpy.array([1,2,3,4,5,6]) b = a[:3] c = a[3:] None of the actual data was copied here. Hmm, that might be worth looking into. Thanks. -- http://mail.python.org/mailman/listinfo/python-list
Efficient way to break up a list into two pieces
Hello, I recently read about augmented assignments and that (with l1, l2 being lists) l1.extend(l2) is more efficient than l1 = l1 + l2 because unnecessary copy operations can be avoided. Now my question is if there's a similar thing for breaking a list into two parts. Let's say I want to remove from l1 everything from and including position 10 and store it in l2. Then I can write l2 = l1[10:] del l1[10:] But since I'm assigning a slice the elements will be copied. Basically, I'm looking for something like l1.pop(10,len(l1)) which returns and removes a whole chunk of data. Is there such a thing (and if not, why not?) Cheers, Martin. -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficient way to break up a list into two pieces
On 21 Feb., 02:30, Steven D'Aprano st...@remove-this- cybersource.com.au wrote: Python lists are arrays of pointers to objects, so copying a slice is fast: it doesn't have to copy the objects, just pointers. Deleting from the end of the list is also quick, because you don't have to move memory, just clear some pointers and change the length field. Splitting such an array without copying data is, essentially, impossible. Python lists aren't linked lists. Well, to split a C array I would simply set l2 to point to l1[10] and then change the length of l1 (which I store somewhere else). No copying of elements needed. I would have assumed that python can do something like this with its internal arrays of pointers, too. Anyway, this was more a question about coding style. I use l1.extend(l2) or l1 += l2 rather than l1 = l1 + l2 because it's as readable and possibly faster. I was simply wondering if something similar exists for splitting lists. -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficient way to break up a list into two pieces
On 21 Feb., 02:41, Steven D'Aprano st...@remove-this- cybersource.com.au wrote: What the OP is doing is quite different: (1) copy l1[:10] (2) assign the name l2 to it (3) resize l1 in place to the first 10 items. What the OP wants is: (1) assign the name l2 to l1[:10] without copying (2) resize l1 in place to the first 10 items without affecting l2. Assuming you meant l1[10:] instead of l1[:10], that's what I wanted, yes. -- http://mail.python.org/mailman/listinfo/python-list
