[issue26167] Improve copy.copy speed for built-in types (list/set/dict)

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26167] Improve copy.copy speed for built-in types (list/set/dict)

[Raymond Hettinger]

Raymond Hettinger added the comment:

Then go ahead with the patch as is.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26167] Improve copy.copy speed for built-in types (list/set/dict)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> This looks nicer and should run faster by taking advantage of the LIST_APPEND 
> opcode.

The difference is insignificantly (less than 1%) for large lists 
(list(range(1))), but it is 3-4% slower for small lists (list(range(10))) 
and 20-25% slower for empty lists.

On 2.7 your code always has advantage, but on 3.x seems it doesn't have any 
advantage (thanks to overhead of using a generator).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26167] Improve copy.copy speed for built-in types (list/set/dict)

[Raymond Hettinger]

Raymond Hettinger added the comment:

One suggestion:

def _deepcopy_list(x, memo, deepcopy=deepcopy):
y = []
memo[id(x)] = y
y[:] = [deepcopy(a, memo) for a in x]
return y

This looks nicer and should run faster by taking advantage of the LIST_APPEND 
opcode.

It should be noted that the core concept of the patch is all about the 
minimizing the calling overhead of the copying operation (the cost to call 
type(x) and the overhead in the type constructors for parsing the positional 
and keyword arguments).  When it comes to actually copying the data in the 
containers, there underlying code to do the copying is the same for both the 
patched and unpatched version (that is why you see a speed-up for copying empty 
containers and almost no change for containers that have data in them).

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26167] Improve copy.copy speed for built-in types (list/set/dict)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

If there are no other comments, I'm going to commit the patch in short time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26167] Improve copy.copy speed for built-in types (list/set/dict)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Oh, sorry. Here is a patch.

--
keywords: +patch
Added file: http://bugs.python.org/file41679/copy_speedup.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26167] Improve copy.copy speed for built-in types (list/set/dict)

[Josh Rosenberg]

Josh Rosenberg added the comment:

Good point. Though I don't see any attached patches...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26167] Improve copy.copy speed for built-in types

[Josh Rosenberg]

New submission from Josh Rosenberg:

copy.copy uses a relatively high overhead approach to copying list, set and 
dict, using:

def _copy_with_constructor(x):
return type(x)(x)

This is despite the fact that all three types implement a .copy() method, and 
there is already a defined method:

def _copy_with_copy_method(x):
return x.copy()

that (in %timeit tests) runs with substantially less overhead, in percentage 
terms; for empty lists, sets and dicts, calling _copy_with_constructor or 
_copy_with_copy_method directly on them, the times on my machine (Python 3.5.0, 
Linux x86-64) are:

empty list: 281 ns (constructor), 137 ns (method)
empty set: 275 ns (constructor), 175 ns (method)
empty dict: 372 ns (constructor), 211 ns (method)

The method costs could be trimmed further if _copy_with_copy_method was changed 
from a Python implementation to using operator.methodcaller, e.g.

try:
# If we have _operator, avoids cost of importing Python code; it's part 
of core modules in CPython, already loaded for free
from _operator import methodcaller  
except ImportError:
from operator import methodcaller

_copy_with_copy_method = methodcaller('copy')

This doesn't save a whole lot more (shaves another 9-17 ns off the times for 
the Python version of _copy_with_copy_method), but it's nice in that it avoids 
reinventing the wheel in the copy module. Combining the two changes (to use 
methodcaller for _copy_with_copy_method and to have list, set and dict use 
_copy_with_copy_method) means we can get rid of both Python defined functions 
in favor of a single use of operator.methodcaller used by all types that 
previously used either of them.

Obviously not a high priority fix, but I noticed this while trying to figure 
out a way to fix zlib's lack of support in the copy module ( #26166 which 
apparently duplicates #25007 ) and how to work around it.

--
components: Library (Lib)
messages: 258704
nosy: josh.r
priority: normal
severity: normal
status: open
title: Improve copy.copy speed for built-in types
type: performance
versions: Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26167] Improve copy.copy speed for built-in types (list/set/dict)

[Josh Rosenberg]

Changes by Josh Rosenberg :


--
title: Improve copy.copy speed for built-in types -> Improve copy.copy speed 
for built-in types (list/set/dict)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26167] Improve copy.copy speed for built-in types (list/set/dict)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

methodcaller is not needed. We can use just list.copy etc.

Proposed patch speeds up copying list, dict, set, bytearray, slice, 
NotImplemented. It makes deepcopying list, tuple, dict a little faster.  It 
makes the code for copying and deepcopying using reduce protocol cleaner and a 
little faster. It cleans up the module and adds new tests for builtin types.

--
assignee:  -> serhiy.storchaka
nosy: +alexandre.vassalotti, serhiy.storchaka
stage:  -> patch review
versions:  -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com