New submission from Giampaolo Rodola' <g.rod...@gmail.com>:
Patch in attachment makes shutil.copytree() use os.scandir() and (differently
from #33414) DirEntry instances are passed around so that cached stat()s are
used also from within copy2() and copystat() functions. The number of times the
filesystem gets accessed via os.stat() is therefore reduced quite consistently.
A similar improvement can be done for rmtree() (but that's for another ticket).
Patch and benchmark script are in attachment.
Linux (+13.5% speedup)
======================
--- without patch:
./python bench.py
Priming the system's cache...
7956 files and dirs, repeat 1/3... min = 0.551s
7956 files and dirs, repeat 2/3... min = 0.548s
7956 files and dirs, repeat 3/3... min = 0.548s
best result = 0.548s
--- with patch:
$ ./python bench.py
Priming the system's cache...
7956 files and dirs, repeat 1/3... min = 0.481s
7956 files and dirs, repeat 2/3... min = 0.479s
7956 files and dirs, repeat 3/3... min = 0.474s
best result = 0.474s
Windows (+17% speedup)
======================
--- without patch:
./python bench.py
Priming the system's cache...
7956 files and dirs, repeat 1/3... min = 9.015s
7956 files and dirs, repeat 2/3... min = 8.747s
7956 files and dirs, repeat 3/3... min = 8.614s
best result = 8.614s
--- with patch:
$ ./python bench.py
Priming the system's cache...
7956 files and dirs, repeat 1/3... min = 7.827s
7956 files and dirs, repeat 2/3... min = 7.369s
7956 files and dirs, repeat 3/3... min = 7.153s
best result = 7.153s
Windows SMB share (+30%)
========================
--- without patch:
C:\Users\user\Desktop\cpython>PCbuild\win32\python.exe bench.py
Priming the system's cache...
7956 files and dirs, repeat 1/3... min = 46.853s
7956 files and dirs, repeat 2/3... min = 46.330s
7956 files and dirs, repeat 3/3... min = 44.720s
best result = 44.720s
--- with patch:
C:\Users\user\Desktop\cpython>PCbuild\win32\python.exe bench.py
Priming the system's cache...
7956 files and dirs, repeat 1/3... min = 31.729s
7956 files and dirs, repeat 2/3... min = 30.936s
7956 files and dirs, repeat 3/3... min = 30.936s
best result = 30.936s
Number of stat() syscalls (-38%)
================================
--- without patch:
$ strace ./python bench.py 2>&1 | grep "stat(" | wc -l
324808
--- with patch:
$ strace ./python bench.py 2>&1 | grep "stat(" | wc -l
198768
----------
components: Library (Lib)
files: bench.py
messages: 318175
nosy: giampaolo.rodola
priority: normal
severity: normal
stage: patch review
status: open
title: Have shutil.copytree(), copy() and copystat() use cached scandir()
stat()s
type: performance
versions: Python 3.8
Added file: https://bugs.python.org/file47624/bench.py
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33695>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
