Hi,
The summary is that I'm motivated to help with work on the performance
of Windows subversion clients, because the speed of 'svn update' is not
great for us on large working copies, and we keep having to re-arrange
our source to work around that.
At work I've used Subversion happily for many years. There are
perennial discussions in the office about the performance of the windows
clients. Several times in the past we have had to carefully prune the
project, especially external modules imported into our code, in order to
keep acceptable performance of 'svn update'. The Linux clients have
always been fine.
I've been hearing rumours of performance improvements to Windows svn
clients with the 'wc-ng' work. A couple of weeks ago I decided to have
a go with subversion trunk to see what it would make of our repository.
I've compiled it, compared the performance with subversion 1.6 from
cygwin, and done some profiling with Intel VTune. The results follow, I
hope they are interesting.
Cygwin session, svn executable from cygwin. OS is Vista 64:
$ uname -a
CYGWIN_NT-6.0-WOW64 brahe 1.7.1(0.218/5/3) 2009-12-07 11:48 i686 Cygwin
$ svn --version
svn, version 1.6.9 (r901367)
compiled Jan 21 2010, 16:47:22
[...]
$ cd avir-cyg
$ svn info
Path: .
URL: svn://morley/repos/trunk/avir
Repository Root: svn://morley/repos
[...]
$ time svn update
At revision 36888.
real 0m19.865s
user 0m0.250s
sys 0m1.468s
$
This is roughly what we've come to expect. TortoiseSVN built with
subversion 1.6.9 has similar performance over an update that has no changes.
So I compiled subversion trunk, r907984:
$ cd ../avir-newsvn
$ /cygdrive/e/subversions/subversion/installdir/bin/svn.exe --version
svn, version 1.7.0 (dev build)
compiled Feb 8 2010, 16:21:13
[...]
$ /cygdrive/e/subversions/subversion/installdir/bin/svn.exe info
Path: .
URL: svn://morley/repos/trunk/avir
Repository Root: svn://morley/repos
$ time /cygdrive/e/subversions/subversion/installdir/bin/svn.exe update
At revision 36891.
real 1m18.007s
user 0m6.609s
sys 0m11.124s
This is disappointing, my build of trunk is a lot slower than cygwin's 1.6.
Rebuilt with -pie option to gcc (required by VTune instrumented
profile), ran a call graph profiling session in VTune. Without dumping
the whole data, I'll try to describe some interesting points.
The profiling run took 172s to do an update with no changes.
The majority (96s) was spent under "svn_wc_crawl_revisions5", another
important routine was "svn_wc__adm_open_anchor_in_context" with 34.6s.
Both ultimately spend almost all of their time in "svn_sqlite__step".
Total time spent under "svn_sqlite__step" was 95s, all spent in
"sqlite3_step".
The other important sqlite routine was "sqlite3_exec", in which 15.6s
was spent.
Total time spent in "Kernel32.dll,CloseHandle" was 32.2s, 31.7s of that
was via calls from sqlite routines.
Total time spent in "Kernel32.dll,CreateFileW" was 75.3s, all via sqlite
routines.
So it looks like the old 'Windows file handling is expensive' issue is
still there.
Digging a bit further, it looks like almost all of the time that sqlite
is spending in opening and closing files is in two related routines,
"sqlite3PagerClose" and "sqlite3PagerBegin". Looking at the sqlite
source, it seems that these are related to journalling.
This document:
http://web.utk.edu/~jplyon/sqlite/SQLite_optimization_FAQ.html#transaction-wrappers
advises using BEGIN_TRANSACTION and END_TRANSACTION to reduce the number
of times that the journal file is opened. These are used in
svn_sqlite__with_transaction, which is used for all sqlite accesses made
under "svn_wc_crawl_revisions5", but not at all for sqlite access under
"svn_wc__adm_open_anchor_in_context", so maybe there is some
optimisation opportunity there.
However, the majority of time is spent under
"svn_sqlite__with_transaction" so how can we speed that up. We could
either increase the size of the transactions (reducing the calls to
BEGIN/END_TRANSACTION) or make journalling faster.
This document:
http://www.sqlite.org/pragma.html
Says that when the 'locking_mode' is set to EXCLUSIVE, "a small number
of filesystem operations are saved by optimizations enabled in this mode".
Inserting "PRAGMA locking_mode=EXCLUSIVE" into svn_sqlite__open gives:
$ time /cygdrive/e/subversions/subversion/profinstalldir/bin/svn.exe update
At revision 36894.
real 0m47.669s
user 0m6.203s
sys 0m7.593s
So considerably faster than the 1m18s that we saw before, but still
nowhere near as fast as subversion 1.6.
That's where I am for now, I'm going to continue noodling with it but I
hope these results are interesting. If anyone has anything that they
would like me to try I would love to help.
All the best,
Matthew Bentham
ART VPS Ltd.
