On Sun, Mar 27, 2011 at 4:51 AM, Mark Phippard <markp...@gmail.com> wrote:
> On Fri, Mar 25, 2011 at 1:33 PM, Mark Phippard <markp...@gmail.com> wrote:
>
> > I have been working on a framework for writing tests to record
> > performance. I have something good enough to share:
> >
> > https://ctf.open.collab.net/sf/projects/csvn
> >
> > It is pretty easy to add new tests if you have ideas on more tests you
> > think we should add. I think I have pretty good coverage of the major
> > functions. The wiki on the site I linked to above has details on how
> > I have constructed the current tests. I am going to put out a call to
> > users for feedback and try to get more people to run the tests and
> > record results.
> >
> > I am not claiming these are anything definitive or even that we will
> > use them to help us make the release decision, but I think it is a
> > start on coming up with some reproducible tests that people can run
> > easily. If after people look at and run the tests they think they are
> > useful or can be tweaked to be useful, then great. If not, then at
> > least I got to write some code for a change :)
> >
> > The tests are written in Java because that is what I know and it gives
> > me good cross platform coverage. However, the Java just drives the
> > command line so all you need to do is have the svn command line in
> > your PATH and that is what it uses for all the work.
>
> These tests are showing some interesting results.
>
> * There are a number of cases where 1.7 shows large performance gains
> over 1.6. For example, a WC with a lot of folders on Windows:
>
>
> https://ctf.open.collab.net/sf/wiki/do/viewPage/projects.csvn/wiki/FolderTests
>
> * Checkout seems to be the biggest remaining hotspot where 1.7 tends
> to be slower, although commit seems to still be slower too.
>
> * Delete and move are slower than I would have expected. Is this
> because we can actually delete the folder now, as opposed to waiting
> until commit time? I actually thought that made would make it faster
> as I thought the folder used to get deleted and then put back (maybe
> just in IDE's).
>
> * There is something seriously bad going on with Anti-Virus on my
> Windows laptop (Symantec Endpoint Protection). I noticed the times
> for commit were way out of whack so I configured my A/V to ignore the
> folder where the benchmarks were running. When I ran the tests again,
> the performance of 1.7 was waaay better. I am about 95% certain the
> A/V is the problem but I need to make sure. While it was great to see
> the good performance, it is still a big concern to see that A/V might
> be having an even bigger impact than it has before. 1.6 also showed
> performance improvements but they were less dramatic than some of the
> 1.7 commands.
>
> Stefan Küng asked me to add svnversion to the benchmark and I have
> done so in the latest release.
>
> I would love to see someone do some tests with the WC on local disk vs
> network mount (1.6 and 1.7). I tried to do it using some virtual
> machines I have access to at CollabNet. The problem is that the
> connection of these boxes to the NetApp with our home folders is too
> slow. Some of the checkouts (even using 1.6) were running for an hour
> and I finally killed the test.
>
> Anyway, overall I am encouraged by the results I am seeing so far. I
> look forward to more people running the tests and sharing the results.
>
>
Attached are my results from the benchmark suite. I've provided two versions
of the 1.7.0 run. One was an older build (prior to the optimizations), and
the last one was a recent build.
These were all run over file:///. I'll try and see if I can re-run it over
svn://.
Cheers,
Daniel B.
=================== TEST RESULTS ==================
SVN Version: 1.7.0-dev
Tests: Basic Tests
Action Time Millis
---------- --------- ---------
Checkout: 0:03.695 3695
Update: 0:03.364 3364
Switch: 0:00.266 266
Proplist: 0:00.071 71
Status: 0:00.099 99
Commit: 0:01.151 1151
Tests: Merge Tests
Action Time Millis
---------- --------- ---------
Merge-all: 0:02.672 2672
Merge-revert: 0:01.229 1229
Merge-synch: 0:02.880 2880
Merge-reintegrate: 0:04.129 4129
Tests: Folder Tests
Action Time Millis
---------- --------- ---------
Folder-co: 0:54.932 54932
Folder-st: 0:01.143 1143
Folder-ci: 0:08.488 8488
Folder-up: 0:03.090 3090
Tests: Binaries Tests
Action Time Millis
---------- --------- ---------
Bin-co: 2:13.671 133671
Bin-up-r25: 0:02.532 2532
Bin-sw: 2:09.494 129494
Bin-cleanup: 0:00.046 46
Bin-rm: 0:03.811 3811
Bin-st: 0:00.341 341
Bin-commit: 0:02.653 2653
Bin-mv: 0:02.852 2852
Bin-st-mv: 0:00.326 326
Bin-commit: 0:03.416 3416
=================== END RESULTS ==================
Total execution time: 6:25.505 385505
Results in wiki format:
Basic Tests:
| 1.7.0-dev | r1086526 | 0:03.695 | 0:03.364 | 0:00.266 | 0:00.071 | 0:00.099 |
0:01.151
Merge Tests:
| 1.7.0-dev | r1086526 | 0:02.672 | 0:01.229 | 0:02.880 | 0:04.129
Folder Tests:
| 1.7.0-dev | r1086526 | 0:54.932 | 0:01.143 | 0:08.488 | 0:03.090
Binaries Tests:
| 1.7.0-dev | r1086526 | 2:13.671 | 0:02.532 | 2:09.494 | 0:00.046 | 0:03.811 |
0:00.341 | 0:02.653 | 0:02.852 | 0:00.326 | 0:03.416
=================== TEST RESULTS ==================
SVN Version: 1.7.0-dev
Tests: Basic Tests
Action Time Millis
---------- --------- ---------
Checkout: 0:05.956 5956
Update: 0:06.505 6505
Switch: 0:03.221 3221
Proplist: 0:03.005 3005
Status: 0:00.100 100
Commit: 0:14.245 14245
Tests: Merge Tests
Action Time Millis
---------- --------- ---------
Merge-all: 0:02.664 2664
Merge-revert: 0:01.225 1225
Merge-synch: 0:02.819 2819
Merge-reintegrate: 0:10.341 10341
Tests: Folder Tests
Action Time Millis
---------- --------- ---------
Folder-co: 8:55.823 535823
Folder-st: 0:01.080 1080
Folder-ci: 30:03.992 1803992
Folder-up: 8:03.296 483296
Tests: Binaries Tests
Action Time Millis
---------- --------- ---------
Bin-co: 3:00.508 180508
Bin-up-r25: 0:05.860 5860
Bin-sw: 2:54.509 174509
Bin-cleanup: 0:00.045 45
Bin-rm: 0:03.883 3883
Bin-st: 0:00.333 333
Bin-commit: 0:46.624 46624
Bin-mv: 0:02.901 2901
Bin-st-mv: 0:00.328 328
Bin-commit: 0:46.894 46894
=================== END RESULTS ==================
Total execution time: 56:30.161 3390161
Results in wiki format:
Basic Tests:
| 1.7.0-dev | r1083961 | 0:05.956 | 0:06.505 | 0:03.221 | 0:03.005 | 0:00.100 |
0:14.245
Merge Tests:
| 1.7.0-dev | r1083961 | 0:02.664 | 0:01.225 | 0:02.819 | 0:10.341
Folder Tests:
| 1.7.0-dev | r1083961 | 8:55.823 | 0:01.080 | 30:03.992 | 8:03.296
Binaries Tests:
| 1.7.0-dev | r1083961 | 3:00.508 | 0:05.860 | 2:54.509 | 0:00.045 | 0:03.883 |
0:00.333 | 0:46.624 | 0:02.901 | 0:00.328 | 0:46.894
