-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
...
>>
>> Diff is fast, getting the commit message is fast, getting file content is
>> fast. It does seem like there is a lot we could be getting from bzr that
>> wouldn't negatively impact render times.
>
> Are they really? Pick a random, cold cache launchpad (itself) branch.
> Get me the diff from tip to -1. Is that subsecond for the whole
> operation? yes? It might be fast; now try on a linux branch.
>
> We may have *hot* cache operations fast subjectively, but that doesn't
> mean we're at all ready to claim its fast when thrown random data
> sets. Our data store massively exceeds main memory on either the
> storage server or web servers for loggerhead: we have a significant
> optimise-and-iterate job ahead of us to make it fast : and its one we
> should do.
>
> bzr and loggerhead have made huge improvements in performance, but web
> services are looking at a different scale again, and I'm not at all
> convinced we're there yet (OTOH I don't think git or any of the other
> DVCSs are there *either* - cold cache performance is hard).
>
> For instance, getting the commit message for a revision from
> postgresql is (at the moment) 8ms *cold*, 0.8ms *hot*.
>
> -Rob
TIMEIT "b = branch.Branch.open('launchpad/devel'); b.lock_read()\
r = b.last_revision(); rev = b.repository.get_revision(r);\
r.message; b.unlock()"
Hot, is 15ms. If I change it to:
TIMEIT -s "b = branch.Branch.open('launchpad/devel');" \
"b.lock_read(); r = b.last_revision();\
rev = b.repository.get_revision(r); r.message; b.unlock()"
It is 0.361ms to lock + read the content + unlock. So most of that time
is spent in opening the branch.
TIMEIT "call('echo 3 | sudo tee -a /proc/sys/vm/drop_caches',
shell=True, stdout=PIPE)"
24ms
TIMEIT "call...; Branch.open(); ..."
49.6ms
TIMEIT "call...; b.repository.get_revision(r)"
26.9ms
So that looks like:
1) 25.6ms cold to Branch.open() + b.last_revision() +
b.repository.get_revsion()
2) 2.9ms cold to b.repository.get_revision()
3) 15ms hot to Branch.open()...
4) 0.361ms hot to b.repository.get_revision()
Now, I believe you said that XMLRPC is in the 50ms range on the local
system, which is a pre-requisite for loggerhead looking up the path to a
branch, and all sorts of extra overhead. (HTTP Branch.open() vs
local-disk Branch.open()
$ TIMEIT "b = branch.Branch.open('devel')
b.lock_read()
r = b.last_revision()
p = b.repository.get_parent_map([r])[r][0]
rt, pt = b.repository.revision_trees([r,p])
l = list(rt.iter_changes(pt))
b.unlock()"
100 loops, best of 3: 10.4 msec per loop
Now, clearly something very strange is going on, because
repo.get_revision() was up at 15ms, but iter_changes is down at 10.4ms.
$ TIMEIT "call('echo 3 | sudo tee -a /proc/sys/vm/drop_caches',
shell=True, stdout=PIPE)
b = branch.Branch.open('devel')
b.lock_read()
r = b.last_revision()
p = b.repository.get_parent_map([r])[r][0]
rt, pt = b.repository.revision_trees([r,p])
l = list(rt.iter_changes(pt))
b.unlock()"
10 loops, best of 3: 44.9 msec per loop
So that is
1) 31ms cold to do iter_changes,
2) 10ms hot to do iter_changes.
That is pretty far underneath your 1s threshold, even cold. Though I'm
leaving out all of the ugly XMLRPC stuff that we have in production.
John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk4J6owACgkQJdeBCYSNAAMdxQCgvoJG0jxviOEkPIWLtbf3wf1i
JDkAn1Hzf6JSPfjMPTjeVX2pMPnlTBvz
=nE5a
-----END PGP SIGNATURE-----
_______________________________________________
Mailing list: https://launchpad.net/~launchpad-dev
Post to : launchpad-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
