On Thu, Sep 04, 2008 at 17:13:02 -0700, Simon Michael wrote: > > I don't see this as a particularly good direction to go. Let's not > > codify this approach to creating a benchmark suite unless someone > > decides it's worth working on. As I understand things, there are > > people working on other benchmarking projects.
For what it's worth, Zooko and I have been working on integrating
performance benchmarking into the darcs buildbot. I have been doing
most of my work on IRC lately, so I do apologise for not keeping
everybody up to speed. So what's the status?
1. The buildbot now reports simple time and memory regressions
between older versions of darcs (currently 2.0.2) and the HEAD.
on a fixed set of tests.
2. Buildbot slave owners should install maybench 0.2.4
via cabal install --global or cabal upgrade. Maybench includes
a tool called darcs-benchmark which times darcs operations.
3. The tests used are a sequence of darcs operations on artificial
repositories (generated via darcs-benchmark), and operations on
a zoo of real world repositories, frozen into tarballs at
http://code.haskell.org/darcs/zoo
The zoo only contains a very small repository at the moment,
because we are still waiting to solidify the infrastructure.
4. http://buildbot.darcs.net is down momentarily, and I hope it
will come back up soonish :-)
5. One of our wishlist items is for people to be able to run this
performance regression suite locally. Currently, the only way
to do this is to install buildbot and
darcs get http://code.haskell.org/darcs/buildbot
Setting up buildbot to run locally is pretty straightforward;
you need only create a master and a slave, and run them both.
The rest of it is a matter of clicking buttons on
http://localhost:10310
6. Where to next? I think there have been some buildbot troubles lately
(or maybe darcs test suite troubles). Zooko has been waiting for
some patches to go into darcs.net (now they are in) to make sure that
everything goes green again. Once everybody has the latest version
of maybench on their slaves, and buildbot is up and running again, I
will slowly start adding creatures to the repository zoo.
Help wanted! I would really love for somebody to take charge of the
performance regression testing infrastructure. I especially need
somebody who can be available up to and including the 25 and 26 October
because I want this infrastructure to serve our hacking sprint. If
you have some basic Python skills, or have some benchmarking ideas,
please take a hat! :-)
Cheers,
Eric
PS: I have heard that is is unfortunately difficult to install on
GHC 6.6 for silly reasons. But I also think that if we're going to
do performance testing, we might as well use the latest stable GHC,
so I am not too concerned about this for now. If people agree that
we should go ahead and do this, I think Ganesh might have a couple of
small patches to send.
PPS: The basic goal of stateless performance regression testing against
older version of darcs is due to Jason Dagit. Thanks!
--
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9
pgp5vMcbEkrRN.pgp
Description: PGP signature
_______________________________________________ darcs-users mailing list [email protected] http://lists.osuosl.org/mailman/listinfo/darcs-users
