On Mon, Jul 17, 2006 at 11:55:46PM -0700, Eric Anderson wrote: > The idea behind the composite benchmark is that one can say: > benchmark(setup-empty-workdir, setup-create-random-files, measure-add, > measure-commit, measure-pull) > > and the benchmark suite will first setup an empty working directory, > create a bunch of random files in that working directory and then > measure (using the instrumenter) the time to add, commit and pull the > resulting repository. This allows one to construct a series of > operations and test them all in sequence. The idea, although not > implemented is that one could change measure- to setup- and the work > would happen during setup not during measurement so that an expensive > instrumenter would only be run over the relevant parts. > > The precise CLI syntax for the above would look like: > -b simple-add='CompositeBenchmark(SetupEmptyWorkdir(), > SetupWorkdirAddFileSource(PartRandomFileSource(1,1024,0.5)), MeasureAdd(), > MeasureCommit(), MeasureWorkingPull())'
I still haven't had a chance to review the patch properly, sorry. Having thought about it a bit, though, I'm feeling uncomfortable about this general approach. In particular, it is reminding me of the ad hoc shell scripts that I was hoping to escape with this framework, because it defines an oddly limited language for specifying procedural computation (why am I writing a list of pre-defined steps, when I could just be writing python code?), and it doesn't make explicit the actual requirements (e.g., the Add benchmark needs a workspace, but nothing here makes that obvious). Finally, as a general note, I am wary of tests that set up the environment for the next test, because they are harder to reason about, harder to run individually or tweak individually, and so on. The general idea of composing mix-and-match parts is very much where I want to go, though! The vague plan I've been working towards is currently illustrated by the "Repo" interface, as implemented by "EmptyRepo" and "ExistingRepo" and, hopefully soon, "SimpleRandomRepo". These all implement particular interface, which is documented, and then the Pull benchmark requires a Repo. So you can say "Pull(EmptyRepo())" or "Pull(SimpleRandomRepo(treesize=10**5))" or whatever. I was planning to create a similar sort of interface for benchmarks that require a workspace, or a workspace + a way to generate edits in the workspace, etc. The overall result is very similar to your example above, but I think better factored -- basically the difference is instead of collapsing things that set up environments and things that actually run tests into one concept, trying to actually identify the different sorts of environmental requirements different tests have, and implementing those seperately. Of course, I haven't actually _done_ most of this yet, so who knows if it will work. But does that general Grand Plan make sense? As for the specific case of porting existing tests, I think the solution might be to just to create a way for tests to give names to particular calls to the vcs object...? -- Nathaniel -- - Don't let your informants burn anything. - Don't grow old. - Be good grad students. -- advice of Murray B. Emeneau on the occasion of his 100th birthday _______________________________________________ Monotone-devel mailing list [email protected] http://lists.nongnu.org/mailman/listinfo/monotone-devel
