Ideally, I'd like to get enough common structure in the output file
formats that I can automate comparing a new bulk run to a previous bulk
run, and highlighting significant changes.
For my recent dissertation research, I needed to compare results of
large numbers of simulation runs. I found XML a reasonable compromise
between human readability and machine processing.
Not an immediate concern - I'd like to have the problem of having too
many performance tests for manual handling.
Do you know of any existing benchmarks we could use?
Patricia
On 12/22/2010 12:30 PM, Dan Creswell wrote:
I agree with the need for performance tests.
From my own experience I'd say you'd want to be able to run those tests in
isolation but also together to get a big picture view of a change because
spaces being what they are, it's incredibly easy for an optimisation that
improves one test to cripple another.
On 22 December 2010 19:08, Patricia Shanahan<p...@acm.org> wrote:
On 12/22/2010 10:57 AM, jgr...@simulexinc.com wrote:
...
This is the biggest concern, I think. As such, I'd be interested in
seeing performance runs, to back up the intuition. Then, at least,
we'd know precisely what trade-off we're talking about.
The test would need to cover both small batches and large, both in
multiples of the batch-size/takeMultipleLimit and for numbers off of
those multiples, with transactions and without.
I think we need a lot of performance tests, some way to organize them, and
some way to retain their results.
I propose adding a "performance" folder to the River trunk, with
subdirectories "src" and "results". src would contain benchmark source code.
result would contain benchmark output.
System level tests could have their own package hierarchy, under
org.apache.impl, but reflecting what is being measured. Unit level tests
would need to follow the package hierarchy for the code being tested, to get
package access. The results hierarchy would mirror that src hierarchy for
the tests.
Any ideas, alternatives, changes, improvements?
Patricia
