Resending this message with one minor correction ( there are currently
2 tests in derbyTesting/functionTests/tests/perf) -- had missed the 'tests'
Sunitha Kambhampati wrote:
As part of DERBY-1970, I have started to work on an extension to
BaseJDBCTestCase that can be used to write performance tests. Just to
get the discussion going, I have also added an example of a simple
performance test that can be run. (see DERBY-1970 for more details).
It is still a work in progress and will need refinements etc but I
thought it would be useful to have something out there to improve and
comment about. I'd appreciate any feedback on this.
Moving forward, I plan to contribute performance tests to derby and so
would like to start the discussion on the hierarchy on where they
should go.
As of today, we have 2 tests in
derbyTesting/functionTests/tests/perf/ , but it is possible that new
benchmarks and tests will be contributed.
1) Are performance tests functional tests and if so, should they run
with the All suite which is part of regression tests for patch
contributions ?
One idea is perf tests could serve double duty as functional test ,
but I think in an ideal world - you would hope that the test
functionality is already covered by an existing 'true' functional
test.. But if we want people to run the performance tests as part of
the regression suite, then the following requirement need to be met --
a) the perf tests that get run with the regression suite must not take
too much resource (ie time, disk, cpu, memory) etc. This can be
achieved by running the performance test for only one iteration and
only once.
b) The timing for the perf tests must be ignored since the purpose of
it being included in the regression suite is just to ensure that we
didnt break anything in the functionality used by the test.
I think (a) can get messy and complicated to achieve - what is
considered as acceptable to be added in All suite etc. For e.g if you
have a lob perf test, it might take quite some time to finish. This
may not be acceptable for a derbyall /All test runs.
So I think, maybe it is ok to not include perf tests as part of the
All suite.
2) Should we have similar sub packages for areas in performance tests
too, like lobs, sql, store,jdbc, etc.
I think yes. It makes sense to have sub packages.
3) Types of tests:
3a)Basic tests: We could have simple basic performance tests that are
not full fledged standard benchmarks like tpc-c or wisconsin can be
added here. E.g Derby1862.java attached to derby-1876. 3b)Benchmark
tests: Benchmarks like tpc-c , wisconsin, something like dots etc.
__________________
*Some Options:
*Option 1
derbyTesting/functionTests/tests/perf - similar to what it is today.
-- also allow for subpackages like lob, sql,jdbc, store.
-- _Suite.java in the perf directory will run all tests in perf suite.
any other benchmark that can be contributed will have to find a new
location to avoid clutter.
derbyTesting/anyotherbenchmark/..
In this option, performance tests are scattered in two places. one is
under functionTests/tests/perf and when someone contributes a
benchmark, it will be at same level as functionTests. ie.
derbyTesting derbyTest/anyotherbenchmark/
Option 2:
derbyTesting/perf/anyperfbenchmark/ -- e.g lets say tpcc
implementation gets contributed.. it could be derbyTesting/perf/tpcc
or some such name
derbyTesting/perf/basic/ -- all basic tests under this directory,
tests can be added to subpackages depending on the area that the test
is testing . so for example, there could be subpackages like lob,
jdbc, sql, store etc. Tests can exist under basic if they are generic
e.g. to add a performance test to test lob, you would have
derbyTesting/perf/basic/lob/LobTest.java
derbyTesting/perf/basic/_Suite will run all tests in basic
Option 3:
Similar to option1 but have instead of perf, perfTests similar to
functionTests. But I thought it has been mentioned on the list that it
is odd that we say tests so many times for a test class.
Right now, I like option 2
-- clear distinction between perf tests and functional tests. .
-- there is potential for contributions to add more performance
benchmarks under one main perf directory.
Comments/Thoughts
Thanks,
Sunitha
