As part of DERBY-1970, I have started to work on an extension to
BaseJDBCTestCase that can be used to write performance tests. Just to
get the discussion going, I have also added an example of a simple
performance test that can be run. (see DERBY-1970 for more details).
It is still a work in progress and will need refinements etc but I
thought it would be useful to have something out there to improve and
comment about. I'd appreciate any feedback on this.
Moving forward, I plan to contribute performance tests to derby and so
would like to start the discussion on the hierarchy on where they should
go.
As of today, we have 2 tests in derbyTesting/functionTests/perf/ , but
it is possible that new benchmarks and tests will be contributed.
1) Are performance tests functional tests and if so, should they run
with the All suite which is part of regression tests for patch
contributions ?
One idea is perf tests could serve double duty as functional test , but
I think in an ideal world - you would hope that the test functionality
is already covered by an existing 'true' functional test.. But if we
want people to run the performance tests as part of the regression
suite, then the following requirement need to be met --
a) the perf tests that get run with the regression suite must not take
too much resource (ie time, disk, cpu, memory) etc. This can be
achieved by running the performance test for only one iteration and only
once.
b) The timing for the perf tests must be ignored since the purpose of
it being included in the regression suite is just to ensure that we
didnt break anything in the functionality used by the test.
I think (a) can get messy and complicated to achieve - what is
considered as acceptable to be added in All suite etc. For e.g if you
have a lob perf test, it might take quite some time to finish. This may
not be acceptable for a derbyall /All test runs.
So I think, maybe it is ok to not include perf tests as part of the All
suite.
2) Should we have similar sub packages for areas in performance tests
too, like lobs, sql, store,jdbc, etc.
I think yes. It makes sense to have sub packages.
3) Types of tests:
3a)Basic tests: We could have simple basic performance tests that are
not full fledged standard benchmarks like tpc-c or wisconsin can be
added here. E.g Derby1862.java attached to derby-1876.
3b)Benchmark tests: Benchmarks like tpc-c , wisconsin, something like
dots etc.
__________________
*Some Options:
*Option 1
derbyTesting/functionTests/tests/perf - similar to what it is today.
-- also allow for subpackages like lob, sql,jdbc, store.
-- _Suite.java in the perf directory will run all tests in perf suite.
any other benchmark that can be contributed will have to find a new
location to avoid clutter.
derbyTesting/anyotherbenchmark/..
In this option, performance tests are scattered in two places. one is
under functionTests/tests/perf and when someone contributes a
benchmark, it will be at same level as functionTests. ie. derbyTesting
derbyTest/anyotherbenchmark/
Option 2:
derbyTesting/perf/anyperfbenchmark/ -- e.g lets say tpcc implementation
gets contributed.. it could be derbyTesting/perf/tpcc or some such name
derbyTesting/perf/basic/ -- all basic tests under this directory, tests
can be added to subpackages depending on the area that the test is
testing . so for example, there could be subpackages like lob, jdbc,
sql, store etc.
Tests can exist under basic if they are generic
e.g. to add a performance test to test lob, you would have
derbyTesting/perf/basic/lob/LobTest.java
derbyTesting/perf/basic/_Suite will run all tests in basic
Option 3:
Similar to option1 but have instead of perf, perfTests similar to
functionTests. But I thought it has been mentioned on the list that it
is odd that we say tests so many times for a test class.
Right now, I like option 2
-- clear distinction between perf tests and functional tests. .
-- there is potential for contributions to add more performance
benchmarks under one main perf directory.
Comments/Thoughts
Thanks,
Sunitha
