Sunitha Kambhampati wrote:
Moving forward, I plan to contribute performance tests to derby and so
would like to start the discussion on the hierarchy on where they should
go.
Thanks for volunteering to contribute performance tests. They will
good addition to derby.
As of today, we have 2 tests in derbyTesting/functionTests/perf/ , but
it is possible that new benchmarks and tests will be contributed.
1) Are performance tests functional tests and if so, should they run
with the All suite which is part of regression tests for patch
contributions ?
One idea is perf tests could serve double duty as functional test , but
I think in an ideal world - you would hope that the test functionality
is already covered by an existing 'true' functional test.. But if we
want people to run the performance tests as part of the regression
suite, then the following requirement need to be met --
a) the perf tests that get run with the regression suite must not take
too much resource (ie time, disk, cpu, memory) etc. This can be
achieved by running the performance test for only one iteration and only
once.
b) The timing for the perf tests must be ignored since the purpose of
it being included in the regression suite is just to ensure that we
didnt break anything in the functionality used by the test.
Well , it depends on whether the tests you are planning to contribute
test any functionality that is not already covered by the
functional tests already. If not, one should add such cases to the
functional test for such cases. I would like to see these test to be
run as part of major changes to the code line, but adding it to the
derby-all might be over-kill. Already most of us feel, they take too
long !
2) Should we have similar sub packages for areas in performance tests
too, like lobs, sql, store,jdbc, etc.
I think yes. It makes sense to have sub packages.
yes. Please separate them into different directories, if you can. It
just makes easier for the eyes to find what test is the relevant test
to run during the development!
*Some Options:
*Option 1
derbyTesting/functionTests/tests/perf - similar to what it is today.
-- also allow for subpackages like lob, sql,jdbc, store.
-- _Suite.java in the perf directory will run all tests in perf suite.
any other benchmark that can be contributed will have to find a new
location to avoid clutter.
derbyTesting/anyotherbenchmark/..
In this option, performance tests are scattered in two places. one is
under functionTests/tests/perf and when someone contributes a
benchmark, it will be at same level as functionTests. ie. derbyTesting
derbyTest/anyotherbenchmark/
Option 2:
derbyTesting/perf/anyperfbenchmark/ -- e.g lets say tpcc implementation
gets contributed.. it could be derbyTesting/perf/tpcc or some such name
derbyTesting/perf/basic/ -- all basic tests under this directory, tests
can be added to subpackages depending on the area that the test is
testing . so for example, there could be subpackages like lob, jdbc,
sql, store etc. Tests can exist under basic if they are generic
e.g. to add a performance test to test lob, you would have
derbyTesting/perf/basic/lob/LobTest.java
derbyTesting/perf/basic/_Suite will run all tests in basic
I like the option 2. It is good to separate functional tests and
performance tests, atleast in theory. If you are not against a
longer name , to make it more clear you can say:
derbyTesting/performance/anyperfbenchmark/
Thanks
-suresh
