On 2/28/2011 9:19 AM, Patricia Shanahan wrote:
Dan Creswell wrote:
On 28 February 2011 14:50, Patricia Shanahan <[email protected]> wrote:
Dennis Reedy wrote:
On Feb 28, 2011, at 1247AM, Patricia Shanahan wrote:
How would you propose handling a case like outrigger.FastList?
It is package access only, so changing its interface to the rest of
outrigger did not affect any public API. Several classes needed to be
changed to handle the interface change.
If I understand your question correctly, I think it should be fairly
straightforward. Following module conventions, we would have a structure
that would look (something) like:
outrigger/src/main/java/org/apache/river/outrigger
outrigger/src/test/java/org/apache/river/outrigger
The test (or benchmark) code would be in the same package, just in a
different directory. You would be able to accommodate your package access
only requirement.
I don't see how that answers the problem of a possible intra-package
interface change that needs to be benchmarked *before* the changes to the
rest of the package that would be needed to integrate the class under test
with the rest of what would be its package if it wins the benchmark.
If I had initially named my new FastList implementation
"com.sun.jini.outrigger.FastList" I could not have compiled outrigger in its
presence. It is not a drop-in replacement for the old FastList.
If it had turned out to be slower than the existing FastList I would still
have wanted to preserve it, and the relevant benchmark, because of the
possibility that future java.util.concurrent changes would make it better.
On the other hand, I would not have done the changes to the rest of
outrigger.
So I think we're coming down to the new FastList implementation having to be
called something else for benchmarking purposes to avoid conflict with old
FastList. Or the new implementation needs to be an inner class of the
benchmark and that could live in the same package as original FastList. Of
course, still packaging and source organisation concerns to conquer.
Giving the new FastList a different name was my original idea for dealing with
this. Now that the discussion has become more active, it may be worth repeating
how I started:
=============================================================
We will need several categories of benchmark code:
1. System level benchmarks. These benchmarks measure some public features, such
as the outrigger JavaSpace implementation. For these, I think a similar
structure to QA may be best. However, I need to understand how the QA harness
links together clients and servers, and whether it has any special performance
implications. We may need, for example, to add network delays to properly score
implementations that involve different amounts of communication.
2. Internal benchmarks. These are more like unit tests, and need to mirror the
main src package structure so that they can access non-public code.
3. Experimental code. In some situations it is useful to do run-offs between two
or more implementations of the same class. We cannot have two classes with the
same fully qualified name at the same time, so this type of test will need
special copies of the classes with modified class names or package names. In
addition to actually doing the tests and picking the implementation to go in the
trunk, it is useful to keep discarded candidates around. One of them may turn
out to be a better basis in a future performance campaign.
I still don't understand how a "branch" or "copy" can't be built parallel to
your work so that you get the "output of the builds" to test with under separate
tests.
Can you help me understand why multiple builds are not working? It just seems
to me that you'd build the existing code "once" to test, and then iterate
through your development practices to work on your "Experiment", testing with
that output as you go.
Gregg Wonderly
