[CONF] Apache Qpid > Proposals for improving the Java tests

[confluence]

Proposals for improving the Java tests Page edited by Phil Harvey Changes (3) ... We suspect that a number of system tests are near-duplicates of each other. To resolve this, we need to: * Re-organise all system tests into a sensible structure, such that closely related tests reside near each other.  We could organise them by category, e.g. ** Areas of the JMS spec ** Separation of JMS vs non-JMS tests.  The former would in theory be able to be run with different JMS clients. * It should then be easy to identify and remove the cruft. This could include replacing some system tests with unit tests. ... If should be easy to create a "fast test suite" that excludes the very slow tests. Developers would run this test suite very frequently when coding. The fact that any tests would be excluded from the suite is not ideal, but at least this is better than not running the tests at all. This would be possible to do using our existing test-profiles approach, but may be easier using JUnit4 categories ([http://kentbeck.github.com/junit/javadoc/latest/org/junit/experimental/categories/Categories.html]). An added benefit of Categories is that they allow us to categorise test classes "in-line" rather than storing their names in a separate file. This reduces the chance of the class name in a test exclusion becoming incorrect after, say, the test's package changes. h5. Consider not restarting the broker between system tests ... Full Content There have been several informal discussions recently about what we should do to improve the Java tests, particularly the system tests. The purpose of this page is to record what has been agreed and to give the wider community an opportunity to participate. The problems Running the whole test suite takes a long time. One consequence of this is that developers have a disincentive to regularly run the full test suite while they're coding. This causes problems to be found late in the development cycle (in some cases, after the changes have been committed to svn). We tolerate the fact that some system tests occasionally fail, on the assumption that the failure is caused by a race condition in the test. For example, some tests wait for x seconds for an expected condition to occur, but this is sometimes insufficient if the server is heavily loaded. QpidBrokerTestCase, the base class from which all our system tests inherit, is hard to work with. It is a large class with multiple concerns (logging, process control, managing system properties, config file management, utility methods for JMS clients such as connecting and sending messages). The solutions Here are some things that have been suggested that would improve matters. Speeding up the test suite Rationalise the system tests and remove redundant ones We suspect that a number of system tests are near-duplicates of each other. To resolve this, we need to: Re-organise all system tests into a sensible structure, such that closely related tests reside near each other.  We could organise them by category, e.g. Areas of the JMS spec Separation of JMS vs non-JMS tests.  The former would in theory be able to be run with different JMS clients. It should then be easy to identify and remove the cruft. This could include replacing some system tests with unit tests. Make it easy to omit slow tests from the test suite We believe that, according the Pareto principle, most of the time is taken up by a small number of tests. For example, some system tests take several minutes but most complete within a few seconds. If should be easy to create a "fast test suite" that excludes the very slow tests. Developers would run this test suite very frequently when coding. The fact that any tests would be excluded from the suite is not ideal, but at least this is better than not running the tests at all. This would be possible to do using our existing test-profiles approach, but may be easier using JUnit4 categories (http://kentbeck.github.com/junit/javadoc/latest/org/junit/experimental/categories/Categories.html). An added benefit of Categories is that they allow us to categorise test classes "in-line" rather than storing their names in a separate file. This reduces the chance of the class name in a test exclusion becoming incorrect after, say, the test's package changes. Consider not restarting the broker between system tests If there's a way to re-use the same broker instance for multiple tests, this would save significant time on the spawned and possibly even the in-process system tests. Removing sporadic test failures Each sporadically-failing test should be examined in turn. The following questions should be asked: Is there a way to re-write the test such that it is less dependent on timing? Is the test really necessary? Bear in mind that most of our testing should be done as conventional unit tests, supplemented by a small number of system tests. Are there any hooks we could add to the broker that would avoid the need for the test to do a Thread.sleep(untilWeThinkBrokerShouldHaveFinishedDoingStuff)? For example, could we add a JMX operation waitUntilFinished(), only intended for use by tests? This would only return when the broker is not processing any messages. I don't yet know how easy it will be to implement this, but I have seen it work well on other applications in the past. Change Notification Preferences View Online | View Changes | Add Comment --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@qpid.apache.org For additional commands, e-mail: commits-h...@qpid.apache.org