Thanks for the list Ram! I haven't had a chance to go through and figure out what tests were hanging regularly (mostly b/c it would take a straight day of consistently running tests), so that is going to be a great start.
Also, I don't think its unreasonable to expect all devs to run with Maven 3. At least I wasn't seeing the issue you are describing with 3, but instead it just goes to the next test w/o printing out timeing stats. -Jesse Yates On Tue, Oct 4, 2011 at 3:43 AM, Ramkrishna S Vasudevan < ramakrish...@huawei.com> wrote: > I think the problem that i mentioned could be due to like i was using maven > 2.4. > In maven 2.9 this problem will not occur. If a testcase gets killed it > will > continue with other testcases. > > Regards > Ram > > -----Original Message----- > From: Ramkrishna S Vasudevan [mailto:ramakrish...@huawei.com] > Sent: Tuesday, October 04, 2011 12:18 PM > To: dev@hbase.apache.org > Subject: RE: Speeding up tests > > Hi Jesse > Thanks for the write up. > I am using the script in HBASE-4480 widely. I have a problem > Sometimes some test cases gets killed by the maven as it took a long time > and those testcases don't have timeout property in them. > > Now if such testcase dont get completed(hanging happens) then maven kills > entirely and we are not able to proceed with other testcases. > Could you help me in this? > > And do you have a list of flaky testcases? I have a list prepared may be > you can add on to it. > <include>**/TestActiveMasterManager*.java</include> > <include>**/TestMasterFailover*.java</include> > <include>**/TestMasterRestartAfterDisablingTable*.java</include> > > <include>**/TestLogsCleaner*.java</include> > <include>**/TestRestartCluster*.java</include> > <include>**/TestMasterAddressManager*.java</include> > <include>**/TestLogRolling*.java</include> > <include>**/TestRegionRebalancing*.java</include> > <include>**/TestZKTable*.java</include> > <include>**/TestZooKeeperNodeTracker*.java</include> > <include>**/TestMergeTool*.java</include> > <include>**/TestMergeTable*.java</include> > <include>**/TestHBaseFsck*.java</include> > <include>**/TestThriftServer*.java</include> > <include>**/TestFullLogReconstruction*.java</include> > <include>**/TestReplicationSink*.java</include> > <include>**/TestReplicationSourceManager*.java</include> > <include>**/TestMasterReplication*.java</include> > <include>**/TestMultiSlaveReplication*.java</include> > <include>**/TestSplitLogWorker*.java</include> > <include>**/TestSplitTransaction*.java</include> > <include>**/TestSplitTransactionOnCluster*.java</include> > <include>**/TestRollingRestart*.java</include> > <include>**/TestSplitLogManager*.java</include> > <include>**/TestAdmin*.java</include> > > > > > > -----Original Message----- > From: Doug Meil [mailto:doug.m...@explorysmedical.com] > Sent: Tuesday, October 04, 2011 6:04 AM > To: dev@hbase.apache.org > Subject: Re: Speeding up tests > > Thanks Jesse. Great write up! > > > > > On 10/3/11 4:55 PM, "Jesse Yates" <jesse.k.ya...@gmail.com> wrote: > > >Hey everyone, > > > >There has been a bunch of work recently on speeding up the testing to make > >it easier for developers to iterate quickly on new features fixes. Part of > >the problem is that the test suite takes anywhere from 1-2 hrs to run and > >have some apparently non-deterministic hanging of tests. > > > >TL;DR To speed it all up, the attack plan would be: > >(1) Move long running tests to be integration tests, > >(2) Use a build server for patches so people only run unit tests locally, > >(3) We add unit tests when integration tests are breaking a lot but unit > >tests pass, > >(4) Go from forked to single jvm unit tests, > >(5) Add in surefire parallelization > >(6) Entertain using HBaseTestingUtilFactory > > > >I recently chatted with Stack and Doug about ways around this. Here is > >what > >we came up with: > > > >(1) Break up long running tests from medium and short (2 mins max) tests > >and > >move the former to be IntegrationTests. This was based on Todd's > >suggestion > >in HBASE-4438. Either naming them integration tests or long running > >functional tests, they would become part of the 'mvn verify' rather than > >the > >'mvn test' suite of tests. Starting point would be to use Doug's > >spreadsheet from HBASE-4448 and when its done, the script from HBASE-4480. > > > >Right now, that means a LOT of tests are going to shift, but it means when > >developers run 'mvn test' the amount of time spent running unit tests will > >be cut down dramatically (hopefully towards the sub 10 -15 mins range) > > > >There is an implicit problem here: if the soon to be integration tests > >capture functionality that is not covered by unit tests, then people may > >incorrectly think that they are not breaking things. Therefore, we would > >do > >(2) and (3): > > > >(2) Add a patch continuous integration server that goes and actually > >builds > >and tests patches as they come in. This would run 'mvn verify' and ensure > >that the patch actually isn't breaking high level/complex functionality. > >It > >would be a requirement before patches are committed that they pass this > >build. > > > >(3) If we find that the unit tests aren't covering a certain level of > >functionality that is constantly breaking on the build server, we add more > >unit tests of the breaking functionality to ensure the unit tests are more > >complete and provide more assurances to developers when running them. > > > >This would be an ongoing process of comparing the integration tests vs. > >the > >unit tests. > > > >(4) Once we have a true unit test suite, we should be able to go from > >'forked' jvm mode back to a single jvm for running tests. Unit tests > >should > >not do crazy fault injection, full failure scenarios, so they should be > >able > >to cleanup after themselves. This means we are going to get some speedup > >from not spinning up a new jvm for each test. > > > >(5) Once we are running in non-forked mode, we can try turning on > >parallelized test execution in surefire, which does parallel runs in a > >single jvm. > > > >(6) Once things all run in a single jvm, using the HBaseTestUtilFactory > >(HBASE-4448) make sense reuse the mini clusters across tests. > > > >What does everyone think of this approach? > > > >Thanks! > > > >-Jesse Yates > >
