+1 on small/medium/large.
> -----Original Message----- > From: Todd Lipcon [mailto:t...@cloudera.com] > Sent: Thursday, October 20, 2011 1:49 PM > To: dev@hbase.apache.org > Subject: Re: [unit tests] small tests vs. others > > +1 on separating into small/medium/large. One of the requirements of > "small" should be that it can run parallel with other tests and in nofork > mode. > > See SUREFIRE-329 for the necessary surefire patch. Until that's integrated > upstream we'd have to have our own patched surefire :( > > -Todd > > On Thu, Oct 20, 2011 at 1:46 PM, N Keywal <nkey...@gmail.com> wrote: > > Hi, > > > > I've done a stat on the tests we have. Many of them are actually quite > > simple: There are 226 test classes in HBase. 100 of them does not > > use a cluster, and takes less than 2 seconds to execute. 24 do not use > > a cluster but need between 2 and 40 seconds. > > > > I tried to execute these with different maven settings. > > fork: once or always, i.e. all the tests are executed in the same JVM > > or there is a new JVM created for every test class parallel; > > deactivated or activated, with 3 threads and a per class > > parallelisation. > > > > The 100 tests executes for the whole maven stuff, including the > > initial dependency & task analysis fork always: 1m50s fork once: 30 > > seconds > > parallel: 30s (=> no difference). > > > > It works quite well, except with TestLruBlockCache that I had to remove. > > > > I tried as well with the 24 'bigger' ones but no cluster, for the > > following > > results: > > fork always: 7m36s > > parallel: 2m32s: 3 times faster. > > > > However, I had to remove many of them, there are some conflicts that I > > have not solved (nor tried to). > > > > The fact that nearly 50% of the tests can be executed in 1 minutes is > > interesting imho. > > > > That makes me wonder if we should try to get a separation of the tests > > in 3 parts, small/medium/large as of > > http://googletesting.blogspot.com/2009/07/software-testing-categorizat > > ion.html > > > > small would be: > > - less than two seconds, can be executed in the same jvm > > - all of them: less than 2 minute > > - no cluster, nothing that can break the environment > > - run by the developers as a part of the build > > > > medium: > > - less than 40 seconds > > - all of them: would last less than 30 minutes, ideally 15 minutes > > - run in an isolated jvm > > - run explicitly by the developers before submitting a fix > > > > large: > > - run in an isolated jvm > > - Should last less than 45minutes > > - run on the central built or explicitly by the developers > > - break the built when it fails. > > > > Only one issue: it's not directly possible with maven, but there are > > known workarounds :-) > > > > > > What do you think? > > > > > > Below the tests set for the 120 tests that seems to work altogether. > > > > > > mvn test > > - > Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hba > > se.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.ap > > ache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client. > > TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.h > > > base.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.fi > > lter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPagina > > tionFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,or > > g.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.fi > > lter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFi > > lter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop. > > hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRando > > > mRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExclude > > Filter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org. > > apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbas > > e.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCac > > > heColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBloc > k > > Queue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache > > .hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hf > > ile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.a > > > pache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hf > > ile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org. > > > apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbas > e > > > .io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable > > > ,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org > > > .apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase > . > > > master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDea > dS > > > erver,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apach > > > e.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase. > > > metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.Test > Me > > > moryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.Test > Tas > > > kMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionH > > > andler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.a > > pache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.h > > > adoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regio > > > nserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKey > V > > > alueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipL > > > istSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apach > e > > > .hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.reg > > > ionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRe > > gionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCoun > > ters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apac > > > he.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.h > > > base.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbas > > > e.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionse > > > rver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLo > > > gBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.a > p > > > ache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.h > > > adoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.mod > e > > > l.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchem > a > > > Model,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.ha > doo > > > p.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.mod > el > > > .TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.Test > > > StorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTabl > > > eInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.a > > > pache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop > .h > > > base.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.mo > de > > l.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apach > > > e.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompar > e, > > > org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.T > e > > > stHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.ha > > doop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org. > > > apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestB > a > > se64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoo > > p.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTes > > t,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoo > > p.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.Test > > > IncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,or > > > g.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.Te > > > stRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorum > P > > > eer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org > .a > > > pache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.had > > oop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter. > > TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.a > > pache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.ha > > doop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.i > > o.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlo > > ckIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.had > > oop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfi > > > le.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache > > > .hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.region > > > server.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestM > e > > > mStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyC > o > > ntrol,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.a > > > pache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop > > .hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.Test > > ReplicationSource > > > > > > -- > Todd Lipcon > Software Engineer, Cloudera
