Hey, Recently I'm working on some hbase test case failures and I think it's useful if we can see a report of all failed test cases from most recent runs. So we can easily see how flaky a test case is. I wrote a tool some time back and below are some reports against different branches from today's run. You can get the tools' source at https://github.com/jeffreyz88/jenkins-tools. If we can run it daily and send out an email then we can quickly notice what may break after recent check-ins.
Notes: 1) " 873 874 875 876 877 878 879 880 881" are builds which has failed test cases in current or previous runs 2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED" HBase-0.95 (from last 10 runs configurable) Failed Test Cases 21 22 23 24 25 27 ======================================================== org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.h adoop.hbase.catalog.testmetamigrationconvertingtopb 0 0 0 0 -1 -1 org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbulkd eletecolumn -1 0 0 0 0 0 org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apache. hadoop.hbase.coprocessor.example.testrowcountendpoint -1 0 0 0 0 0 org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver. org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver -1 0 0 0 0 0 org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock edritondeadrs 1 -1 -1 0 1 1 org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshoul dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback 1 1 1 1 1 -1 org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutd ownfixupwhendaughterhassplit 1 1 1 -1 -1 -1 org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0 1 1 1 -1 0 org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicreplicat ion 1 1 1 1 1 -1 org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailov er 1 -1 0 1 -1 0 org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.q ueuefailover 0 1 -1 0 -1 0 org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.hado op.hbase.security.access.testaccesscontroller 0 -1 0 0 0 0 As you can see we have a few test cases don't run successfully at all or recently. HBase-TRUNK (from last 10 runs) Failed Test Cases 3908 3909 3910 3912 3913 3914 3915 3916 ======================================================== org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.h adoop.hbase.catalog.testmetamigrationconvertingtopb 0 -1 0 0 0 0 -1 -1 org.apache.hadoop.hbase.client.testadmin.testcloseregionwhenservernameisempt y 1 1 1 1 1 1 -1 0 org.apache.hadoop.hbase.client.testscannertimeout.test3686a 1 1 -1 0 1 1 1 1 org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testofflinesnap shotregionoperationsindependent 0 1 -1 0 1 1 1 1 org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testonlinesnaps hotregionoperationsindependent 1 1 -1 0 1 1 1 1 org.apache.hadoop.hbase.master.testassignmentmanageroncluster.testmoveregion 1 1 -1 0 1 1 1 1 org.apache.hadoop.hbase.master.testdistributedlogsplitting.testdelayeddelete onfailure 1 1 -1 0 1 1 1 1 org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock edritondeadrs 1 1 1 1 1 -1 0 1 org.apache.hadoop.hbase.master.testtablelockmanager.testdelete 1 1 -1 0 1 1 1 1 org.apache.hadoop.hbase.procedure.testzkprocedurecontrollers.testsimplezkcoh ortmembercontroller 1 1 1 1 1 1 -1 0 org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testexist ingznodeblockssplitandwerollback 1 1 1 1 1 -1 0 1 org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshoul dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback 1 1 -1 0 1 1 1 1 org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutd ownfixupwhendaughterhassplit -1 -1 -1 0 -1 0 -1 0 org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0 -1 -1 0 1 1 1 1 org.apache.hadoop.hbase.replication.testmultislavereplication.testmultislave replication 1 1 -1 0 1 1 1 1 org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailov er 1 1 1 -1 0 -1 0 1 org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.q ueuefailover 1 1 1 -1 0 -1 0 1 org.apache.hadoop.hbase.security.access.testaccesscontrolfilter.testqualifie raccess -1 0 1 -1 0 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testdegenerateregions 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testfixassignmentsandnohdfschecki ng 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testfixbytable 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testhbckthreadpooling 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testlingeringreferencefile 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testmissingfirstregion 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testmissinglastregion 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testnotinhdfs 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testnotinmetaordeployedhole 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testoverlapandorphan 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testregionhole 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testregionshouldnotbedeployed 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testsidelineoverlapregion 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.util.testhbasefsck.testsplitdaughtersnotinmeta 1 -1 0 1 1 1 1 1 More test cases are not in good shape. HBase-0.94(from last 10 runs) Failed Test Cases 873 874 875 876 877 878 879 880 881 ======================================================== org.apache.hadoop.hbase.backup.testhfilearchiving.testcleaningrace -1 0 1 1 1 1 1 1 1 org.apache.hadoop.hbase.catalog.testmetareadereditor.testgetregion 1 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.catalog.testmetareadereditor.testretrying 1 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.catalog.testmetareadereditor.testscanmetafortable 1 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.catalog.testmetareadereditor.testtableexists 1 1 -1 0 1 1 1 1 1 org.apache.hadoop.hbase.client.testmetamigrationremovinghtd.org.apache.hadoo p.hbase.client.testmetamigrationremovinghtd 0 -1 0 0 0 0 0 0 0 org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitsnoabort 1 1 1 1 1 -1 0 1 1 org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitswithabort 1 1 1 1 1 -1 0 1 1 org.apache.hadoop.hbase.client.testshell.testrunshelltests 1 1 1 1 1 1 -1 0 1 org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock edritondeadrs 1 1 -1 -1 0 1 1 1 1 org.apache.hadoop.hbase.regionserver.testatomicoperation.testmultirowmutatio nmultithreads 1 1 1 -1 0 1 1 1 1 org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutd ownfixupwhendaughterhassplit 1 -1 0 1 1 1 1 1 1 As you can see 0.94 is stable and testmetamigrationremovinghtd hasn't been run for a while either being removed or due to other failures. Some test cases aren't stable but not too bad. Thanks, -Jeffrey
