Jeffrey Zhong created HBASE-8018:
------------------------------------
Summary: Add "Flaky Testcase Dector" tool into dev-tools
Key: HBASE-8018
URL: https://issues.apache.org/jira/browse/HBASE-8018
Project: HBase
Issue Type: Bug
Components: util
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Fix For: 0.98.0
Hey,
Recently I'm working on some hbase test case failures and I think it's useful
if we can see a report of all failed test cases from most recent runs. So we
can easily see how flaky a test case is. I wrote a tool some time back and
below are some reports against different branches from today's run. You can get
the tools' source at https://github.com/jeffreyz88/jenkins-tools. If we can run
it daily and send out an email then we can quickly notice what may break after
recent check-ins.
Notes:
1) " 873 874 875 876 877 878 879 880 881" are builds which has failed
test cases in current or previous runs
2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"
HBase-0.95 (from last 10 runs configurable)
Failed Test Cases 21 22 23 24 25 27
========================================================
org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb
0 0 0 0 -1 -1
org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbulkdeletecolumn
-1 0 0 0 0 0
org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint
-1 0 0 0 0 0
org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver.org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver
-1 0 0 0 0 0
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmockedritondeadrs
1 -1 -1 0 1 1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshouldthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
1 1 1 1 1 -1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutdownfixupwhendaughterhassplit
1 1 1 -1 -1 -1
org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0 1
1 1 -1 0
org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicreplication
1 1 1 1 1 -1
org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover
1 -1 0 1 -1 0
org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.queuefailover
0 1 -1 0 -1 0
org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.hadoop.hbase.security.access.testaccesscontroller
0 -1 0 0 0 0
As you can see we have a few test cases don't run successfully at all or
recently.
HBase-TRUNK (from last 10 runs)
Failed Test Cases 3908 3909 3910 3912 3913 3914 3915 3916
========================================================
org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb
0 -1 0 0 0 0 -1 -1
org.apache.hadoop.hbase.client.testadmin.testcloseregionwhenservernameisempty
1 1 1 1 1 1 -1 0
org.apache.hadoop.hbase.client.testscannertimeout.test3686a 1 1 -1 0
1 1 1 1
org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testofflinesnapshotregionoperationsindependent
0 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testonlinesnapshotregionoperationsindependent
1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.master.testassignmentmanageroncluster.testmoveregion
1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.master.testdistributedlogsplitting.testdelayeddeleteonfailure
1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmockedritondeadrs
1 1 1 1 1 -1 0 1
org.apache.hadoop.hbase.master.testtablelockmanager.testdelete 1 1 -1
0 1 1 1 1
org.apache.hadoop.hbase.procedure.testzkprocedurecontrollers.testsimplezkcohortmembercontroller
1 1 1 1 1 1 -1 0
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testexistingznodeblockssplitandwerollback
1 1 1 1 1 -1 0 1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshouldthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutdownfixupwhendaughterhassplit
-1 -1 -1 0 -1 0 -1 0
org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0 -1
-1 0 1 1 1 1
org.apache.hadoop.hbase.replication.testmultislavereplication.testmultislavereplication
1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover
1 1 1 -1 0 -1 0 1
org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.queuefailover
1 1 1 -1 0 -1 0 1
org.apache.hadoop.hbase.security.access.testaccesscontrolfilter.testqualifieraccess
-1 0 1 -1 0 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testdegenerateregions 1 -1 0
1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testfixassignmentsandnohdfschecking
1 -1 0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testfixbytable 1 -1 0 1
1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testhbckthreadpooling 1 -1 0
1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testlingeringreferencefile 1 -1
0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testmissingfirstregion 1 -1
0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testmissinglastregion 1 -1 0
1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testnotinhdfs 1 -1 0 1
1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testnotinmetaordeployedhole 1
-1 0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testoverlapandorphan 1 -1 0
1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testregionhole 1 -1 0 1
1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testregionshouldnotbedeployed 1
-1 0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testsidelineoverlapregion 1 -1
0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testsplitdaughtersnotinmeta 1
-1 0 1 1 1 1 1
More test cases are not in good shape.
HBase-0.94(from last 10 runs)
Failed Test Cases 873 874 875 876 877 878 879 880 881
========================================================
org.apache.hadoop.hbase.backup.testhfilearchiving.testcleaningrace -1 0
1 1 1 1 1 1 1
org.apache.hadoop.hbase.catalog.testmetareadereditor.testgetregion 1 1
-1 0 1 1 1 1 1
org.apache.hadoop.hbase.catalog.testmetareadereditor.testretrying 1 1
-1 0 1 1 1 1 1
org.apache.hadoop.hbase.catalog.testmetareadereditor.testscanmetafortable 1
1 -1 0 1 1 1 1 1
org.apache.hadoop.hbase.catalog.testmetareadereditor.testtableexists 1 1
-1 0 1 1 1 1 1
org.apache.hadoop.hbase.client.testmetamigrationremovinghtd.org.apache.hadoop.hbase.client.testmetamigrationremovinghtd
0 -1 0 0 0 0 0 0 0
org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitsnoabort 1
1 1 1 1 -1 0 1 1
org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitswithabort 1
1 1 1 1 -1 0 1 1
org.apache.hadoop.hbase.client.testshell.testrunshelltests 1 1 1 1
1 1 -1 0 1
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmockedritondeadrs
1 1 -1 -1 0 1 1 1 1
org.apache.hadoop.hbase.regionserver.testatomicoperation.testmultirowmutationmultithreads
1 1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutdownfixupwhendaughterhassplit
1 -1 0 1 1 1 1 1 1
As you can see 0.94 is stable and testmetamigrationremovinghtd hasn't been run
for a while either being removed or due to other failures. Some test cases
aren't stable but not too bad.
Thanks,
-Jeffrey
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
