[
https://issues.apache.org/jira/browse/HBASE-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeffrey Zhong updated HBASE-8018:
---------------------------------
Description:
jenkins-tools
=============
A tool which pulls test case results from Jenkins server. It displays a union
of failed test cases
from the last 15(by default and actual number of jobs can be less depending on
availablity) runs
recorded in Jenkins sever and track how each of them are performed for all the
last 15 runs(passed,
not run or failed)
*Pre-requirement(run under folder jenkins-tools)*
Please download jenkins-client from
https://github.com/cosmin/jenkins-client
1) git clone git://github.com/cosmin/jenkins-client.git
2) make sure the dependency jenkins-client version in
./buildstats/pom.xml matches the
downloaded jenkins-client(current value is 0.1.6-SNAPSHOT)
Build command(run under folder jenkins-tools):
{code}
mvn clean package
{code}
Usage are:
{code}
java -jar ./buildstats/target/buildstats.jar <Jenkins HTTP URL> <Job
Name> [number of last most recent jobs to check]
{code}
Sample commands are:
{code}
java -jar ./buildstats/target/buildstats.jar https://builds.apache.org
HBase-TRUNK
{code}
Sample output(where 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means
"FAILED"):
Failed Test Cases Stats 4360 4361 4362 4363 4364 4365 4366 4367 4368 4369
========================================================
org.apache.hadoop.hbase.backup.testhfilearchiving.testcleaningrace 1 1
1 1 1 1 1 1 -1 0
org.apache.hadoop.hbase.migration.testnamespaceupgrade.testrenameusingsnapshots
1 1 1 -1 0 1 1 1 1 1
Skipped Test Cases Stats
======= 4360 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.replication.testreplicationkillmasterrscompressed
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
======= 4361 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
======= 4362 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
======= 4363 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
======= 4368 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.client.testadmin
org.apache.hadoop.hbase.client.testclonesnapshotfromclient
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
was:
Recently I'm working on some hbase test case failures and I think it's useful
if we can see a report of all failed test cases from most recent runs. So we
can easily see how flaky a test case is. I wrote a tool some time back and
below are some reports against different branches from today's run. You can get
the tools' source at https://github.com/jeffreyz88/jenkins-tools. If we can run
it daily and send out an email then we can quickly notice what may break after
recent check-ins.
Notes:
1) " 873 874 875 876 877 878 879 880 881" are builds which has failed
test cases in current or previous runs
2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"
HBase-0.95 (from last 10 runs configurable)
Failed Test Cases 21 22 23 24 25 27
========================================================
org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb
0 0 0 0 -1 -1
org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbulkdeletecolumn
-1 0 0 0 0 0
org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint
-1 0 0 0 0 0
org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver.org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver
-1 0 0 0 0 0
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmockedritondeadrs
1 -1 -1 0 1 1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshouldthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
1 1 1 1 1 -1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutdownfixupwhendaughterhassplit
1 1 1 -1 -1 -1
org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0 1
1 1 -1 0
org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicreplication
1 1 1 1 1 -1
org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover
1 -1 0 1 -1 0
org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.queuefailover
0 1 -1 0 -1 0
org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.hadoop.hbase.security.access.testaccesscontroller
0 -1 0 0 0 0
As you can see we have a few test cases don't run successfully at all or
recently.
HBase-TRUNK (from last 10 runs)
Failed Test Cases 3908 3909 3910 3912 3913 3914 3915 3916
========================================================
org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb
0 -1 0 0 0 0 -1 -1
org.apache.hadoop.hbase.client.testadmin.testcloseregionwhenservernameisempty
1 1 1 1 1 1 -1 0
org.apache.hadoop.hbase.client.testscannertimeout.test3686a 1 1 -1 0
1 1 1 1
org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testofflinesnapshotregionoperationsindependent
0 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testonlinesnapshotregionoperationsindependent
1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.master.testassignmentmanageroncluster.testmoveregion
1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.master.testdistributedlogsplitting.testdelayeddeleteonfailure
1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmockedritondeadrs
1 1 1 1 1 -1 0 1
org.apache.hadoop.hbase.master.testtablelockmanager.testdelete 1 1 -1
0 1 1 1 1
org.apache.hadoop.hbase.procedure.testzkprocedurecontrollers.testsimplezkcohortmembercontroller
1 1 1 1 1 1 -1 0
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testexistingznodeblockssplitandwerollback
1 1 1 1 1 -1 0 1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshouldthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutdownfixupwhendaughterhassplit
-1 -1 -1 0 -1 0 -1 0
org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0 -1
-1 0 1 1 1 1
org.apache.hadoop.hbase.replication.testmultislavereplication.testmultislavereplication
1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover
1 1 1 -1 0 -1 0 1
org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.queuefailover
1 1 1 -1 0 -1 0 1
org.apache.hadoop.hbase.security.access.testaccesscontrolfilter.testqualifieraccess
-1 0 1 -1 0 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testdegenerateregions 1 -1 0
1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testfixassignmentsandnohdfschecking
1 -1 0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testfixbytable 1 -1 0 1
1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testhbckthreadpooling 1 -1 0
1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testlingeringreferencefile 1 -1
0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testmissingfirstregion 1 -1
0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testmissinglastregion 1 -1 0
1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testnotinhdfs 1 -1 0 1
1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testnotinmetaordeployedhole 1
-1 0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testoverlapandorphan 1 -1 0
1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testregionhole 1 -1 0 1
1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testregionshouldnotbedeployed 1
-1 0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testsidelineoverlapregion 1 -1
0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testsplitdaughtersnotinmeta 1
-1 0 1 1 1 1 1
More test cases are not in good shape.
HBase-0.94(from last 10 runs)
Failed Test Cases 873 874 875 876 877 878 879 880 881
========================================================
org.apache.hadoop.hbase.backup.testhfilearchiving.testcleaningrace -1 0
1 1 1 1 1 1 1
org.apache.hadoop.hbase.catalog.testmetareadereditor.testgetregion 1 1
-1 0 1 1 1 1 1
org.apache.hadoop.hbase.catalog.testmetareadereditor.testretrying 1 1
-1 0 1 1 1 1 1
org.apache.hadoop.hbase.catalog.testmetareadereditor.testscanmetafortable 1
1 -1 0 1 1 1 1 1
org.apache.hadoop.hbase.catalog.testmetareadereditor.testtableexists 1 1
-1 0 1 1 1 1 1
org.apache.hadoop.hbase.client.testmetamigrationremovinghtd.org.apache.hadoop.hbase.client.testmetamigrationremovinghtd
0 -1 0 0 0 0 0 0 0
org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitsnoabort 1
1 1 1 1 -1 0 1 1
org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitswithabort 1
1 1 1 1 -1 0 1 1
org.apache.hadoop.hbase.client.testshell.testrunshelltests 1 1 1 1
1 1 -1 0 1
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmockedritondeadrs
1 1 -1 -1 0 1 1 1 1
org.apache.hadoop.hbase.regionserver.testatomicoperation.testmultirowmutationmultithreads
1 1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutdownfixupwhendaughterhassplit
1 -1 0 1 1 1 1 1 1
As you can see 0.94 is stable and testmetamigrationremovinghtd hasn't been run
for a while either being removed or due to other failures. Some test cases
aren't stable but not too bad.
Thanks,
-Jeffrey
> Add "Flaky Testcase Detector" tool into dev-tools
> -------------------------------------------------
>
> Key: HBASE-8018
> URL: https://issues.apache.org/jira/browse/HBASE-8018
> Project: HBase
> Issue Type: Bug
> Components: util
> Reporter: Jeffrey Zhong
> Assignee: Jeffrey Zhong
> Fix For: 0.98.0
>
> Attachments: hbase-8018.patch, hbase-8018-v1.patch,
> hbase-8018-v2.patch
>
>
> jenkins-tools
> =============
> A tool which pulls test case results from Jenkins server. It displays a union
> of failed test cases
> from the last 15(by default and actual number of jobs can be less depending
> on availablity) runs
> recorded in Jenkins sever and track how each of them are performed for all
> the last 15 runs(passed,
> not run or failed)
> *Pre-requirement(run under folder jenkins-tools)*
> Please download jenkins-client from
> https://github.com/cosmin/jenkins-client
> 1) git clone git://github.com/cosmin/jenkins-client.git
> 2) make sure the dependency jenkins-client version in
> ./buildstats/pom.xml matches the
> downloaded jenkins-client(current value is 0.1.6-SNAPSHOT)
>
> Build command(run under folder jenkins-tools):
> {code}
> mvn clean package
> {code}
> Usage are:
> {code}
> java -jar ./buildstats/target/buildstats.jar <Jenkins HTTP URL> <Job
> Name> [number of last most recent jobs to check]
> {code}
> Sample commands are:
> {code}
> java -jar ./buildstats/target/buildstats.jar https://builds.apache.org
> HBase-TRUNK
> {code}
> Sample output(where 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means
> "FAILED"):
> Failed Test Cases Stats 4360 4361 4362 4363 4364 4365 4366 4367 4368
> 4369
> ========================================================
> org.apache.hadoop.hbase.backup.testhfilearchiving.testcleaningrace 1 1
> 1 1 1 1 1 1 -1 0
> org.apache.hadoop.hbase.migration.testnamespaceupgrade.testrenameusingsnapshots
> 1 1 1 -1 0 1 1 1 1 1
> Skipped Test Cases Stats
> ======= 4360 skipped(Or don't have) following test suites =======
> org.apache.hadoop.hbase.replication.testreplicationkillmasterrscompressed
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
> org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
> ======= 4361 skipped(Or don't have) following test suites =======
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
> org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
> ======= 4362 skipped(Or don't have) following test suites =======
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
> org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
> ======= 4363 skipped(Or don't have) following test suites =======
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
> org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
> ======= 4368 skipped(Or don't have) following test suites =======
> org.apache.hadoop.hbase.client.testadmin
> org.apache.hadoop.hbase.client.testclonesnapshotfromclient
> org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
