[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15248766#comment-15248766
]
stack commented on HBASE-15651:
-------------------------------
I tried it:
{code}
kalashnikov:hbase.git stack$ python ./dev-support/report-flakies.py
urls missing.
Usage: ./dev-support/report-flakies.py [options] urls
urls Space separated list of urls (single/multi-configuration project) to
analyze
Options:
-h Prints this help message
-r Number of runs to analyze for each job (if available in jenkins)
-mvn Writes two strings for including/excluding these flaky tests using
maven
flags. These strings are written to files so they can be saved as
artifacts
and easily imported in other projects.
kalashnikov:hbase.git stack$
kalashnikov:hbase.git stack$ python ./dev-support/report-flakies.py
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/
Analyzing job:
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/856/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/855/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/854/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/853/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/852/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/851/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/850/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/849/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/848/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/847/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/846/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/845/
No test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/845/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/844/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/843/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/842/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/841/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/840/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/839/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/838/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/837/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/745/
Test Name Failed Total Runs Flakyness
regionserver.TestDateTieredCompactionPolicy#negativeForMajor 1
19 5%
client.TestMetaCache#testPreserveMetaCacheOnException 1 16 6%
regionserver.TestFailedAppendAndSync#testLockupAroundBadAssignSync 1
17 6%
security.visibility.TestVisibilityLabelsWithACL#testScanForUserWithFewerLabelAuthsThanLabelsInScanAuthorizations
1 17 6%
regionserver.TestRegionMergeTransactionOnCluster#testMergeWithReplicas
2 17 12%
security.visibility.TestVisibilityLabelsWithDeletes#testDeleteColumnWithLatestTimeStampWhenNoVersionMatches
1 17 6%
ipc.TestAsyncIPC#testRTEDuringAsyncConnectionSetup[0] 2 20 10%
io.hfile.TestScannerFromBucketCache#testBasicScanWithOnheapBucketCache
1 17 6%
regionserver.TestHRegion#testWritesWhileGetting 1 17 6%
master.procedure.TestMasterFailoverWithProcedures#testWALfencingWithWALRolling
3 17 18%
client.TestBlockEvictionFromClient#testReverseScanWithCompaction 1
17 6%
coprocessor.TestMasterObserver#testStarted 1 17 6%
regionserver.throttle.TestFlushWithThroughputController#testFlushControlForStripedStore
1 17 6%
coprocessor.TestMasterObserver#testTableNamesEnumeration 1 17
6%
master.balancer.TestStochasticLoadBalancer2#testRegionReplicasOnMidClusterHighReplication
1 17 6%
master.procedure.TestMasterFailoverWithProcedures#testCreateWithFailover
3 17 18%
coprocessor.TestMasterObserver#testTableOperations 1 17 6%
Analyzing job:
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/856/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/855/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/854/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/853/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/852/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/851/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/850/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/849/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/848/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/847/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/846/
No test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/846/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/845/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/844/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/843/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/842/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/841/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/840/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/839/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/838/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/837/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/745/
Test Name Failed Total Runs Flakyness
regionserver.TestFailedAppendAndSync#testLockupAroundBadAssignSync 1
11 9%
regionserver.TestDateTieredCompactionPolicy#negativeForMajor 2
19 11%
ipc.TestAsyncIPC#testRTEDuringAsyncConnectionSetup[0] 4 20 20%
replication.regionserver.TestReplicationThrottler#testThrottling 5
20 25%
namespace.TestNamespaceAuditor#testRegionOperations 1 11 9%
set([u'regionserver.TestDateTieredCompactionPolicy#negativeForMajor',
u'client.TestMetaCache#testPreserveMetaCacheOnException',
u'regionserver.TestFailedAppendAndSync#testLockupAroundBadAssignSync',
u'security.visibility.TestVisibilityLabelsWithACL#testScanForUserWithFewerLabelAuthsThanLabelsInScanAuthorizations',
u'regionserver.TestRegionMergeTransactionOnCluster#testMergeWithReplicas',
u'security.visibility.TestVisibilityLabelsWithDeletes#testDeleteColumnWithLatestTimeStampWhenNoVersionMatches',
u'ipc.TestAsyncIPC#testRTEDuringAsyncConnectionSetup[0]',
u'namespace.TestNamespaceAuditor#testRegionOperations',
u'replication.regionserver.TestReplicationThrottler#testThrottling',
u'io.hfile.TestScannerFromBucketCache#testBasicScanWithOnheapBucketCache',
u'regionserver.TestHRegion#testWritesWhileGetting',
u'master.procedure.TestMasterFailoverWithProcedures#testWALfencingWithWALRolling',
u'client.TestBlockEvictionFromClient#testReverseScanWithCompaction',
u'coprocessor.TestMasterObserver#testStarted',
u'regionserver.throttle.TestFlushWithThroughputController#testFlushControlForStripedStore',
u'coprocessor.TestMasterObserver#testTableNamesEnumeration',
u'master.balancer.TestStochasticLoadBalancer2#testRegionReplicasOnMidClusterHighReplication',
u'master.procedure.TestMasterFailoverWithProcedures#testCreateWithFailover',
u'coprocessor.TestMasterObserver#testTableOperations'])
{code}
Want to fix the bit on the end before commit?
> Track our flaky tests and use them to improve our build environment
> -------------------------------------------------------------------
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
> Issue Type: Improvement
> Reporter: Appy
> Assignee: Appy
> Attachments: HBASE-15651-master.patch, flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix]
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We
> can either directly pull these results into other jobs using curl on this
> job's artifacts, or commit the list of flaky test to repo (idk if it's
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using
> --exclude maven flag. Someone familiar with yetus might be able to do it
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
