[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249153#comment-15249153
]
Dima Spivak commented on HBASE-15651:
-------------------------------------
Looks so awesome, Appy! For anyone who wants to see what this looks like:
{noformat}
dspivak-MBP:~ dspivak$ ./appytest.py --max_runs=15
https://builds.apache.org/job/HBase-TRUNK/
WARNING:root:Analyzing job: https://builds.apache.org/job/HBase-TRUNK/
WARNING:root:
Test Name Failed Total Runs Flakyness
WARNING:root:
regionserver.TestRegionReplicaFailover#testSecondaryRegionKillWhilePrimaryIsAcceptingWrites[0]
1 13 8%
WARNING:root:
util.TestRegionMover#testLoadWithoutAck 1 9 11%
WARNING:root:
regionserver.TestSplitTransactionOnCluster#testSSHCleanupDaugtherRegionsOfAbortedSplit
1 13 8%
WARNING:root:
regionserver.TestRegionReplicaFailover#testPrimaryRegionKill[0] 1
13 8%
{noformat}
Just a few more minor tweaks, but then I'm +1:
- I ran this with {{-v}} and got a lot of
{{INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection
(1): builds.apache.org}} lines because using {{logging.basicConfig}} will trip
verbosity for every module that uses logging (i.e. and not just yours). A
better way is probably to use {{logging.getLogger()}} and then work with that
{{Logger}} instance.
- The output at the end can probably just use {{print}} since the intended
output is stdout. That'd also save some precious characters for those of us
working on laptops. :)
- This same output should probably suppress the {{Test Name Failed Total Runs
Flakyness}} header if no flaky tests were detected.
- I think there's a typo in the padding of the results (with the {{:10}} and
{{:6}} reversed).
- Please change {{\-\-max_build}} to {{--max-build}} (argparse automatically
makes this {{args.max_build}}, but it pretties up the command line.
- May want to output a list of which builds of a particular job were actually
inspected at the end (as it stands, we'd have to guess which ones are picked up
by the script when run in non-verbose mode).
- As a potential improvement to the user interface, we might want to entertain
allowing specification of {{--builds}} where a user would give a list of builds
to inspect (instead of just going off of the last {{n}} for a project). This
doesn't block my +1 from this one, though, if it's too much work.
> Track our flaky tests and use them to improve our build environment
> -------------------------------------------------------------------
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
> Issue Type: Improvement
> Reporter: Appy
> Assignee: Appy
> Attachments: HBASE-15651-master-v2.patch, HBASE-15651-master.patch,
> flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix]
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We
> can either directly pull these results into other jobs using curl on this
> job's artifacts, or commit the list of flaky test to repo (idk if it's
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using
> --exclude maven flag. Someone familiar with yetus might be able to do it
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
