[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253306#comment-15253306 ] Hudson commented on HBASE-15651: FAILURE: Integrated in HBase-Trunk_matrix #865 (See [https://builds.apache.org/job/HBase-Trunk_matrix/865/]) HBASE-15651 Script to report flaky tests. (Apekshit) (stack: rev 57e1dbc8a65071df3ccd4e0c1f7d124d6a03f0ed) * dev-support/report-flakies.py > Track our flaky tests and use them to improve our build environment > --- > > Key: HBASE-15651 > URL: https://issues.apache.org/jira/browse/HBASE-15651 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy > Fix For: master > > Attachments: HBASE-15651-master-v2.patch, > HBASE-15651-master-v3.patch, HBASE-15651-master-v4.patch, > HBASE-15651-master.patch, flakies.py, hbase-personality-master.patch > > > So i have written this simple script (attached) which looks at history of the > [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] > and outputs a list of flaky tests with some numbers. > Next steps: > 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We > can either directly pull these results into other jobs using curl on this > job's artifacts, or commit the list of flaky test to repo (idk if it's > possible to commit something from jenkins job). > We'll collect results from both *post-commit* job (to add new flakies) and > *flaky-tests* job (to delete tests which are no more flaky). > 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using > --exclude maven flag. Someone familiar with yetus might be able to do it > easily. > 3. Setup a new job (say *flaky-tests*) to run only these flaky tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252982#comment-15252982
]
Hadoop QA commented on HBASE-15651:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 3s
{color} | {color:blue} Precommit patch detected. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 3s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m 3s
{color} | {color:red} The applied patch generated 4 new + 416 unchanged - 0
fixed = 420 total (was 416) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 40s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12800108/hbase-personality-master.patch
|
| JIRA Issue | HBASE-15651 |
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux proserpina.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4c05871 |
| shellcheck | v0.3.3 (This is an old version that has serious bugs. Consider
upgrading.) |
| shellcheck |
https://builds.apache.org/job/PreCommit-HBASE-Build/1541/artifact/patchprocess/diff-patch-shellcheck.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/1541/artifact/patchprocess/whitespace-eol.txt
|
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1541/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> Track our flaky tests and use them to improve our build environment
> ---
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
> Issue Type: Task
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15651-master-v2.patch,
> HBASE-15651-master-v3.patch, HBASE-15651-master-v4.patch,
> HBASE-15651-master.patch, flakies.py, hbase-personality-master.patch
>
>
> So i have written this simple script (attached) which looks at history of the
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix]
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We
> can either directly pull these results into other jobs using curl on this
> job's artifacts, or commit the list of flaky test to repo (idk if it's
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using
> --exclude maven flag. Someone familiar with yetus might be able to do it
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252943#comment-15252943 ] Hadoop QA commented on HBASE-15651: --- (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HBASE-Build/1541/console in case of problems. > Track our flaky tests and use them to improve our build environment > --- > > Key: HBASE-15651 > URL: https://issues.apache.org/jira/browse/HBASE-15651 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy > Attachments: HBASE-15651-master-v2.patch, > HBASE-15651-master-v3.patch, HBASE-15651-master-v4.patch, > HBASE-15651-master.patch, flakies.py, hbase-personality-master.patch > > > So i have written this simple script (attached) which looks at history of the > [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] > and outputs a list of flaky tests with some numbers. > Next steps: > 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We > can either directly pull these results into other jobs using curl on this > job's artifacts, or commit the list of flaky test to repo (idk if it's > possible to commit something from jenkins job). > We'll collect results from both *post-commit* job (to add new flakies) and > *flaky-tests* job (to delete tests which are no more flaky). > 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using > --exclude maven flag. Someone familiar with yetus might be able to do it > easily. > 3. Setup a new job (say *flaky-tests*) to run only these flaky tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252936#comment-15252936 ] Appy commented on HBASE-15651: -- https://issues.apache.org/jira/browse/HBASE-15689 > Track our flaky tests and use them to improve our build environment > --- > > Key: HBASE-15651 > URL: https://issues.apache.org/jira/browse/HBASE-15651 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy > Attachments: HBASE-15651-master-v2.patch, > HBASE-15651-master-v3.patch, HBASE-15651-master-v4.patch, > HBASE-15651-master.patch, flakies.py, hbase-personality-master.patch > > > So i have written this simple script (attached) which looks at history of the > [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] > and outputs a list of flaky tests with some numbers. > Next steps: > 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We > can either directly pull these results into other jobs using curl on this > job's artifacts, or commit the list of flaky test to repo (idk if it's > possible to commit something from jenkins job). > We'll collect results from both *post-commit* job (to add new flakies) and > *flaky-tests* job (to delete tests which are no more flaky). > 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using > --exclude maven flag. Someone familiar with yetus might be able to do it > easily. > 3. Setup a new job (say *flaky-tests*) to run only these flaky tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252929#comment-15252929 ] Appy commented on HBASE-15651: -- Let me create a separate jira for this. > Track our flaky tests and use them to improve our build environment > --- > > Key: HBASE-15651 > URL: https://issues.apache.org/jira/browse/HBASE-15651 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-15651-master-v2.patch, > HBASE-15651-master-v3.patch, HBASE-15651-master-v4.patch, > HBASE-15651-master.patch, flakies.py, hbase-personality-master.patch > > > So i have written this simple script (attached) which looks at history of the > [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] > and outputs a list of flaky tests with some numbers. > Next steps: > 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We > can either directly pull these results into other jobs using curl on this > job's artifacts, or commit the list of flaky test to repo (idk if it's > possible to commit something from jenkins job). > We'll collect results from both *post-commit* job (to add new flakies) and > *flaky-tests* job (to delete tests which are no more flaky). > 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using > --exclude maven flag. Someone familiar with yetus might be able to do it > easily. > 3. Setup a new job (say *flaky-tests*) to run only these flaky tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250799#comment-15250799 ] Dima Spivak commented on HBASE-15651: - Fair enough, [~appy]. Would be nice to cleanup all the Pylint, but otherwise +1 from me. > Track our flaky tests and use them to improve our build environment > --- > > Key: HBASE-15651 > URL: https://issues.apache.org/jira/browse/HBASE-15651 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-15651-master-v2.patch, > HBASE-15651-master-v3.patch, HBASE-15651-master-v4.patch, > HBASE-15651-master.patch, flakies.py > > > So i have written this simple script (attached) which looks at history of the > [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] > and outputs a list of flaky tests with some numbers. > Next steps: > 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We > can either directly pull these results into other jobs using curl on this > job's artifacts, or commit the list of flaky test to repo (idk if it's > possible to commit something from jenkins job). > We'll collect results from both *post-commit* job (to add new flakies) and > *flaky-tests* job (to delete tests which are no more flaky). > 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using > --exclude maven flag. Someone familiar with yetus might be able to do it > easily. > 3. Setup a new job (say *flaky-tests*) to run only these flaky tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250659#comment-15250659
]
Hadoop QA commented on HBASE-15651:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} pylint {color} | {color:blue} 0m 6s
{color} | {color:blue} Pylint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
45m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 14s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12799805/HBASE-15651-master-v4.patch
|
| JIRA Issue | HBASE-15651 |
| Optional Tests | asflicense pylint |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / fa215a6 |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1519/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> Track our flaky tests and use them to improve our build environment
> ---
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15651-master-v2.patch,
> HBASE-15651-master-v3.patch, HBASE-15651-master-v4.patch,
> HBASE-15651-master.patch, flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix]
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We
> can either directly pull these results into other jobs using curl on this
> job's artifacts, or commit the list of flaky test to repo (idk if it's
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using
> --exclude maven flag. Someone familiar with yetus might be able to do it
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250555#comment-15250555 ] Appy commented on HBASE-15651: -- Thanks again. Attached v4 addressing above comments. The reason am doing [24:] is to get rid of just the redundant "org.apache.hadoop.hbase" prefix but keep the remaining package name for context about where the test case lives. > Track our flaky tests and use them to improve our build environment > --- > > Key: HBASE-15651 > URL: https://issues.apache.org/jira/browse/HBASE-15651 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-15651-master-v2.patch, > HBASE-15651-master-v3.patch, HBASE-15651-master-v4.patch, > HBASE-15651-master.patch, flakies.py > > > So i have written this simple script (attached) which looks at history of the > [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] > and outputs a list of flaky tests with some numbers. > Next steps: > 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We > can either directly pull these results into other jobs using curl on this > job's artifacts, or commit the list of flaky test to repo (idk if it's > possible to commit something from jenkins job). > We'll collect results from both *post-commit* job (to add new flakies) and > *flaky-tests* job (to delete tests which are no more flaky). > 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using > --exclude maven flag. Someone familiar with yetus might be able to do it > easily. > 3. Setup a new job (say *flaky-tests*) to run only these flaky tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250353#comment-15250353
]
Hadoop QA commented on HBASE-15651:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m 47s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 17s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} pylint {color} | {color:red} 0m 3s {color}
| {color:red} The applied patch generated 20 new + 0 unchanged - 0 fixed = 20
total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 12s
{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 15s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-04-20 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12799663/HBASE-15651-master-v3.patch
|
| JIRA Issue | HBASE-15651 |
| Optional Tests | asflicense pylint |
| uname | Linux d658e995e916 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / fa215a6 |
| hbaseprotoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/1518/artifact/patchprocess/patch-hbaseprotoc-root.txt
|
| pylint | v1.5.5 |
| pylint |
https://builds.apache.org/job/PreCommit-HBASE-Build/1518/artifact/patchprocess/diff-patch-pylint.txt
|
| hbaseprotoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/1518/artifact/patchprocess/patch-hbaseprotoc-root.txt
|
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1518/console |
| Powered by | Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Track our flaky tests and use them to improve our build environment
> ---
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15651-master-v2.patch,
> HBASE-15651-master-v3.patch, HBASE-15651-master.patch, flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix]
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We
> can either directly pull these results into other jobs using curl on this
> job's artifacts, or commit the list of flaky test to repo (idk if it's
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using
> --exclude maven flag. Someone familiar with yetus might be able to do it
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250191#comment-15250191
]
Dima Spivak commented on HBASE-15651:
-
One more round of nits and then I'm done, I promise. :)
- Line length of 31 is huge, bring that down to 100 chars.
- For the {{\-\-max-builds}} arg, set {{metavar='n'}} to pretty up the help
message. Also consider setting a default value there so that it's clear from
the help message how many jobs will be analyzed if it's left out.
- Alphabetize the order of the args.
- On line 46, have logging as follows to make Pylint happy: {{logger.info("No
test results for %s", build_url)}}
- Just noticed this, but on line 54, truncating test names using {{\[24:\]}}
can get a little hairy. A simpler way would be:
{noformat}
test_name = "{0}#{1}".format(test['className'].rsplit('.',1)[-1], test['name'])
{noformat}
- In lines 69-70, instead of logging an error and then raising an empty
Exception, just put the error in the Exception: {{raise Exception("Bad url
(\{0\}).".format(url))}}.
That's all I got. Sorry for being so type-A, this is an awesome contribution,
[~appy].
> Track our flaky tests and use them to improve our build environment
> ---
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15651-master-v2.patch,
> HBASE-15651-master-v3.patch, HBASE-15651-master.patch, flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix]
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We
> can either directly pull these results into other jobs using curl on this
> job's artifacts, or commit the list of flaky test to repo (idk if it's
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using
> --exclude maven flag. Someone familiar with yetus might be able to do it
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249246#comment-15249246
]
Hadoop QA commented on HBASE-15651:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} pylint {color} | {color:blue} 0m 4s
{color} | {color:blue} Pylint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
39m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 40s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12799663/HBASE-15651-master-v3.patch
|
| JIRA Issue | HBASE-15651 |
| Optional Tests | asflicense pylint |
| uname | Linux pomona.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT
Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 954a417 |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1501/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> Track our flaky tests and use them to improve our build environment
> ---
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15651-master-v2.patch,
> HBASE-15651-master-v3.patch, HBASE-15651-master.patch, flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix]
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We
> can either directly pull these results into other jobs using curl on this
> job's artifacts, or commit the list of flaky test to repo (idk if it's
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using
> --exclude maven flag. Someone familiar with yetus might be able to do it
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249226#comment-15249226 ] Appy commented on HBASE-15651: -- Thanks again for the review. Attaching patch v3 with fixes. Is there a scenario where --builds flag will be useful in our flaky setup? If yes, i'll add it, otherwise we can postpone it to when the real need arises. > Track our flaky tests and use them to improve our build environment > --- > > Key: HBASE-15651 > URL: https://issues.apache.org/jira/browse/HBASE-15651 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-15651-master-v2.patch, HBASE-15651-master.patch, > flakies.py > > > So i have written this simple script (attached) which looks at history of the > [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] > and outputs a list of flaky tests with some numbers. > Next steps: > 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We > can either directly pull these results into other jobs using curl on this > job's artifacts, or commit the list of flaky test to repo (idk if it's > possible to commit something from jenkins job). > We'll collect results from both *post-commit* job (to add new flakies) and > *flaky-tests* job (to delete tests which are no more flaky). > 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using > --exclude maven flag. Someone familiar with yetus might be able to do it > easily. > 3. Setup a new job (say *flaky-tests*) to run only these flaky tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249153#comment-15249153
]
Dima Spivak commented on HBASE-15651:
-
Looks so awesome, Appy! For anyone who wants to see what this looks like:
{noformat}
dspivak-MBP:~ dspivak$ ./appytest.py --max_runs=15
https://builds.apache.org/job/HBase-TRUNK/
WARNING:root:Analyzing job: https://builds.apache.org/job/HBase-TRUNK/
WARNING:root:
Test Name Failed Total Runs Flakyness
WARNING:root:
regionserver.TestRegionReplicaFailover#testSecondaryRegionKillWhilePrimaryIsAcceptingWrites[0]
1 13 8%
WARNING:root:
util.TestRegionMover#testLoadWithoutAck 1 9 11%
WARNING:root:
regionserver.TestSplitTransactionOnCluster#testSSHCleanupDaugtherRegionsOfAbortedSplit
1 13 8%
WARNING:root:
regionserver.TestRegionReplicaFailover#testPrimaryRegionKill[0] 1
13 8%
{noformat}
Just a few more minor tweaks, but then I'm +1:
- I ran this with {{-v}} and got a lot of
{{INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection
(1): builds.apache.org}} lines because using {{logging.basicConfig}} will trip
verbosity for every module that uses logging (i.e. and not just yours). A
better way is probably to use {{logging.getLogger()}} and then work with that
{{Logger}} instance.
- The output at the end can probably just use {{print}} since the intended
output is stdout. That'd also save some precious characters for those of us
working on laptops. :)
- This same output should probably suppress the {{Test Name Failed Total Runs
Flakyness}} header if no flaky tests were detected.
- I think there's a typo in the padding of the results (with the {{:10}} and
{{:6}} reversed).
- Please change {{\-\-max_build}} to {{--max-build}} (argparse automatically
makes this {{args.max_build}}, but it pretties up the command line.
- May want to output a list of which builds of a particular job were actually
inspected at the end (as it stands, we'd have to guess which ones are picked up
by the script when run in non-verbose mode).
- As a potential improvement to the user interface, we might want to entertain
allowing specification of {{--builds}} where a user would give a list of builds
to inspect (instead of just going off of the last {{n}} for a project). This
doesn't block my +1 from this one, though, if it's too much work.
> Track our flaky tests and use them to improve our build environment
> ---
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15651-master-v2.patch, HBASE-15651-master.patch,
> flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix]
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We
> can either directly pull these results into other jobs using curl on this
> job's artifacts, or commit the list of flaky test to repo (idk if it's
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using
> --exclude maven flag. Someone familiar with yetus might be able to do it
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249147#comment-15249147
]
Hadoop QA commented on HBASE-15651:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} pylint {color} | {color:blue} 0m 4s
{color} | {color:blue} Pylint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 26s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12799648/HBASE-15651-master-v2.patch
|
| JIRA Issue | HBASE-15651 |
| Optional Tests | asflicense pylint |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 954a417 |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1498/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> Track our flaky tests and use them to improve our build environment
> ---
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15651-master-v2.patch, HBASE-15651-master.patch,
> flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix]
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We
> can either directly pull these results into other jobs using curl on this
> job's artifacts, or commit the list of flaky test to repo (idk if it's
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using
> --exclude maven flag. Someone familiar with yetus might be able to do it
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249112#comment-15249112 ] Appy commented on HBASE-15651: -- Thanks for the great review [~dimaspivak]. Addressed all the comments in the new patch (v2). bq. Can you add a comment describing how you know whether a particular job is a multi-configuration job or not? In my experience, it's been through looking for a runs key in the JSON of a particular build. yes that's the way, except that the key name is 'activeConfigurations'. > Track our flaky tests and use them to improve our build environment > --- > > Key: HBASE-15651 > URL: https://issues.apache.org/jira/browse/HBASE-15651 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-15651-master-v2.patch, HBASE-15651-master.patch, > flakies.py > > > So i have written this simple script (attached) which looks at history of the > [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] > and outputs a list of flaky tests with some numbers. > Next steps: > 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We > can either directly pull these results into other jobs using curl on this > job's artifacts, or commit the list of flaky test to repo (idk if it's > possible to commit something from jenkins job). > We'll collect results from both *post-commit* job (to add new flakies) and > *flaky-tests* job (to delete tests which are no more flaky). > 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using > --exclude maven flag. Someone familiar with yetus might be able to do it > easily. > 3. Setup a new job (say *flaky-tests*) to run only these flaky tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248805#comment-15248805
]
Dima Spivak commented on HBASE-15651:
-
Great stuff overall, [~appy]. Some thoughts:
- Best practice is to set shebang to {{/usr/bin/env python}} to handle any
virtualenvs someone might wanna use.
- With three options and any number of positional arguments, you're better off
using argparse for command line parsing as opposed to manually manipulating
sys.argv. Will make it more resilient and maybe even shorter, too.
- Remove space between function name and parentheses on line 63.
- Instead of prints, import logging module and use throughout for status
updates. This makes automation easier in the future.
- Speaking of making life easier with modules, requests tends to be way easier
to read than urllib2
- Good stuff using Jenkins' tree endpoint. Not many people know how awesome it
is. :)
- Instead of using {{sys.exit(1)}} when errors happen, raise an Exception to
let people use this as a library in the future.
- Maybe add {{max-runs}} as an optional argument to let this be controlled
during invocation.
- Can you add a comment describing how you know whether a particular job is a
multi-configuration job or not? In my experience, it's been through looking for
a {{runs}} key in the JSON of a particular build.
- Use with statement when writing file to handle problems that might arise.
- You import {{sets}} and {{os}}, but I don't see you using either.
> Track our flaky tests and use them to improve our build environment
> ---
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15651-master.patch, flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix]
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We
> can either directly pull these results into other jobs using curl on this
> job's artifacts, or commit the list of flaky test to repo (idk if it's
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using
> --exclude maven flag. Someone familiar with yetus might be able to do it
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248768#comment-15248768
]
stack commented on HBASE-15651:
---
Talking about the bit on the end...
{code}set([u'regionserver.TestDateTieredCompactionPolicy#negativeForMajor',
u'client.TestMetaCache#testPreserveMetaCacheOnException',
u'regionserver.TestFailedAppendAndSync#testLockupAroundBadAssignSync',
...
{code}
> Track our flaky tests and use them to improve our build environment
> ---
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15651-master.patch, flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix]
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We
> can either directly pull these results into other jobs using curl on this
> job's artifacts, or commit the list of flaky test to repo (idk if it's
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using
> --exclude maven flag. Someone familiar with yetus might be able to do it
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248766#comment-15248766
]
stack commented on HBASE-15651:
---
I tried it:
{code}
kalashnikov:hbase.git stack$ python ./dev-support/report-flakies.py
urls missing.
Usage: ./dev-support/report-flakies.py [options] urls
urls Space separated list of urls (single/multi-configuration project) to
analyze
Options:
-hPrints this help message
-rNumber of runs to analyze for each job (if available in jenkins)
-mvn Writes two strings for including/excluding these flaky tests using
maven
flags. These strings are written to files so they can be saved as
artifacts
and easily imported in other projects.
kalashnikov:hbase.git stack$
kalashnikov:hbase.git stack$ python ./dev-support/report-flakies.py
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/
Analyzing job:
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/856/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/855/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/854/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/853/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/852/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/851/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/850/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/849/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/848/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/847/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/846/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/845/
No test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/845/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/844/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/843/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/842/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/841/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/840/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/839/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/838/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/837/
Getting test results for
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/745/
Test Name Failed Total Runs Flakyness
regionserver.TestDateTieredCompactionPolicy#negativeForMajor 1
19 5%
client.TestMetaCache#testPreserveMetaCacheOnException 1 16 6%
regionserver.TestFailedAppendAndSync#testLockupAroundBadAssignSync 1
17 6%
security.visibility.TestVisibilityLabelsWithACL#testScanForUserWithFewerLabelAuthsThanLabelsInScanAuthorizations
1 17 6%
regionserver.TestRegionMergeTransactionOnCluster#testMergeWithReplicas
2 17 12%
security.visibility.TestVisibilityLabelsWithDeletes#testDeleteColumnWithLatestTimeStampWhenNoVersionMatches
1 17 6%
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15246651#comment-15246651 ] Appy commented on HBASE-15651: -- Good to commit now. > Track our flaky tests and use them to improve our build environment > --- > > Key: HBASE-15651 > URL: https://issues.apache.org/jira/browse/HBASE-15651 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-15651-master.patch, flakies.py > > > So i have written this simple script (attached) which looks at history of the > [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] > and outputs a list of flaky tests with some numbers. > Next steps: > 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We > can either directly pull these results into other jobs using curl on this > job's artifacts, or commit the list of flaky test to repo (idk if it's > possible to commit something from jenkins job). > We'll collect results from both *post-commit* job (to add new flakies) and > *flaky-tests* job (to delete tests which are no more flaky). > 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using > --exclude maven flag. Someone familiar with yetus might be able to do it > easily. > 3. Setup a new job (say *flaky-tests*) to run only these flaky tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244016#comment-15244016
]
Hadoop QA commented on HBASE-15651:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} pylint {color} | {color:blue} 0m 12s
{color} | {color:blue} Pylint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
160m 17s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
7s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 162m 17s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12799058/HBASE-15651-master.patch
|
| JIRA Issue | HBASE-15651 |
| Optional Tests | asflicense pylint |
| uname | Linux priapus.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT
Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d815211 |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1441/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> Track our flaky tests and use them to improve our build environment
> ---
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15651-master.patch, flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix]
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We
> can either directly pull these results into other jobs using curl on this
> job's artifacts, or commit the list of flaky test to repo (idk if it's
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using
> --exclude maven flag. Someone familiar with yetus might be able to do it
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242470#comment-15242470
]
stack commented on HBASE-15651:
---
Script is missing apache license.
Does this if I pass it general Trunk URL:
kalashnikov:hbase.git stack$ python ./dev-support/flakies.py
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/848/
Traceback (most recent call last):
File "./dev-support/flakies.py", line 36, in
run_id_to_results[run["number"]] = get_build_results(run["url"])
File "./dev-support/flakies.py", line 20, in get_build_results
for test_cases in json_response["suites"]:
KeyError: 'suites'
Otherwise, it is excellent. I can check it in and fix the apache license if
you want or do you want to do it and address the above (tell user what URL you
are expecting?)
We should probably remove the ./dev-tools/jenkins-tools/ They used to do
something like this:
{code}
22 A tool which pulls test case results from Jenkins server. It displays a
union of failed test cases
23 from the last 15(by default and actual number of jobs can be less depending
on availablity) runs
24 recorded in Jenkins sever and track how each of them are performed for all
the last 15 runs(passed,
25 not run or failed)
{code}
Here is an example run:
{code}
kalashnikov:hbase.git stack$ python ./dev-support/flakies.py
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/848/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/847/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/846/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/845/
No test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/845/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/844/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/843/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/842/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/841/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/840/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/839/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/838/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/837/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/836/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/835/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/834/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/833/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/832/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/831/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/830/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/829/
Getting test results for
https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/745/
Test Name Failed Total Runs Flakyness
client.TestMetaCache#testPreserveMetaCacheOnException 1 18 6%
ipc.TestAsyncIPC#testRTEDuringAsyncConnectionSetup[0] 1 20 5%
security.visibility.TestVisibilityLabelsWithACL#testVisibilityLabelsForUserWithNoAuths
1 19 5%
regionserver.TestRegionMergeTransactionOnCluster#testWholesomeMerge 2
19 11%
[jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
[ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240657#comment-15240657 ] Appy commented on HBASE-15651: -- Script attached. > Track our flaky tests and use them to improve our build environment > --- > > Key: HBASE-15651 > URL: https://issues.apache.org/jira/browse/HBASE-15651 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: flakies.py > > > So i have written this simple script which looks at history of the [post > commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] and > outputs a list of flaky tests with some numbers. > Next steps: > 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We > can either directly pull these results into other jobs using curl on this > job's artifacts, or commit the list of flaky test to repo (idk if it's > possible to commit something from jenkins job). > We'll collect results from both *post-commit* job (to add new flakies) and > *flaky-tests* job (to delete tests which are no more flaky). > 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using > --exclude maven flag. Someone familiar with yetus might be able to do it > easily. > 3. Setup a new job (say *flaky-tests*) to run only these flaky tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
