[jira] [Commented] (HBASE-18540) Ad-hoc job for checking if a test is flaky

2017-08-08 Thread Jonathan Hsieh (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-18540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118915#comment-16118915
 ] 

Jonathan Hsieh commented on HBASE-18540:


I use this script on my lappy.  Making it job sounds like it could be useful.

http://github.mtv.cloudera.com/jon/bin/blob/master/loopfail.sh

> Ad-hoc job for checking if a test is flaky
> --
>
> Key: HBASE-18540
> URL: https://issues.apache.org/jira/browse/HBASE-18540
> Project: HBase
>  Issue Type: New Feature
>  Components: test
>Reporter: Sean Busbey
>
> It would be great if we had an ad-hoc job that could run a given test a bunch 
> of times and tell me if it is flaky.
> I'm trying to evaluate the state of our new nightly jobs, and I'm not sure if 
> hte failures I'm seeing are a) things I need to get fixed soon, b) things 
> that are flaky but previously unrecognized, c) things that are flaky but 
> missed because our current flaky list is only based on the behavior of the 
> master branch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-18540) Ad-hoc job for checking if a test is flaky

2017-08-08 Thread Mike Drob (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-18540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118914#comment-16118914
 ] 

Mike Drob commented on HBASE-18540:
---

Last time I had to do this, I used basically this recipe:

{code}
mvn clean package -DskipTests
for i in {1..100}; do
  mvn test -q -Dtest=$TEST
  echo $? >> results
done
echo $TEST is $(grep -c 1 results) % flaky.
{code}

> Ad-hoc job for checking if a test is flaky
> --
>
> Key: HBASE-18540
> URL: https://issues.apache.org/jira/browse/HBASE-18540
> Project: HBase
>  Issue Type: New Feature
>  Components: test
>Reporter: Sean Busbey
>
> It would be great if we had an ad-hoc job that could run a given test a bunch 
> of times and tell me if it is flaky.
> I'm trying to evaluate the state of our new nightly jobs, and I'm not sure if 
> hte failures I'm seeing are a) things I need to get fixed soon, b) things 
> that are flaky but previously unrecognized, c) things that are flaky but 
> missed because our current flaky list is only based on the behavior of the 
> master branch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-18540) Ad-hoc job for checking if a test is flaky

2017-08-08 Thread Sean Busbey (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-18540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118803#comment-16118803
 ] 

Sean Busbey commented on HBASE-18540:
-

/cc [~mdrob], because I remember him talking about this kind of issue before so 
maybe I've just filed a dup.

> Ad-hoc job for checking if a test is flaky
> --
>
> Key: HBASE-18540
> URL: https://issues.apache.org/jira/browse/HBASE-18540
> Project: HBase
>  Issue Type: New Feature
>  Components: test
>Reporter: Sean Busbey
>
> It would be great if we had an ad-hoc job that could run a given test a bunch 
> of times and tell me if it is flaky.
> I'm trying to evaluate the state of our new nightly jobs, and I'm not sure if 
> hte failures I'm seeing are a) things I need to get fixed soon, b) things 
> that are flaky but previously unrecognized, c) things that are flaky but 
> missed because our current flaky list is only based on the behavior of the 
> master branch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)