Thanks for the update Shane.
As a point of process, for things like this where we re debugging
specific issues - can we use JIRA instead of notifying everyone on the
spark-dev list?
I'd prefer if ops/infra announcements on the dev list are restricted
to things that are widely applicable to developers (e.g. planned or
unplanned maintenance on jenkins), since this last has hundreds of
people on it.
- Patrick
On Fri, Oct 24, 2014 at 1:32 PM, shane knapp <skn...@berkeley.edu> wrote:
> so, things look like they've stabilized significantly over the past 10 days,
> and without any changes on our end:
> <snip>
> $ /root/tools/get_timeouts.sh 10
> timeouts by date:
> 2014-10-14 -- 2
> 2014-10-16 -- 1
> 2014-10-19 -- 1
> 2014-10-20 -- 2
> 2014-10-23 -- 5
>
> timeouts by project:
> 5 NewSparkPullRequestBuilder
> 5 SparkPullRequestBuilder
> 1 Tachyon-Pull-Request-Builder
> total builds (excepting aborted by a user):
> 602
>
> total percentage of builds timing out:
> 01
> </snip>
>
> the NewSparkPullRequestBuilder failures are spread over five different days
> (10-14 through 10-20), and the SparkPullRequestBuilder failures all happened
> yesterday. there were a LOT of SparkPullRequestBuilder builds yesterday
> (60), and the failures happened during these hours (first number == number
> of builds failed, second number == hour of the day):
> <snip>
> $ cat timeouts-102414-130817 | grep SparkPullRequestBuilder | grep
> 2014-10-23 | awk '{print$3}' | awk -F":" '{print$1'} | sort | uniq -c
> 1 03
> 2 20
> 1 22
> 1 23
> </snip>
>
> however, the number of total SparkPullRequestBuilder builds during these
> times don't seem egregious:
> <snip>
> 4 03
> 9 20
> 4 22
> 9 23
> </snip>
>
> nor does the total for ALL builds at those times:
> <snip>
> 5 03
> 9 20
> 7 22
> 11 23
> </snip>
>
> 9 builds was the largest number of SparkPullRequestBuilder builds per hour,
> but there were other hours with 5, 6 or 7 builds/hour that didn't have a
> timeout issue.
>
> in fact, hour 16 (4pm) had the most builds running total yesterday, which
> includes 7 SparkPullRequestBuilder builds, and nothing timed out.
>
> most of the pull request builder hits on github are authenticated w/an oauth
> token. this gives us 5000 hits/hour, and unauthed gives us 60/hour.
>
> in conclusion: there is no way are we hitting github often enough to be
> rate limited. i think i've finally ruled that out completely. :)
>
> --
> You received this message because you are subscribed to the Google Groups
> "amp-infra" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to amp-infra+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
