yeah, at this point it might be worth trying. :) the absolutely irritating thing is that i am not seeing this happen w/any other jobs other that the spark prb, nor does it seem to correlate w/time of day, network or system load, or what slave it runs on. nor are we hitting our limit of connections on github. i really, truly hate non-deterministic failures.
i'm also going to write an email to support@github and see if they have any insight in to this as well. On Thu, Oct 16, 2014 at 12:51 PM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > Thanks for continuing to look into this, Shane. > > One suggestion that Patrick brought up, if we have trouble getting to the > bottom of this, is doing the git checkout ourselves in the > run-tests-jenkins script and cutting out the Jenkins git plugin entirely. > That way we can script retries and post friendlier messages about timeouts > if they still occur by ourselves. > > Do you think that’s worth trying at some point? > > Nick > > > On Thu, Oct 16, 2014 at 2:04 PM, shane knapp <skn...@berkeley.edu> wrote: > >> the bad news is that we've had a couple more failures due to timeouts, >> but the good news is that the frequency that these happen has decreased >> significantly (3 in the past ~18hr). >> >> seems like the git plugin downgrade has helped relieve the problem, but >> hasn't fixed it. i'll be looking in to this more today. >> >> On Wed, Oct 15, 2014 at 7:05 PM, Nicholas Chammas < >> nicholas.cham...@gmail.com> wrote: >> >>> A quick scan through the Spark PR board <https://spark-prs.appspot.com/> >>> shows >>> no recent failures related to this git checkout problem. >>> >>> Looks promising! >>> >>> Nick >>> >>> On Wed, Oct 15, 2014 at 6:10 PM, shane knapp <skn...@berkeley.edu> >>> wrote: >>> >>>> ok, we've had about 10 spark pull request builds go through w/o any git >>>> timeouts. it seems that the git timeout issue might be licked. >>>> >>>> i will be definitely be keeping an eye on this for the next few days. >>>> >>>> thanks for being patient! >>>> >>>> shane >>>> >>>> On Wed, Oct 15, 2014 at 2:27 PM, shane knapp <skn...@berkeley.edu> >>>> wrote: >>>> >>>> > four builds triggered.... and no timeouts. :crossestoes: :) >>>> > >>>> > On Wed, Oct 15, 2014 at 2:19 PM, shane knapp <skn...@berkeley.edu> >>>> wrote: >>>> > >>>> >> ok, we're up and building... :crossesfingersfortheumpteenthtime: >>>> >> >>>> >> On Wed, Oct 15, 2014 at 1:59 PM, Nicholas Chammas < >>>> >> nicholas.cham...@gmail.com> wrote: >>>> >> >>>> >>> I support this effort. :thumbsup: >>>> >>> >>>> >>> On Wed, Oct 15, 2014 at 4:52 PM, shane knapp <skn...@berkeley.edu> >>>> >>> wrote: >>>> >>> >>>> >>>> i'm going to be downgrading our git plugin (from 2.2.7 to 2.2.2) >>>> to see >>>> >>>> if >>>> >>>> that helps w/the git fetch timeouts. >>>> >>>> >>>> >>>> this will require a short downtime (~20 mins for builds to finish, >>>> ~20 >>>> >>>> mins >>>> >>>> to downgrade), and will hopefully give us some insight in to wtf is >>>> >>>> going >>>> >>>> on. >>>> >>>> >>>> >>>> thanks for your patience... >>>> >>>> >>>> >>>> shane >>>> >>>> >>>> >>> >>>> >>> -- >>>> >>> You received this message because you are subscribed to the Google >>>> >>> Groups "amp-infra" group. >>>> >>> To unsubscribe from this group and stop receiving emails from it, >>>> send >>>> >>> an email to amp-infra+unsubscr...@googlegroups.com. >>>> >>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>>> >> >>>> >> >>>> > >>>> >>> >>> >> >
