Thanks for continuing to look into this, Shane. One suggestion that Patrick brought up, if we have trouble getting to the bottom of this, is doing the git checkout ourselves in the run-tests-jenkins script and cutting out the Jenkins git plugin entirely. That way we can script retries and post friendlier messages about timeouts if they still occur by ourselves.
Do you think that’s worth trying at some point? Nick On Thu, Oct 16, 2014 at 2:04 PM, shane knapp <skn...@berkeley.edu> wrote: > the bad news is that we've had a couple more failures due to timeouts, but > the good news is that the frequency that these happen has decreased > significantly (3 in the past ~18hr). > > seems like the git plugin downgrade has helped relieve the problem, but > hasn't fixed it. i'll be looking in to this more today. > > On Wed, Oct 15, 2014 at 7:05 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> A quick scan through the Spark PR board <https://spark-prs.appspot.com/> >> shows >> no recent failures related to this git checkout problem. >> >> Looks promising! >> >> Nick >> >> On Wed, Oct 15, 2014 at 6:10 PM, shane knapp <skn...@berkeley.edu> wrote: >> >>> ok, we've had about 10 spark pull request builds go through w/o any git >>> timeouts. it seems that the git timeout issue might be licked. >>> >>> i will be definitely be keeping an eye on this for the next few days. >>> >>> thanks for being patient! >>> >>> shane >>> >>> On Wed, Oct 15, 2014 at 2:27 PM, shane knapp <skn...@berkeley.edu> >>> wrote: >>> >>> > four builds triggered.... and no timeouts. :crossestoes: :) >>> > >>> > On Wed, Oct 15, 2014 at 2:19 PM, shane knapp <skn...@berkeley.edu> >>> wrote: >>> > >>> >> ok, we're up and building... :crossesfingersfortheumpteenthtime: >>> >> >>> >> On Wed, Oct 15, 2014 at 1:59 PM, Nicholas Chammas < >>> >> nicholas.cham...@gmail.com> wrote: >>> >> >>> >>> I support this effort. :thumbsup: >>> >>> >>> >>> On Wed, Oct 15, 2014 at 4:52 PM, shane knapp <skn...@berkeley.edu> >>> >>> wrote: >>> >>> >>> >>>> i'm going to be downgrading our git plugin (from 2.2.7 to 2.2.2) to >>> see >>> >>>> if >>> >>>> that helps w/the git fetch timeouts. >>> >>>> >>> >>>> this will require a short downtime (~20 mins for builds to finish, >>> ~20 >>> >>>> mins >>> >>>> to downgrade), and will hopefully give us some insight in to wtf is >>> >>>> going >>> >>>> on. >>> >>>> >>> >>>> thanks for your patience... >>> >>>> >>> >>>> shane >>> >>>> >>> >>> >>> >>> -- >>> >>> You received this message because you are subscribed to the Google >>> >>> Groups "amp-infra" group. >>> >>> To unsubscribe from this group and stop receiving emails from it, >>> send >>> >>> an email to amp-infra+unsubscr...@googlegroups.com. >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >> >>> >> >>> > >>> >> >> >
