+1, i think we should start looking into that. https://code.google.com/p/gerrit/wiki/Scaling
eyal. ----- Original Message ----- > From: "David Caro" <[email protected]> > To: "Eyal Edri" <[email protected]> > Cc: [email protected], "Michal Skrivanek" <[email protected]> > Sent: Thursday, June 5, 2014 6:20:12 PM > Subject: Re: Jenkins issue (findbugs) > > On Thu 05 Jun 2014 05:10:48 PM CEST, Eyal Edri wrote: > > i think this happens when gerrit server is too overloaded, and can't return > > an answer for more than 10min (default value). > > latest git plugin allows us to set a different value than 10min. > > > > for 1st action, i'd change those failing jobs to use maybe 20min instead, > > and see if it's fixed. > > can you provide list of links for the failing jobs you saw? > > > > for longer run, i'd think on how to reduce/manage load on gerrit server, > > maybe using zuul. [1] > > david - any thoughts? > > > > [1] http://ci.openstack.org/zuul/ > > > > ----- Original Message ----- > >> From: "Michal Skrivanek" <[email protected]> > >> To: [email protected] > >> Sent: Thursday, June 5, 2014 5:36:59 PM > >> Subject: Jenkins issue (findbugs) > >> > >> > >>> ----- Forwarded Message ----- > >>> From: "Frantisek Kobzik" <[email protected]> > >>> To: [email protected] > >>> Sent: Thursday, June 5, 2014 11:23:12 AM > >>> Subject: Jenkins issue (findbugs) > >>> > >>> Hi guys, > >>> > >>> I have an issue with jenkins build and I was advised to contact infra. > >>> The console output is below (Similar issue also happens with other kinds > >>> of > >>> jobs). > >>> > >>> I'm seeing this issue quite often (~50 % of pushes), is there anything > >>> you > >>> can do about it? > >>> > >>> Thanks, > >>> Franta. > >>> > >>> > >>> 9:47:36 ERROR: Timeout after 10 minutes > >>> 09:47:36 FATAL: Failed to fetch from git://gerrit.ovirt.org/ovirt-engine > >>> 09:47:37 hudson.plugins.git.GitException: Failed to fetch from > >>> git://gerrit.ovirt.org/ovirt-engine > >>> 09:47:37 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623) > >>> 09:47:37 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855) > >>> 09:47:37 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880) > >>> 09:47:37 at > >>> org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:118) > >>> 09:47:37 at > >>> hudson.model.AbstractProject.checkout(AbstractProject.java:1320) > >>> 09:47:37 at > >>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609) > >>> 09:47:37 at > >>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) > >>> 09:47:37 at > >>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518) > >>> 09:47:37 at hudson.model.Run.execute(Run.java:1700) > >>> 09:47:37 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) > >>> 09:47:37 at > >>> hudson.model.ResourceController.execute(ResourceController.java:88) > >>> 09:47:37 at hudson.model.Executor.run(Executor.java:231) > >>> 09:47:37 Caused by: hudson.plugins.git.GitException: Command "git fetch > >>> --tags --progress git://gerrit.ovirt.org/ovirt-engine > >>> refs/changes/14/28314/3 --prune" returned status code 143: > >>> 09:47:37 stdout: > >>> 9:47:37 stderr: remote: Counting objects: 136 > >>> remote: Counting objects: 375 > >>> remote: Counting objects: 582 > >>> . > >>> etc. > >>> . > >>> 09:47:37 at > >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1276) > >>> 09:47:37 at > >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1146) > >>> 09:47:37 at > >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87) > >>> 09:47:37 at > >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:254) > >>> 09:47:37 at > >>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) > >>> 09:47:37 at > >>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) > >>> 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:118) > >>> 09:47:37 at hudson.remoting.UserRequest.perform(UserRequest.java:48) > >>> 09:47:37 at hudson.remoting.Request$2.run(Request.java:328) > >>> 09:47:37 at > >>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > >>> 09:47:37 at java.util.concurrent.FutureTask.run(FutureTask.java:262) > >>> 09:47:37 at > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >>> 09:47:37 at > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >>> 09:47:37 at java.lang.Thread.run(Thread.java:744) > >>> . > >> > >> _______________________________________________ > >> Infra mailing list > >> [email protected] > >> http://lists.ovirt.org/mailman/listinfo/infra > >> > > Maybe we can start investigating on create mirrors for gerrit to pull > from the slaves (also downstream if able) at least for read operations, > that will leverage a lot of load from the jenkins slaves (if not all of > it) > > -- > David Caro > > Red Hat S.L. > Continuous Integration Engineer - EMEA ENG Virtualization R&D > > Email: [email protected] > Web: www.redhat.com > RHT Global #: 82-62605 > > _______________________________________________ Infra mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/infra
