Hi Artur, Thanks for reply.
The machine is a server with 256G RAM and 48 cores I think. The FPGA build application runs standalone fine so I don't think it's a machine problem. I tried running on a slave initially (with similiar large specs) but this also ran slowly. The first 5 or so builds of 10 in parallel seems to work fine, the other 5 don't seem to be "getting in" (as they don't timeout). I'm wondering is it something to do with logging that after a whole the logging slows things down. The log file is ~10MB. Mike. On Friday, March 3, 2017 at 11:56:34 AM UTC, Artur Szostak wrote: > > What size machine are you trying to run this on? > From your description if you are trying to run 10 jobs each using 1 CPU > and 10GB of RAM you need a machine that has > 10 cores and > 100GB of RAM. > If your machine too small then you will likely experience thrashing and > stalls. > Also, default setups of Jenkins have scalability issues. For scalable > Jenkins, never run the builds on the master, only on build slaves (unless > you have carefully calculated and considered the required resources) > > ------------------------------ > *From:* [email protected] <javascript:> [ > [email protected] <javascript:>] on behalf of mike crowley [ > [email protected] <javascript:>] > *Sent:* 03 March 2017 11:03 > *To:* “[email protected] <javascript:>” > *Subject:* Jenkins running large multi-CPU memory-intensive jobs > > Hi, > > I'm trying to use Jenkins to automate some hardware build and sim jobs. > One of these jobs is to build an FPGA. I use Jenkins to call an FPGA build > tool. The FPGA build tool runs processes in parallel (~10 jobs each hogging > a CPU and ~10GB of RAM) in order to speedup build time. When I run the tool > standalone it runs in ~10 hours. On Jenkins it can run for >24 hours (which > is no use, I need this to run overnight). If I only have 1 run it runs OK > in 10 hours. > > When I look at what is going on it seems that some of the "parallel" jobs > stall. There is a timeout (within the FPGA build tool - TCL internal to the > tool) which isn't getting hit so it looks like Jenkins somehow stalls these > jobs. > > I'm running the job on the same server that the Jenkins instance is > running on. > > Does Jenkins somehow throttle jobs that consume large amount of memory? I > mean is there some limit that I can change within Jenkins? > > There is one thing within the parallel builds in that when 1 finishes the > stdout from the FPGA build tool switches from 1 of the parallel builds to > another. There is also quite a lot of console output 10MB or so. > > Any help/thoughts appreciated, > Mike. > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <http://redir.aspx?REF=dHcTJFbq_5PhtuX51IQBxy65TDZHqBwqLBPIpWSANc-0zP2hK2LUCAFtYWlsdG86amVua2luc2NpLXVzZXJzK3Vuc3Vic2NyaWJlQGdvb2dsZWdyb3Vwcy5jb20.> > . > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/CAK2oGcrmAsW%2BUe1MJk%3Dk6PcisUMNYuaWSuWUVZCTYgX0R7EH0Q%40mail.gmail.com > > <http://redir.aspx?REF=96A7eIdm-ZNvb5B9vk_eT4CutYzC-shzba4PepnPT0m0zP2hK2LUCAFodHRwczovL2dyb3Vwcy5nb29nbGUuY29tL2QvbXNnaWQvamVua2luc2NpLXVzZXJzL0NBSzJvR2NybUFzVyUyQlVlMU1KayUzRGs2UGNpc1VNTll1YVdTdVdVVlpDVFlnWDBSN0VIMFElNDBtYWlsLmdtYWlsLmNvbT91dG1fbWVkaXVtPWVtYWlsJnV0bV9zb3VyY2U9Zm9vdGVy> > . > For more options, visit https://groups.google.com/d/optout > <http://redir.aspx?REF=n4jphIfoRRk33X-w3z0e4SYWA-i74F-_8S1tJ28B1x0OLwCiK2LUCAFodHRwczovL2dyb3Vwcy5nb29nbGUuY29tL2Qvb3B0b3V0> > . > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a8c8a2cf-d7ce-4402-8abb-455862ad42b2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
