What size machine are you trying to run this on? >From your description if you are trying to run 10 jobs each using 1 CPU and >10GB of RAM you need a machine that has > 10 cores and > 100GB of RAM. If your >machine too small then you will likely experience thrashing and stalls. Also, default setups of Jenkins have scalability issues. For scalable Jenkins, never run the builds on the master, only on build slaves (unless you have carefully calculated and considered the required resources)
________________________________ From: [email protected] [[email protected]] on behalf of mike crowley [[email protected]] Sent: 03 March 2017 11:03 To: “[email protected]” Subject: Jenkins running large multi-CPU memory-intensive jobs Hi, I'm trying to use Jenkins to automate some hardware build and sim jobs. One of these jobs is to build an FPGA. I use Jenkins to call an FPGA build tool. The FPGA build tool runs processes in parallel (~10 jobs each hogging a CPU and ~10GB of RAM) in order to speedup build time. When I run the tool standalone it runs in ~10 hours. On Jenkins it can run for >24 hours (which is no use, I need this to run overnight). If I only have 1 run it runs OK in 10 hours. When I look at what is going on it seems that some of the "parallel" jobs stall. There is a timeout (within the FPGA build tool - TCL internal to the tool) which isn't getting hit so it looks like Jenkins somehow stalls these jobs. I'm running the job on the same server that the Jenkins instance is running on. Does Jenkins somehow throttle jobs that consume large amount of memory? I mean is there some limit that I can change within Jenkins? There is one thing within the parallel builds in that when 1 finishes the stdout from the FPGA build tool switches from 1 of the parallel builds to another. There is also quite a lot of console output 10MB or so. Any help/thoughts appreciated, Mike. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<redir.aspx?REF=dHcTJFbq_5PhtuX51IQBxy65TDZHqBwqLBPIpWSANc-0zP2hK2LUCAFtYWlsdG86amVua2luc2NpLXVzZXJzK3Vuc3Vic2NyaWJlQGdvb2dsZWdyb3Vwcy5jb20.>. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAK2oGcrmAsW%2BUe1MJk%3Dk6PcisUMNYuaWSuWUVZCTYgX0R7EH0Q%40mail.gmail.com<redir.aspx?REF=96A7eIdm-ZNvb5B9vk_eT4CutYzC-shzba4PepnPT0m0zP2hK2LUCAFodHRwczovL2dyb3Vwcy5nb29nbGUuY29tL2QvbXNnaWQvamVua2luc2NpLXVzZXJzL0NBSzJvR2NybUFzVyUyQlVlMU1KayUzRGs2UGNpc1VNTll1YVdTdVdVVlpDVFlnWDBSN0VIMFElNDBtYWlsLmdtYWlsLmNvbT91dG1fbWVkaXVtPWVtYWlsJnV0bV9zb3VyY2U9Zm9vdGVy>. For more options, visit https://groups.google.com/d/optout<redir.aspx?REF=n4jphIfoRRk33X-w3z0e4SYWA-i74F-_8S1tJ28B1x0OLwCiK2LUCAFodHRwczovL2dyb3Vwcy5nb29nbGUuY29tL2Qvb3B0b3V0>. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b05cd86c99a74bf2850c152dd818b6c2%40partner.eso.org. For more options, visit https://groups.google.com/d/optout.
