Thanks Scott for the quick reply. After talking to DevOps it seems as there is no real process that's the culprit. As of right now, the issue is also instability as builds don't always take 2h20m, but sometimes, for some reason it lasts 5-6 hours. I asked them if there is any specific process that's maxing out that CPU and it seems to be random. If two builds went on for longer, one time it was process "x" the other time it was process "y".
Is there anything at the JVM level that needs to be tweaked? Max, min memory levels? I am not familiar with that but i heard it during conversations. We are looking into adding more slaves. As of right now, it is my understanding we have: - one master - two slaves - one slave that writes the build results to a cifs share It is a possibility to soon host on a solid state drive infrastructure as i am working to implement that. Also, i am suggesting to write the build results locally rather than a cifs share hosted on the network. The 100% is during certain times of the build. On Wednesday, September 17, 2014 2:02:38 PM UTC-7, SA Evans wrote: > > Alex, > > I'm not a VMWare admin, so your experience would probably be better than > anything I could suggest for setting up VMware nodes. Is your build set up > (such as with a multiconfiguration / matrix build) to use all 3 nodes for > its work? If your build already uses all 3 slaves and all 3 are maxing out > their CPU for 2+ hours, then I doubt there's much that you can do, as-is. > If you need to reduce the build time by that much, you probably need to > look at splitting up the build into more parallel pieces that can run > simultaneously on more nodes. If your current build is all serial on one > slave and the other two are idle, you need to look at how you can split up > your build into multiple pieces that can run simultaneously on more than > one slave at a time. For example, if you are building for multiple targets > one at a time, see if you can change your build setup to build targets in > parallel on different slaves rather than all of them sequential in one > build. > > In absence of other solutions, the only other thing that will likely give > you significantly better performance is looking at your hardware > configuration in regards to disk speed, processing power, etc. Make sure > your slaves are set for unlimited CPU slices, etc. We've seen significant > build speedups on some of our builds when going with solid state disks, but > I don't know if that's feasible in your situation. Also look to see if you > have other stuff running on your slave nodes that are slowing things down > such as active virus scanners, inventory management tools, live disk > backups, etc. > > When you say you're maxing out the cpu, is that on the slave node, or host > system that's running the VM's? > > One thing that will help you figure out what's going on in regards to CPU > usage is to look at the slave nodes during a build and seeing what > process(es) are taking up the CPU, and see what you can do to optimize > that. Is the 100% CPU usage continuous during the build, or only at brief > times during the build? You need to look at your whole build and see what > parts of the build are taking the most time. Is it the pull from Perforce, > the build, the packaging, the verification, or some other step(s) that are > taking a lot of time? > > Scott > > > > On Wed, Sep 17, 2014 at 3:41 PM, Alex Demitri <[email protected] > <javascript:>> wrote: > >> What are good points of say, VMware nodes set up for good performance? >> >> Also, what would you recommend for reaching 1hr? >> >> On Wednesday, September 17, 2014 1:38:57 PM UTC-7, SA Evans wrote: >>> >>> Alex, >>> >>> The first question I'd ask is what part of the build is running at 100% >>> cpu, and can you tell what processes are consuming the cpu at that time? >>> You need to figure out the bottlenecks and where your pain points are >>> first, before trying to solve your performance issues (or perceived >>> issues). If your build is doing a ton of CPU-intensive work that's all >>> legitimate, then I'd think your 1 hour goal is not realistic, given the >>> information you've provided and assuming you've already set up your VMware >>> nodes properly for good performance. >>> >>> Scott >>> >>> On Wed, Sep 17, 2014 at 3:24 PM, Alex Demitri <[email protected]> >>> wrote: >>> >>>> Hi! We just started using Jenkins for continuous intergration. The code >>>> is pulled from Perforce. We have one jenkins master (Windows VM) and 3 >>>> slaves (Windows VMs). I am more the VMware admin than a programmer. >>>> >>>> I have been trying to tweak more and more the Jenkins slave setup. Now >>>> they are configured as 16vCPUs + 48GB of RAM per slave. Each time during a >>>> build, the CPU is always spiking at 100%. We are closing the build in >>>> 2h20m >>>> but the goal is to reach 1hr. >>>> >>>> What is the best way to do so? What type of tweaks in VMware? How can >>>> we push through the build faster? >>>> >>>> Thanks! >>>> Alex >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
