On Thu, Dec 14, 2017 at 6:53 PM Nathaniel Irons <[email protected]> wrote:
> When a stash is being performed on a non-master agent, is the stash > operation really CPU-intensive on the master? I wasn't sure from that > description whether it assumed the master was also executing the stash > phase. We did try tripling the CPU allocated to the master's VM, and saw no > change in stash performance. > > As recently as three weeks ago, this same stash pipeline stage on the same > repository was executing in 30-45 seconds. The repo's only grown > fractionally since then. If I'm at the limit of the stash implementation, I > hit a heck of a wall. > > That sounds like something else has changed. If it was working well before, and has only grown a little since then, it seems like it is worth investigating other things that might have changed. Has the disc I/O capability on the master or the agent changed significantly? If you perform a smaller stash from the same workspace, does it perform significantly better? Are there competing processes which might be slowing throughput? Has the Jenkins memory use increased significantly? Mark Waite > Thanks, > > -nat > > > On Thu, Dec 14, 2017, at 03:58 PM, Mark Waite wrote: > > > > On Thu, Dec 14, 2017 at 4:13 PM Nathaniel Irons <[email protected]> wrote: > > Jenkins is building an iOS project with five similarly-sized targets. > The git repository is 251 MB, of which the .git dir is 130 MB. A > pipeline stage handles the checkout, injects build numbers, and stashes > the modified sources, minus the unnecessary .git dir, for the other > build nodes to consume. > > stage ("Checkout and modify") { > agent { label 'xcode-9' } > steps { > checkout scm > sh ("agvtool new-version -all $BUILD_ID") > stash (excludes: '.git', name: "modified-sources") > } > } > > This stash operation consistently takes 15 minutes. Unstashing it on any > of the build nodes then takes about ten minutes per node. Something is > wrong, but I'm not sure what steps to take. > > > Unfortunately, I don't have a solution to offer. > > The online help for stash (in the Pipeline syntax when "Stash - Stash some > files to be used later in the build" is selected), hints that you may be at > the limits of the stash implementation. It says: > > Note that the stash and unstash steps are designed for use with small > files. For large data transfers, use the External Workspace Manager plugin, > or use an external repository manager such as Nexus or Artifactory. This is > because stashed files are archived in a compressed TAR, and with large > files this demands considerable on-master resources, particularly CPU time. > There's not a hard stash size limit, but between 5-100 MB you should > probably consider alternatives. > > Mark Waite > > > > The build environment is vsphere on an older mac pro, with some Mac VMs > for Xcode and a (dockerized) jenkins master in a linux VM. The storage > is all SSD, and we're not seeing any issues with I/O performance other > than this stash/unstash problem. Zipping this same source directory > takes six seconds, and tar+bzip2 takes 15 seconds. > > An individual build of this project runs in under three minutes, so the > unstash alone quadruples our build times. Any suggestions for how to > investigate? > > Thanks, > > -nat > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/1513293192.3586726.1205509440.0B960302%40webmail.messagingengine.com > . > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtE0k5MswmNzRhTRfcHCdq3Ry%2BXbognyX6km1GQ_c8-eEw%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtE0k5MswmNzRhTRfcHCdq3Ry%2BXbognyX6km1GQ_c8-eEw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/1513302770.3633224.1205623960.21FCC9DC%40webmail.messagingengine.com > <https://groups.google.com/d/msgid/jenkinsci-users/1513302770.3633224.1205623960.21FCC9DC%40webmail.messagingengine.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtF3ueKhOO3CtmU1Tb7VxnbTMRJRAZZcVn%2BoKOLrTXYaTw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
