Hi Vincent, Well, I think Stephen already answered, and I guess you didn't file a RFE like he was talking about? Passing Docker image to work around stash speed is a hack, since you're forced to change your pipeline script when in the logical sense of it you want to stash/unstash some build data.
But, here I'm not even sure what you're doing. If you're building a Docker image of an image targeting customers later or something, which then you're going to test somewhere else, then you want a Docker registry IMO. Because docker save + stash/unstash is a waste compared to pushing/pulling only the modified layers of a Docker image you're building. And if you write your Dockerfile in an interesting oriented way for example, the data need to be transfered could go down from hundreds of MB, if not only keeping a few MB of changes to be pushed around each time... But to answer all that, we probably need to know more about your use case and setup. Cheers 2016-11-22 0:47 GMT+01:00 Vincent Brouillet <[email protected]>: > Stephen, > > We build a docker image in the first build stage and use it to spin up a > temporary environment for automated testing with docker compose on jenkins > nodes. > We rely on docker save and stash: > > - docker save --output="./my-image.tar" image_name' > - > - stash includes: 'my-image.tar', name: 'project-image' > > On the nodes (6 of them in parallel) we unstash. the docker image is > 1.9GB. It takes 4 min to unstash. > We thought of using a private docker repo as an artifact manager, but that > is one more dependency. A docker repo is faster than stash though. > > What are your thoughts on passing docker images to nodes? > > > On Tuesday, 5 July 2016 01:50:15 UTC+10, Stephen Connolly wrote: >> >> I haven't looked into this closely... it being mostly Jesse's baby... but: >> >> The point of using stash and unstash is to enable Jenkins to be smarter >> about the files. If Jenkins knows you are stashing and unstashing then >> Jenkins can be smarter about what it does... e.g. using local copies where >> that is possible, etc. >> >> You also should pay attention to parallel builds and interleaved >> scheduling... which could render an external workspace into an invalid >> state. >> >> So I would encourage you to just use stash and unstash and file concrete >> bugs or RFEs against the current implementations (which my guess is that >> they are the naïve implementation) outlining how less naïve implementations >> could work. >> >> If you build yourself a layer of hacks using external workspaces then: >> >> 1. You will not be able to switch as easily to stash/unstash when >> they are improved >> 2. You will not be a data point towards improving them >> >> Now I can understand if you have pressing needs that force you to make a >> choice away from using stash/unstash (such as - for example - you are >> creating ISO images of DVDs) >> >> Most people should typically not be falling into that use case, and the >> trade-off of concurrent build safety vs speed increase by zero-copy is >> likely not worth it... after all building the right thing slowly will >> always trump building the wrong thing fast. >> >> So, if it were me... I would use stash/unstash for now (unless I was >> dealing with 1gb+ of stuff)... I would concentrate on ensuring that I stash >> / unstash only that which is needed (as it makes the pipeline better... not >> just the stash/unstash faster)... I would file RFEs for some improvements >> around stash/unstash: >> >> - stash/unstash implementation that reuses the file in-place if it >> has the same checksum as the stashed version >> - stash/unstash implementation that copies from local disk if a local >> workspace has the same checksum as the stashed version >> - unstash implementation that creates read-only links rather than copy >> - node implementation that allows for affinity to the stash >> - etc >> >> -Stephen >> >> On 4 July 2016 at 13:35, Christian Ditscher <[email protected]> >> wrote: >> >>> Hi, >>> >>> @Baptiste: Your suggestion sound more me like my last suggestion. >>> Checking out, stashing and then unstashing again. (This could be a solution >>> if it were clear how exactly the stash/unstash works. --> What is >>> transferred? Where are stashed files located/saved?) In general I would >>> like to reduce the a mount of data that has to be transferred over network >>> connections to a minimum. Therefore I would prefer to only transfer the >>> needed files only once and then use them again. (Something like Björn >>> suggested). >>> >>> >>> @ Björn: the External Workspace Manager plugin sounds really >>> interesting. This would be a great thing to reduce data transfer. Sadly it >>> is not a "proven" plugin. Using it in my application is therefore not >>> possible at this moment. I will have to wait some time until it is stable. >>> >>> >>> Am Freitag, 1. Juli 2016 10:05:35 UTC+2 schrieb Björn Pedersen: >>>> >>>> Hi, >>>> >>>> Maybe https://groups.google.com/d/msg/jenkinsci-dev/nAZD0-upB68/Yw >>>> YjqmKgAwAJ can be of help for you? >>>> >>>> Björn >>>> >>>> Am Donnerstag, 30. Juni 2016 18:03:19 UTC+2 schrieb Christian Ditscher: >>>>> >>>>> Hello, >>>>> >>>>> I was wondering how exactly the stash unstash commands work. When are >>>>> the files transferred? >>>>> >>>>> - stash transfers selected files to master and unstash loads them >>>>> from master to a slave? >>>>> - stash attaches a label to the files which are transferred >>>>> between the nodes when unstashed? >>>>> - something else? >>>>> >>>>> >>>>> Also I am asking myself how I can minimize the be transferred data during >>>>> the build. >>>>> >>>>> One Idea I have is the following (example): I have 10 nodes which >>>>> should be used to build and run tests and some other nodes to do something >>>>> else. To minimize the number of files to be transferred I'd like to check >>>>> out the sources only once. >>>>> >>>>> - checkout and build on one of the 10 nodes. >>>>> - do something with the build result on another node (stash -> >>>>> unstash). >>>>> - do something with the sources already on the node from before >>>>> (without checking out again) >>>>> >>>>> If another build is started during the first is still running any of >>>>> the remaining nodes is used for the next build. >>>>> >>>>> How could I achieve that? Is this procedure useful? >>>>> >>>>> If I don't use the procedure described above I think have to stash >>>>> all the sources and then unstash when I need them again. If checked out >>>>> from source control again the sources already might already have changed. >>>>> --> Is this correct? >>>>> >>>>> >>>>> Thanks! >>>>> Chris >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/jenkinsci-users/48395472-6a7c-4fea-8860-8197e47643e9% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/jenkinsci-users/48395472-6a7c-4fea-8860-8197e47643e9%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/jenkinsci-users/0b739be7-9b74-4382-8554-eb1cd0f8bbb2%40googlegroups. > com > <https://groups.google.com/d/msgid/jenkinsci-users/0b739be7-9b74-4382-8554-eb1cd0f8bbb2%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS6bG90wcMa8ajyNeNexqGTr63OzfT-Xaw6DnfT865O7tQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
