On Wed, Nov 30, 2016 at 10:18 AM, Peter Hayes <[email protected]> wrote: > each time you run a job, you > start with a fresh container without any previously cached dependencies (we > use gradle generally). This increases the length of the build and adds > network traffic to our Artifactory instance. I looked around for existing > plugins but didn't find any so I have started a plugin[1] based on > SimpleBuildWrapper that stores a configured set of files on the master at > the end of the build and then on the next build downloads them to master in > the original location.
This seems like a poor approach; rather than overloading Artifactory, you will be overloading the Jenkins master. Archiving artifacts via the Remoting channel can already wreck performance; you are talking about potentially orders of magnitude more traffic than that. There are two basic approaches to this kind of problem. One, which assumes that the agents reuse workspaces between builds, is to set the local repository/cache location to a workspace location. The `docker-workflow` demo does this: https://github.com/jenkinsci/docker-workflow-plugin/blob/46432bbe36af17dac93cfedcc93ffa51beba1343/demo/repo/flow.groovy#L20-L22 The other approach is to mount a volume containing the cache, letting the Docker daemon handle the storage, which the `parallel-test-executor` demo does: https://github.com/jenkinsci/parallel-test-executor-plugin/blob/3961df3784045df1f6f285bc2b685ead4bc8593b/demo/Makefile#L3-L27 The volume-based approach is probably the more scalable, though there are two points to beware: at least Maven’s `install:install` will dump locally built artifacts into the repository alongside downloaded releases (probably Gradle does something similar); and Maven’s Aether repository manager is by default not thread-safe (Takari fixes this). Maven 5 may allow the cache to be properly separated (again I am not sure how Gradle fares here); in the meantime you may need to ensure that there is a distinct volume for every potentially concurrent build, for example keyed by `${JOB_NAME}/${EXECUTOR_NUMBER}`. At any rate the exact solution chosen is going to depend on details of how agents are provisioned and workspaces managed, so at root this might simply be an RFE for CJP-PSE. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3pkMzJ9MzMFhUPNRkePJCM3EeyDd3C1KrgAtXvnnZnWg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
