Hey, Thanks for all the help guys. I slept on this idea for a few days because, to be honest I really didn't want to write my own package manager <https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527#.jieuao7e5> and re-invent the wheel. I took a step back and looked at how Jenkins solved this problem for Plugin developers, and I think that we could just piggy-back on top of what they use <https://github.com/jenkinsci/gradle-jpi-plugin>.
Basically what I've done is specify the plugins I want to install in a build.gradle file on my Jenkins server. The build.gradle file lets me specify exactly what versions of the plugins I want for some, and get the latest for the rest. My install task then goes and copies just the runtime hpi files to the $JENKINS_HOME/plugins folder (after clearing out whatever is in there). After restarting my Jenkins server, all plugins are installed, with the correct versions. I've included a plugin management section in my blog post: You Don't Know Jenkins - Part 1 <http://blog.thesparktree.com/post/149039600544/you-dont-know-jenkins-part-1> which goes into more detail on how it all works, and includes an example build.gradle file. Things to note: - The plugin.lock file isn't perfect, its just a STDOUT redirect of `gradle dependencies` which is great for visually checking which versions are installed, but committing it to git gets you nothing, subsequent installs wont be locked to the same transient dependencies. I think I can solve this by using https://github.com/nebula-plugins/gradle-dependency-lock-plugin - Since the build.gradle file uses repo.jenkins-ci.org instead of updates.jenkins-ci.org it does pick up the occassional beta/alpha version that gets pushed to the releases repo by developers. I'm working to fix this using a filter in the gradle dependency solver configuration. On Thursday, August 11, 2016 at 6:03:12 PM UTC-7, Michael Kobit wrote: > > We are looking at doing something similar (actually talking about this > with colleagues today). The idea is to basically build an immutable Jenkins > instance that can't be modified. Or at least severely limit any kinds of > modifications to it so that we have an easily deployable "Jenkins as a > service". > > I've looked at possibly doing an "unpack and install" execution with the > *jenkins.war > *, but it doesn't look like an easy route. The other pain-point I see is > effectively treating the correct files as "data" that should be persisted > over time, rather than at "Jenkins build time". I am considering trying out > the Docker-type approach. I think for plugin resolution, we are probably > going to have to go the route that you are talking about for doing the > resolution ourselves. > > For security type issues, I think we could still handle it with the Docker > approach. Build whatever restrictions into the next "immutable" image and > making that deployable. Then, we can have a "staging" area and easily > rollback if we effectively control all the things we need to control. We > are experimenting with pipelines right now, and are waiting to see how > https://issues.jenkins-ci.org/browse/JENKINS-33507 will work for us to > get as much of the job configuration out of Jenkins as possible. > > We are still in the brainstorming phase, so I'm interested to see who else > has ran into this and what they have done. > > On Thursday, August 11, 2016 at 5:47:45 PM UTC-5, Jason Kulatunga wrote: >> >> Hey, >> Thanks for all the feedback :) >> >> @Daniel Beck: >> Yup, I'm familiar with the limitations of the >> https://updates.jenkins-ci.org/current/update-center.json file. Thats >> why I'm thinking of creating a plugin/dependency resolution system that >> will have to directly download the specific version of a plugin file from >> update site folder structure >> https://updates.jenkins-ci.org/download/plugins/*/ or use >> https://updates.jenkins-ci.org/latest/ >> if no version restriction is found. >> >> I wasn't aware that pinning was pointless in 2.x so that'll be an >> interesting problem to deal with. It seems that I'll have to restrict all >> access to the UpdateCenter for idea #1, or do a hybrid approach with a >> UpdateCenter subclass as well. >> >> @Baptiste Mathus >> Unfortunately just using an image with locked plugins isn't a long term >> solution, because we'll have to occasionally update our Jenkins due to >> required security updates in plugins or the main application. So being able >> to update plugins, creating a new *.lock file, test the plugin interactions >> and deploying the *.lock file to existing Jenkins servers is a requirement. >> >> I was hoping to stay away from a hybrid approach that used both an >> executable and a subclass as it makes development and deployment more >> complicated, decreasing adoption with Jenkins users. >> >> Honestly the goal was to create a tool like Bundler/Pip which would just >> work out of the box for 99% of use cases. >> >> Are there other people experiencing the same issue? I'm more than happy >> to create my own open source solution, but I'd love to base it on an >> existing (even unmaintained) project. >> >> -Jason >> >> >> On Thursday, August 11, 2016 at 4:51:07 PM UTC-4, Baptiste Mathus wrote: >>> >>> IMO a Docker image with the right set of plugins you've tested, plus the >>> security config you're talking about about forbidding any upgrade would >>> seem a simpler way. And probably it would your life simpler if you somehow >>> have to support all those different instances which can currently be >>> actually quite different. >>> >>> HTH >>> >>> Le 11 août 2016 3:14 PM, "Jason Kulatunga" <[email protected]> a >>> écrit : >>> >>> Hey Jenkins-Users, >>> >>> I manage almost a dozen Jenkins servers and our team has been having >>> some issues with plugin management: such as locking our new installations >>> to known working versions of some troublesome Jenkins plugins. >>> We use chef + Jenkins DSL to completely automate the initial >>> installation of Jenkins, but we're not exactly thrilled with the way the >>> Chef cookbook handles plugin installation and we've also verified that >>> 'installNecessaryPlugins' does not actually respect the version parameter. >>> >>> curl -XPOST >>> http://192.150.23.105:8080/pluginManager/installNecessaryPlugins -d >>> '<install plugin="[email protected]" />' >>> >>> As such I've started looking into alternative means of locking plugins >>> in an automated way during installation and I've come up with the following >>> ideas: >>> >>> # An External Dependency Management Tool, eg Bundler, Pip, Berkshelf >>> Basically an executable that would: >>> >>> 1. retrieve a list of all plugins installed in a specific Jenkins >>> server using the API, and add them to a dependency graph (with metadata: >>> installed, pinned, enabled, version) >>> 2. look for a dependency config file (like Gemfile, Berksfile, >>> requirements.txt) >>> 3. iterate through all the uninstalled plugins in the dep config >>> file and add them (and their dependencies) to the dependency graph >>> 4. solve the graph by ensuring that no pinned/locked version >>> conflicts occur. >>> 5. download all uninstalled plugins directly from >>> https://updates.jenkins-ci.org/ >>> 6. using the Jenkins api, pin any version locked plugins specified >>> in the dependency config file. >>> 7. write the solved dependency graph to the filesystem (eg >>> Berksfile.lock, Gemfile.lock) (and use it for any subsequent installs if >>> no >>> plugins have changed) >>> 8. disable all permissions to the update center in Jenkins so no >>> users enable/update plugins manually. >>> >>> # UpdateCenter Override >>> >>> 1. subclass the default Jenkins UpdateCenter, and tell Jenkins to >>> use it using a JVM property >>> 2. override the UpdateCenter.InstallationJob constructor to download >>> the plugin version specified from the dependency config lock file if it >>> exists or install like normal and then generate/update a dependency >>> config >>> lock file with every operation. >>> 3. listen to the pin event in the PluginCenter and update the >>> dependency config lock file. >>> >>> I'm not sure if anyone has done something similar but I wanted to get >>> some feedback before I spent too much time investigating either idea. >>> >>> Any and all feedback is welcome >>> >>> -Jason >>> Build Automation Engineer >>> Adobe >>> >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/2d4f0e32-7d6a-4159-9635-51df7ff83643%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/jenkinsci-users/2d4f0e32-7d6a-4159-9635-51df7ff83643%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/5381d2f3-c032-4581-a3bd-9d8ece4cc6f8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
