Re: Automated Jenkins Plugin/Dependency Management

[Daniel Beck]

> On 15.10.2016, at 20:47, Daniel Beck  wrote:
> 
> As the update center JSON tells you, its groupId is not 
> org.jenkins-ci.plugins.
> 
> https://repo.jenkins-ci.org/releases/com/sonyericsson/hudson/plugins/rebuild/rebuild/

Missed the newest response, sorry about that.


> On 14.10.2016, at 22:12, Maciej Wiczynski  wrote:
> 
> is there an easy way to find the group for all the plugins that don't use 
> 'org.jenkins-ci.plugins'? I think I can look in 
> https://updates.jenkins-ci.org/current/update-center.json and find the group 
> for each plugin. If this isn't correct way to go about it I would appreciate 
> tips on what I should be doing.

Extracting the group from the 'gav' elements is probably the best (automated) 
approach. Otherwise, repo.jenkins-ci.org has a simple search interface, look 
for the artifactId and version and it'll show you the full path.

The problem with the former, of course, is when plugins change groupId, which 
can happen occasionally (so a specific version may actually not be present in 
the groupId used for the newest version of the plugin).

It's not clear to me what kind of problem is solved here. Why not just download 
specific plugin versions from the well-known URLs on the update site? It's 
linked from every plugin wiki page…


-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/2DD569A4-AFDA-4DB0-A3F0-7585B3CD4DEB%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.

Re: Automated Jenkins Plugin/Dependency Management

[Daniel Beck]

> On 14.10.2016, at 22:03, Maciej Wiczynski  wrote:
> 
> I looked in https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ and 
> don't see rebuild plugin listed.
> I did find it in https://updates.jenkins-ci.org/current/update-center.json 
> but don't see how to set deps in build.gradle file.
> I have tried adding another maven url block under repositories block using 
> different urls, didn't work.  

As the update center JSON tells you, its groupId is not org.jenkins-ci.plugins.

https://repo.jenkins-ci.org/releases/com/sonyericsson/hudson/plugins/rebuild/rebuild/

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/34712D92-BDCF-4A32-B441-7E216FE3250A%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.

Re: Automated Jenkins Plugin/Dependency Management

[Maciej Wiczynski]

So.. figures once I posted I found a possible solution.  If I set the 
group(via cookbook attribute) I get a build.gradle that seems to work, 
snippet of rebuild dependency: 
jenkinsPlugins([
group: 'com.sonyericsson.hudson.plugins.rebuild',
name: 'rebuild',
version: 'latest.release'
  ])

So I guess my question is, is there an easy way to find the group for all 
the plugins that don't use 'org.jenkins-ci.plugins'? I think I can look 
in https://updates.jenkins-ci.org/current/update-center.json and find the 
group for each plugin. If this isn't correct way to go about it I would 
appreciate tips on what I should be doing.
Thanks.
-Maciej


On Friday, October 14, 2016 at 4:03:17 PM UTC-4, Maciej Wiczynski wrote:
>
> I found this thread via 
> http://blog.thesparktree.com/post/149039600544/you-dont-know-jenkins-part-1. 
>  I'm new to gradle/jenkins and was trying to do something similar with my 
> setup.  I'm running into problem where some plugins are not found.  I'm not 
> sure how to find or specify where plugins can be found.. this is one error 
> I'm getting about rebuild plugin as example:
>  * What went wrong:
>A problem occurred configuring root project 'jenkins'.
>> Failed to notify project evaluation listener.
>   > Could not resolve all dependencies for configuration 
> ':jenkinsPlugins'.
>  > Could not find any matches for 
> org.jenkins-ci.plugins:rebuild:latest.release as no versions of 
> org.jenkins-ci.plugins:rebuild are available.
> Searched in the following locations:
> 
> http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/rebuild/maven-metadata.xml
> 
> http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/rebuild/
> 
> https://repo1.maven.org/maven2/org/jenkins-ci/plugins/rebuild/maven-metadata.xml
> https://repo1.maven.org/maven2/org/jenkins-ci/plugins/rebuild/
> 
> file:/var/lib/jenkins/.m2/repository/org/jenkins-ci/plugins/rebuild/maven-metadata.xml
> 
> file:/var/lib/jenkins/.m2/repository/org/jenkins-ci/plugins/rebuild/
> 
> https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/rebuild/maven-metadata.xml
> 
> https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/rebuild/
> Required by:
> :jenkins:unspecified
>   > Could not resolve all dependencies for configuration 
> ':jenkinsPlugins'.
>  > Could not find any matches for 
> org.jenkins-ci.plugins:rebuild:latest.release as no versions of 
> org.jenkins-ci.plugins:rebuild are available.
> Searched in the following locations:
> 
> http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/rebuild/maven-metadata.xml
> 
> http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/rebuild/
> 
> https://repo1.maven.org/maven2/org/jenkins-ci/plugins/rebuild/maven-metadata.xml
> https://repo1.maven.org/maven2/org/jenkins-ci/plugins/rebuild/
> 
> file:/var/lib/jenkins/.m2/repository/org/jenkins-ci/plugins/rebuild/maven-metadata.xml
> 
> file:/var/lib/jenkins/.m2/repository/org/jenkins-ci/plugins/rebuild/
> 
> https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/rebuild/maven-metadata.xml
> 
> https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/rebuild/
>
> I looked in https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ and 
> don't see rebuild plugin listed.
> I did find it in https://updates.jenkins-ci.org/current/update-center.json 
> but don't see how to set deps in build.gradle file.
> I have tried adding another maven url block under repositories block using 
> different urls, didn't work.  
>
> Thanks.
> -Maciej
>
> On Tuesday, August 30, 2016 at 12:12:27 PM UTC-4, Michael Kobit wrote:
>>
>> My fault explaining that
>>
>> I know that I can extract it using the *jar* command, but I don't get 
>> the same "unzipped" output as running *java -jar jenkins.war*.
>>
>> It looks like this happens because of some magic in 
>> https://github.com/jenkinsci/extras-executable-war that handles the 
>> unpacking and bootstrapping.
>>
>> I'm wondering, is there a similar way to unpack the *jenkins.war* without 
>> actually running the service, so that I can then programmatically configure 
>> the JENKINS_HOME.
>>
>> This might be the wrong approach or the totally wrong idea. I was 
>> probably going to move in the same direction that you said with using the 
>> https://github.com/jenkinsci/gradle-jpi-plugin to handle plugin 
>> dependency resolution, but plugins are not the only thing I want to 
>> configure. The Groovy init.d type scripts work, but it requires Jenkins to 
>> hit a certain lifecycle stage to run.
>>
>> Jenkins just doesn't seem to lend itself well to configuration as code, 
>> but maybe I'm missing something. 
>>
>> On Saturday, August 27, 2016 at 12:43:51 PM UTC-5, Jason Kulatunga 

Re: Automated Jenkins Plugin/Dependency Management

[Maciej Wiczynski]

I found this thread via 
http://blog.thesparktree.com/post/149039600544/you-dont-know-jenkins-part-1. 
 I'm new to gradle/jenkins and was trying to do something similar with my 
setup.  I'm running into problem where some plugins are not found.  I'm not 
sure how to find or specify where plugins can be found.. this is one error 
I'm getting about rebuild plugin as example:
 * What went wrong:
   A problem occurred configuring root project 'jenkins'.
   > Failed to notify project evaluation listener.
  > Could not resolve all dependencies for configuration 
':jenkinsPlugins'.
 > Could not find any matches for 
org.jenkins-ci.plugins:rebuild:latest.release as no versions of 
org.jenkins-ci.plugins:rebuild are available.
Searched in the following locations:

http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/rebuild/maven-metadata.xml

http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/rebuild/

https://repo1.maven.org/maven2/org/jenkins-ci/plugins/rebuild/maven-metadata.xml
https://repo1.maven.org/maven2/org/jenkins-ci/plugins/rebuild/

file:/var/lib/jenkins/.m2/repository/org/jenkins-ci/plugins/rebuild/maven-metadata.xml

file:/var/lib/jenkins/.m2/repository/org/jenkins-ci/plugins/rebuild/

https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/rebuild/maven-metadata.xml

https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/rebuild/
Required by:
:jenkins:unspecified
  > Could not resolve all dependencies for configuration 
':jenkinsPlugins'.
 > Could not find any matches for 
org.jenkins-ci.plugins:rebuild:latest.release as no versions of 
org.jenkins-ci.plugins:rebuild are available.
Searched in the following locations:

http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/rebuild/maven-metadata.xml

http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/rebuild/

https://repo1.maven.org/maven2/org/jenkins-ci/plugins/rebuild/maven-metadata.xml
https://repo1.maven.org/maven2/org/jenkins-ci/plugins/rebuild/

file:/var/lib/jenkins/.m2/repository/org/jenkins-ci/plugins/rebuild/maven-metadata.xml

file:/var/lib/jenkins/.m2/repository/org/jenkins-ci/plugins/rebuild/

https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/rebuild/maven-metadata.xml

https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/rebuild/

I looked in 
https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ and don't see 
rebuild plugin listed.
I did find it in https://updates.jenkins-ci.org/current/update-center.json 
but don't see how to set deps in build.gradle file.
I have tried adding another maven url block under repositories block using 
different urls, didn't work.  

Thanks.
-Maciej

On Tuesday, August 30, 2016 at 12:12:27 PM UTC-4, Michael Kobit wrote:
>
> My fault explaining that
>
> I know that I can extract it using the *jar* command, but I don't get the 
> same "unzipped" output as running *java -jar jenkins.war*.
>
> It looks like this happens because of some magic in 
> https://github.com/jenkinsci/extras-executable-war that handles the 
> unpacking and bootstrapping.
>
> I'm wondering, is there a similar way to unpack the *jenkins.war* without 
> actually running the service, so that I can then programmatically configure 
> the JENKINS_HOME.
>
> This might be the wrong approach or the totally wrong idea. I was probably 
> going to move in the same direction that you said with using the 
> https://github.com/jenkinsci/gradle-jpi-plugin to handle plugin 
> dependency resolution, but plugins are not the only thing I want to 
> configure. The Groovy init.d type scripts work, but it requires Jenkins to 
> hit a certain lifecycle stage to run.
>
> Jenkins just doesn't seem to lend itself well to configuration as code, 
> but maybe I'm missing something. 
>
> On Saturday, August 27, 2016 at 12:43:51 PM UTC-5, Jason Kulatunga wrote:
>>
>> Yep, the command is `jar xvf jenkins.war`, that will explode the war into 
>> the current directory. 
>>
>> On Friday, August 26, 2016 at 12:27:00 PM UTC-7, Michael Kobit wrote:
>>>
>>> Is there a way to basically "unzip" the *jenkins.war* so that the 
>>> plugins, workflow-libs, and other parts can be configured before actually 
>>> running the service?
>>>
>>> On Wednesday, August 17, 2016 at 9:36:08 AM UTC-5, Jason Kulatunga wrote:

 Hey,
 Thanks for all the help guys.
 I slept on this idea for a few days because, to be honest I really 
 didn't want to write my own package manager 
 
  and 
 re-invent the wheel. I took a step back and looked at how Jenkins solved 
 this problem for Plugin developers, and I think that we could just 
 piggy-back on 

Re: Automated Jenkins Plugin/Dependency Management

[Michael Kobit]

My fault explaining that

I know that I can extract it using the *jar* command, but I don't get the 
same "unzipped" output as running *java -jar jenkins.war*.

It looks like this happens because of some magic 
in https://github.com/jenkinsci/extras-executable-war that handles the 
unpacking and bootstrapping.

I'm wondering, is there a similar way to unpack the *jenkins.war* without 
actually running the service, so that I can then programmatically configure 
the JENKINS_HOME.

This might be the wrong approach or the totally wrong idea. I was probably 
going to move in the same direction that you said with using the 
https://github.com/jenkinsci/gradle-jpi-plugin to handle plugin dependency 
resolution, but plugins are not the only thing I want to configure. The 
Groovy init.d type scripts work, but it requires Jenkins to hit a certain 
lifecycle stage to run.

Jenkins just doesn't seem to lend itself well to configuration as code, but 
maybe I'm missing something. 

On Saturday, August 27, 2016 at 12:43:51 PM UTC-5, Jason Kulatunga wrote:
>
> Yep, the command is `jar xvf jenkins.war`, that will explode the war into 
> the current directory. 
>
> On Friday, August 26, 2016 at 12:27:00 PM UTC-7, Michael Kobit wrote:
>>
>> Is there a way to basically "unzip" the *jenkins.war* so that the 
>> plugins, workflow-libs, and other parts can be configured before actually 
>> running the service?
>>
>> On Wednesday, August 17, 2016 at 9:36:08 AM UTC-5, Jason Kulatunga wrote:
>>>
>>> Hey,
>>> Thanks for all the help guys.
>>> I slept on this idea for a few days because, to be honest I really 
>>> didn't want to write my own package manager 
>>> 
>>>  and 
>>> re-invent the wheel. I took a step back and looked at how Jenkins solved 
>>> this problem for Plugin developers, and I think that we could just 
>>> piggy-back on top of what they use 
>>> .
>>>
>>> Basically what I've done is specify the plugins I want to install in a 
>>> build.gradle file on my Jenkins server. The build.gradle file lets me 
>>> specify exactly what versions of the plugins I want for some, and get the 
>>> latest for the rest. My install task then goes and copies just the runtime 
>>> hpi files to the $JENKINS_HOME/plugins folder (after clearing out whatever 
>>> is in there). After restarting my Jenkins server, all plugins are 
>>> installed, with the correct versions.
>>>
>>> I've included a plugin management section in my blog post: You Don't 
>>> Know Jenkins - Part 1 
>>> 
>>>  which 
>>> goes into more detail on how it all works, and includes an example 
>>> build.gradle file. 
>>>
>>> Things to note:
>>> - The plugin.lock file isn't perfect, its just a STDOUT redirect of 
>>> `gradle dependencies` which is great for visually checking which versions 
>>> are installed, but committing it to git gets you nothing, subsequent 
>>> installs wont be locked to the same transient dependencies. I think I can 
>>> solve this by using 
>>> https://github.com/nebula-plugins/gradle-dependency-lock-plugin
>>> - Since the build.gradle file uses repo.jenkins-ci.org instead of 
>>> updates.jenkins-ci.org it does pick up the occassional beta/alpha 
>>> version that gets pushed to the releases repo by developers. I'm working to 
>>> fix this using a filter in the gradle dependency solver configuration. 
>>>
>>>
>>>
>>> On Thursday, August 11, 2016 at 6:03:12 PM UTC-7, Michael Kobit wrote:

 We are looking at doing something similar (actually talking about this 
 with colleagues today). The idea is to basically build an immutable 
 Jenkins 
 instance that can't be modified. Or at least severely limit any kinds of 
 modifications to it so that we have an easily deployable "Jenkins as a 
 service".

 I've looked at possibly doing an "unpack and install" execution with 
 the *jenkins.war *, but it doesn't look like an easy route. The other 
 pain-point I see is effectively treating the correct files as "data" that 
 should be persisted over time, rather than at "Jenkins build time". I am 
 considering trying out the Docker-type approach. I think for plugin 
 resolution, we are probably going to have to go the route that you are 
 talking about for doing the resolution ourselves.

 For security type issues, I think we could still handle it with the 
 Docker approach. Build whatever restrictions into the next "immutable" 
 image and making that deployable. Then, we can have a "staging" area and 
 easily rollback if we effectively control all the things we need to 
 control. We are experimenting with pipelines right now, and are waiting to 
 see how https://issues.jenkins-ci.org/browse/JENKINS-33507 will work 
 for us to get as much of the job configuration 

Re: Automated Jenkins Plugin/Dependency Management

[Jason Kulatunga]

Yep, the command is `jar xvf jenkins.war`, that will explode the war into 
the current directory. 

On Friday, August 26, 2016 at 12:27:00 PM UTC-7, Michael Kobit wrote:
>
> Is there a way to basically "unzip" the *jenkins.war* so that the 
> plugins, workflow-libs, and other parts can be configured before actually 
> running the service?
>
> On Wednesday, August 17, 2016 at 9:36:08 AM UTC-5, Jason Kulatunga wrote:
>>
>> Hey,
>> Thanks for all the help guys.
>> I slept on this idea for a few days because, to be honest I really didn't 
>> want to write my own package manager 
>> 
>>  and 
>> re-invent the wheel. I took a step back and looked at how Jenkins solved 
>> this problem for Plugin developers, and I think that we could just 
>> piggy-back on top of what they use 
>> .
>>
>> Basically what I've done is specify the plugins I want to install in a 
>> build.gradle file on my Jenkins server. The build.gradle file lets me 
>> specify exactly what versions of the plugins I want for some, and get the 
>> latest for the rest. My install task then goes and copies just the runtime 
>> hpi files to the $JENKINS_HOME/plugins folder (after clearing out whatever 
>> is in there). After restarting my Jenkins server, all plugins are 
>> installed, with the correct versions.
>>
>> I've included a plugin management section in my blog post: You Don't 
>> Know Jenkins - Part 1 
>> 
>>  which 
>> goes into more detail on how it all works, and includes an example 
>> build.gradle file. 
>>
>> Things to note:
>> - The plugin.lock file isn't perfect, its just a STDOUT redirect of 
>> `gradle dependencies` which is great for visually checking which versions 
>> are installed, but committing it to git gets you nothing, subsequent 
>> installs wont be locked to the same transient dependencies. I think I can 
>> solve this by using 
>> https://github.com/nebula-plugins/gradle-dependency-lock-plugin
>> - Since the build.gradle file uses repo.jenkins-ci.org instead of 
>> updates.jenkins-ci.org it does pick up the occassional beta/alpha 
>> version that gets pushed to the releases repo by developers. I'm working to 
>> fix this using a filter in the gradle dependency solver configuration. 
>>
>>
>>
>> On Thursday, August 11, 2016 at 6:03:12 PM UTC-7, Michael Kobit wrote:
>>>
>>> We are looking at doing something similar (actually talking about this 
>>> with colleagues today). The idea is to basically build an immutable Jenkins 
>>> instance that can't be modified. Or at least severely limit any kinds of 
>>> modifications to it so that we have an easily deployable "Jenkins as a 
>>> service".
>>>
>>> I've looked at possibly doing an "unpack and install" execution with the 
>>> *jenkins.war 
>>> *, but it doesn't look like an easy route. The other pain-point I see 
>>> is effectively treating the correct files as "data" that should be 
>>> persisted over time, rather than at "Jenkins build time". I am considering 
>>> trying out the Docker-type approach. I think for plugin resolution, we are 
>>> probably going to have to go the route that you are talking about for doing 
>>> the resolution ourselves.
>>>
>>> For security type issues, I think we could still handle it with the 
>>> Docker approach. Build whatever restrictions into the next "immutable" 
>>> image and making that deployable. Then, we can have a "staging" area and 
>>> easily rollback if we effectively control all the things we need to 
>>> control. We are experimenting with pipelines right now, and are waiting to 
>>> see how https://issues.jenkins-ci.org/browse/JENKINS-33507 will work 
>>> for us to get as much of the job configuration out of Jenkins as possible.
>>>
>>> We are still in the brainstorming phase, so I'm interested to see who 
>>> else has ran into this and what they have done.
>>>
>>> On Thursday, August 11, 2016 at 5:47:45 PM UTC-5, Jason Kulatunga wrote:

 Hey,
 Thanks for all the feedback :)

 @Daniel Beck:
 Yup, I'm familiar with the limitations of the 
 https://updates.jenkins-ci.org/current/update-center.json file. Thats 
 why I'm thinking of creating a plugin/dependency resolution system that 
 will have to directly download the specific version of a plugin file from 
 update site folder structure 
 https://updates.jenkins-ci.org/download/plugins/*/ or use 
 https://updates.jenkins-ci.org/latest/ 
 if no version restriction is found.

 I wasn't aware that pinning was pointless in 2.x so that'll be an 
 interesting problem to deal with. It seems that I'll have to restrict all 
 access to the UpdateCenter for idea #1, or do a hybrid approach with a 
 UpdateCenter subclass as well.

 @Baptiste Mathus 
 Unfortunately just using an image with locked 

Re: Automated Jenkins Plugin/Dependency Management

[Michael Kobit]

Is there a way to basically "unzip" the *jenkins.war* so that the plugins, 
workflow-libs, and other parts can be configured before actually running 
the service?

On Wednesday, August 17, 2016 at 9:36:08 AM UTC-5, Jason Kulatunga wrote:
>
> Hey,
> Thanks for all the help guys.
> I slept on this idea for a few days because, to be honest I really didn't 
> want to write my own package manager 
> 
>  and 
> re-invent the wheel. I took a step back and looked at how Jenkins solved 
> this problem for Plugin developers, and I think that we could just 
> piggy-back on top of what they use 
> .
>
> Basically what I've done is specify the plugins I want to install in a 
> build.gradle file on my Jenkins server. The build.gradle file lets me 
> specify exactly what versions of the plugins I want for some, and get the 
> latest for the rest. My install task then goes and copies just the runtime 
> hpi files to the $JENKINS_HOME/plugins folder (after clearing out whatever 
> is in there). After restarting my Jenkins server, all plugins are 
> installed, with the correct versions.
>
> I've included a plugin management section in my blog post: You Don't Know 
> Jenkins - Part 1 
>  
> which 
> goes into more detail on how it all works, and includes an example 
> build.gradle file. 
>
> Things to note:
> - The plugin.lock file isn't perfect, its just a STDOUT redirect of 
> `gradle dependencies` which is great for visually checking which versions 
> are installed, but committing it to git gets you nothing, subsequent 
> installs wont be locked to the same transient dependencies. I think I can 
> solve this by using 
> https://github.com/nebula-plugins/gradle-dependency-lock-plugin
> - Since the build.gradle file uses repo.jenkins-ci.org instead of 
> updates.jenkins-ci.org it does pick up the occassional beta/alpha version 
> that gets pushed to the releases repo by developers. I'm working to fix 
> this using a filter in the gradle dependency solver configuration. 
>
>
>
> On Thursday, August 11, 2016 at 6:03:12 PM UTC-7, Michael Kobit wrote:
>>
>> We are looking at doing something similar (actually talking about this 
>> with colleagues today). The idea is to basically build an immutable Jenkins 
>> instance that can't be modified. Or at least severely limit any kinds of 
>> modifications to it so that we have an easily deployable "Jenkins as a 
>> service".
>>
>> I've looked at possibly doing an "unpack and install" execution with the 
>> *jenkins.war 
>> *, but it doesn't look like an easy route. The other pain-point I see is 
>> effectively treating the correct files as "data" that should be persisted 
>> over time, rather than at "Jenkins build time". I am considering trying out 
>> the Docker-type approach. I think for plugin resolution, we are probably 
>> going to have to go the route that you are talking about for doing the 
>> resolution ourselves.
>>
>> For security type issues, I think we could still handle it with the 
>> Docker approach. Build whatever restrictions into the next "immutable" 
>> image and making that deployable. Then, we can have a "staging" area and 
>> easily rollback if we effectively control all the things we need to 
>> control. We are experimenting with pipelines right now, and are waiting to 
>> see how https://issues.jenkins-ci.org/browse/JENKINS-33507 will work for 
>> us to get as much of the job configuration out of Jenkins as possible.
>>
>> We are still in the brainstorming phase, so I'm interested to see who 
>> else has ran into this and what they have done.
>>
>> On Thursday, August 11, 2016 at 5:47:45 PM UTC-5, Jason Kulatunga wrote:
>>>
>>> Hey,
>>> Thanks for all the feedback :)
>>>
>>> @Daniel Beck:
>>> Yup, I'm familiar with the limitations of the 
>>> https://updates.jenkins-ci.org/current/update-center.json file. Thats 
>>> why I'm thinking of creating a plugin/dependency resolution system that 
>>> will have to directly download the specific version of a plugin file from 
>>> update site folder structure 
>>> https://updates.jenkins-ci.org/download/plugins/*/ or use 
>>> https://updates.jenkins-ci.org/latest/ 
>>> if no version restriction is found.
>>>
>>> I wasn't aware that pinning was pointless in 2.x so that'll be an 
>>> interesting problem to deal with. It seems that I'll have to restrict all 
>>> access to the UpdateCenter for idea #1, or do a hybrid approach with a 
>>> UpdateCenter subclass as well.
>>>
>>> @Baptiste Mathus 
>>> Unfortunately just using an image with locked plugins isn't a long term 
>>> solution, because we'll have to occasionally update our Jenkins due to 
>>> required security updates in plugins or the main application. So being able 
>>> to update plugins, creating a new *.lock file, test the plugin interactions 
>>> and 

Re: Automated Jenkins Plugin/Dependency Management

[Jason Kulatunga]

Hey,
Thanks for all the help guys.
I slept on this idea for a few days because, to be honest I really didn't 
want to write my own package manager 

 and 
re-invent the wheel. I took a step back and looked at how Jenkins solved 
this problem for Plugin developers, and I think that we could just 
piggy-back on top of what they use 
.

Basically what I've done is specify the plugins I want to install in a 
build.gradle file on my Jenkins server. The build.gradle file lets me 
specify exactly what versions of the plugins I want for some, and get the 
latest for the rest. My install task then goes and copies just the runtime 
hpi files to the $JENKINS_HOME/plugins folder (after clearing out whatever 
is in there). After restarting my Jenkins server, all plugins are 
installed, with the correct versions.

I've included a plugin management section in my blog post: You Don't Know 
Jenkins - Part 1 
 
which 
goes into more detail on how it all works, and includes an example 
build.gradle file. 

Things to note:
- The plugin.lock file isn't perfect, its just a STDOUT redirect of `gradle 
dependencies` which is great for visually checking which versions are 
installed, but committing it to git gets you nothing, subsequent installs 
wont be locked to the same transient dependencies. I think I can solve this 
by using https://github.com/nebula-plugins/gradle-dependency-lock-plugin
- Since the build.gradle file uses repo.jenkins-ci.org instead 
of updates.jenkins-ci.org it does pick up the occassional beta/alpha 
version that gets pushed to the releases repo by developers. I'm working to 
fix this using a filter in the gradle dependency solver configuration. 



On Thursday, August 11, 2016 at 6:03:12 PM UTC-7, Michael Kobit wrote:
>
> We are looking at doing something similar (actually talking about this 
> with colleagues today). The idea is to basically build an immutable Jenkins 
> instance that can't be modified. Or at least severely limit any kinds of 
> modifications to it so that we have an easily deployable "Jenkins as a 
> service".
>
> I've looked at possibly doing an "unpack and install" execution with the 
> *jenkins.war 
> *, but it doesn't look like an easy route. The other pain-point I see is 
> effectively treating the correct files as "data" that should be persisted 
> over time, rather than at "Jenkins build time". I am considering trying out 
> the Docker-type approach. I think for plugin resolution, we are probably 
> going to have to go the route that you are talking about for doing the 
> resolution ourselves.
>
> For security type issues, I think we could still handle it with the Docker 
> approach. Build whatever restrictions into the next "immutable" image and 
> making that deployable. Then, we can have a "staging" area and easily 
> rollback if we effectively control all the things we need to control. We 
> are experimenting with pipelines right now, and are waiting to see how 
> https://issues.jenkins-ci.org/browse/JENKINS-33507 will work for us to 
> get as much of the job configuration out of Jenkins as possible.
>
> We are still in the brainstorming phase, so I'm interested to see who else 
> has ran into this and what they have done.
>
> On Thursday, August 11, 2016 at 5:47:45 PM UTC-5, Jason Kulatunga wrote:
>>
>> Hey,
>> Thanks for all the feedback :)
>>
>> @Daniel Beck:
>> Yup, I'm familiar with the limitations of the 
>> https://updates.jenkins-ci.org/current/update-center.json file. Thats 
>> why I'm thinking of creating a plugin/dependency resolution system that 
>> will have to directly download the specific version of a plugin file from 
>> update site folder structure 
>> https://updates.jenkins-ci.org/download/plugins/*/ or use 
>> https://updates.jenkins-ci.org/latest/ 
>> if no version restriction is found.
>>
>> I wasn't aware that pinning was pointless in 2.x so that'll be an 
>> interesting problem to deal with. It seems that I'll have to restrict all 
>> access to the UpdateCenter for idea #1, or do a hybrid approach with a 
>> UpdateCenter subclass as well.
>>
>> @Baptiste Mathus 
>> Unfortunately just using an image with locked plugins isn't a long term 
>> solution, because we'll have to occasionally update our Jenkins due to 
>> required security updates in plugins or the main application. So being able 
>> to update plugins, creating a new *.lock file, test the plugin interactions 
>> and deploying the *.lock file to existing Jenkins servers is a requirement. 
>>
>> I was hoping to stay away from a hybrid approach that used both an 
>> executable and a subclass as it makes development and deployment more 
>> complicated, decreasing adoption with Jenkins users. 
>>
>> Honestly the goal was to create a tool like Bundler/Pip which would just 
>> work out 

Re: Automated Jenkins Plugin/Dependency Management

[Michael Kobit]

We are looking at doing something similar (actually talking about this with 
colleagues today). The idea is to basically build an immutable Jenkins 
instance that can't be modified. Or at least severely limit any kinds of 
modifications to it so that we have an easily deployable "Jenkins as a 
service".

I've looked at possibly doing an "unpack and install" execution with the 
*jenkins.war 
*, but it doesn't look like an easy route. The other pain-point I see is 
effectively treating the correct files as "data" that should be persisted 
over time, rather than at "Jenkins build time". I am considering trying out 
the Docker-type approach. I think for plugin resolution, we are probably 
going to have to go the route that you are talking about for doing the 
resolution ourselves.

For security type issues, I think we could still handle it with the Docker 
approach. Build whatever restrictions into the next "immutable" image and 
making that deployable. Then, we can have a "staging" area and easily 
rollback if we effectively control all the things we need to control. We 
are experimenting with pipelines right now, and are waiting to see 
how https://issues.jenkins-ci.org/browse/JENKINS-33507 will work for us to 
get as much of the job configuration out of Jenkins as possible.

We are still in the brainstorming phase, so I'm interested to see who else 
has ran into this and what they have done.

On Thursday, August 11, 2016 at 5:47:45 PM UTC-5, Jason Kulatunga wrote:
>
> Hey,
> Thanks for all the feedback :)
>
> @Daniel Beck:
> Yup, I'm familiar with the limitations of the 
> https://updates.jenkins-ci.org/current/update-center.json file. Thats why 
> I'm thinking of creating a plugin/dependency resolution system that will 
> have to directly download the specific version of a plugin file from update 
> site folder structure https://updates.jenkins-ci.org/download/plugins/*/ or 
> use https://updates.jenkins-ci.org/latest/ if no version restriction is 
> found.
>
> I wasn't aware that pinning was pointless in 2.x so that'll be an 
> interesting problem to deal with. It seems that I'll have to restrict all 
> access to the UpdateCenter for idea #1, or do a hybrid approach with a 
> UpdateCenter subclass as well.
>
> @Baptiste Mathus 
> Unfortunately just using an image with locked plugins isn't a long term 
> solution, because we'll have to occasionally update our Jenkins due to 
> required security updates in plugins or the main application. So being able 
> to update plugins, creating a new *.lock file, test the plugin interactions 
> and deploying the *.lock file to existing Jenkins servers is a requirement. 
>
> I was hoping to stay away from a hybrid approach that used both an 
> executable and a subclass as it makes development and deployment more 
> complicated, decreasing adoption with Jenkins users. 
>
> Honestly the goal was to create a tool like Bundler/Pip which would just 
> work out of the box for 99% of use cases. 
>
> Are there other people experiencing the same issue? I'm more than happy to 
> create my own open source solution, but I'd love to base it on an existing 
> (even unmaintained) project. 
>
> -Jason
>
>
> On Thursday, August 11, 2016 at 4:51:07 PM UTC-4, Baptiste Mathus wrote:
>>
>> IMO a Docker image with the right set of plugins you've tested, plus the 
>> security config you're talking about about forbidding any upgrade would 
>> seem a simpler way. And probably it would your life simpler if you somehow 
>> have to support all those different instances which can currently be 
>> actually quite different.
>>
>> HTH
>>
>> Le 11 août 2016 3:14 PM, "Jason Kulatunga"  a écrit :
>>
>> Hey Jenkins-Users,
>>
>> I manage almost a dozen Jenkins servers and our team has been having some 
>> issues with plugin management: such as locking our new installations to 
>> known working versions of some troublesome Jenkins plugins.
>> We use chef + Jenkins DSL to completely automate the initial installation 
>> of Jenkins, but we're not exactly thrilled with the way the Chef cookbook 
>> handles plugin installation and we've also verified that 
>> 'installNecessaryPlugins' does not actually respect the version parameter. 
>>
>> curl -XPOST 
>> http://192.150.23.105:8080/pluginManager/installNecessaryPlugins -d 
>> ''
>>
>> As such I've started looking into alternative means of locking plugins in 
>> an automated way during installation and I've come up with the following 
>> ideas:
>>
>> # An External Dependency Management Tool, eg Bundler, Pip, Berkshelf
>> Basically an executable that would:
>>
>>1. retrieve a list of all plugins installed in a specific Jenkins 
>>server using the API, and add them to a dependency graph (with metadata: 
>>installed, pinned, enabled, version)
>>2. look for a dependency config file (like Gemfile, Berksfile, 
>>requirements.txt)
>>3. iterate through all the uninstalled plugins in the dep config file 
>>and add them 

Re: Automated Jenkins Plugin/Dependency Management

[Jason Kulatunga]

Hey,
Thanks for all the feedback :)

@Daniel Beck:
Yup, I'm familiar with the limitations of the 
https://updates.jenkins-ci.org/current/update-center.json file. Thats why 
I'm thinking of creating a plugin/dependency resolution system that will 
have to directly download the specific version of a plugin file from update 
site folder structure https://updates.jenkins-ci.org/download/plugins/*/ or 
use https://updates.jenkins-ci.org/latest/ if no version restriction is 
found.

I wasn't aware that pinning was pointless in 2.x so that'll be an 
interesting problem to deal with. It seems that I'll have to restrict all 
access to the UpdateCenter for idea #1, or do a hybrid approach with a 
UpdateCenter subclass as well.

@Baptiste Mathus 
Unfortunately just using an image with locked plugins isn't a long term 
solution, because we'll have to occasionally update our Jenkins due to 
required security updates in plugins or the main application. So being able 
to update plugins, creating a new *.lock file, test the plugin interactions 
and deploying the *.lock file to existing Jenkins servers is a requirement. 

I was hoping to stay away from a hybrid approach that used both an 
executable and a subclass as it makes development and deployment more 
complicated, decreasing adoption with Jenkins users. 

Honestly the goal was to create a tool like Bundler/Pip which would just 
work out of the box for 99% of use cases. 

Are there other people experiencing the same issue? I'm more than happy to 
create my own open source solution, but I'd love to base it on an existing 
(even unmaintained) project. 

-Jason


On Thursday, August 11, 2016 at 4:51:07 PM UTC-4, Baptiste Mathus wrote:
>
> IMO a Docker image with the right set of plugins you've tested, plus the 
> security config you're talking about about forbidding any upgrade would 
> seem a simpler way. And probably it would your life simpler if you somehow 
> have to support all those different instances which can currently be 
> actually quite different.
>
> HTH
>
> Le 11 août 2016 3:14 PM, "Jason Kulatunga"  > a écrit :
>
> Hey Jenkins-Users,
>
> I manage almost a dozen Jenkins servers and our team has been having some 
> issues with plugin management: such as locking our new installations to 
> known working versions of some troublesome Jenkins plugins.
> We use chef + Jenkins DSL to completely automate the initial installation 
> of Jenkins, but we're not exactly thrilled with the way the Chef cookbook 
> handles plugin installation and we've also verified that 
> 'installNecessaryPlugins' does not actually respect the version parameter. 
>
> curl -XPOST 
> http://192.150.23.105:8080/pluginManager/installNecessaryPlugins -d 
> ''
>
> As such I've started looking into alternative means of locking plugins in 
> an automated way during installation and I've come up with the following 
> ideas:
>
> # An External Dependency Management Tool, eg Bundler, Pip, Berkshelf
> Basically an executable that would:
>
>1. retrieve a list of all plugins installed in a specific Jenkins 
>server using the API, and add them to a dependency graph (with metadata: 
>installed, pinned, enabled, version)
>2. look for a dependency config file (like Gemfile, Berksfile, 
>requirements.txt)
>3. iterate through all the uninstalled plugins in the dep config file 
>and add them (and their dependencies) to the dependency graph
>4. solve the graph by ensuring that no pinned/locked version conflicts 
>occur. 
>5. download all uninstalled plugins directly from 
>https://updates.jenkins-ci.org/
>6. using the Jenkins api, pin any version locked plugins specified in 
>the dependency config file. 
>7. write the solved dependency graph to the filesystem (eg 
>Berksfile.lock, Gemfile.lock) (and use it for any subsequent installs if 
> no 
>plugins have changed)
>8. disable all permissions to the update center in Jenkins so no users 
>enable/update plugins manually. 
>
> # UpdateCenter Override
>
>1. subclass the default Jenkins UpdateCenter, and tell Jenkins to use 
>it using a JVM property
>2. override the UpdateCenter.InstallationJob constructor to download 
>the plugin version specified from the  dependency config lock file if it 
>exists or install like normal and then generate/update a dependency config 
>lock file with every operation.
>3. listen to the pin event in the PluginCenter and update the 
>dependency config lock file. 
>
> I'm not sure if anyone has done something similar but I wanted to get some 
> feedback before I spent too much time investigating either idea. 
>
> Any and all feedback is welcome
>
> -Jason
> Build Automation Engineer
> Adobe
>
>
>
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to 

Re: Automated Jenkins Plugin/Dependency Management

[Baptiste Mathus]

IMO a Docker image with the right set of plugins you've tested, plus the
security config you're talking about about forbidding any upgrade would
seem a simpler way. And probably it would your life simpler if you somehow
have to support all those different instances which can currently be
actually quite different.

HTH

Le 11 août 2016 3:14 PM, "Jason Kulatunga"  a écrit :

Hey Jenkins-Users,

I manage almost a dozen Jenkins servers and our team has been having some
issues with plugin management: such as locking our new installations to
known working versions of some troublesome Jenkins plugins.
We use chef + Jenkins DSL to completely automate the initial installation
of Jenkins, but we're not exactly thrilled with the way the Chef cookbook
handles plugin installation and we've also verified that
'installNecessaryPlugins' does not actually respect the version parameter.

curl -XPOST http://192.150.23.105:8080/pluginManager/installNecessaryPlugins
-d ''

As such I've started looking into alternative means of locking plugins in
an automated way during installation and I've come up with the following
ideas:

# An External Dependency Management Tool, eg Bundler, Pip, Berkshelf
Basically an executable that would:

   1. retrieve a list of all plugins installed in a specific Jenkins server
   using the API, and add them to a dependency graph (with metadata:
   installed, pinned, enabled, version)
   2. look for a dependency config file (like Gemfile, Berksfile,
   requirements.txt)
   3. iterate through all the uninstalled plugins in the dep config file
   and add them (and their dependencies) to the dependency graph
   4. solve the graph by ensuring that no pinned/locked version conflicts
   occur.
   5. download all uninstalled plugins directly from
   https://updates.jenkins-ci.org/ 
   6. using the Jenkins api, pin any version locked plugins specified in
   the dependency config file.
   7. write the solved dependency graph to the filesystem (eg
   Berksfile.lock, Gemfile.lock) (and use it for any subsequent installs if no
   plugins have changed)
   8. disable all permissions to the update center in Jenkins so no users
   enable/update plugins manually.

# UpdateCenter Override

   1. subclass the default Jenkins UpdateCenter, and tell Jenkins to use it
   using a JVM property
   2. override the UpdateCenter.InstallationJob constructor to download the
   plugin version specified from the  dependency config lock file if it exists
   or install like normal and then generate/update a dependency config lock
   file with every operation.
   3. listen to the pin event in the PluginCenter and update the dependency
   config lock file.

I'm not sure if anyone has done something similar but I wanted to get some
feedback before I spent too much time investigating either idea.

Any and all feedback is welcome

-Jason
Build Automation Engineer
Adobe




-- 
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/jenkinsci-users/2d4f0e32-7d6a-4159-9635-51df7ff83643%40googlegroups.
com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS54gzCcyVtMd1z4UXGmdeLoNccqJm3G4tU3pOmOKWALQA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Automated Jenkins Plugin/Dependency Management

[Slide]

Kind of off topic, but it would really be nice to be able to "pin" a plugin
to a specific version that won't get upgraded.

On Thu, Aug 11, 2016 at 11:08 AM Daniel Beck  wrote:

>
> > On 11.08.2016, at 15:14, Jason Kulatunga  wrote:
> >
> > I'm not sure if anyone has done something similar but I wanted to get
> some feedback before I spent too much time investigating either idea.
>
> Note that pinning doesn't prevent a plugin from being upgraded -- it's
> related to Jenkins 1.x plugin bundling, and means that a plugin that's also
> bundled (e.g. subversion) won't get overwritten by the version bundled with
> Jenkins on startup. In Jenkins 2, it's basically irrelevant.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/3C393994-86E4-4048-9D99-2A4A258706BE%40beckweb.net
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CAPiUgVdCMKhe0ka_j1OMOU9XxkaXBVihJDhvjg207BgoQ%2BtBEQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Automated Jenkins Plugin/Dependency Management

[Daniel Beck]

> On 11.08.2016, at 15:14, Jason Kulatunga  wrote:
> 
> I'm not sure if anyone has done something similar but I wanted to get some 
> feedback before I spent too much time investigating either idea. 

Note that pinning doesn't prevent a plugin from being upgraded -- it's related 
to Jenkins 1.x plugin bundling, and means that a plugin that's also bundled 
(e.g. subversion) won't get overwritten by the version bundled with Jenkins on 
startup. In Jenkins 2, it's basically irrelevant.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/3C393994-86E4-4048-9D99-2A4A258706BE%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.

Re: Automated Jenkins Plugin/Dependency Management

[Daniel Beck]

> On 11.08.2016, at 15:14, Jason Kulatunga  wrote:
> 
> we've also verified that 'installNecessaryPlugins' does not actually respect 
> the version parameter. 

As the plugin update site metadata does not support arbitrary versions, it'll 
always install the one present in the JSON -- which is the latest compatible 
with your release.

This may work for you, but AFAIK still requires manual dependency resolution 
(we added it for UI uploads, but not for the CLI):

https://ci.jenkins.io/cli/command/install-plugin

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/262E4CEC-919A-49C2-ABDC-DB28849957BB%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.