Re: New Plugin hosting request: Docker Ephemeral Cloud

[Daniel Beck]

I'm not sure where this thread is now -- are you still asking for hosting of 
this plugin?

If so, could anyone speak up wrt whether there are legitimate reasons not to do 
this? So far the number of existing similar implementations has been mentioned 
-- how similar are they? Would an implementation switch like the Git Plugin has 
be a reasonable approach to support multiple APIs/libs?

> On 23.11.2015, at 17:56, kmbulebu  wrote:
> 
> The Docker landscape in Jenkins is kind of a mess, but I do feel this is a 
> good thing for an extensible open source project. The situation is 
> exaggerated here because there is so much interest and momentum in Docker. 
> We've discussed two Docker client libraries in this thread. There is a third, 
> used by Jenkins/Cloudbees, that relies on calling the docker client from the 
> command line. I know they're using it in the Docker Custom Build Environments 
> plugin, but I believe it's used in all of the new Docker plugins that were 
> released this year.
> 
> The approach calls the docker command line client and I think it is flawed 
> approach. I would much rather see the docker-commons plugin become a focal 
> point for a pure Java docker client, credentials management, and reusable 
> jelly components. I believe this has worked out well for launchers and other 
> 'common' plugins. Here's some of the issues I've encountered while working 
> with the plugins that rely on the docker cli:
>   • Requires an existing executor, on master or a slave, for use in 
> invoking the docker client. A pure docker build environment isn't possible.
>   • Docker clients are currently married by version to the server. 
> Connecting to a 1.7.1 server, requires a 1.7.1 client. This breaks down 
> quickly from a maintenance perspective.
>   • The REST apis are built with API stability and backwards 
> compatibility in mind, where as the cli makes no such guarantees. 
> If we want to address divergent strategies, I think we should start with the 
> above. 
> 
> 
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/c77af750-7924-4a02-a6e6-07e2fb3343d7%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/D6DABFFA-D218-49C0-8D41-1275774A93A3%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.

Re: New Plugin hosting request: Docker Ephemeral Cloud

[kmbulebu]

The Docker landscape in Jenkins is kind of a mess, but I do feel this is a 
good thing for an extensible open source project. The situation is 
exaggerated here because there is so much interest and momentum in Docker. 
We've discussed two Docker client libraries in this thread. There is a 
third, used by Jenkins/Cloudbees, that relies on calling the docker client 
from the command line. I know they're using it in the Docker Custom Build 
Environments plugin, but I believe it's used in all of the new Docker 
plugins that were released this year.

The approach calls the docker command line client and I think it is flawed 
approach. I would much rather see the docker-commons plugin become a focal 
point for a pure Java docker client, credentials management, and reusable 
jelly components. I believe this has worked out well for launchers and 
other 'common' plugins. Here's some of the issues I've encountered while 
working with the plugins that rely on the docker cli:

   1. Requires an existing executor, on master or a slave, for use in 
   invoking the docker client. A pure docker build environment isn't possible.
   2. Docker clients are currently married by version to the server. 
   Connecting to a 1.7.1 server, requires a 1.7.1 client. This breaks down 
   quickly from a maintenance perspective.
   3. The REST apis are built with API stability and backwards 
   compatibility in mind, where as the cli makes no such guarantees. 

If we want to address divergent strategies, I think we should start with 
the above. 





-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/c77af750-7924-4a02-a6e6-07e2fb3343d7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: New Plugin hosting request: Docker Ephemeral Cloud

['Jesse Glick' via Jenkins Developers]

On Mon, Nov 23, 2015 at 11:56 AM, kmbulebu  wrote:
> 'Here's some of the issues I've encountered while working
> with the plugins that rely on the docker cli:

All valid points (with some workarounds), but the flip side is that
when calling a CLI you can reproduce the exact command Jenkins was
running and try it yourself from a shell if anything goes wrong. Also
I am not sure where the Docker landscape is in this regard, but our
experience with SCM plugins using native Java clients has been that
performance is often poor and there are endless subtle behavioral
differences from the “golden standard” CLI. Users are rightly
frustrated when something that works fine from a shell fails with a
mysterious error in Jenkins.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2g2YX5aPhahKerxfKb%2BFT77b2TECM_VqVNbT-g1ME9XA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: New Plugin hosting request: Docker Ephemeral Cloud

[David Karlsen]

This is the problem about jenkins - 1000 half backed plugins.
Would be better with 100 well curated :)
That's where I'd be willing to pay for support.

2015-11-18 16:48 GMT+01:00 Kirill Merkushev :
> Think plugins can be merged without any problem with saving of current
> functionality for both. (With shading of both libs).
> One will use one type of cloud, another - own type of cloud. They can be
> located in separate packages and don't mix in different cases.
>
> (But please use org.jenkinsci.plugins.* prefix for new packages)
>
>
> суббота, 14 ноября 2015 г., 6:07:15 UTC+3 пользователь kmbulebu написал:
>>
>> Hi,
>>
>> I'd like to introduce a new plugin for hosting on jenkins-ci.org, the
>> docker ephemeral cloud plugin.
>>
>> Github Home: https://github.com/kmbulebu/docker-ephemeral-cloud
>> Github and Jenkins username: kmbulebu
>> Name: docker-ephemeral-cloud-plugin
>>
>> Description:
>>
>> The Docker Ephemeral Cloud plugin provides an elastic pool of slaves,
>> using Docker containers, that live only as long as the job. When a job is
>> scheduled, Jenkins checks for Docker image configurations that match the
>> labels of the job. When one is found, Docker is invoked via REST APIs to
>> pull the image and run the container. Within the container, the Jenkins JNLP
>> slave jar is downloaded and run. Communication is established to the Jenkins
>> master and the new slave node performs the requested job. When complete, the
>> container is stopped and removed.
>>
>> The key differentiators here are that this plugin enables stateless slaves
>> on containers, without the need for any port mappings, SSH servers, docker
>> clients, or existing slaves. The focus is on stateless jobs and builds and
>> not continuous deployment via containers.
>>
>> More on how this plugin differs from similar Docker plugins:
>> https://github.com/kmbulebu/docker-ephemeral-cloud#comparative-to-other-docker-plugins
>>
>> Thanks,
>>
>> Kevin
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/33893b7f-b1bb-49d5-9276-70bd3b787a46%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.



-- 
--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAGO7Ob31%3D7ZKR4cPON-2PEsxCCL-sNObfzjjP%2BqPdHHG_b%2Bijg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: New Plugin hosting request: Docker Ephemeral Cloud

[Kanstantsin Shautsou]

> 
> On Nov 18, 2015, at 17:32, domi  wrote:
> 
> To be honest, the whole list of jenkins docker plugins feels like a zoo and 
> there is no way a normal user can keep up and make the right choice.
Technically it’s a question to plugin manager and providing ratings & 
statistics. That afair will be enhanced soon.
> I think this work should be coordinated better and an uptodate comparison 
> should be kept at a central place.
> ..my 2cents
@Domi, i thought the same and even tried to go through processes, but they are 
ends into project organisation (or miss organisation).
What should i do if plugin author don’t want to use PRs or static analysing 
tools? Right, leave into fork. 

What should you do if, for example, you make business around docker and jenkins 
and enough power to support yourself this code? You will of course make your 
own company internally decided plugins hierarchy. And project organisation of 
course allows you to make your development under jenkinsci hosting. That what i 
understand too late after personal talk with Kohsuke - his project is “bazaar”. 
You solve your cases and share sources and it’s usage. If you disagree you can 
do your plugin :). 

> /Domi
> 
> 
>> On 18 Nov 2015, at 15:26, Kanstantsin Shautsou > > wrote:
>> 
>> 
>> 
>> On Tuesday, November 17, 2015 at 7:28:00 PM UTC+3, Jesse Glick wrote:
>> On Mon, Nov 16, 2015 at 10:02 PM, kmbulebu > wrote: 
>> > I believe the real horse race is between the 
>> > underlying docker libraries. In the end, we'll likely have a clear winner, 
>> > and can standardize a group of plugins around that. Perhaps docker-commons 
>> > becomes that focal point, and we have smaller plugins that deliver slaves, 
>> > build steps, workflow DSL, etc. 
>> 
>> Seems reasonable to me to have smaller plugins with focused features, 
>> and a little competition. I would not object to hosting as is, 
>> provided that plugin wikis clearly link to alternatives. 
>> 
>> IIUC the main differences between your plugin and the `Cloud` portion 
>> of the `docker` plugin are 
>> 
>> · Different client libraries. No clear winner yet. 
>> I created small thread between docker-java and docker-client to unify 
>> efforts, but obviously it will be long time action and probably will have no 
>> results. Both libraries missing different things, but technically 
>> docker-client doesn't support callbacks for long running commands. 
>> 
>> · Slave launcher: yours uses JNLP; `docker` plugin currently ships 
>> SSH, has JNLP support in code but disabled. 
>> It was disabled because of lack of integration tests, ssh launching 
>> stabilisation was very hard and i didn't want to have broken instances with 
>> one more untested launcher. Those PR that tried enable JNLP was not exactly 
>> how it was planned to work from design view point.
>> 
>> If you can later reach consensus with the `docker` plugin devs on the 
>> approach to take for a general-purpose Docker cloud provider, it 
>> should be possible to unify code into a new plugin release. Automatic 
>> migration of user settings will be a bit trickier but is possible. 
>> As soon as @magnayn likes "master development" without test cases he will of 
>> course merge any changes. So you are free to submit any PRs to docker-plugin 
>> now.  
>> 
>> By the way your comparison chart neglected to mention 
>> 
>> https://github.com/ndeloof/docker-slaves-plugin 
>>  
>> 
>> which is a novel approach that I think is very promising. 
>> Very promising thing will be finally implement normal Cloud API in core (or 
>> in plugin but provide normal extension points) because docker is not only 
>> the single cloud provider.
>> 
>> PS i'm co-maintainer of docker-java atm and working on one more docker cloud 
>> plugin for jenkins which from my viewpoint will have the best implementation 
>> of currently possible state. 
>> TLS support in docker-plugin is also possible (or even already works) 
>> because docker-java resolves system variables for connection builder (though 
>> there is no right implementation for this field).
>> PPS From tech viewpoint i see that "ephemeral cloud" has queue lock issues 
>> and shading. But if this plugin solves user issue, then it should be hosted 
>> and allow people experiment on it.
>> 
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Jenkins Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to jenkinsci-dev+unsubscr...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/jenkinsci-dev/d10a18f3-1c8c-4398-aecb-b9e474818ff4%40googlegroups.com
>>  
>> 

Re: New Plugin hosting request: Docker Ephemeral Cloud

[Nigel Magnay]

>
>
> To be honest, the whole list of jenkins docker plugins feels like a zoo
> and there is no way a normal user can keep up and make the right choice.
> I think this work should be coordinated better and an uptodate comparison
> should be kept at a central place.
> ..my 2cents
> /Domi
>
>
I think some duplication occurs accidentally (e.g: closed-source plugin
becomes open), some that is scope-creep (we started here and wanted this
feature too - I didn't know plugin  already also did that), and some
that is 'I just want to play with a different library / technique /
sandpit'.

​Other duplication though is for less healthy reasons. Needing to do TLS
and JNLP rose to the top of my stack last week, and I was pleasantly
surprised that most of the initial work had already been done for the
latter. I was rather less pleasantly surprised that - though there were
bugs and someone had submitted a PR
 that effectively made
the feature work (​maybe not in all eventualities, but sufficient for many
cases) over 2 months ago, but had effectively been told to 'go away'. To
me, that's not the right approach for a whole host of reasons - the
pertinent one being that it drives a 'not invented here' culture that leads
to duplication because dealing with the gatekeeper becomes too painful.

It could well be that some plugins could be folded together.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAPYP83SZYeCs19OQchpFzpvv%3DTR3kJa7p-kL27P_9ek5975VvA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: New Plugin hosting request: Docker Ephemeral Cloud

[Kanstantsin Shautsou]

> On Nov 18, 2015, at 18:25, Nigel Magnay  wrote:
> 
> 
> To be honest, the whole list of jenkins docker plugins feels like a zoo and 
> there is no way a normal user can keep up and make the right choice.
> I think this work should be coordinated better and an uptodate comparison 
> should be kept at a central place.
> ..my 2cents
> /Domi
> 
> 
> I think some duplication occurs accidentally (e.g: closed-source plugin 
> becomes open), some that is scope-creep (we started here and wanted this 
> feature too - I didn't know plugin  already also did that), and some 
> that is 'I just want to play with a different library / technique / sandpit'.
> 
> ​Other duplication though is for less healthy reasons. Needing to do TLS and 
> JNLP rose to the top of my stack last week, and I was pleasantly surprised 
> that most of the initial work had already been done for the latter. I was 
> rather less pleasantly surprised that - though there were bugs and someone 
> had submitted a PR  that 
> effectively made the feature work (​maybe not in all eventualities, but 
> sufficient for many cases) over 2 months ago, but had effectively been told 
> to 'go away’.
It wasn’t “go away”, it was “it needs changes” instead “it my rapid hack (or 
checkbox) for my case”. This was affected in open for everybody 
https://github.com/jenkinsci/docker-plugin/issues/235 
 roadmap. Everybody can 
join and help with development, but people do only small hacky PRs and nobody 
wants spend their time on real development. Docker allows flexibility and any 
change without integration tests ends in broken places. As soon as SSH was 
stabilised (with enough blood) i started working on ITs and delayed untested 
JNLP launcher (that btw may work in 3 different ways). Saying that it “shame” 
is harmful. Like ephemeral plugin it just single user case that will be hardly 
changeable after releasing in bad state. 
I made the most hard work and now you can:
 - continue and create ITs with releasing tested and stable features
 - pick untested (implemented) work and release as is, then deal (or not deal) 
with tones of bugs.

ITs in general is not well powered in Jenkins in comparison to maven-its, so i 
end with digging into ATH spaghetti and making own simple ITs. You can find my 
parallel emails about tests implementation.

> To me, that's not the right approach for a whole host of reasons - the 
> pertinent one being that it drives a 'not invented here' culture that leads 
> to duplication because dealing with the gatekeeper becomes too painful.
> 
> It could well be that some plugins could be folded together.
> 
> 
> 
> 
> 
>  
> 
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "Jenkins Developers" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/jenkinsci-dev/S06tNF53NxA/unsubscribe 
> .
> To unsubscribe from this group and all its topics, send an email to 
> jenkinsci-dev+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/CAPYP83SZYeCs19OQchpFzpvv%3DTR3kJa7p-kL27P_9ek5975VvA%40mail.gmail.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/21D29E6A-8B72-4B7B-83C7-69D4C234C12B%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: New Plugin hosting request: Docker Ephemeral Cloud

[Kirill Merkushev]

Think plugins can be merged without any problem with saving of current 
functionality for both. (With shading of both libs).
One will use one type of cloud, another - own type of cloud. They can be 
located in separate packages and don't mix in different cases.

(But please use org.jenkinsci.plugins.* prefix for new packages)


суббота, 14 ноября 2015 г., 6:07:15 UTC+3 пользователь kmbulebu написал:
>
> Hi,
>
> I'd like to introduce a new plugin for hosting on jenkins-ci.org, the 
> docker ephemeral cloud plugin. 
>
> Github Home: https://github.com/kmbulebu/docker-ephemeral-cloud
> Github and Jenkins username: kmbulebu
> Name: docker-ephemeral-cloud-plugin 
>
> Description: 
>
> The Docker Ephemeral Cloud plugin provides an elastic pool of slaves, 
> using Docker containers, that live only as long as the job. When a job is 
> scheduled, Jenkins checks for Docker image configurations that match the 
> labels of the job. When one is found, Docker is invoked via REST APIs to 
> pull the image and run the container. Within the container, the Jenkins 
> JNLP slave jar is downloaded and run. Communication is established to the 
> Jenkins master and the new slave node performs the requested job. When 
> complete, the container is stopped and removed.
>
> The key differentiators here are that this plugin enables stateless slaves 
> on containers, without the need for any port mappings, SSH servers, docker 
> clients, or existing slaves. The focus is on stateless jobs and builds and 
> not continuous deployment via containers. 
>
> More on how this plugin differs from similar Docker plugins: 
> https://github.com/kmbulebu/docker-ephemeral-cloud#comparative-to-other-docker-plugins
>
> Thanks,
>
> Kevin
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/33893b7f-b1bb-49d5-9276-70bd3b787a46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: New Plugin hosting request: Docker Ephemeral Cloud

[Kanstantsin Shautsou]

On Tuesday, November 17, 2015 at 7:28:00 PM UTC+3, Jesse Glick wrote:
>
> On Mon, Nov 16, 2015 at 10:02 PM, kmbulebu  > wrote: 
> > I believe the real horse race is between the 
> > underlying docker libraries. In the end, we'll likely have a clear 
> winner, 
> > and can standardize a group of plugins around that. Perhaps 
> docker-commons 
> > becomes that focal point, and we have smaller plugins that deliver 
> slaves, 
> > build steps, workflow DSL, etc. 
>
> Seems reasonable to me to have smaller plugins with focused features, 
> and a little competition. I would not object to hosting as is, 
> provided that plugin wikis clearly link to alternatives. 
>
> IIUC the main differences between your plugin and the `Cloud` portion 
> of the `docker` plugin are 
>
> · Different client libraries. No clear winner yet. 
>
I created small thread between docker-java and docker-client to unify 
efforts, but obviously it will be long time action and probably will have 
no results. Both libraries missing different things, but technically 
docker-client doesn't support callbacks for long running commands. 

· Slave launcher: yours uses JNLP; `docker` plugin currently ships 
> SSH, has JNLP support in code but disabled. 
>
It was disabled because of lack of integration tests, ssh launching 
stabilisation was very hard and i didn't want to have broken instances with 
one more untested launcher. Those PR that tried enable JNLP was not exactly 
how it was planned to work from design view point.

>
> If you can later reach consensus with the `docker` plugin devs on the 
> approach to take for a general-purpose Docker cloud provider, it 
> should be possible to unify code into a new plugin release. Automatic 
> migration of user settings will be a bit trickier but is possible. 
>
As soon as @magnayn likes "master development" without test cases he will 
of course merge any changes. So you are free to submit any PRs to 
docker-plugin now.  

>
> By the way your comparison chart neglected to mention 
>
> https://github.com/ndeloof/docker-slaves-plugin 
>
> which is a novel approach that I think is very promising. 
>
Very promising thing will be finally implement normal Cloud API in core (or 
in plugin but provide normal extension points) because docker is not only 
the single cloud provider.

PS i'm co-maintainer of docker-java atm and working on one more docker 
cloud plugin for jenkins which from my viewpoint will have the best 
implementation of currently possible state. 
TLS support in docker-plugin is also possible (or even already works) 
because docker-java resolves system variables for connection builder 
(though there is no right implementation for this field).
PPS From tech viewpoint i see that "ephemeral cloud" has queue lock issues 
and shading. But if this plugin solves user issue, then it should be hosted 
and allow people experiment on it.


-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/d10a18f3-1c8c-4398-aecb-b9e474818ff4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: New Plugin hosting request: Docker Ephemeral Cloud

[domi]

To be honest, the whole list of jenkins docker plugins feels like a zoo and 
there is no way a normal user can keep up and make the right choice.
I think this work should be coordinated better and an uptodate comparison 
should be kept at a central place.
..my 2cents
/Domi


> On 18 Nov 2015, at 15:26, Kanstantsin Shautsou  
> wrote:
> 
> 
> 
> On Tuesday, November 17, 2015 at 7:28:00 PM UTC+3, Jesse Glick wrote:
> On Mon, Nov 16, 2015 at 10:02 PM, kmbulebu  
> wrote: 
> > I believe the real horse race is between the 
> > underlying docker libraries. In the end, we'll likely have a clear winner, 
> > and can standardize a group of plugins around that. Perhaps docker-commons 
> > becomes that focal point, and we have smaller plugins that deliver slaves, 
> > build steps, workflow DSL, etc. 
> 
> Seems reasonable to me to have smaller plugins with focused features, 
> and a little competition. I would not object to hosting as is, 
> provided that plugin wikis clearly link to alternatives. 
> 
> IIUC the main differences between your plugin and the `Cloud` portion 
> of the `docker` plugin are 
> 
> · Different client libraries. No clear winner yet. 
> I created small thread between docker-java and docker-client to unify 
> efforts, but obviously it will be long time action and probably will have no 
> results. Both libraries missing different things, but technically 
> docker-client doesn't support callbacks for long running commands. 
> 
> · Slave launcher: yours uses JNLP; `docker` plugin currently ships 
> SSH, has JNLP support in code but disabled. 
> It was disabled because of lack of integration tests, ssh launching 
> stabilisation was very hard and i didn't want to have broken instances with 
> one more untested launcher. Those PR that tried enable JNLP was not exactly 
> how it was planned to work from design view point.
> 
> If you can later reach consensus with the `docker` plugin devs on the 
> approach to take for a general-purpose Docker cloud provider, it 
> should be possible to unify code into a new plugin release. Automatic 
> migration of user settings will be a bit trickier but is possible. 
> As soon as @magnayn likes "master development" without test cases he will of 
> course merge any changes. So you are free to submit any PRs to docker-plugin 
> now.  
> 
> By the way your comparison chart neglected to mention 
> 
> https://github.com/ndeloof/docker-slaves-plugin 
>  
> 
> which is a novel approach that I think is very promising. 
> Very promising thing will be finally implement normal Cloud API in core (or 
> in plugin but provide normal extension points) because docker is not only the 
> single cloud provider.
> 
> PS i'm co-maintainer of docker-java atm and working on one more docker cloud 
> plugin for jenkins which from my viewpoint will have the best implementation 
> of currently possible state. 
> TLS support in docker-plugin is also possible (or even already works) because 
> docker-java resolves system variables for connection builder (though there is 
> no right implementation for this field).
> PPS From tech viewpoint i see that "ephemeral cloud" has queue lock issues 
> and shading. But if this plugin solves user issue, then it should be hosted 
> and allow people experiment on it.
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-dev+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/d10a18f3-1c8c-4398-aecb-b9e474818ff4%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/127A6A84-F1F2-42C6-9195-922972EA3D68%40fortysix.ch.
For more options, visit https://groups.google.com/d/optout.

Re: New Plugin hosting request: Docker Ephemeral Cloud

['Jesse Glick' via Jenkins Developers]

On Mon, Nov 16, 2015 at 10:02 PM, kmbulebu  wrote:
> I believe the real horse race is between the
> underlying docker libraries. In the end, we'll likely have a clear winner,
> and can standardize a group of plugins around that. Perhaps docker-commons
> becomes that focal point, and we have smaller plugins that deliver slaves,
> build steps, workflow DSL, etc.

Seems reasonable to me to have smaller plugins with focused features,
and a little competition. I would not object to hosting as is,
provided that plugin wikis clearly link to alternatives.

IIUC the main differences between your plugin and the `Cloud` portion
of the `docker` plugin are

· Different client libraries. No clear winner yet.
· Slave launcher: yours uses JNLP; `docker` plugin currently ships
SSH, has JNLP support in code but disabled.

If you can later reach consensus with the `docker` plugin devs on the
approach to take for a general-purpose Docker cloud provider, it
should be possible to unify code into a new plugin release. Automatic
migration of user settings will be a bit trickier but is possible.

By the way your comparison chart neglected to mention

https://github.com/ndeloof/docker-slaves-plugin

which is a novel approach that I think is very promising.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1hTuSsTQ8RWL%3DdqSZLDH_JRZ6jPpMBQ51o2Hrv%2BKCNUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: New Plugin hosting request: Docker Ephemeral Cloud

[Nigel Magnay]

FWIW, I have pushed changes for both TLS and JNLP in docker-plugin, so the
comparison can now be reduced to

Docker Plugin 


   - Provides additional capabilities beyond slaves, such as Docker build
  steps.







On Tue, Nov 17, 2015 at 4:27 PM, 'Jesse Glick' via Jenkins Developers <
jenkinsci-dev@googlegroups.com> wrote:

> On Mon, Nov 16, 2015 at 10:02 PM, kmbulebu  wrote:
> > I believe the real horse race is between the
> > underlying docker libraries. In the end, we'll likely have a clear
> winner,
> > and can standardize a group of plugins around that. Perhaps
> docker-commons
> > becomes that focal point, and we have smaller plugins that deliver
> slaves,
> > build steps, workflow DSL, etc.
>
> Seems reasonable to me to have smaller plugins with focused features,
> and a little competition. I would not object to hosting as is,
> provided that plugin wikis clearly link to alternatives.
>
> IIUC the main differences between your plugin and the `Cloud` portion
> of the `docker` plugin are
>
> · Different client libraries. No clear winner yet.
> · Slave launcher: yours uses JNLP; `docker` plugin currently ships
> SSH, has JNLP support in code but disabled.
>
> If you can later reach consensus with the `docker` plugin devs on the
> approach to take for a general-purpose Docker cloud provider, it
> should be possible to unify code into a new plugin release. Automatic
> migration of user settings will be a bit trickier but is possible.
>
> By the way your comparison chart neglected to mention
>
> https://github.com/ndeloof/docker-slaves-plugin
>
> which is a novel approach that I think is very promising.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1hTuSsTQ8RWL%3DdqSZLDH_JRZ6jPpMBQ51o2Hrv%2BKCNUA%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAPYP83QwyEVY2kVw9_F6D9hz%2Ba_QVGoHZ5wk69%3Dc%3DFxXq-hBCA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: New Plugin hosting request: Docker Ephemeral Cloud

[R. Tyler Croy]

(replies inline)

On Fri, 13 Nov 2015, kmbulebu wrote:

> I'd like to introduce a new plugin for hosting on jenkins-ci.org, the 
> docker ephemeral cloud plugin. 
> 
> Github Home: https://github.com/kmbulebu/docker-ephemeral-cloud
> Github and Jenkins username: kmbulebu
> Name: docker-ephemeral-cloud-plugin 
> 
> Description: 
> 
> The Docker Ephemeral Cloud plugin provides an elastic pool of slaves, using 
> Docker containers, that live only as long as the job. When a job is 
> scheduled, Jenkins checks for Docker image configurations that match the 
> labels of the job. When one is found, Docker is invoked via REST APIs to 
> pull the image and run the container. Within the container, the Jenkins 
> JNLP slave jar is downloaded and run. Communication is established to the 
> Jenkins master and the new slave node performs the requested job. When 
> complete, the container is stopped and removed.
> 
> The key differentiators here are that this plugin enables stateless slaves 
> on containers, without the need for any port mappings, SSH servers, docker 
> clients, or existing slaves. The focus is on stateless jobs and builds and 
> not continuous deployment via containers. 
> 
> More on how this plugin differs from similar Docker 
> plugins: 
> https://github.com/kmbulebu/docker-ephemeral-cloud#comparative-to-other-docker-plugins



The differences between this plugin and the Docker plugin don't seem very
major to me, is there an objection you would have to merging this with the
Docker plugin? That plugin already has many thousands of installations and the
TLS support in particular I think would be useful as well.

Have you also seen the Docker build step plugin, I believe that might overlap
with your work as well, and already has a few thousand installations.


There's some really good work here and by merging efforts with the existing
Jenkins/Docker plugins I think we can improve the usage for many existing users
and help attract new ones.



Let me know what you think, if you're open to it I'm happy to help where I can.


If you're not keen on merging, then I suppose I'll make a frowney face and
setup the hosting, etc.



Cheers
- R. Tyler Croy

--
 Code: 
  Chatter: 

  % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F
--

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/20151116153241.GV23766%40blackberry.coupleofllamas.com.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: Digital signature

Re: New Plugin hosting request: Docker Ephemeral Cloud

[kmbulebu]

Thanks for your kind words :). 

I understand. At a high level, they both aim to deliver slaves using 
Docker. This is an old problem; do you break away and create new, or revise 
in place? I think we should move forward with the new one, and let it work 
itself out. I'm not opposed to merging, but I don't want to delay 
delivering immediate value to users. I created the plugin out of urgency (I 
couldn't get the firewall rules for 100+ SSH ports approved in my 
organization) and I feel others have similar needs. Before looking at 
merging the two, I would like to gain some more learning through user 
experience.

I recall we once had two Gerrit plugins, but have since seen the two 
combine into one excellent plugin. I believe the real horse race is between 
the underlying docker libraries. In the end, we'll likely have a clear 
winner, and can standardize a group of plugins around that. Perhaps 
docker-commons becomes that focal point, and we have smaller plugins that 
deliver slaves, build steps, workflow DSL, etc. For now, I think it's too 
early to settle. 

I don't want you to frown, but can we go ahead and move forward with this 
as-is?

Thanks,

Kevin



On Monday, November 16, 2015 at 10:34:07 AM UTC-5, R Tyler Croy wrote:
>
> (replies inline) 
>
> On Fri, 13 Nov 2015, kmbulebu wrote: 
>
> > I'd like to introduce a new plugin for hosting on jenkins-ci.org, the 
> > docker ephemeral cloud plugin. 
> > 
> > Github Home: https://github.com/kmbulebu/docker-ephemeral-cloud 
> > Github and Jenkins username: kmbulebu 
> > Name: docker-ephemeral-cloud-plugin 
> > 
> > Description: 
> > 
> > The Docker Ephemeral Cloud plugin provides an elastic pool of slaves, 
> using 
> > Docker containers, that live only as long as the job. When a job is 
> > scheduled, Jenkins checks for Docker image configurations that match the 
> > labels of the job. When one is found, Docker is invoked via REST APIs to 
> > pull the image and run the container. Within the container, the Jenkins 
> > JNLP slave jar is downloaded and run. Communication is established to 
> the 
> > Jenkins master and the new slave node performs the requested job. When 
> > complete, the container is stopped and removed. 
> > 
> > The key differentiators here are that this plugin enables stateless 
> slaves 
> > on containers, without the need for any port mappings, SSH servers, 
> docker 
> > clients, or existing slaves. The focus is on stateless jobs and builds 
> and 
> > not continuous deployment via containers. 
> > 
> > More on how this plugin differs from similar Docker 
> > plugins: 
> https://github.com/kmbulebu/docker-ephemeral-cloud#comparative-to-other-docker-plugins
>  
>
>
>
> The differences between this plugin and the Docker plugin don't seem very 
> major to me, is there an objection you would have to merging this with the 
> Docker plugin? That plugin already has many thousands of installations and 
> the 
> TLS support in particular I think would be useful as well. 
>
> Have you also seen the Docker build step plugin, I believe that might 
> overlap 
> with your work as well, and already has a few thousand installations. 
>
>
> There's some really good work here and by merging efforts with the 
> existing 
> Jenkins/Docker plugins I think we can improve the usage for many existing 
> users 
> and help attract new ones. 
>
>
>
> Let me know what you think, if you're open to it I'm happy to help where I 
> can. 
>
>
> If you're not keen on merging, then I suppose I'll make a frowney face and 
> setup the hosting, etc. 
>
>
>
> Cheers 
> - R. Tyler Croy 
>
> -- 
>  Code:  
>   Chatter:  
>
>   % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F 
> -- 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/c9ba5f44-73db-41ed-b6de-1f7719d01ab6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

New Plugin hosting request: Docker Ephemeral Cloud

[kmbulebu]

Hi,

I'd like to introduce a new plugin for hosting on jenkins-ci.org, the 
docker ephemeral cloud plugin. 

Github Home: https://github.com/kmbulebu/docker-ephemeral-cloud
Github and Jenkins username: kmbulebu
Name: docker-ephemeral-cloud-plugin 

Description: 

The Docker Ephemeral Cloud plugin provides an elastic pool of slaves, using 
Docker containers, that live only as long as the job. When a job is 
scheduled, Jenkins checks for Docker image configurations that match the 
labels of the job. When one is found, Docker is invoked via REST APIs to 
pull the image and run the container. Within the container, the Jenkins 
JNLP slave jar is downloaded and run. Communication is established to the 
Jenkins master and the new slave node performs the requested job. When 
complete, the container is stopped and removed.

The key differentiators here are that this plugin enables stateless slaves 
on containers, without the need for any port mappings, SSH servers, docker 
clients, or existing slaves. The focus is on stateless jobs and builds and 
not continuous deployment via containers. 

More on how this plugin differs from similar Docker 
plugins: 
https://github.com/kmbulebu/docker-ephemeral-cloud#comparative-to-other-docker-plugins

Thanks,

Kevin


-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/7c63ca60-7262-4005-a767-da2ef43c9f6d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.