subject:"Docker cloud plugin and pipeline using docker"

Re: [solved] Re: Docker cloud plugin and pipeline using docker

2018-02-27 Thread Christian Gagneraud

On 27 February 2018 at 04:55, nicolas de loof  wrote:
> For a dockerized agent to be able to run docker pipeline, it has to use a
> volume for workspace directory so this one can be shared with other
> containers it creates.
> Your docker agent template should be configured to use "/home/jenkins/agent"
> as agent root directory, as this one is a volume and as such workspace
> sub-folder will be shareable with other containers.

Thanks a lot for the tip, greatly appreciated!

Chris

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CABxGUTj7dV1DDcgmBMd6rc7KxO767%3DSZ%2BqZLeE%3DSqHhXWGvBaQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [solved] Re: Docker cloud plugin and pipeline using docker

2018-02-26 Thread nicolas de loof

For a dockerized agent to be able to run docker pipeline, it has to use a
volume for workspace directory so this one can be shared with other
containers it creates.
Your docker agent template should be configured to use
"/home/jenkins/agent" as agent root directory, as this one is a volume and
as such workspace sub-folder will be shareable with other containers.

2018-02-26 5:14 GMT+01:00 Christian Gagneraud :

> On 26 February 2018 at 15:29, Christian Gagneraud 
> wrote:
> > On 26 February 2018 at 03:10, nicolas de loof 
> wrote:
> >> docker support in declarative pipeline relies on docker-workfow plugin,
> >> which require the jenkins slave to have access to a LOCAL docker daemon,
> >> which means the docker agent you are provisioning on computer B using
> docker
> >> plugin HAS to be configured with /var/run/docker.sock (so, the one from
> B)
> >> bind mounted. Doing so it can then create some side container to host
> the
> >> build steps.
> >
> > Hi Nicolas,
> >
> > Thanks for the tips, i think i'm getting there, but i'm still missing
> some bits:
> > - I have created a docker image based on jenkinsci/jnlp-slave, the
> > only customisation is installing docker in it
> > - The pipeline now get checkout inside tmy slave (on machine B)
> > - The docker image from the Jenkinsfile is pulled and executed.
> >
> > But i get this weird error (see full log at the end of this email):
> > 
> > docker-44734997459c on work-leader seems to be running inside
> > container e7f7a4381e32c456b806695b9b7334751d88f1a3985867bb964e9fd3d2e8
> c948
> > but /home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YK
> HSBXYJPJLEX7A2AM6K5M6Q
> > could not be found among [/home/jenkins/.jenkins, /home/jenkins/agent]
>
> Google didn't return much with "could not be found among"
>
> https://stackoverflow.com/questions/47559469/jenkins-
> declarative-pipeline-docker-agent-timeout-run?rq=1
> https://tutel.me/c/programming/questions/46530963/run+jenkins+agent+in+
> docker+container+issue+with+jenkins+pipeline+and+varrundockersocket
>
> Not very useful, but using google's "repeat the search with the
> omitted results included.", I got a few extra results, one of them, an
> IRD log, that gave me the answer:
> https://echelog.com/logs/browse/jenkins/1497304800
>
> I had to add /home/jenkins in the "Volumes" of the "Docker Agent
> templates" (as well as "Remoting Filling System Root", but this was
> set by default).
> I noticed as well that if no "user" is specified in the "Connect
> method"/"Attach Docker Container", the Python image script will be run
> with id 1000:1000, which does not exists. But this doesn't seem to be
> a problem. (user=root is not required, nor is "Run container
> privileged").
>
> Thanks!
> Chris
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/jenkinsci-users/CABxGUTjm_P7nc0pk2WntGjq1BX2Eoacn6tD9xqw
> skjLpxg2_ZA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CANMVJz%3DL%3DP3ZGcPTEdEJi2eEfyM1Su-0sHDUoPQHN95qrHcAuA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[solved] Re: Docker cloud plugin and pipeline using docker

2018-02-25 Thread Christian Gagneraud

On 26 February 2018 at 15:29, Christian Gagneraud  wrote:
> On 26 February 2018 at 03:10, nicolas de loof  
> wrote:
>> docker support in declarative pipeline relies on docker-workfow plugin,
>> which require the jenkins slave to have access to a LOCAL docker daemon,
>> which means the docker agent you are provisioning on computer B using docker
>> plugin HAS to be configured with /var/run/docker.sock (so, the one from B)
>> bind mounted. Doing so it can then create some side container to host the
>> build steps.
>
> Hi Nicolas,
>
> Thanks for the tips, i think i'm getting there, but i'm still missing some 
> bits:
> - I have created a docker image based on jenkinsci/jnlp-slave, the
> only customisation is installing docker in it
> - The pipeline now get checkout inside tmy slave (on machine B)
> - The docker image from the Jenkinsfile is pulled and executed.
>
> But i get this weird error (see full log at the end of this email):
> 
> docker-44734997459c on work-leader seems to be running inside
> container e7f7a4381e32c456b806695b9b7334751d88f1a3985867bb964e9fd3d2e8c948
> but 
> /home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
> could not be found among [/home/jenkins/.jenkins, /home/jenkins/agent]

Google didn't return much with "could not be found among"

https://stackoverflow.com/questions/47559469/jenkins-declarative-pipeline-docker-agent-timeout-run?rq=1
https://tutel.me/c/programming/questions/46530963/run+jenkins+agent+in+docker+container+issue+with+jenkins+pipeline+and+varrundockersocket

Not very useful, but using google's "repeat the search with the
omitted results included.", I got a few extra results, one of them, an
IRD log, that gave me the answer:
https://echelog.com/logs/browse/jenkins/1497304800

I had to add /home/jenkins in the "Volumes" of the "Docker Agent
templates" (as well as "Remoting Filling System Root", but this was
set by default).
I noticed as well that if no "user" is specified in the "Connect
method"/"Attach Docker Container", the Python image script will be run
with id 1000:1000, which does not exists. But this doesn't seem to be
a problem. (user=root is not required, nor is "Run container
privileged").

Thanks!
Chris

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CABxGUTjm_P7nc0pk2WntGjq1BX2Eoacn6tD9xqwskjLpxg2_ZA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Docker cloud plugin and pipeline using docker

2018-02-25 Thread Christian Gagneraud

On 26 February 2018 at 03:10, nicolas de loof  wrote:
> docker support in declarative pipeline relies on docker-workfow plugin,
> which require the jenkins slave to have access to a LOCAL docker daemon,
> which means the docker agent you are provisioning on computer B using docker
> plugin HAS to be configured with /var/run/docker.sock (so, the one from B)
> bind mounted. Doing so it can then create some side container to host the
> build steps.

Hi Nicolas,

Thanks for the tips, i think i'm getting there, but i'm still missing some bits:
- I have created a docker image based on jenkinsci/jnlp-slave, the
only customisation is installing docker in it
- The pipeline now get checkout inside tmy slave (on machine B)
- The docker image from the Jenkinsfile is pulled and executed.

But i get this weird error (see full log at the end of this email):

docker-44734997459c on work-leader seems to be running inside
container e7f7a4381e32c456b806695b9b7334751d88f1a3985867bb964e9fd3d2e8c948
but 
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
could not be found among [/home/jenkins/.jenkins, /home/jenkins/agent]
but 
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp
could not be found among [/home/jenkins/.jenkins, /home/jenkins/agent]
$ docker run -t -d -u 1:1 -w
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
-v 
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q:/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q:rw,z
-v 
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp:/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp:rw,z
-e  -e  -e  -e  -e  -e
 -e  -e  -e  -e  -e 
-e  -e  -e  -e  -e  -e
 -e  -e  -e  -e  -e 
-e  -e  -e  -e  -e  -e
 python:2.7 cat
$ docker top 20801bdc7f7f4777b89a864c2cd52b03e3906d02ad3c708df4c4cdc5ebe4f45b
-eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (dummy)
[Pipeline] sh

[aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q]
Running shell script
sh: 1: cannot create
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp/durable-f19371c8/jenkins-log.txt:
Directory nonexistent
sh: 1: cannot create
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp/durable-f19371c8/jenkins-result.txt:
Directory nonexistent

process apparently never started in
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp/durable-f19371c8
---

work-leader is machine B, docker-44734997459c is, I believe, an
instance of my custom jnlp-slave, as you can see the slave fails to
run the python container

I wonder if this could be some permissions issues, for now machine B
is a VBox managed using vagrant, on it I can see that:
vagrant@leader:~$ ls -l /home/jenkins/workspace/
total 8
drwxr-xr-x 2 root root 4096 Feb 26 02:02
aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
drwxr-xr-x 2 root root 4096 Feb 26 02:02
aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp
vagrant@leader:~$ ls -la /home/jenkins/workspace/*
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q:
total 8
drwxr-xr-x 2 root root 4096 Feb 26 02:02 .
drwxr-xr-x 4 root root 4096 Feb 26 02:02 ..

/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp:
total 8
drwxr-xr-x 2 root root 4096 Feb 26 02:02 .
drwxr-xr-x 4 root root 4096 Feb 26 02:02 ..

Any idea?
Chris

Here is my Dockerfile (jenkins-slave-agent):
-
FROM jenkinsci/jnlp-slave
USER root
curl -sfL -o docker.tgz https:// && tar -C /usr/bin -xzf
docker.tgz && rm docker.tgz
USER jenkins
-

And the project's Jenkinsfile:
--
pipeline {
agent { docker 'python:2.7'}
stages { stage('dummy') { steps { sh 'python --version' } } }
}
--

Full build log:
--
Started by user admin
Obtained Jenkinsfile from 0be91558399793412462578e0061fef50a7919c3
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on docker-44734997459c on work-leader in
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning

Re: Docker cloud plugin and pipeline using docker

2018-02-25 Thread nicolas de loof

docker support in declarative pipeline relies on docker-workfow plugin,
which require the jenkins slave to have access to a LOCAL docker daemon,
which means the docker agent you are provisioning on computer B using
docker plugin HAS to be configured with /var/run/docker.sock (so, the one
from B) bind mounted. Doing so it can then create some side container to
host the build steps.

2018-02-25 11:16 GMT+01:00 Christian Gagneraud :

> Hi there,
>
> I have a machine A that runs Jenkins and Bitbucket-server in docker
> containers.
> I have a machine B, that just has docker listening over tcp.
> I have a project that uses Declarative pipeline with "agent { docker
> 'foobar:X.Y'}"
> I'm using "jenkinsci/blueocean:latest" docker image with
> "docker-workflow v1.15" plugin.
>
> Ignoring machine B (for now), my setup is working:
> - 1. A dev open a pull-request on bitbucket
> - 2. The declarative pipline is executed by jenkins
> - 3. The docker container(s) specified in the Jenkinsfile are executed
> by machine A
> - 4. The pipeline finished
> - 5. Jenkins notify Bitbucket.
>
> From a docker point of view, for the above to work, I simply need to
> run the Jenkins container with --volume
> /var/run/docker.sock:/var/run/docker.sock
>
> Now, I would like to achieve 3 things:
> - Leave machine A runs non-build services only
> - Use machine B to spawn containers as directed by declarative pipelines.
> - Eventually make machine B a docker swarm manager, and let other
> docker swarm workers join in if/when needed.
>
> The first modification I did to the jenkins container is to remove the
> "/var/run/docker.sock" volume. This is to make sure jenkins can only
> talk to the remote docker I configure.
>
> I have then tried 3 plugins:
> - "docker plugin" https://plugins.jenkins.io/docker-plugin
> - "docker swarm" https://plugins.jenkins.io/docker-swarm
> - "Yet another docker plugin"
> https://plugins.jenkins.io/yet-another-docker-plugin
>
> All of them add a type of "cloud providers", but I never managed to
> get my pipeline executed using one.
>
> The "docker plugin" is the only one that gave me the feeling I was almost
> there:
> The job was handled by a docker container spawn on machine B, which i
> had to specify the image name (From examples and tutorials, i settled
> down on "jenkinsci/jnlp-slave" in "Launch attached" mode).
> But at that stage, I got an error "docker: command not found" emitted
> from the jnlp-slave container.
> The reason being that Jenkins master contacted docker on machine B to
> execute a "jenkinsci/jnlp-slave", and then ask this
> "jenkinsci/jnlp-slave" to execute the pipeline, which is an operation
> that requires to instantiate the docker containers specified in the
> project's pipeline...
>
> So I would like to know if the "docker-plugin" is the right tool for
> my job or should I head in another direction.
> In case "docker-plugin" is the way to go, then how do I tell my
> customised jnlp-slave to connect to the right docker daemon? Is there
> an off-the-shelf docker image to play that role? I couldn't find
> documentation on how to create it, and the specificites of having
> docker inside it.
>
> As a side note:
> I found a bit inefficient to have to instantiate Jenkins slave
> container whose sole purpose is to read the pipeline and spawn
> containers as directed by the pipeline.
> Ideally I would like the pipeline to be parsed by Machine A (jenkins
> master) and the required containers to be executed on machine B, I do
> not know if volumes (jenkins workspace) can be shared across 2 docker
> daemons thought...
>
> Any opinion or advice are very welcome, point-out of any sort would be
> greatly appreciated too.
>
> Thanks,
> Chris
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/jenkinsci-users/CABxGUTgbu2BPOO1MRcPVPaxqqGhe3
> 06kU_LBAWf3H2jO2RooFA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CANMVJz%3DSW65OKUj7pK9xbdSjZGegEySRXTY8BfrnsmLoVNFmzQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Docker cloud plugin and pipeline using docker

2018-02-25 Thread Christian Gagneraud

Hi there,

I have a machine A that runs Jenkins and Bitbucket-server in docker containers.
I have a machine B, that just has docker listening over tcp.
I have a project that uses Declarative pipeline with "agent { docker
'foobar:X.Y'}"
I'm using "jenkinsci/blueocean:latest" docker image with
"docker-workflow v1.15" plugin.

Ignoring machine B (for now), my setup is working:
- 1. A dev open a pull-request on bitbucket
- 2. The declarative pipline is executed by jenkins
- 3. The docker container(s) specified in the Jenkinsfile are executed
by machine A
- 4. The pipeline finished
- 5. Jenkins notify Bitbucket.

>From a docker point of view, for the above to work, I simply need to
run the Jenkins container with --volume
/var/run/docker.sock:/var/run/docker.sock

Now, I would like to achieve 3 things:
- Leave machine A runs non-build services only
- Use machine B to spawn containers as directed by declarative pipelines.
- Eventually make machine B a docker swarm manager, and let other
docker swarm workers join in if/when needed.

The first modification I did to the jenkins container is to remove the
"/var/run/docker.sock" volume. This is to make sure jenkins can only
talk to the remote docker I configure.

I have then tried 3 plugins:
- "docker plugin" https://plugins.jenkins.io/docker-plugin
- "docker swarm" https://plugins.jenkins.io/docker-swarm
- "Yet another docker plugin"
https://plugins.jenkins.io/yet-another-docker-plugin

All of them add a type of "cloud providers", but I never managed to
get my pipeline executed using one.

The "docker plugin" is the only one that gave me the feeling I was almost there:
The job was handled by a docker container spawn on machine B, which i
had to specify the image name (From examples and tutorials, i settled
down on "jenkinsci/jnlp-slave" in "Launch attached" mode).
But at that stage, I got an error "docker: command not found" emitted
from the jnlp-slave container.
The reason being that Jenkins master contacted docker on machine B to
execute a "jenkinsci/jnlp-slave", and then ask this
"jenkinsci/jnlp-slave" to execute the pipeline, which is an operation
that requires to instantiate the docker containers specified in the
project's pipeline...

So I would like to know if the "docker-plugin" is the right tool for
my job or should I head in another direction.
In case "docker-plugin" is the way to go, then how do I tell my
customised jnlp-slave to connect to the right docker daemon? Is there
an off-the-shelf docker image to play that role? I couldn't find
documentation on how to create it, and the specificites of having
docker inside it.

As a side note:
I found a bit inefficient to have to instantiate Jenkins slave
container whose sole purpose is to read the pipeline and spawn
containers as directed by the pipeline.
Ideally I would like the pipeline to be parsed by Machine A (jenkins
master) and the required containers to be executed on machine B, I do
not know if volumes (jenkins workspace) can be shared across 2 docker
daemons thought...

Any opinion or advice are very welcome, point-out of any sort would be
greatly appreciated too.

Thanks,
Chris

--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/CABxGUTgbu2BPOO1MRcPVPaxqqGhe306kU_LBAWf3H2jO2RooFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [solved] Re: Docker cloud plugin and pipeline using docker

Re: [solved] Re: Docker cloud plugin and pipeline using docker

[solved] Re: Docker cloud plugin and pipeline using docker

Re: Docker cloud plugin and pipeline using docker

Re: Docker cloud plugin and pipeline using docker

Docker cloud plugin and pipeline using docker

6 matches

Site Navigation

Mail list logo

Footer information