thanks for reporting this. Most probably I've only ran tests for various launchers only once :P
2017-10-20 17:05 GMT+02:00 Martin Heg <[email protected]>: > Sounds good - if it helps with your analysis I've noticed that first build > after a restart always seems to work fine with the executor being destroyed > correctly. Subsequent builds then suffer from the 'offline'/non-terminated > executor. > Thanks. > > On 20 October 2017 at 15:59, nicolas de loof <[email protected]> > wrote: > >> looks like the build termination isn't successfully caught to terminate >> this node. >> I'm trying to stabilize the "launch" mechanism in 1.0.x, then will try to >> address the more general lifecycle for docker agents, being created as a >> task enter the queue and destroyed on completion. My experiments on >> one-shot-executor give me some ideas on how to improve this process without >> being too much hack-ish :D >> >> 2017-10-20 16:16 GMT+02:00 Martin Heg <[email protected]>: >> >>> Aha - yes I have 'Docker Host URI' set to 'unix:///var/run/docker.sock', >>> but what I was probably missing was to set the 'Docker Hostname' under the >>> Advanced section in the Cloud settings... once I set that to the ip address >>> of the docker host I am having some more success... >>> >>> Now slave containers are being spawned and builds are running in them >>> :), but all is not quite right.. >>> >>> Even though the container is created and destroyed after the build the >>> UI is still showing the 'node' as offline. See attached screenshot. >>> >>> Looking at the log for any of these shows that the docker host ip >>> address is now being used for routing instead of 0.0.0.0 but the log still >>> shows the 'connection refused' error... (I've masked out ip address in the >>> paste below).. >>> >>> Despite this error, the build history for the node shows that it ran the >>> build successfully.. >>> >>> *[10/20/17 14:06:44] [SSH] Opening SSH connection to 10.x.x.x:32842. >>> Connection refused (Connection refused) >>> SSH Connection failed with IOException: "Connection refused (Connection >>> refused)". >>> java.io.IOException: There was a problem while connecting to 10.x.x.x:32842 >>> at com.trilead.ssh2.Connection.co >>> <http://com.trilead.ssh2.Connection.co>nnect(Connection.java:834) >>> at com.trilead.ssh2.Connection.co >>> <http://com.trilead.ssh2.Connection.co>nnect(Connection.java:703) >>> at com.trilead.ssh2.Connection.co >>> <http://com.trilead.ssh2.Connection.co>nnect(Connection.java:617) >>> at >>> hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1284) >>> at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:804) >>> at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:793) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> Caused by: java.net.ConnectException: Connection refused (Connection >>> refused) >>> at java.net.PlainSocketImpl.socketConnect(Native Method) >>> at java.net >>> <http://java.net>.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) >>> at java.net >>> <http://java.net>.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) >>> at java.net >>> <http://java.net>.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) >>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >>> at java.net.Socket.connect(Socket.java:589) >>> at >>> com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:367) >>> at >>> com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:480) >>> at com.trilead.ssh2.Connection.co >>> <http://com.trilead.ssh2.Connection.co>nnect(Connection.java:774) >>> ... 9 more >>> [10/20/17 14:06:44] Launch failed - cleaning up connection >>> [10/20/17 14:06:44] [SSH] Connection closed.* >>> >>> >>> >>> >>> Thank you so much for your help Nicolas. >>> >>> On 20 October 2017 at 14:11, nicolas de loof <[email protected]> >>> wrote: >>> >>>> issue here is 0.0.0.0 (localhost) from jenkins master is a container, >>>> not the docker host, so can't connect to ssh slave. >>>> I assume you have master to access docker.sock to run this sibling >>>> container ? So master has no way to guess the external IP >>>> >>>> 2017-10-20 14:51 GMT+02:00 Martin Heg <[email protected]>: >>>> >>>>> What I should add once more just to clarify is that the slave >>>>> container does actually get started (and hence why I am able to try >>>>> manually ssh-ing into it) - it stays running for a few minutes and then >>>>> exits. >>>>> >>>>> >>>>> On 20 October 2017 at 13:34, Martin Heg <[email protected]> wrote: >>>>> >>>>>> Frustrating thing is that I definitely had this same exact setup >>>>>> working previously (jenkins master in docker container spawning slaves in >>>>>> sibling containers), albeit in the context of docker toolbox >>>>>> (boot2docker) >>>>>> running on windows, whereas I am now running against standard docker on >>>>>> linux. >>>>>> >>>>>> On 20 October 2017 at 13:25, Martin Heg <[email protected]> wrote: >>>>>> >>>>>>> Yes - jenkins master running in container. >>>>>>> From the docker host I can manually ssh into the slave container >>>>>>> with: >>>>>>> ssh -p *ephmeral_port* [email protected] >>>>>>> If I docker exec into the jenkins master I can also ssh into the >>>>>>> slave container using the docker host ip: >>>>>>> ssh -p *ephmeral_port* jenkins@*hostip* >>>>>>> >>>>>>> hmmmm..... >>>>>>> >>>>>>> On 20 October 2017 at 13:05, nicolas de loof < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Is your jenkins master running in a docker container as well ? >>>>>>>> >>>>>>>> stacktrace seems to demonstrate failure to establish a route to >>>>>>>> 0.0.0.0:32791, even before any SSH authentication attempt >>>>>>>> >>>>>>>> 2017-10-20 12:16 GMT+02:00 Martin Heg <[email protected]>: >>>>>>>> >>>>>>>>> Thanks for your reply Nicolas. Yes I saw that suggestion in one of >>>>>>>>> your other posts and have already tried it but still seeing the same >>>>>>>>> issue >>>>>>>>> unfortunately. >>>>>>>>> >>>>>>>>> On 20 October 2017 at 11:11, nicolas de loof < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> to use your own SSH credentials you need to disable the "SSH key >>>>>>>>>> management" option (title is unclear) >>>>>>>>>> with this option set the container is configured to rely on >>>>>>>>>> jenkins master Identity ssh key >>>>>>>>>> >>>>>>>>>> I'm working on a UI refactoring to avoid such confusing options >>>>>>>>>> >>>>>>>>>> 2017-10-20 11:38 GMT+02:00 <[email protected]>: >>>>>>>>>> >>>>>>>>>>> Just to add that the Docker plugin also recognises the slave >>>>>>>>>>> container as running as shown in this attachment. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> <https://lh3.googleusercontent.com/-dwvQgB3E6S4/WenD9OpnTbI/AAAAAAAAAAM/zOJHAj-aEuQ32teM26AFPsAELCX8QAVvwCLcBGAs/s1600/screenshot.9.jpg> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Friday, 20 October 2017 10:03:45 UTC+1, [email protected] >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> I am getting this error on various versions of Jenkins and the >>>>>>>>>>>> docker-plugin including latest of both. >>>>>>>>>>>> My Jenkins master is running in a container. It successfully >>>>>>>>>>>> launches a docker container for the slave (using the >>>>>>>>>>>> evarga/jenkins-slave >>>>>>>>>>>> image) but then fails to connect with the error below. >>>>>>>>>>>> I can see my slave running with a docker ps and can even >>>>>>>>>>>> successfully connect to it via ssh on 0.0.0.0:32791 (or >>>>>>>>>>>> whatever the given ephmeral port is) using the same credentials >>>>>>>>>>>> configured >>>>>>>>>>>> in the docker agent template section. >>>>>>>>>>>> Any ideas really welcome as I'm now going round in circles >>>>>>>>>>>> trying to get this working. >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [10/20/17 08:13:45] [SSH] Opening SSH connection to 0.0.0.0:32791. >>>>>>>>>>>> Connection refused (Connection refused) >>>>>>>>>>>> SSH Connection failed with IOException: "Connection refused >>>>>>>>>>>> (Connection refused)". >>>>>>>>>>>> java.io.IOException: There was a problem while connecting to >>>>>>>>>>>> 0.0.0.0:32791 >>>>>>>>>>>> at com.trilead.ssh2.Connection.connect(Connection.java:834) >>>>>>>>>>>> at com.trilead.ssh2.Connection.connect(Connection.java:703) >>>>>>>>>>>> at com.trilead.ssh2.Connection.connect(Connection.java:617) >>>>>>>>>>>> at >>>>>>>>>>>> hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1284) >>>>>>>>>>>> at >>>>>>>>>>>> hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:804) >>>>>>>>>>>> at >>>>>>>>>>>> hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:793) >>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>>>>>> at >>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>>>>>>>>> at >>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>>>>>>>> Caused by: java.net.ConnectException: Connection refused >>>>>>>>>>>> (Connection refused) >>>>>>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method) >>>>>>>>>>>> at >>>>>>>>>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) >>>>>>>>>>>> at >>>>>>>>>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204) >>>>>>>>>>>> at >>>>>>>>>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) >>>>>>>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >>>>>>>>>>>> at java.net.Socket.connect(Socket.java:589) >>>>>>>>>>>> at >>>>>>>>>>>> com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:367) >>>>>>>>>>>> at >>>>>>>>>>>> com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:480) >>>>>>>>>>>> at com.trilead.ssh2.Connection.connect(Connection.java:774) >>>>>>>>>>>> ... 9 more >>>>>>>>>>>> [10/20/17 08:13:45] Launch failed - cleaning up connection >>>>>>>>>>>> [10/20/17 08:13:45] [SSH] Connection closed. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "Jenkins Users" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to jenkinsci-users+unsubscribe@go >>>>>>>>>>> oglegroups.com. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/jenkinsci-users/9359318d-f >>>>>>>>>>> 78f-432a-bd06-4c10768f2cdc%40googlegroups.com >>>>>>>>>>> <https://groups.google.com/d/msgid/jenkinsci-users/9359318d-f78f-432a-bd06-4c10768f2cdc%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>> in the Google Groups "Jenkins Users" group. >>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>> https://groups.google.com/d/topic/jenkinsci-users/cEKuC4GwP5 >>>>>>>>>> I/unsubscribe. >>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>> to [email protected]. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/jenkinsci-users/CANMVJzkXM >>>>>>>>>> Ljco0tzp_rcZpcyQ-4xFMQNi9%3DKAAEK6-Z8r%3DLYMg%40mail.gmail.com >>>>>>>>>> <https://groups.google.com/d/msgid/jenkinsci-users/CANMVJzkXMLjco0tzp_rcZpcyQ-4xFMQNi9%3DKAAEK6-Z8r%3DLYMg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Jenkins Users" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/jenkinsci-users/CAHigaWxJM >>>>>>>>> bmwMoMdXADbmwsYwgwsX8RpixHfqfZ-nJZbj1rFjA%40mail.gmail.com >>>>>>>>> <https://groups.google.com/d/msgid/jenkinsci-users/CAHigaWxJMbmwMoMdXADbmwsYwgwsX8RpixHfqfZ-nJZbj1rFjA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "Jenkins Users" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/jenkinsci-users/cEKuC4GwP5 >>>>>>>> I/unsubscribe. >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/jenkinsci-users/CANMVJznY_ >>>>>>>> yeVFHw6GY0kj1Oj6Zrg04N9ws6D%2BEMXdfunYfe6Hg%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/jenkinsci-users/CANMVJznY_yeVFHw6GY0kj1Oj6Zrg04N9ws6D%2BEMXdfunYfe6Hg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Jenkins Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/jenkinsci-users/CAHigaWxa3 >>>>> BdDpYd4or8PvGBvRx8Je5qgyR5Gx%2BBHbTvCUzrEvA%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/jenkinsci-users/CAHigaWxa3BdDpYd4or8PvGBvRx8Je5qgyR5Gx%2BBHbTvCUzrEvA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Jenkins Users" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>> pic/jenkinsci-users/cEKuC4GwP5I/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit https://groups.google.com/d/ms >>>> gid/jenkinsci-users/CANMVJzmqUE4kBYVVCHtoHJqvuVCf06SXMN1cDE4 >>>> VwrCAMjearw%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/jenkinsci-users/CANMVJzmqUE4kBYVVCHtoHJqvuVCf06SXMN1cDE4VwrCAMjearw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/jenkinsci-users/CAHigaWwfiCjw_xX2LsXbuh3_rtdPpejVFqEawLA >>> fFa7qy0KQSw%40mail.gmail.com >>> <https://groups.google.com/d/msgid/jenkinsci-users/CAHigaWwfiCjw_xX2LsXbuh3_rtdPpejVFqEawLAfFa7qy0KQSw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Jenkins Users" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/to >> pic/jenkinsci-users/cEKuC4GwP5I/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/jenkinsci-users/CANMVJzmjjvRDpQVJG%3Dmn8eFYCa%3DDDkLmtpw >> %2BPVt5EPT6UiR5FQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/jenkinsci-users/CANMVJzmjjvRDpQVJG%3Dmn8eFYCa%3DDDkLmtpw%2BPVt5EPT6UiR5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/jenkinsci-users/CAHigaWzes4nar48s-%3DYm3_ > TRLZJ2n_c5gbMSwVLWLtrOuJgkrA%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-users/CAHigaWzes4nar48s-%3DYm3_TRLZJ2n_c5gbMSwVLWLtrOuJgkrA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CANMVJzm7XAaoY9rK2sbr0_08Rx02n4xsK9GeQv8CQM5Q7N_g7A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
