[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Andrew Grimberg commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes Joshua Harlow it works pretty well actually. The one I linked is one of our simpler setups. Our most complex one is at: https://github.com/opendaylight/releng-builder/tree/master/jenkins-scripts Though I think the README is a little outdated since we're using the jenkins-init-script.sh pattern that was mentioned in the previous one I linked. It gives us some extra flexibility in our instance management so we don't have have to be regularly respinning packer builds for small configuration tweaks. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Oliver Gondža commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes Joshua Harlow, No good news from cloud-init land? Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Joshua Harlow commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes Ya, that idea is similar to what I'm doing at godaddy as well. Its not ideal, but basically using a shell script to add the user (and making sure adding the user last) will at least ensure jenkins can't login until the rest of the user script is finished... Best I've seen so far (though IMHO far from the best it could be). Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Andrew Grimberg commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes Donald Morton Here's how we do it in our various environments. We pass a user-data script that does what is explained in the README in the following location. https://github.com/zephyrproject-rtos/ci-management/tree/master/jenkins-scripts I'll note we do this for several different Jenkins systems that we support < /td> Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Donald Morton commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes How do you control when cloud-init creates a user? Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Joshua Harlow commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes Right, I've got another way to do this also, Where basically u make the user that is being used to log-in really late in the cloud-init process. Then the external ssh connection won't be possible until the final user adding has actually happened (which is then after all the other stuff I want to occur has finished). Its not perfect, and as being one of the cloud-init authors this is a known thing; and since this plug-in is not really tying itself to cloud-init tightly the solutions to it become a little bit harder (especially say with windows). Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title
Donald Morton commented on JENKINS-30183
Re: Openstack Cloud Plugin does not wait until the user data script completes
I have run into this problem as well. It takes about 20 minutes for our init script to finish running. I was able to work around it by using the jenkins-cli.jar to reconnect the node at the end of the script. Here is an example: curl --insecure https:///jnlpJars/jenkins-cli.jar > jenkins-cli.jar 2>> $ {LOGFILE} #Fix SSL Cert so jenkins-cli works properly openssl x509 -in <(openssl s_client -connect :443 -prexit 2>/dev/null) > jenkins.cer 2>> ${LOGFILE} /opt/java/jdk1.8.0_25/bin/keytool -keystore /opt/java/jdk1.8.0_25/jre/lib/security/cacerts -import -file jenkins.cer -alias -storepass changeit -noprompt >> $ {LOGFILE} 2>&1 #Tell Jenkins master to connect to the slave runuser -l cloud-user -c "/opt/java/jdk1.8.0_25/bin/java -jar jenkins-cli.jar -s https:// connect-node `hostname`" >> ${LOGFILE} 2>&1
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Joshua Harlow commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes Do u know what image u were trying this on, a image with systemd? Knowing that will help me figure out how to adjust the module order in cloud-init to better suit what u want to do. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Joshua Harlow commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes Let me see what I can dig up, part of this feels like a out of order system daemon configuration (where sshd is starting up before cloud-init); which may just be a packaging issue. Let me look that over and get back to u. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Oliver Gondža commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes JNLP is easy as the connection is generated from slave side. It is enough to start the agent once the setup is done from cloud-init. SSH connection is that causes this problem. The problem is that SSH starts responding before cloud-init completes so jobs can observe instance not fully setup. Which is something you can help us to clarify: Is there a way to tell cloud-init to reject the connection until it is done in a reasonable way (not like https://github.com/jenkinsci/acceptance-test-harness/blob/master/src/test/resources/openstack_plugin/cloud-init#L16)? That would narrow the problem down to properly wait for ssh connection to start working. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Joshua Harlow commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes So the other option IMHO is to just depend on the JNLP or ssh and use that as the detection that if u can't access the server over SSH (I don't know much about JNLP) then the slave isn't going to work anyway so if u can't connect to it over those mechanisms after X period of time, then u might as well as declare the instance dead (because jenkins won't be able to use it anyway if those don't work). Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Oliver Gondža edited a comment on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes I have considered the phone home feature though there are several caveats:- Cloud-init can be user written and therefore the directive might be missing. Adding it at the end of the script sounds problematic.- Cloud init does not necessarily have to use #cloud-config format, it can be bash, powershell or other script. I do not thing think plugin should manipulate all of them.- Not all images have that feature enabled. If user-data are sent and not supported by image - nothing really happens. If Jenkins would wait for a signal to arrive it would be a problem for such images. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Joshua Harlow commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes Agreed to the caveats, btw. I'm one of the cloud-init developers (one of the like three)! There is a feature that folks have used that is called multi-part messages into cloud-init, this would allow u to combine user provided scripts and jenkins provided ones. So that could handle having both scripts and ... http://cloudinit.readthedocs.io/en/latest/topics/format.html#mime-multi-part-archive So as for the question, its not running ahead of time, its installed via a cloud-init script then started by cloud-init as close to being done as it can be. The other option that I've seen (below in ansible) and I guess its similar to what jenkins is doing (when trying to setup the slave?) is wait for ssh to get started, and if that never happens, assume the server is dead. https://github.com/kubespray/kargo-cli/blob/master/src/kargo/cloud.py#L382 (for example), It'd be nice to pick a strategy, or just use the existing jenkins ssh retries and such (and/or make that more robust)? Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Oliver Gondža commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes My current thinking is using marker entries in openstack server console log. Iff there is OPENSTACK_USER_DATA_START logged, the plugin would wait for OPENSTACK_USER_DATA_DONE. Existing scripts would work as before since the first marker would never appear. So does images/setups without cloud-init/cloud-config. If someone is doing complicated setup, they will log extra 2 lines to the output from the script they prefer. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Oliver Gondža assigned an issue to Oliver Gondža Jenkins / JENKINS-30183 Openstack Cloud Plugin does not wait until the user data script completes Change By: Oliver Gondža Assignee: Marat Mavlyutov Oliver Gondža Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Oliver Gondža commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes Why do you need a web server running on your slaves ahead of time? Please elaborate further. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Oliver Gondža commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes I have considered the phone home feature though there are several caveats: Cloud-init can be user written and therefore the directive might be missing. Adding it at the end of the script sounds problematic. Cloud init does not necessarily have to use #cloud-config format, it can be bash, powershell or other script. I do not thing plugin should manipulate all of them. Not all images have that feature enabled. If user-data are sent and not supported by image - nothing really happens. If Jenkins would wait for a signal to arrive it would be a problem for such images. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Joshua Harlow commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes So over at godaddy we are moving towards this plugin and our existing solution has been to spin-up a tiny webserver (either via apache) or via python via something like the following (this would happen as the last thing cloud-init via user data script does): $ python -m SimpleHTTPServer Then have the code that provisioned that server (ie, the openstack cloud plugin here), poll for a given period of time trying to access the above port, if the port doesn't become accessible after some configurable amount of time, then assume the server is busted (and the VM is dead); otherwise assume everything worked out. Another option (and I'm not really sure how/if it could work) is to take advvantage of the cloud-init feature to callback to a URL as the last thing that it does when running (ie after all the other user data scripts have ran): example @ https://github.com/number5/cloud-init/blob/master/doc/examples/cloud-config-phone-home.txt#L3 If that could be used to somehow post back to the jenkins master that the 'slave is now ready for usage' that'd be even better (less polling required). Anyone know which approach people would be ok with? Both would require a version of cloud-init installed (which is in most/all images), the later would be nice, but unsure if jenkins has such an ability in the first place (how does it know slaves are ready to be used?). Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-30183) Openstack Cloud Plugin does not wait until the user data script completes
Title: Message Title Andrew Grimberg commented on JENKINS-30183 Re: Openstack Cloud Plugin does not wait until the user data script completes What we do is have our Jenkins user created as the very last step of the user data scripts. That keeps Jenkins from connecting into the node until it's actually ready. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
