Unfortunately it doesn't work this way. If there is a long running job, "wait-node-offline" will not wait for it to finish.
# time java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass offline-node jenkins-node real 0m1.249s user 0m0.834s sys 0m0.049s At this point, I can see the node is set to offline in Build Executor Status in Jenkins webgui, and the job which was running before starting the command continues to run. # time java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass wait-node-offline jenkins-node real 0m1.949s user 0m0.817s sys 0m0.061s At this point, I can still see the node is set to offline in Build Executor Status in Jenkins webgui, and the job which was running before starting the command continues to run. So yeah, that sucks, because there doesn't seem to be a good way to offline a jenkins node gracefully. On Friday, October 27, 2017 at 2:47:37 PM UTC+9, Robert Hales wrote: > > I think wait-offline-node is supposed to just wait until the node is > actually offline. So you are supposed to run offline-node, then in your > script you can wait-offline-node before you do whatever what you wanted to > do after the node was offline. I would assume the node is not configured > offline until there are no jobs running on it, but that may be a bad > assumption. > > Try kicking off a long-running job (with a sh"sleep 500") and then try to > offline the node and then run wait-offline-node to see if it shows it > offline immediately, or if it waits until the job finishes. > > If it isn't actually waiting for the jobs to finish, well, that sucks. But > the wait-offline-node is still probably supposed to wait for a node to go > offline. There may be other things that cause the node to take a long time > to go offline. > > > On Thursday, October 26, 2017 at 11:03:04 PM UTC-6, Tomasz Chmielewski > wrote: >> >> Actually, I still didn't find a solution to my problem. >> >> 1) I don't understand the purpose of "wait-node-offline" >> >> java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass >> wait-node-offline some-node >> >> This will never exit, unless I use "offline-node" first. But >> "offline-node" exits immediately. >> >> To be precise: actually "wait-node-offline" will crash after some time >> and will never exit if used through a reverse proxy like nginx. But it will >> never exit if connecting to jenkins directly. >> >> >> 2) "offline-node" exits immediately and the node is set to "offline": >> >> java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass >> offline-node some-node >> >> The builds which were running will continue to run until finished >> (existing builds are not interrupted) - this is expected. >> >> >> 3) So my reasoning is that I need some kind of: >> >> java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass >> list-jobs some-view >> >> But only showing the builds which are currently running (for a given >> view, or for a given node). Then parse it, and decide if the node is really >> offline (as in, not accepting any new builds, and not running any builds >> right now) or not. >> >> >> Unfortunately, I don't see such an option in >> https://jenkins.my-domain/cli/ >> >> >> >> On Friday, October 27, 2017 at 7:25:43 AM UTC+9, Devin Nusbaum wrote: >>> >>> Great, thanks for confirming! I’ll update the docs. >>> >>> Thanks, >>> Devin >>> >>> On Oct 26, 2017, at 6:06 PM, 'Tomasz Chmielewski' via Jenkins Users < >>> [email protected]> wrote: >>> >>> That was it! Adding proxy_buffering off helped. >>> >>> For reference, here is my SSL vhost definition: >>> >>> server { >>> >>> listen 443 ssl; >>> >>> server_name jenkins.my-domain; >>> >>> ssl_certificate ssl/my-domain.crt; >>> ssl_certificate_key ssl/my-domain.key; >>> ssl_dhparam ssl/dhparam-2048.pem; >>> add_header Strict-Transport-Security "max-age=31536000; >>> includeSubDomains"; >>> add_header X-Frame-Options SAMEORIGIN; >>> >>> access_log /var/log/nginx/jenkins.my-domain.access.log; >>> error_log /var/log/nginx/jenkins.my-domain.error.log; >>> >>> location / { >>> proxy_set_header Host $host:$server_port; >>> proxy_set_header X-Real-IP $remote_addr; >>> proxy_set_header X-Forwarded-For >>> $proxy_add_x_forwarded_for; >>> proxy_set_header X-Forwarded-Proto $scheme; >>> proxy_redirect http:// https://; >>> proxy_pass http://jenkins; >>> >>> # Required for new HTTP-based CLI >>> proxy_http_version 1.1; >>> proxy_request_buffering off; >>> proxy_buffering off; >>> >>> # workaround for >>> https://issues.jenkins-ci.org/browse/JENKINS-45651 >>> # not used for this installation >>> #add_header 'X-SSH-Endpoint' 'jenkins.my-domain:22' always; >>> } >>> } >>> >>> >>> On Friday, October 27, 2017 at 6:56:55 AM UTC+9, Devin Nusbaum wrote: >>>> >>>> Some comments on https://issues.jenkins-ci.org/browse/JENKINS-43666 >>>> suggest >>>> that proxy_buffering off; is needed for SSL connections. (I was also >>>> testing nginx reverse proxies in the past and have that setting in my SSL >>>> config; might have forgotten to update the SSL docs with my findings) Try >>>> setting that option, and if it works I can update the SSL section of the >>>> wiki page. >>>> >>>> On Oct 26, 2017, at 5:43 PM, 'Tomasz Chmielewski' via Jenkins Users < >>>> jenkins...@googlegroups. <http://googlegroups.com/>com >>>> <http://googlegroups.com/>> wrote: >>>> >>>> nginx vhost is almost the exact copy of the vhost on >>>> https://wiki.jenkins.io/display/JENKINS/Jenkins+behind+an+NGinX+reverse+proxy: >>>> >>>> upstream jenkins { >>>> server 127.0.0.1:8080 fail_timeout=0; >>>> } >>>> >>>> server { >>>> >>>> listen 80; >>>> server_name jenkins.my-domain; >>>> >>>> add_header X-Frame-Options SAMEORIGIN; >>>> include /etc/nginx/release.conf; >>>> >>>> access_log /var/log/nginx/redirects-access.log vhosts; >>>> error_log /var/log/nginx/redirects-error.log; >>>> >>>> rewrite ^ https://$host$request_uri? permanent; >>>> } >>>> >>>> >>>> server { >>>> >>>> listen 443 ssl; >>>> >>>> server_name jenkins.my-domain; >>>> >>>> ssl_certificate ssl/my-domain.crt; >>>> ssl_certificate_key ssl/my-domain.key; >>>> ssl_dhparam ssl/dhparam-2048.pem; >>>> add_header Strict-Transport-Security "max-age=31536000; >>>> includeSubDomains"; >>>> add_header X-Frame-Options SAMEORIGIN; >>>> >>>> access_log /var/log/nginx/jenkins.my-domain.access.log; >>>> error_log /var/log/nginx/jenkins.my-domain.error.log; >>>> >>>> location / { >>>> proxy_set_header Host $host:$server_port; >>>> proxy_set_header X-Real-IP $remote_addr; >>>> proxy_set_header X-Forwarded-For >>>> $proxy_add_x_forwarded_for; >>>> proxy_set_header X-Forwarded-Proto $scheme; >>>> proxy_redirect http:// https://; >>>> proxy_pass http://jenkins; >>>> # Required for new HTTP-based CLI >>>> proxy_http_version 1.1; >>>> proxy_request_buffering off; >>>> # workaround for >>>> https://issues.jenkins-ci.org/browse/JENKINS-45651 >>>> add_header 'X-SSH-Endpoint' 'jenkins.my-domain:22' always; >>>> } >>>> } >>>> >>>> >>>> So either I'm blind, or the documentation is somehow wrong? >>>> >>>> And indeed, I can see "java.io.IOException: HTTP full-duplex channel >>>> timeout" in jenkins log. >>>> >>>> This one indeed works: >>>> >>>> java -jar jenkins-cli.jar -s http://localhost:8080 >>>> <https://jenkins-url/> -auth user:pass help offline-node >>>> >>>> But since I need to execute it from remote, I'd rather connect to >>>> https://jenkins.my-domain >>>> >>>> >>>> On Friday, October 27, 2017 at 6:26:39 AM UTC+9, Devin Nusbaum wrote: >>>>> >>>>> Make sure to follow >>>>> https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Nginx if >>>>> Nginx is configured as a a reverse proxy. >>>>> >>>>> Notably proxy_http_version 1.1; and proxy_request_buffering off; are >>>>> required for your version of Jenkins. (If your Jenkins logs at the time >>>>> you >>>>> try to connect via CLI have errors that say something to the effect >>>>> of “Full-duplex channel timeout” then I expect those settings to fix it.) >>>>> >>>>> On Oct 26, 2017, at 5:18 PM, 'Tomasz Chmielewski' via Jenkins Users < >>>>> jenkins...@googlegroups. <http://googlegroups.com/>com >>>>> <http://googlegroups.com/>> wrote: >>>>> >>>>> Except... it doesn't seem to work. >>>>> >>>>> $ java -jar jenkins-cli.jar -s https://jenkins-url -auth user:pass >>>>> help offline-node >>>>> $ echo $? >>>>> 255 >>>>> >>>>> In nginx log: >>>>> >>>>> 10.11.0.8 - user [26/Oct/2017:21:11:51 +0000] "GET / HTTP/1.1" 200 >>>>> 150393 "-" "Java/1.8.0_131" >>>>> 10.11.0.8 - user [26/Oct/2017:21:11:52 +0000] "GET >>>>> /crumbIssuer/api/xml/?xpath=concat(//crumbRequestField,\x22:\x22,//crumb) >>>>> HTTP/1.1" 404 335 "-" "Java/1.8.0_131" >>>>> 10.11.0.8 - user [26/Oct/2017:21:12:07 +0000] "POST >>>>> /cli?remoting=false HTTP/1.1" 200 11 "-" "Java/1.8.0_131" >>>>> 10.11.0.8 - user [26/Oct/2017:21:12:07 +0000] "POST >>>>> /cli?remoting=false HTTP/1.1" 500 13912 "-" "Java/1.8.0_131" >>>>> >>>>> How do I debug this? >>>>> >>>>> >>>>> >>>>> On Friday, October 27, 2017 at 6:07:03 AM UTC+9, Tomasz Chmielewski >>>>> wrote: >>>>>> >>>>>> Got it, thanks: >>>>>> >>>>>> https://wiki.jenkins.io/display/JENKINS/Jenkins+CLI >>>>>> >>>>>> On Friday, October 27, 2017 at 5:57:18 AM UTC+9, Robert Hales wrote: >>>>>>> >>>>>>> You have to use the Jenkins CLI. I guess that can be a bit >>>>>>> confusing. It isn't a script available to run at the command line. >>>>>>> Jenkins >>>>>>> has their own CLI. If you google for it, you will find the details >>>>>>> pretty >>>>>>> easily. >>>>>>> >>>>>>> On Thursday, October 26, 2017 at 2:55:07 PM UTC-6, Tomasz >>>>>>> Chmielewski wrote: >>>>>>>> >>>>>>>> Hmm, where do I find "offline-node" command? >>>>>>>> >>>>>>>> root@jenkins:~# dpkg -L jenkins >>>>>>>> /. >>>>>>>> /usr >>>>>>>> /usr/share >>>>>>>> /usr/share/doc >>>>>>>> /usr/share/doc/jenkins >>>>>>>> /usr/share/doc/jenkins/changelog.gz >>>>>>>> /usr/share/doc/jenkins/copyright >>>>>>>> /usr/share/jenkins >>>>>>>> /usr/share/jenkins/jenkins.war >>>>>>>> /etc >>>>>>>> /etc/logrotate.d >>>>>>>> /etc/logrotate.d/jenkins >>>>>>>> /etc/default >>>>>>>> /etc/default/jenkins >>>>>>>> /etc/init.d >>>>>>>> /etc/init.d/jenkins >>>>>>>> /var >>>>>>>> /var/cache >>>>>>>> /var/cache/jenkins >>>>>>>> /var/lib >>>>>>>> /var/lib/jenkins >>>>>>>> /var/log >>>>>>>> /var/log/jenkins >>>>>>>> >>>>>>>> root@jenkins:~# find / -name offline-node >>>>>>>> >>>>>>>> root@jenkins:~# >>>>>>>> >>>>>>>> root@jenkins:~# dpkg -l | grep jenkins >>>>>>>> ii jenkins 2.73.2 (...) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Friday, October 27, 2017 at 12:21:17 AM UTC+9, Robert Hales >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> In the CLI, use the 'offline-node' command. Another useful command >>>>>>>>> in what it looks like you want to do might be "wait-offline-node". >>>>>>>>> >>>>>>>>> You could also create a groovy script to do it and run that from >>>>>>>>> the REST API. >>>>>>>>> >>>>>>>>> On Thursday, October 26, 2017 at 3:35:29 AM UTC-6, Tomasz >>>>>>>>> Chmielewski wrote: >>>>>>>>>> >>>>>>>>>> Is there a CLI/scripted way to stop scheduling any new builds on >>>>>>>>>> a given node? >>>>>>>>>> >>>>>>>>>> Basically, any builds currently running on a given node should >>>>>>>>>> continue to run until they are finished -- and no new builds should >>>>>>>>>> be >>>>>>>>>> started. >>>>>>>>>> >>>>>>>>>> Think of "retiring" a node, and replacing it with a new one -- >>>>>>>>>> but allowing any existing jobs to finish gracefully. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Tomasz Chmielewski >>>>>>>>>> https://lxadm.com >>>>>>>>>> >>>>>>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Jenkins Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/jenkinsci-users/a19302b1-6ed1-44bb-b65b-28868a64708b%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/jenkinsci-users/a19302b1-6ed1-44bb-b65b-28868a64708b%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-users/476c0ec2-753f-45bd-944b-2f9dcf60deae%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/jenkinsci-users/476c0ec2-753f-45bd-944b-2f9dcf60deae%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/cb454333-dddf-4186-ac97-66550f856454%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/jenkinsci-users/cb454333-dddf-4186-ac97-66550f856454%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/888e579a-7b9d-4255-87c7-31e78c28f614%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
