Looks like you will have to look at writing a groovy script, maybe. You could offline it in a groovy script, then there must be some way to get the status of the node. Can probably put that in a 'while' loop until the node is idle. Then do your other work. You can send a groovy script to the script console through http on the rest API. I can shove you in the right direction, but that's about all the more help I will probably be, though I'd be interested in seeing your script when/if you do it. These links should get you started.
https://wiki.jenkins.io/display/JENKINS/Jenkins+Script+Console https://stackoverflow.com/questions/37227562/how-to-know-whether-the-jenkins-build-executors-are-idle-or-not-in-jenkins-using https://wiki.jenkins.io/display/JENKINS/Add+a+new+label+to+slaves+meeting+a+condition https://wiki.jenkins.io/display/JENKINS/Display+Information+About+Nodes https://wiki.jenkins.io/display/JENKINS/Monitor+and+Restart+Offline+Slaves https://wiki.jenkins.io/display/JENKINS/Wipe+workspaces+for+a+set+of+jobs+on+all+nodes You could look into the rest API, too. You can offline the node in the api: http://jenkins/computer/${NODE}/toggleOffline?offlineMessage=Taken_down I can't find a way to see the executor status that works. I would think http://jenkins/computer/${NODE}/api/json?pretty=true would do it, but in a quick test, it didn't seem to show different if the executors were busy or not. Let me know what you come up with. On Friday, October 27, 2017 at 5:29:00 AM UTC-6, Tomasz Chmielewski wrote: > > Unfortunately it doesn't work this way. If there is a long running job, > "wait-node-offline" will not wait for it to finish. > > # time java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth > user:pass offline-node jenkins-node > > real 0m1.249s > user 0m0.834s > sys 0m0.049s > > > At this point, I can see the node is set to offline in Build Executor > Status in Jenkins webgui, and the job which was running before starting the > command continues to run. > > > # time java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth > user:pass wait-node-offline jenkins-node > > real 0m1.949s > user 0m0.817s > sys 0m0.061s > > > At this point, I can still see the node is set to offline in Build > Executor Status in Jenkins webgui, and the job which was running before > starting the command continues to run. > > > So yeah, that sucks, because there doesn't seem to be a good way to > offline a jenkins node gracefully. > > > > On Friday, October 27, 2017 at 2:47:37 PM UTC+9, Robert Hales wrote: >> >> I think wait-offline-node is supposed to just wait until the node is >> actually offline. So you are supposed to run offline-node, then in your >> script you can wait-offline-node before you do whatever what you wanted to >> do after the node was offline. I would assume the node is not configured >> offline until there are no jobs running on it, but that may be a bad >> assumption. >> >> Try kicking off a long-running job (with a sh"sleep 500") and then try to >> offline the node and then run wait-offline-node to see if it shows it >> offline immediately, or if it waits until the job finishes. >> >> If it isn't actually waiting for the jobs to finish, well, that sucks. >> But the wait-offline-node is still probably supposed to wait for a node to >> go offline. There may be other things that cause the node to take a long >> time to go offline. >> >> >> On Thursday, October 26, 2017 at 11:03:04 PM UTC-6, Tomasz Chmielewski >> wrote: >>> >>> Actually, I still didn't find a solution to my problem. >>> >>> 1) I don't understand the purpose of "wait-node-offline" >>> >>> java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass >>> wait-node-offline some-node >>> >>> This will never exit, unless I use "offline-node" first. But >>> "offline-node" exits immediately. >>> >>> To be precise: actually "wait-node-offline" will crash after some time >>> and will never exit if used through a reverse proxy like nginx. But it will >>> never exit if connecting to jenkins directly. >>> >>> >>> 2) "offline-node" exits immediately and the node is set to "offline": >>> >>> java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass >>> offline-node some-node >>> >>> The builds which were running will continue to run until finished >>> (existing builds are not interrupted) - this is expected. >>> >>> >>> 3) So my reasoning is that I need some kind of: >>> >>> java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass >>> list-jobs some-view >>> >>> But only showing the builds which are currently running (for a given >>> view, or for a given node). Then parse it, and decide if the node is really >>> offline (as in, not accepting any new builds, and not running any builds >>> right now) or not. >>> >>> >>> Unfortunately, I don't see such an option in >>> https://jenkins.my-domain/cli/ >>> >>> >>> >>> On Friday, October 27, 2017 at 7:25:43 AM UTC+9, Devin Nusbaum wrote: >>>> >>>> Great, thanks for confirming! I’ll update the docs. >>>> >>>> Thanks, >>>> Devin >>>> >>>> On Oct 26, 2017, at 6:06 PM, 'Tomasz Chmielewski' via Jenkins Users < >>>> [email protected]> wrote: >>>> >>>> That was it! Adding proxy_buffering off helped. >>>> >>>> For reference, here is my SSL vhost definition: >>>> >>>> server { >>>> >>>> listen 443 ssl; >>>> >>>> server_name jenkins.my-domain; >>>> >>>> ssl_certificate ssl/my-domain.crt; >>>> ssl_certificate_key ssl/my-domain.key; >>>> ssl_dhparam ssl/dhparam-2048.pem; >>>> add_header Strict-Transport-Security "max-age=31536000; >>>> includeSubDomains"; >>>> add_header X-Frame-Options SAMEORIGIN; >>>> >>>> access_log /var/log/nginx/jenkins.my-domain.access.log; >>>> error_log /var/log/nginx/jenkins.my-domain.error.log; >>>> >>>> location / { >>>> proxy_set_header Host $host:$server_port; >>>> proxy_set_header X-Real-IP $remote_addr; >>>> proxy_set_header X-Forwarded-For >>>> $proxy_add_x_forwarded_for; >>>> proxy_set_header X-Forwarded-Proto $scheme; >>>> proxy_redirect http:// https://; >>>> proxy_pass http://jenkins; >>>> >>>> # Required for new HTTP-based CLI >>>> proxy_http_version 1.1; >>>> proxy_request_buffering off; >>>> proxy_buffering off; >>>> >>>> # workaround for >>>> https://issues.jenkins-ci.org/browse/JENKINS-45651 >>>> # not used for this installation >>>> #add_header 'X-SSH-Endpoint' 'jenkins.my-domain:22' always; >>>> } >>>> } >>>> >>>> >>>> On Friday, October 27, 2017 at 6:56:55 AM UTC+9, Devin Nusbaum wrote: >>>>> >>>>> Some comments on https://issues.jenkins-ci.org/browse/JENKINS-43666 >>>>> suggest >>>>> that proxy_buffering off; is needed for SSL connections. (I was also >>>>> testing nginx reverse proxies in the past and have that setting in my SSL >>>>> config; might have forgotten to update the SSL docs with my findings) Try >>>>> setting that option, and if it works I can update the SSL section of the >>>>> wiki page. >>>>> >>>>> On Oct 26, 2017, at 5:43 PM, 'Tomasz Chmielewski' via Jenkins Users < >>>>> jenkins...@googlegroups. <http://googlegroups.com/>com >>>>> <http://googlegroups.com/>> wrote: >>>>> >>>>> nginx vhost is almost the exact copy of the vhost on >>>>> https://wiki.jenkins.io/display/JENKINS/Jenkins+behind+an+NGinX+reverse+proxy: >>>>> >>>>> upstream jenkins { >>>>> server 127.0.0.1:8080 fail_timeout=0; >>>>> } >>>>> >>>>> server { >>>>> >>>>> listen 80; >>>>> server_name jenkins.my-domain; >>>>> >>>>> add_header X-Frame-Options SAMEORIGIN; >>>>> include /etc/nginx/release.conf; >>>>> >>>>> access_log /var/log/nginx/redirects-access.log vhosts; >>>>> error_log /var/log/nginx/redirects-error.log; >>>>> >>>>> rewrite ^ https://$host$request_uri? permanent; >>>>> } >>>>> >>>>> >>>>> server { >>>>> >>>>> listen 443 ssl; >>>>> >>>>> server_name jenkins.my-domain; >>>>> >>>>> ssl_certificate ssl/my-domain.crt; >>>>> ssl_certificate_key ssl/my-domain.key; >>>>> ssl_dhparam ssl/dhparam-2048.pem; >>>>> add_header Strict-Transport-Security "max-age=31536000; >>>>> includeSubDomains"; >>>>> add_header X-Frame-Options SAMEORIGIN; >>>>> >>>>> access_log /var/log/nginx/jenkins.my-domain.access.log; >>>>> error_log /var/log/nginx/jenkins.my-domain.error.log; >>>>> >>>>> location / { >>>>> proxy_set_header Host $host:$server_port; >>>>> proxy_set_header X-Real-IP $remote_addr; >>>>> proxy_set_header X-Forwarded-For >>>>> $proxy_add_x_forwarded_for; >>>>> proxy_set_header X-Forwarded-Proto $scheme; >>>>> proxy_redirect http:// https://; >>>>> proxy_pass http://jenkins; >>>>> # Required for new HTTP-based CLI >>>>> proxy_http_version 1.1; >>>>> proxy_request_buffering off; >>>>> # workaround for >>>>> https://issues.jenkins-ci.org/browse/JENKINS-45651 >>>>> add_header 'X-SSH-Endpoint' 'jenkins.my-domain:22' always; >>>>> } >>>>> } >>>>> >>>>> >>>>> So either I'm blind, or the documentation is somehow wrong? >>>>> >>>>> And indeed, I can see "java.io.IOException: HTTP full-duplex channel >>>>> timeout" in jenkins log. >>>>> >>>>> This one indeed works: >>>>> >>>>> java -jar jenkins-cli.jar -s http://localhost:8080 >>>>> <https://jenkins-url/> -auth user:pass help offline-node >>>>> >>>>> But since I need to execute it from remote, I'd rather connect to >>>>> https://jenkins.my-domain >>>>> >>>>> >>>>> On Friday, October 27, 2017 at 6:26:39 AM UTC+9, Devin Nusbaum wrote: >>>>>> >>>>>> Make sure to follow >>>>>> https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Nginx if >>>>>> Nginx is configured as a a reverse proxy. >>>>>> >>>>>> Notably proxy_http_version 1.1; and proxy_request_buffering off; are >>>>>> required for your version of Jenkins. (If your Jenkins logs at the time >>>>>> you >>>>>> try to connect via CLI have errors that say something to the effect >>>>>> of “Full-duplex channel timeout” then I expect those settings to fix it.) >>>>>> >>>>>> On Oct 26, 2017, at 5:18 PM, 'Tomasz Chmielewski' via Jenkins Users < >>>>>> jenkins...@googlegroups. <http://googlegroups.com/>com >>>>>> <http://googlegroups.com/>> wrote: >>>>>> >>>>>> Except... it doesn't seem to work. >>>>>> >>>>>> $ java -jar jenkins-cli.jar -s https://jenkins-url -auth user:pass >>>>>> help offline-node >>>>>> $ echo $? >>>>>> 255 >>>>>> >>>>>> In nginx log: >>>>>> >>>>>> 10.11.0.8 - user [26/Oct/2017:21:11:51 +0000] "GET / HTTP/1.1" 200 >>>>>> 150393 "-" "Java/1.8.0_131" >>>>>> 10.11.0.8 - user [26/Oct/2017:21:11:52 +0000] "GET >>>>>> /crumbIssuer/api/xml/?xpath=concat(//crumbRequestField,\x22:\x22,//crumb) >>>>>> >>>>>> HTTP/1.1" 404 335 "-" "Java/1.8.0_131" >>>>>> 10.11.0.8 - user [26/Oct/2017:21:12:07 +0000] "POST >>>>>> /cli?remoting=false HTTP/1.1" 200 11 "-" "Java/1.8.0_131" >>>>>> 10.11.0.8 - user [26/Oct/2017:21:12:07 +0000] "POST >>>>>> /cli?remoting=false HTTP/1.1" 500 13912 "-" "Java/1.8.0_131" >>>>>> >>>>>> How do I debug this? >>>>>> >>>>>> >>>>>> >>>>>> On Friday, October 27, 2017 at 6:07:03 AM UTC+9, Tomasz Chmielewski >>>>>> wrote: >>>>>>> >>>>>>> Got it, thanks: >>>>>>> >>>>>>> https://wiki.jenkins.io/display/JENKINS/Jenkins+CLI >>>>>>> >>>>>>> On Friday, October 27, 2017 at 5:57:18 AM UTC+9, Robert Hales wrote: >>>>>>>> >>>>>>>> You have to use the Jenkins CLI. I guess that can be a bit >>>>>>>> confusing. It isn't a script available to run at the command line. >>>>>>>> Jenkins >>>>>>>> has their own CLI. If you google for it, you will find the details >>>>>>>> pretty >>>>>>>> easily. >>>>>>>> >>>>>>>> On Thursday, October 26, 2017 at 2:55:07 PM UTC-6, Tomasz >>>>>>>> Chmielewski wrote: >>>>>>>>> >>>>>>>>> Hmm, where do I find "offline-node" command? >>>>>>>>> >>>>>>>>> root@jenkins:~# dpkg -L jenkins >>>>>>>>> /. >>>>>>>>> /usr >>>>>>>>> /usr/share >>>>>>>>> /usr/share/doc >>>>>>>>> /usr/share/doc/jenkins >>>>>>>>> /usr/share/doc/jenkins/changelog.gz >>>>>>>>> /usr/share/doc/jenkins/copyright >>>>>>>>> /usr/share/jenkins >>>>>>>>> /usr/share/jenkins/jenkins.war >>>>>>>>> /etc >>>>>>>>> /etc/logrotate.d >>>>>>>>> /etc/logrotate.d/jenkins >>>>>>>>> /etc/default >>>>>>>>> /etc/default/jenkins >>>>>>>>> /etc/init.d >>>>>>>>> /etc/init.d/jenkins >>>>>>>>> /var >>>>>>>>> /var/cache >>>>>>>>> /var/cache/jenkins >>>>>>>>> /var/lib >>>>>>>>> /var/lib/jenkins >>>>>>>>> /var/log >>>>>>>>> /var/log/jenkins >>>>>>>>> >>>>>>>>> root@jenkins:~# find / -name offline-node >>>>>>>>> >>>>>>>>> root@jenkins:~# >>>>>>>>> >>>>>>>>> root@jenkins:~# dpkg -l | grep jenkins >>>>>>>>> ii jenkins 2.73.2 (...) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Friday, October 27, 2017 at 12:21:17 AM UTC+9, Robert Hales >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> In the CLI, use the 'offline-node' command. Another useful >>>>>>>>>> command in what it looks like you want to do might be >>>>>>>>>> "wait-offline-node". >>>>>>>>>> >>>>>>>>>> You could also create a groovy script to do it and run that from >>>>>>>>>> the REST API. >>>>>>>>>> >>>>>>>>>> On Thursday, October 26, 2017 at 3:35:29 AM UTC-6, Tomasz >>>>>>>>>> Chmielewski wrote: >>>>>>>>>>> >>>>>>>>>>> Is there a CLI/scripted way to stop scheduling any new builds on >>>>>>>>>>> a given node? >>>>>>>>>>> >>>>>>>>>>> Basically, any builds currently running on a given node should >>>>>>>>>>> continue to run until they are finished -- and no new builds should >>>>>>>>>>> be >>>>>>>>>>> started. >>>>>>>>>>> >>>>>>>>>>> Think of "retiring" a node, and replacing it with a new one -- >>>>>>>>>>> but allowing any existing jobs to finish gracefully. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Tomasz Chmielewski >>>>>>>>>>> https://lxadm.com >>>>>>>>>>> >>>>>>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Jenkins Users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/jenkinsci-users/a19302b1-6ed1-44bb-b65b-28868a64708b%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/jenkinsci-users/a19302b1-6ed1-44bb-b65b-28868a64708b%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Jenkins Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/jenkinsci-users/476c0ec2-753f-45bd-944b-2f9dcf60deae%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/jenkinsci-users/476c0ec2-753f-45bd-944b-2f9dcf60deae%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-users/cb454333-dddf-4186-ac97-66550f856454%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/jenkinsci-users/cb454333-dddf-4186-ac97-66550f856454%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/4e8dcbbe-225b-4d59-b021-e196a3e94a63%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
