[JIRA] (JENKINS-37719) Build cannot be interrupted if `docker stop` hangs

[jgl...@cloudbees.com (JIRA)]

Title: Message Title

Jesse Glick created an issue   Jenkins / JENKINS-37719 Build cannot be interrupted if `docker stop` hangs Issue Type: Bug Assignee: Jesse Glick Components: docker-workflow-plugin Created: 2016/Aug/26 3:15 PM Labels: robustness threads Priority: Major Reporter: Jesse Glick Cyrille Le Clerc found a problem in a build doing basically docker.image('...').inside { sh '...' } The shell command appeared to complete, but then the build seemed to hang, and did not respond to anything short of a hard kill. A virtual thread dump just showed WorkflowScript on the last line of the closure. A physical thread dump of the master showed the problem: at ... at hudson.Launcher$ProcStarter.join(Launcher.java:388) at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:260) at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:241) at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:238) at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:133) at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:104) at ... On the agent, stream copiers were still active for the docker exec call associated with the shell step, as well as a docker stop call on the container. Indeed process inspection showed that both docker commands were running, and docker ps just hanged: for reasons TBD, the Docker daemon was not responding to requests (though docker version worked). Docker did not work again until sudo service docker restart was run. Was not clear if killing the docker stop process would have resumed the build, since by that point a sequence of kill escalations had already been attempted. The most straightforward fix in the plugin would be to use .start().joinWithTimeout(...) rather than .join() in DockerClient. Thus, a failure to clean up a container due to an outage in the Docker daemon would at worst result in a few minutes' delay followed by a (more or less) comprehensible error. (All Docker commands in the plugin that are expected to take a long time, because they might contact a registry, are run from sh steps in Groovy code so they are durable and cleanly interruptible. Docker commands which we do not need to wait for, like docker exec to run processes in the container, are just launched and the process handle discarded. DockerClient with its blocking join call is only used for commands which under normal conditions should be close to instantaneous: docker run -d after checking for local availability of the image, docker inspect, docker stop, etc. But clearly this assumption is not completely reliable.) Add Comment   This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.