[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Ahmed Kamel commented on JENKINS-55527 Re: Builds fail randomly when running sh in container For whoever stumbles onto this thread. Our security team reached out to Twistlock to try and figure out the root cause of this issue. They told us they are aware of the issue and are working on updates. In the meantime here is a nice blog post that explains the issue and how it was found https://www.twistlock.com/2018/12/04/advanced-runc-debugging-fun-profit/ Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Ahmed Kamel commented on JENKINS-55527 Re: Builds fail randomly when running sh in container Awesome thank you Andy Powell for tracking this down. We have disabled it and saw the error count drop significantly. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Carlos Sanchez closed an issue as Not A Defect Thanks for figuring it out Jenkins / JENKINS-55527 Builds fail randomly when running sh in container Change By: Carlos Sanchez Status: Open Closed Resolution: Not A Defect Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Andy Powell commented on JENKINS-55527 Re: Builds fail randomly when running sh in container yes it was Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Ahmed Kamel commented on JENKINS-55527 Re: Builds fail randomly when running sh in container Go it. We have a very similar setup here. Feel free not to comment but was this 3rd party tool Twistlock by any chance? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Andy Powell commented on JENKINS-55527 Re: Builds fail randomly when running sh in container Ahmed Kamel it was not GCP, but the 3rd party product that was causing the issue. GKE is running Jenkins after we turned off the 3rd party service. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Ahmed Kamel commented on JENKINS-55527 Re: Builds fail randomly when running sh in container Andy Powell are you talking about a GCP service or was it a 3rd party security scanner that was causing the issue? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Andy Powell commented on JENKINS-55527 Re: Builds fail randomly when running sh in container Update: We were able to isolate this to a security scanner within our GKE cluster. Turning it off made the problems go away. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Ahmed Kamel commented on JENKINS-55527 Re: Builds fail randomly when running sh in container Got it. Thanks for posting these. I'll watch the issue over on github. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Carlos Sanchez commented on JENKINS-55527 Re: Builds fail randomly when running sh in container This looks like https://github.com/moby/moby/issues/31230 and the fix could be in runc v1.0.0-rc6 https://github.com/opencontainers/runc/pull/1916 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Andy Powell commented on JENKINS-55527 Re: Builds fail randomly when running sh in container We are seeing similar errors. rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3559144 to cgroups caused \"failed to write 3559144 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod03460050-19f2-11e9-beb6-42010a8e01b2/f4978b3f515bcf1f942bd0ed21ce084ca2039e6ae56f357870cbbe55517ed151/cgroup.procs: invalid argument\"" command terminated with non-zero exit code: Error executing in Docker Container: 126process apparently never started in /home/jenkins/workspace/le_platform-nodejs-hello_sandbox@tmp/durable-11cdeba1 Jenkins version = 2.150.1, running on GKE Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Ahmed Kamel updated an issue Jenkins / JENKINS-55527 Builds fail randomly when running sh in container Change By: Ahmed Kamel Priority: Major Critical Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Ahmed Kamel updated an issue Jenkins / JENKINS-55527 Builds fail randomly when running sh in container Change By: Ahmed Kamel My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in a container in the pods running the job. Here is the error message I see. ``` [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed... ... ...ERROR: script returned exit code 1 Finished: FAILURE ``` Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Ahmed Kamel updated an issue Jenkins / JENKINS-55527 Builds fail randomly when running sh in container Change By: Ahmed Kamel My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in a container in the pods running the job. Here is the error message I see. [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed... ... ... ERROR: script returned exit code 1 Finished: FAILURE Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Ahmed Kamel updated an issue Jenkins / JENKINS-55527 Builds fail randomly when running sh in container Change By: Ahmed Kamel My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in a container in the pods running the job. Here is the error message I see. ``` [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed... ... ...ERROR: script returned exit code 1 Finished: FAILURE ``` Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Ahmed Kamel updated an issue Jenkins / JENKINS-55527 Builds fail randomly when running sh in container Change By: Ahmed Kamel My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in a container in the cluster pods running the job .Here is the error message I see. [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed.ERROR: script returned exit code 1Finished: FAILURE Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-55527) Builds fail randomly when running sh in container
Title: Message Title Ahmed Kamel created an issue Jenkins / JENKINS-55527 Builds fail randomly when running sh in container Issue Type: Bug Assignee: Carlos Sanchez Components: kubernetes-plugin Created: 2019-01-11 01:05 Environment: Running jenkins in a Kubernetes cluster on GCP Labels: bug Priority: Major Reporter: Ahmed Kamel My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in the cluster. Here is the error message I see. [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed... ... ...ERROR: script returned exit code 1 Finished: FAILURE
