[JIRA] (JENKINS-59137) Connectivity issues with ec2 slaves

[email protected] (JIRA) Thu, 29 Aug 2019 00:16:31 -0700

Title: Message Title

Issue Type:	Bug
Assignee:	FABRIZIO MANFREDI
Components:	ec2-plugin
Created:	2019-08-29 07:15
Environment:	Jenkins: 2.191 ec2-plugin: 1.44.1 ec2-plugin: 1.45
Priority:	Blocker
Reporter:	Toni Van de Voorde

Hi guys,

Since yesterday (28/08/19) we are facing a strange issue with our builds on jenkins. Either our builds are failing immediately, either the builds perform all steps but fail at the end. And each time it seems to be related with a connectivity issue with the slaves. I tried to figure out what the issue is but the it's too cryptic :/

The funny part is that I don't remember upgrading anything. So I did upgrade to latest version (jenkins and plugin) but the problem persists.

Hopefully the following stacktraces tell you more.

This is a stacktrace I found in the jenkins log server:

 
                                                                2019-08-29 06:28:05.986+0000 [id=257] WARNING o.j.p.w.s.s.ExecutorStepExecution$PlaceholderTask$2#run: failed to shut down 75c2e395-96e2-4e6d-85d4-be32482bfdec2019-08-29 06:28:05.986+0000 [id=257] WARNING o.j.p.w.s.s.ExecutorStepExecution$PlaceholderTask$2#run: failed to shut down 75c2e395-96e2-4e6d-85d4-be32482bfdecAlso:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxxxxxxxxxxx) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) at hudson.remoting.Channel.call(Channel.java:955) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1086) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$2.run(ExecutorStepExecution.java:738) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)java.lang.NoClassDefFoundError: Could not initialize class hudson.slaves.SlaveComputer at hudson.util.ProcessTree.get(ProcessTree.java:414) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)Caused: java.io.IOException: Remote call on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxxxxxxxxxxx) failed at hudson.remoting.Channel.call(Channel.java:961) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1086) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$2.run(ExecutorStepExecution.java:738) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

This is the stacktrace I found in a job execution failing in the very beginning:

 
                                                                Started by user tonivdvStarted by user tonivdvObtained Jenkinsfile from 5c95f59dd9dee0c3c77979501d40bfcba1f6ae53Running in Durability level: MAX_SURVIVABILITY[Pipeline] Start of Pipeline[Pipeline] stage[Pipeline] { (php 5.6 and php 7)[Pipeline] parallel[Pipeline] { (Branch: php 5.6)[Pipeline] { (Branch: php 7)[Pipeline] stage[Pipeline] { (php 5.6)[Pipeline] stage[Pipeline] { (php 7)[Pipeline] node[Pipeline] nodeRunning on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxxxxxxxxxxx) in /home/ec2-user/workspace/adsdaq_portal_master[Pipeline] {[Pipeline] checkout[Pipeline] }[Pipeline] // node[Pipeline] }[Pipeline] // stage[Pipeline] }Failed in branch php 5.6Running on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxxxxxxxxxxx) in /home/ec2-user/workspace/adsdaq/portal/master[Pipeline] {[Pipeline] checkout[Pipeline] }[Pipeline] // node[Pipeline] }[Pipeline] // stage[Pipeline] }Failed in branch php 7[Pipeline] // parallel[Pipeline] }[Pipeline] // stage[Pipeline] stage[Pipeline] { (Deploy)Stage "Deploy" skipped due to earlier failure(s)[Pipeline] }[Pipeline] // stage[Pipeline] stage[Pipeline] { (Declarative: Post Actions)[Pipeline] script[Pipeline] {[Pipeline] slackSendSlack Send Pipeline step running, values are - baseUrl: <empty>, teamDomain: adsdaq-team, channel: ci-alerts, color: <empty>, botUser: false, tokenCredentialId: slack-token, notifyCommitters: false, iconEmoji: <empty>, username: <empty>[Pipeline] }[Pipeline] // script[Pipeline] }[Pipeline] // stage[Pipeline] End of Pipeline[Bitbucket] Notifying commit build result[Bitbucket] Build result notifiedAlso:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxxxxxxxxxxx) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) at hudson.remoting.Channel.call(Channel.java:955) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy8 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739) at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:167) at hudson.remoting.Channel.export(Channel.java:768) at hudson.remoting.Channel.export(Channel.java:731) at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:204) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1230) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at hudson.remoting.UserRequest._serialize(UserRequest.java:263) at hudson.remoting.UserRequest.serialize(UserRequest.java:272) at hudson.remoting.UserRequest.perform(UserRequest.java:222) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)Also:   java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-0.amzn2.x86_64/jre/lib/currency.data (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at java.util.Currency$1.run(Currency.java:221) Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxxxxxxxxxxx) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) at hudson.remoting.Channel.call(Channel.java:955) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) Caused: java.lang.InternalError at java.util.Currency$1.run(Currency.java:247) at java.util.Currency$1.run(Currency.java:214) at java.security.AccessController.doPrivileged(Native Method) at java.util.Currency.<clinit>(Currency.java:214) at java.text.DecimalFormatSymbols.initialize(DecimalFormatSymbols.java:638) at java.text.DecimalFormatSymbols.<init>(DecimalFormatSymbols.java:113) at sun.util.locale.provider.DecimalFormatSymbolsProviderImpl.getInstance(DecimalFormatSymbolsProviderImpl.java:85) at java.text.DecimalFormatSymbols.getInstance(DecimalFormatSymbols.java:180) at sun.util.locale.provider.NumberFormatProviderImpl.getInstance(NumberFormatProviderImpl.java:178) at sun.util.locale.provider.NumberFormatProviderImpl.getIntegerInstance(NumberFormatProviderImpl.java:131) at java.text.NumberFormat.getInstance(NumberFormat.java:884) at java.text.NumberFormat.getInstance(NumberFormat.java:861) at java.text.NumberFormat.getIntegerInstance(NumberFormat.java:508) at java.text.SimpleDateFormat.initialize(SimpleDateFormat.java:639) at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:605) at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:580) at com.thoughtworks.xstream.core.JVM.<clinit>(JVM.java:147) at com.thoughtworks.xstream.core.util.CompositeClassLoader.<clinit>(CompositeClassLoader.java:56) at com.thoughtworks.xstream.XStream.<init>(XStream.java:411) at com.thoughtworks.xstream.XStream.<init>(XStream.java:378) at hudson.util.XStream2.<init>(XStream2.java:113) at hudson.ProxyConfiguration.<clinit>(ProxyConfiguration.java:344) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.sun.proxy.$Proxy8.<clinit>(Unknown Source) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739) at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:167) at hudson.remoting.Channel.export(Channel.java:768) at hudson.remoting.Channel.export(Channel.java:731) at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:204) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1230) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at hudson.remoting.UserRequest._serialize(UserRequest.java:263) at hudson.remoting.UserRequest.serialize(UserRequest.java:272) at hudson.remoting.UserRequest.perform(UserRequest.java:222) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) Caused: java.io.IOException: Remote call on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxxxxxxxxxxx) failedCaused: java.io.IOException: Remote call on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxxxxxxxxxxx) failed at hudson.remoting.Channel.call(Channel.java:961) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Finished: FAILURE

And finally the stacktrace we get after successful execution of all pipeline steps:

 
                                                                Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxxxxxxxxxxx)Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxxxxxxxxxxx) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) at hudson.remoting.Channel.call(Channel.java:955) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy8 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739) at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:167) at hudson.remoting.Channel.export(Channel.java:768) at hudson.remoting.Channel.export(Channel.java:731) at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:204) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1230) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at hudson.remoting.UserRequest._serialize(UserRequest.java:263) at hudson.remoting.UserRequest.serialize(UserRequest.java:272) at hudson.remoting.UserRequest.perform(UserRequest.java:222) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)Caused: java.io.IOException: Remote call on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxxxxxxxxxxx) failed at hudson.remoting.Channel.call(Channel.java:961) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Add Comment

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201569.1567062958000.2499.1567062960272%40Atlassian.JIRA.

[JIRA] (JENKINS-59137) Connectivity issues with ec2 slaves

Reply via email to