Hello all, I was able to solve my problem. If you are interested find the full explanation on the ticket itself: https://issues.jenkins-ci.org/browse/JENKINS-59137?focusedCommentId=374533&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-374533
Cheers On Thursday, August 29, 2019 at 1:05:20 PM UTC+2, Toni Van de Voorde wrote: > > Hey everyone, > > I suddenly have issues with my master slave setup on jenkins. It seems > there is a connectivity issue each time a job wants to run on it. I already > opened a ticket on Jira for ec2-plugin ( > https://issues.jenkins-ci.org/browse/JENKINS-59137) but not knowing if > the problem is really in that plugin, or if they uberhaubt are going to > answer quickly, I'm trying my luck here, in the hope someone already > encountered this. > > As previously mentioned, AFAIK the problem suddenly happened ... I did > upgrade to latest versions when the problem started, but the it didn't > solve it. > > So what's really happening is that whenever a job needs to a run, the > master starts a slave and executes the job ... but the job fails in 2 use > cases: > > *1. The job fails immediately* > > Branch event > Obtained Jenkinsfile from 140cfe60aa7963ef13cb23c18f07bf70ee97d061 > Running in Durability level: MAX_SURVIVABILITY > [Pipeline] Start of Pipeline > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] stage > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] { (php 5.6 and php 7) > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] parallel > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] { (Branch: php 5.6) > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] { (Branch: php 7) > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] stage > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] { (php 5.6) > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] stage > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] { (php 7) > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] node > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] node > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > Running on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx) > <https://ci.adsdaq.eu/computer/EC2%20(jenkins-slave)%20-%20Adsdaq%20Jenkins%20Slave%20(i-0c2875a63156eab4c)/> > in /home/ec2-user/workspace/adsdaq_portal_master > [Pipeline] { > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] checkout > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] } > [Pipeline] // node > [Pipeline] } > [Pipeline] // stage > Running on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx) > <https://ci.adsdaq.eu/computer/EC2%20(jenkins-slave)%20-%20Adsdaq%20Jenkins%20Slave%20(i-0c2875a63156eab4c)/> > in /home/ec2-user/workspace/adsdaq_portal_master > [Pipeline] { > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] } > Failed in branch php 5.6 > [Pipeline] checkout > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] } > [Pipeline] // node > [Pipeline] } > [Pipeline] // stage > [Pipeline] } > Failed in branch php 7 > [Pipeline] // parallel > [Pipeline] } > > > This job above runs in parallel and even executes 2 slaves. It happens > that both fail (like here), but sometimes one passes. > > The additional stack trace we get in this use case is: > > Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 > (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx) > at > hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) > at > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) > at hudson.remoting.Channel.call(Channel.java:955) > at hudson.FilePath.act(FilePath.java:1072) > at hudson.FilePath.act(FilePath.java:1061) > at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) > at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) > at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) > at > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > java.lang.NoClassDefFoundError: Could not initialize class > com.sun.proxy.$Proxy8 > at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739) > at > hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:167) > at hudson.remoting.Channel.export(Channel.java:768) > at hudson.remoting.Channel.export(Channel.java:731) > at > org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:204) > at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1230) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136) > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > at hudson.remoting.UserRequest._serialize(UserRequest.java:263) > at hudson.remoting.UserRequest.serialize(UserRequest.java:272) > at hudson.remoting.UserRequest.perform(UserRequest.java:222) > at hudson.remoting.UserRequest.perform(UserRequest.java:54) > at hudson.remoting.Request$2.run(Request.java:369) > at > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > Also: Also: hudson.remoting.Channel$CallSiteStackTrace: Remote > call to EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx) > at > hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) > at > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) > at hudson.remoting.Channel.call(Channel.java:955) > at hudson.FilePath.act(FilePath.java:1072) > at hudson.FilePath.act(FilePath.java:1061) > at > org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) > at > hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) > at > hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) > at > hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) > at > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > java.lang.NoClassDefFoundError: Could not initialize class > com.sun.proxy.$Proxy8 > at > sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739) > at > hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:167) > at hudson.remoting.Channel.export(Channel.java:768) > at hudson.remoting.Channel.export(Channel.java:731) > at > org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:204) > at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown > Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1230) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136) > at > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > at > hudson.remoting.UserRequest._serialize(UserRequest.java:263) > at hudson.remoting.UserRequest.serialize(UserRequest.java:272) > at hudson.remoting.UserRequest.perform(UserRequest.java:222) > at hudson.remoting.UserRequest.perform(UserRequest.java:54) > at hudson.remoting.Request$2.run(Request.java:369) > at > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > Caused: java.io.IOException: Remote call on EC2 (jenkins-slave) - > Adsdaq Jenkins Slave (i-xxxx) failed > Caused: java.io.IOException: Remote call on EC2 (jenkins-slave) - Adsdaq > Jenkins Slave (i-xxxx) failed > at hudson.remoting.Channel.call(Channel.java:961) > at hudson.FilePath.act(FilePath.java:1072) > at hudson.FilePath.act(FilePath.java:1061) > at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) > at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) > at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) > at > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Finished: FAILURE > > > *2. The job executes but fails when recording the junits* > > Another scenario which happens (randomly) is that the execution starts > correctly, but when all steps were performed, the job fails from the moment > the junits need to be recorded: > > [30;42mOK (715 tests, 2105 assertions) [0m > > Generating code coverage report in Clover XML format ... done > > Generating code coverage report in HTML format ... done > > Remaining indirect deprecation notices (17) > > Other deprecation notices (7) > [Pipeline] step > <https://ci.adsdaq.eu/job/adsdaq/job/print/job/engine/job/PR-258/8/console#> > Publishing Clover coverage report... > Publishing Clover HTML report... > Publishing Clover XML report... > Processing Clover XML report ... > Publishing Clover coverage results... > [Pipeline] junit > <https://ci.adsdaq.eu/job/adsdaq/job/print/job/engine/job/PR-258/8/console#> > Recording test results > Remote call on EC2 (jenkins-slave) - Adsdaq Jenkins Slave > (i-0617d5ac254d28fce) failed > > > The stack trace provide after is: > > java.io.FileNotFoundException: > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-0.amzn2.x86_64/jre/lib/resources.jar > at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:938) > at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:801) > at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:886) > at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:879) > at java.security.AccessController.doPrivileged(Native Method) > at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:878) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050) > Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 > (jenkins-slave) - Adsdaq Jenkins Slave (i-xxx) > at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) > at > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) > at hudson.remoting.Channel.call(Channel.java:955) > at hudson.FilePath.act(FilePath.java:1072) > at hudson.FilePath.act(FilePath.java:1061) > at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114) > at > hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137) > at > hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167) > at > hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52) > at > hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25) > at > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > Caused: java.lang.InternalError > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1052) > at sun.misc.URLClassPath.getResource(URLClassPath.java:249) > at sun.misc.URLClassPath.getResource(URLClassPath.java:302) > at java.lang.ClassLoader.getBootstrapResource(ClassLoader.java:1264) > at java.lang.ClassLoader.getResource(ClassLoader.java:1093) > at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:233) > at > java.util.ResourceBundle$RBClassLoader.getResourceAsStream(ResourceBundle.java:524) > at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2686) > at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2671) > at java.security.AccessController.doPrivileged(Native Method) > at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2670) > at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1510) > at java.util.ResourceBundle.findBundle(ResourceBundle.java:1474) > at java.util.ResourceBundle.findBundle(ResourceBundle.java:1428) > at java.util.ResourceBundle.findBundle(ResourceBundle.java:1428) > at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1370) > at java.util.ResourceBundle.getBundle(ResourceBundle.java:854) > at > com.sun.org.apache.xerces.internal.utils.SecuritySupport$7.run(SecuritySupport.java:169) > at > com.sun.org.apache.xerces.internal.utils.SecuritySupport$7.run(SecuritySupport.java:166) > at java.security.AccessController.doPrivileged(Native Method) > at > com.sun.org.apache.xerces.internal.utils.SecuritySupport.getResourceBundle(SecuritySupport.java:166) > at > com.sun.org.apache.xerces.internal.util.SAXMessageFormatter.formatMessage(SAXMessageFormatter.java:58) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.getProperty(AbstractSAXParser.java:2063) > at > com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.setProperty(SAXParserImpl.java:574) > at org.dom4j.io.SAXHelper.setParserProperty(SAXHelper.java:34) > at org.dom4j.io.SAXReader.configureReader(SAXReader.java:911) > at org.dom4j.io.SAXReader.read(SAXReader.java:463) > at org.dom4j.io.SAXReader.read(SAXReader.java:343) > at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:178) > at hudson.tasks.junit.TestResult.parse(TestResult.java:348) > at hudson.tasks.junit.TestResult.parsePossiblyEmpty(TestResult.java:281) > at hudson.tasks.junit.TestResult.parse(TestResult.java:206) > at hudson.tasks.junit.TestResult.parse(TestResult.java:178) > at hudson.tasks.junit.TestResult.<init>(TestResult.java:143) > at > hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:146) > at > hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:118) > at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3052) > at hudson.remoting.UserRequest.perform(UserRequest.java:211) > at hudson.remoting.UserRequest.perform(UserRequest.java:54) > at hudson.remoting.Request$2.run(Request.java:369) > at > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > Caused: java.io.IOException: Remote call on EC2 (jenkins-slave) - Adsdaq > Jenkins Slave (i-xxx) failed > at hudson.remoting.Channel.call(Channel.java:961) > at hudson.FilePath.act(FilePath.java:1072) > at hudson.FilePath.act(FilePath.java:1061) > at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114) > at > hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137) > at > hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167) > at > hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52) > at > hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25) > at > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Finished: FAILURE > > > As you can see these are very strange errors. I repeat that we didn't > change anything in the structure of the configuration. Not sure if it's a > problem with ec2-plugin, or a change somewhere in amazon. > > If anybody has or had this issue and knows how to solve it, I'm all ears, > as this is blocking our CD flow :/ > > Thanks in advance > > Cheers > > Toni > > > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2d48a635-54eb-4b31-8a05-10cad3446ea0%40googlegroups.com.
