Hello

   I have about 20 nodes of various RHEL versions and one team's nodes 
always pose certain problems and cause their pipelines to fail.  For a 
while, I was able to get them to pass by 'bringing nodes online on-demand' 
because it would refresh the connection.  But lately that strategy has also 
failed.  The slaves had a higher version of Java than the master and after 
updating the master the problem seemed to go away only to come back again 
like this.  If I disconnect, offline the node and bring it back, the logs 
are clean. But once the pipeline starts running again, it fails.  Any help 
would be much appreciated.  Thanks. 


   I'm running Jenkins master version: 2.91.  

JDK version on master RHEL-7
[support@10.0.156.53 ~]$ java -version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)

Slave - 
RHEL 7.3
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

Pipeline fails with such errors:

[Pipeline] End of Pipelinejava.io.EOFException
        at 
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at 
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at 
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at 
hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at 
hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
        at 
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
        at 
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused: hudson.remoting.ChannelClosedException: Remote call on 10.0.135.15 
failed. The channel is closing down or has closed down
        at hudson.remoting.Channel.call(Channel.java:901)
        at hudson.FilePath.act(FilePath.java:986)
        at hudson.FilePath.act(FilePath.java:975)
        at hudson.FilePath.mkdirs(FilePath.java:1158)
        at 
org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:113)
        at 
org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:198)
        at 
org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:190)
        at 
org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:111)
        at 
org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:64)
        at 
org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:177)
        at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:224)
        at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:150)
        at 
org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
        at sun.reflect.GeneratedMethodAccessor5286.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
        at 
org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
        at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
        at 
org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
        at 
org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:133)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:153)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:157)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
        at 
com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
Caused: java.io.IOException: remote file operation failed: 
/mnt/grandprix/homes/axsbuild/workspace/Sensor/Sensor_4.x_Groovy at 
hudson.remoting.Channel@3735998a:10.0.135.15
        at hudson.FilePath.act(FilePath.java:993)
        at hudson.FilePath.act(FilePath.java:975)
        at hudson.FilePath.mkdirs(FilePath.java:1158)
        at 
org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:113)
        at 
org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:198)
        at 
org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:190)
        at 
org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:111)
        at 
org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:64)
        at 
org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:177)
        at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:224)
        at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:150)
        at 
org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
        at sun.reflect.GeneratedMethodAccessor5286.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
        at 
org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
        at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
        at 
org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
        at 
org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:133)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:153)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:157)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
        at 
com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
        at WorkflowScript.run(WorkflowScript:343)
        at ___cps.transform___(Native Method)
        at 
com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
        at 
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
        at 
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
        at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
        at 
com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
        at 
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
        at 
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
        at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
        at 
com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:39)
        at 
com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
        at 
com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:28)
        at 
com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
        at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
        at com.cloudbees.groovy.cps.Next.step(Next.java:83)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
        at 
org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
        at 
org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
        at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
        at 
org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
        at 
org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
        at 
org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
        at 
org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
        at 
org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
        at 
org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
        at 
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330)
        at 
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
        at 
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
        at 
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
        at 
org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
        at 
jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE


And the slave has logs like this:

Log from the node 10.0.135.15:

Mar 01, 2018 9:21:54 PM hudson.util.ProcessTree getKillers

WARNING: Failed to obtain killers

java.io.IOException: Remote call on channel failed

        at hudson.remoting.Channel.call(Channel.java:916)

        at hudson.util.ProcessTree.getKillers(ProcessTree.java:159)

        at hudson.util.ProcessTree$OSProcess.killByKiller(ProcessTree.java:218)

        at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:660)

        at 
hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:668)

        at 
hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:667)

        at 
hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:667)

        at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:589)

        at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)

        at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1082)

        at hudson.remoting.UserRequest.perform(UserRequest.java:207)

        at hudson.remoting.UserRequest.perform(UserRequest.java:53)

        at hudson.remoting.Request$2.run(Request.java:358)

        at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.NoClassDefFoundError: 
hudson/remoting/RemoteClassLoader$RemoteIClassLoader

        at hudson.remoting.RemoteClassLoader.export(RemoteClassLoader.java:744)

        at hudson.remoting.UserRequest.<init>(UserRequest.java:104)

        at hudson.remoting.Channel.call(Channel.java:906)

        ... 17 more

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/2ae6ec10-330f-4bc1-bfc4-f1d5a7eb3782%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to