Hi Carlos, I am running into this same issue when I try to run pods in parallel within the same job. As the number of pods launched by the job increases, the number of timeouts/disconnects increases.
See attached images for the failure details. I believe it is the same issue. Regards, Art On Thursday, April 25, 2019 at 9:39:36 AM UTC-7, Carlos Sanchez wrote: > > what flavor of kubernetes ? cloud? on prem? how was it built? what > networking is it using? > there are a lot of variations so the best chance to find out is to relate > with other people's environment and try to isolate > > On Thu, Apr 25, 2019 at 6:36 PM Gopi <[email protected] <javascript:>> > wrote: > >> Hi >> Using latest Kubernetes 1.13 with 3 master nodes. >> >> On Thu, Apr 25, 2019 at 2:15 PM Carlos Sanchez <[email protected] >> <javascript:>> wrote: >> >>> what kubernetes cluster are you using and with what networking? >>> my guess is that the underlying network layer is killing inactive >>> connections >>> >>> On Thu, Apr 25, 2019 at 4:24 AM Gopi Naidu <[email protected] >>> <javascript:>> wrote: >>> >>>> I have got into the same issue. Could you please post the solution if >>>> anything is found? >>>> >>>> On Sunday, 26 August 2018 04:28:47 UTC+5:30, [email protected] >>>> wrote: >>>>> >>>>> I'm using Kubernetes plugin 1.12.3 >>>>> >>>>> The plugin works well, but whenever the agent does streams the log to >>>>> the master, it causes a connection exception as follows. >>>>> To reproduce, simply create a free style job, restrict to run on the >>>>> kubernetes pod, and add a step to sleep for 300 seconds. >>>>> >>>>> >>>>> FATAL: command execution failed >>>>> java.nio.channels.ClosedChannelException >>>>> at >>>>> org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) >>>>> at >>>>> org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142) >>>>> at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795) >>>>> at >>>>> jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) >>>>> at >>>>> jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>> at java.lang.Thread.run(Thread.java:748) >>>>> Caused: java.io.IOException: Backing channel 'JNLP4-connect connection >>>>> from ip-192-168-247-3.us-west-2.compute.internal/192.168.247.3:23684' is >>>>> disconnected. >>>>> at >>>>> hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214) >>>>> at >>>>> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283) >>>>> at com.sun.proxy.$Proxy110.isAlive(Unknown Source) >>>>> at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1137) >>>>> at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1129) >>>>> at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155) >>>>> at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109) >>>>> at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) >>>>> at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) >>>>> at >>>>> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) >>>>> at hudson.model.Build$BuildExecution.build(Build.java:206) >>>>> at hudson.model.Build$BuildExecution.doRun(Build.java:163) >>>>> at >>>>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) >>>>> at hudson.model.Run.execute(Run.java:1798) >>>>> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >>>>> at hudson.model.ResourceController.execute(ResourceController.java:97) >>>>> at hudson.model.Executor.run(Executor.java:429) >>>>> FATAL: Unable to delete script file /tmp/jenkins5788393309854271460.sh >>>>> java.nio.channels.ClosedChannelException >>>>> at >>>>> org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) >>>>> at >>>>> org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142) >>>>> at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795) >>>>> at >>>>> jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) >>>>> at >>>>> jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>> at java.lang.Thread.run(Thread.java:748) >>>>> Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote >>>>> call on JNLP4-connect connection from >>>>> ip-192-168-247-3.us-west-2.compute.internal/192.168.247.3:23684 failed. >>>>> The channel is closing down or has closed down >>>>> at hudson.remoting.Channel.call(Channel.java:948) >>>>> at hudson.FilePath.act(FilePath.java:1036) >>>>> at hudson.FilePath.act(FilePath.java:1025) >>>>> at hudson.FilePath.delete(FilePath.java:1511) >>>>> at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:123) >>>>> at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) >>>>> at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) >>>>> at >>>>> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) >>>>> at hudson.model.Build$BuildExecution.build(Build.java:206) >>>>> at hudson.model.Build$BuildExecution.doRun(Build.java:163) >>>>> at >>>>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) >>>>> at hudson.model.Run.execute(Run.java:1798) >>>>> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >>>>> at hudson.model.ResourceController.execute(ResourceController.java:97) >>>>> at hudson.model.Executor.run(Executor.java:429) >>>>> Build step 'Execute shell' marked build as failure >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected] <javascript:>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-users/cb6a14e9-c8d0-411a-bd3a-1b1481482df4%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/jenkinsci-users/cb6a14e9-c8d0-411a-bd3a-1b1481482df4%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/CALHFn6PefYwsUuZHwaaGONXkQ3C45xC8nmi%2BeJ-bS6SCH_4JFA%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/jenkinsci-users/CALHFn6PefYwsUuZHwaaGONXkQ3C45xC8nmi%2BeJ-bS6SCH_4JFA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> Thanks & Regards >> Gopi. Ch >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/CACSzXNtm6uuZfirZ3sRT1reLpF6yXhYgiY%2Bn1L%3DPADw0u-1%2Bdw%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/jenkinsci-users/CACSzXNtm6uuZfirZ3sRT1reLpF6yXhYgiY%2Bn1L%3DPADw0u-1%2Bdw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/23a9268a-df6f-477b-bb5c-777fbc86ff98%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
