Hi Team,
I am using AWS Instance as a Jenkins server which has different kind of
nodes ( AWS EC2 Instance, Linux and Windows Nodes )
We are using this Jenkins server to execute UT and Functional test to
qualify the builds.
We are facing "Slave went offline during the build" issue frequently. (
Note For EC2 instance its less frequency )
Jenkins version : 1.658
OS : Ubuntu 14.04.2 LTS
I need some help on fixing "Slave went offline during the build" issue.
Observed different error messages in log during the slave went offline, In
that following 2 messages we observed frequently
1) SEVERE: I/O error in channel followed by Unexpected termination of the
channel
2) TcpSlaveAgentListener$ConnectionHandler error followed by
Protocol:JNLP3-connect ( This is mostly in Windows machine )
Whenever the Job failed due to slave went offline issue we are seeing
either one of the message as above.
Please refer the following logs for reference.
Note : 1) We have raised Internal IT Ticket to monitor Packet loss of AWS
Instance to my Company network and we confirmed that there were no packet
loss
2) We Raised AWS Support ticket to make sure do we need to tune
anything specific with respect in AWS Side and confirmed that everything is
perfect.
1) We are connecting the nodes on-demand with 1 minute of delay
2) Disabled "Response Time" in preventive node monitoring settings
But still we are observing this issue.
It would be great if somebody guides me to fix this issue.
Thanks,
Gopal
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/0fdc68f8-c9c0-4b8f-8036-7909feafef75%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
SEVERE: I/O error in channel followed by Unexpected termination of the channel
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:42:22 PM
hudson.remoting.SynchronousCommandTransport$ReaderThread run
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: SEVERE: I/O error in channel
10_39_19_5_Client_Node
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: java.io.IOException: Unexpected
termination of the channel
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: Caused by: java.io.EOFException
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]:
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:42:22 PM
net.bull.javamelody.JavaLogger warn
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: WARNING: exception while collecting
data
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]:
hudson.remoting.ChannelClosedException: channel is already closed
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.Channel.send(Channel.java:578)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.Request.callAsync(Request.java:205)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.Channel.callAsync(Channel.java:807)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
net.bull.javamelody.RemoteCallHelper.collectDataByNodeName(RemoteCallHelper.java:159)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
net.bull.javamelody.RemoteCallHelper.collectJavaInformationsListByName(RemoteCallHelper.java:183)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
net.bull.javamelody.NodesCollector.collectWithoutErrorsNow(NodesCollector.java:154)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
net.bull.javamelody.NodesCollector.collectWithoutErrors(NodesCollector.java:143)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
net.bull.javamelody.NodesCollector$2.run(NodesCollector.java:111)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
java.util.TimerThread.mainLoop(Timer.java:555)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
java.util.TimerThread.run(Timer.java:505)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: Caused by: java.io.IOException
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.Channel.close(Channel.java:1163)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:118)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.PingThread.ping(PingThread.java:126)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.PingThread.run(PingThread.java:85)
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: Caused by:
java.util.concurrent.TimeoutException: Ping started at 1491175712681 hasn't
completed by 1491175952681
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]: #011... 2 more
Apr 2 16:42:22 ip-172-28-5-15 [jenkins]:
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:42:24 PM
hudson.remoting.SynchronousCommandTransport$ReaderThread run
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: SEVERE: I/O error in channel gkrishnan
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: java.io.IOException: Unexpected
termination of the channel
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: Caused by: java.io.EOFException
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: #011at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335)
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: #011at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804)
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: #011at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: #011at
java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]:
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:42:24 PM
hudson.remoting.SynchronousCommandTransport$ReaderThread run
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: SEVERE: I/O error in channel
FC23_sai_workstation
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: java.io.IOException: Unexpected
termination of the channel
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: #011at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: Caused by: java.io.EOFException
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: #011at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335)
Apr 2 16:42:24 ip-172-28-5-15 [jenkins]: #011at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804)
TcpSlaveAgentListener$ConnectionHandler error followed by
Protocol:JNLP3-connect ( This is mostly in Windows machine )
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: INFO: Accepted connection #1,013 from
/10.36.199.17:4584
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:27:03 PM
hudson.TcpSlaveAgentListener$ConnectionHandler error
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: WARNING: Connection #1013 is aborted:
Unknown protocol:Protocol:JNLP3-connect
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:27:03 PM
hudson.TcpSlaveAgentListener$ConnectionHandler run
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: INFO: Accepted connection #1,014 from
/10.36.199.17:4585
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:27:03 PM
jenkins.slaves.DefaultJnlpSlaveReceiver handle
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: INFO: Disconnecting
win_strawman_instance as we are reconnected from the current peer
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:27:03 PM
hudson.TcpSlaveAgentListener$ConnectionHandler run
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: INFO: Accepted connection #1,015 from
/10.36.200.16:60078
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:27:03 PM
hudson.TcpSlaveAgentListener$ConnectionHandler error
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: WARNING: Connection #1015 is aborted:
Unknown protocol:Protocol:JNLP3-connect
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:27:03 PM
hudson.TcpSlaveAgentListener$ConnectionHandler run
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: INFO: Accepted connection #1,016 from
/10.36.200.16:60079
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:27:03 PM
jenkins.slaves.DefaultJnlpSlaveReceiver handle
Apr 2 16:27:03 ip-172-28-5-15 [jenkins]: INFO: Disconnecting
ATEP_UI_Windows_8_10_36_200_16 as we are reconnected from the current peer
Apr 2 16:27:09 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:27:09 PM
hudson.TcpSlaveAgentListener$ConnectionHandler run
Apr 2 16:27:09 ip-172-28-5-15 [jenkins]: INFO: Accepted connection #1,017 from
/10.36.200.17:60446
Apr 2 16:27:09 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:27:09 PM
hudson.TcpSlaveAgentListener$ConnectionHandler run
Apr 2 16:27:09 ip-172-28-5-15 [jenkins]: INFO: Accepted connection #1,018 from
/10.36.198.17:64140
Apr 2 16:27:09 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:27:09 PM
hudson.TcpSlaveAgentListener$ConnectionHandler error
Apr 2 16:27:09 ip-172-28-5-15 [jenkins]: WARNING: Connection #1017 is aborted:
Unknown protocol:Protocol:JNLP3-connect
Apr 2 16:27:09 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:27:09 PM
hudson.TcpSlaveAgentListener$ConnectionHandler error
Apr 2 16:27:09 ip-172-28-5-15 [jenkins]: WARNING: Connection #1018 is aborted:
Unknown protocol:Protocol:JNLP3-connect
Apr 2 16:27:09 ip-172-28-5-15 [jenkins]: Apr 02, 2017 4:27:09 PM
hudson.TcpSlaveAgentListener$ConnectionHandler run
