[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Ivan Fernandez Calvo commented on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute I usually request this info Common info needed to troubleshooting a bug to try to replicate the issue, but in this case, I guess that the 4th point would give you the solution directly when the issue happens should be an entry on the SSHD logs that said something that explains why the agent disconnect Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.187765.1516088319000.128.1580745840354%40Atlassian.JIRA.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Ian Katz commented on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute I see that this has been closed as "can't reproduce". Assuming that you could reproduce the issue, what data would you record? I can try to do that and open a new issue with the requested data next time I experience this issue (which happens regularly) Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.187765.1516088319000.10750.1580735340490%40Atlassian.JIRA.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Ivan Fernandez Calvo updated JENKINS-48955 Jenkins / JENKINS-48955 master-slave connection getting terminated once in every 12 hours and recovered after 1 minute Change By: Ivan Fernandez Calvo Status: Fixed but Unreleased Closed Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.187765.1516088319000.10343.1580577121088%40Atlassian.JIRA.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Eugene Chepurniy commented on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute Ivan Fernandez Calvo I'm going to give this solution a chance and provide a feedback here. Thanks for staying in touch. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Ivan Fernandez Calvo commented on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute On the latest version, you can disable the TCP_NODELAY in the UI, try this too. I am working on a version that uses the native ssh client, this will fix this kind of issues, it will take a couple of months. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Eugene Chepurniy edited a comment on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute We still experiencing described problems. What was done among other actions: 1. Ping thread was disabled in Jenkins. ([https://wiki.jenkins.io/display/JENKINS/Ping+Thread)]2. SELinux was completely disabled on slaves (getenforce outputs `Disabled`)3. All possible timeouts were increased. 4. Java version was set to be same on agents and server. The most helpful action was disabling of SELinux - the number of ssh failures decreased by 10 times.[~stibi] FYI. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Eugene Chepurniy commented on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute We still experiencing described problems. What was done among other actions: 1. Ping thread was disabled in Jenkins. (https://wiki.jenkins.io/display/JENKINS/Ping+Thread) 2. SELinux was completely disabled on slaves (getenforce outputs `Disabled`) 3. All possible timeouts were increased. The most helpful action was disabling of SELinux - the number of ssh failures decreased by 10 times. Martin Stiborský FYI. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title
Martin Stiborský commented on JENKINS-48955
Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Ivan Fernandez Calvo yes, I found that. There is the same error as logged on master: Sep 03, 2018 12:56:07 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel channel java.io.IOException: Unexpected termination of the channel {{ at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)}} Caused by: java.io.EOFException {{ at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681)}} {{ at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)}} {{ at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)}} {{ at java.io.ObjectInputStream.(ObjectInputStream.java:358)}} {{ at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:49)}} {{ at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)}} {{ at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)}}
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title
Martin Stiborský edited a comment on JENKINS-48955
Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
[~ifernandezcalvo] yes, I found that. There is the same error as logged on master: { { code:java} Sep 03, 2018 12:56:07 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run }} {{ SEVERE: I/O error in channel channel }} {{ java.io.IOException: Unexpected termination of the channel }} {{ at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) }} {{ Caused by: java.io.EOFException }} {{ at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681) }} {{ at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156) }} {{ at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862) }} {{ at java.io.ObjectInputStream.(ObjectInputStream.java:358) }} {{ at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:49) }} {{ at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36) }} {{ at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) {code } }
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Ivan Fernandez Calvo commented on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute Martin Stiborský if you are using the latest version, you could check the remoting logs, on the working agent folder should be a folder named remoting, there would be the log files Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Martin Stiborský commented on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute Hi, I'd like to join this party. We have exactly the same problem as described here. All matches, our agents EC2 instances also run on m4.xlarge (and some other instance types), all the details are the same as our environment. From time to time, some slaves just disconnects, leaving build failed and developers frustrated. We are trying to find out what's wrong for pretty long couple of weeks. Any hints appreciated. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Eugene Chepurniy commented on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute Ivan Fernandez Calvo thanks for your responses. any kind of network issues are excluded (or have very low possibility) - both server and agents are in same AWS VPC and have 10Gigs network enabled. In most of the time (99.99%) agents are performing well w/o any issues. there are only 2 executors per agent and agent is m4.xlarge (with 16G of RAM) instance. Jenkins agent is starting with default config. There were no OOMs/agent crushes spotted. I'm going to follow your suggestion and turn Agents logs on to see if some additional information can be gathered. And yes - we have a pretty high load on agents but I'm not sure it is so huge to make ssh connection interruptions. Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Ivan Fernandez Calvo updated JENKINS-48955 Jenkins / JENKINS-48955 master-slave connection getting terminated once in every 12 hours and recovered after 1 minute Change By: Ivan Fernandez Calvo Status: Reopened Fixed but Unreleased Resolution: Cannot Reproduce Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Ivan Fernandez Calvo commented on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute Eugene Chepurniy this issue is not related at all to the trace you attach, please open a new issue with those details if the problem still persists. The problem seems a timeout between the Jenkins instance and the Agent, these could be the root causes: Network issue between Jenkins instance and Agent, if both are in different networks check with your IT Team that there are no issues The Jenkins instance has a heavy load then the agent lost connections with it. The Agent has a heavy load then the Jenkins instance lost connections with it. The Agent process die (slave.jar) check the latest build logs on that Agent, and Agent logs, if you do not have Agent logs check https://github.com/jenkinsci/remoting/blob/master/docs/workDir.md to enable them An OOM error on the agent, check that there are no hs_err_pid files on the workdir of the Agent, if you Agents has low memory try to pass -Xmx256m and -Xms256m parameters to the Agent JVM options to fix the memory, 128MB/256MB used to be enough to and Agent with 10 executors. Check the Agent syslog and kernel log for network issue or another kind of performance issues. In any case without the logs on the Agent side, it is complicated to know what happens. Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues"
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Ivan Fernandez Calvo assigned an issue to Ivan Fernandez Calvo Jenkins / JENKINS-48955 master-slave connection getting terminated once in every 12 hours and recovered after 1 minute Change By: Ivan Fernandez Calvo Assignee: Oleg Nenashev Ivan Fernandez Calvo Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Eugene Chepurniy commented on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute Prakash G, thanks for the comment but in my case, IPs are managed by AWS and overlapping of addresses is impossible. Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Prakash G commented on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute The problem is because of duplicate IP in the network, Slave IP I used was consumed by some other device on that particular time and during that time master resolved ARP with that unknown host rather than with slave. Please check your network and monitor connectivity during that time with master Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title
Eugene Chepurniy edited a comment on JENKINS-48955
Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
The same approach found:{code:java}ERROR: Connection terminatedjava.io.EOFExceptionat java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:861)at java.io.ObjectInputStream.(ObjectInputStream.java:357)at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:48)at hudson.remoting.Command.readFrom(Command.java:140)at hudson.remoting.Command.readFrom(Command.java:126)at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)Caused: java.io.IOException: Unexpected termination of the channelat hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)ERROR: Socket connection to SSH server was lostjava.net.SocketTimeoutException: The connect timeout expiredat com.trilead.ssh2.Connection$1.run(Connection.java:762)at com.trilead.ssh2.util.TimeoutService$TimeoutThread.run(TimeoutService.java:91)Slave JVM has not reported exit code before the socket was lost[08/15/18 06:37:34] [SSH] Connection closed.{code}Jenkins app: 2.136SSH Agent Plugin: 1.16 SSH Slaves Plugin: 1.26 EC2 Fleet Plugin (v.1.1.7) was used to prepare agents. No possible network issues found, reproduces frequently in 20-30 minutes after successful agent startup. AWS spot instances weren't stopped/terminated during spotted fails.For the time of bug investigation is there any possibility to make failed stages transparently restart in case of agent connectivity issues?
Add Comment
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Eugene Chepurniy reopened an issue Jenkins / JENKINS-48955 master-slave connection getting terminated once in every 12 hours and recovered after 1 minute Change By: Eugene Chepurniy Resolution: Cannot Reproduce Status: Resolved Reopened Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48955) master-slave connection getting terminated once in every 12 hours and recovered after 1 minute
Title: Message Title Eugene Chepurniy commented on JENKINS-48955 Re: master-slave connection getting terminated once in every 12 hours and recovered after 1 minute The same approach found: ERROR: Connection terminated java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:861) at java.io.ObjectInputStream.(ObjectInputStream.java:357) at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:48) at hudson.remoting.Command.readFrom(Command.java:140) at hudson.remoting.Command.readFrom(Command.java:126) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) ERROR: Socket connection to SSH server was lost java.net.SocketTimeoutException: The connect timeout expired at com.trilead.ssh2.Connection$1.run(Connection.java:762) at com.trilead.ssh2.util.TimeoutService$TimeoutThread.run(TimeoutService.java:91) Slave JVM has not reported exit code before the socket was lost [08/15/18 06:37:34] [SSH] Connection closed. Jenkins app: 2.136 SSH Agent Plugin: 1.16 EC2 Fleet Plugin (v.1.1.7) was used to prepare agents. No possible network issues found, reproduces frequently in 20-30 minutes after successful agent startup. AWS spot instances weren't stopped/terminated during spotted fails. For the time of bug investigation is there any possibility to make failed stages transparently restart in case of agent connectivity issues? Add Comment
