This thing caught my eye:
Feb 11, 2019 8:15:03 AM hudson.remoting.ProxyOutputStream$Chunk$1 run
WARNING: Failed to *ack *the stream
That indicates a TCP issue over there. Try this
[root@ip-172-31-2-59 ec2-user]# ethtool -K eth0 sg off
Actual changes:
scatter-gather: off
tx-scatter-gather: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
generic-segmentation-offload: off [requested on]
https://github.com/scala/scala-jenkins-infra/issues/26#issuecomment-73825006
On Mon, Feb 11, 2019 at 9:16 AM Ivan Fernandez Calvo <
[email protected]> wrote:
> It could be tons of things, Probably, if you go to the node log
> (JENKINS_URL/computer/NODE_NAME/log) we could see if the agent connect or
> not, then if the communication is broken before/after connection and in
> which point.
>
> El miércoles, 6 de febrero de 2019, 16:39:17 (UTC+1), Glenn Burkhardt
> escribió:
>>
>> We've been seeing job termination due to slave disconnects, and would
>> like help in solving this issue.
>>
>> Jenkins 2.140
>> ssh-slaves-plugin 1.29.4
>> Ubuntu 16.04.5
>> Hypervisor, plugin Libvirt 1.8.6
>> Slave guest: Ubuntu 18.04.1
>>
>> Jenkins and the VMs are all running on the same machine, so network
>> activity shouldn't be an issue.
>>
>> I've been looking at the wiki note here:
>> https://wiki.jenkins.io/display/JENKINS/Remoting+issue
>>
>> and the anomaly I've noticed is repeated in the slave.log file created by
>> Jenkins (SocketTimeoutException):
>>
>> Feb 06, 2019 8:37:58 AM org.jenkinsci.remoting.engine.WorkDirManager
>> initializeWorkDir
>> INFO: Using /home/jenkins/remoting as a remoting work directory
>> Both error and output logs will be printed to /home/jenkins/remoting
>> <===[JENKINS REMOTING CAPACITY]===>channel started
>> Remoting version: 3.25
>> This is a Unix agent
>> Evacuated stdout
>> Agent successfully connected and online
>> ERROR: Connection terminated
>> java.io.EOFException
>> at
>> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681)
>> at
>> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
>> at
>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
>> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
>> at
>> hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
>> at hudson.remoting.Command.readFrom(Command.java:140)
>> at hudson.remoting.Command.readFrom(Command.java:126)
>> at
>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
>> at
>> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
>> Caused: java.io.IOException: Unexpected termination of the channel
>> at
>> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
>> ERROR: Socket connection to SSH server was lost
>> java.net.SocketTimeoutException: The connect timeout expired
>> at com.trilead.ssh2.Connection$1.run(Connection.java:762)
>> at
>> com.trilead.ssh2.util.TimeoutService$TimeoutThread.run(TimeoutService.java:91)
>> Slave JVM has not reported exit code before the socket was lost
>> [02/06/19 08:41:05] [SSH] Connection closed.
>>
>> The remoting log on the slave has
>>
>> Feb 06, 2019 8:41:05 AM
>> hudson.remoting.SynchronousCommandTransport$ReaderThread run
>> SEVERE: I/O error in channel channel
>> java.io.IOException: Unexpected termination of the channel
>> at
>> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
>> Caused by: java.io.EOFException
>> at
>> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2671)
>> at
>> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3146)
>> at
>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
>> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)
>> at
>> hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
>> at hudson.remoting.Command.readFrom(Command.java:140)
>> at hudson.remoting.Command.readFrom(Command.java:126)
>> at
>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
>> at
>> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
>>
>> and the /var/log/jenkins/jenkins.log has:
>>
>> Feb 06, 2019 8:41:05 AM
>> hudson.remoting.SynchronousCommandTransport$ReaderThread run
>> SEVERE: I/O error in channel ubuntu-122-2
>> java.io.IOException: Unexpected termination of the channel
>> at
>> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
>> Caused by: java.io.EOFException
>> at
>> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681)
>> at
>> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
>> at
>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
>> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
>> at
>> hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
>> at hudson.remoting.Command.readFrom(Command.java:140)
>> at hudson.remoting.Command.readFrom(Command.java:126)
>> at
>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
>> at
>> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
>>
>> Feb 06, 2019 8:41:12 AM hudson.slaves.RetentionStrategy$Demand check
>> INFO: Disconnecting computer ubuntu-122-1 as it has been idle for 1 min 2
>> sec
>> Feb 06, 2019 8:41:12 AM
>> hudson.plugins.libvirt.VirtualMachineSlaveComputer disconnect
>> INFO: Virtual machine "ubuntu-20170406-122-1" (slave "ubuntu-122-1") is
>> to be shut down.reason: Offline because computer was idle; it will be
>> relaunched when needed. (hudson.slaves.OfflineCause$IdleOfflineCause)
>>
>> The syslog file on the slave doesn't indicate any anomalies between the
>> startup and termination initiated by Jenkins:
>>
>> Feb 6 08:39:31 VirtualBox systemd[1]: Starting Clean php session files...
>> Feb 6 08:39:31 VirtualBox systemd[1]: Started Clean php session files.
>> Feb 6 08:41:43 VirtualBox systemd[1]: Stopping User Manager for UID
>> 998...
>> Feb 6 08:41:43 VirtualBox systemd[1294]: Stopped target Default.
>>
>> Thanks.
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/0ffc8193-e585-4821-9aec-3b4eec70e629%40googlegroups.com
> <https://groups.google.com/d/msgid/jenkinsci-users/0ffc8193-e585-4821-9aec-3b4eec70e629%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
Regards
nirish okram
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/CAPzcO4ibDezMCL9T9AwcRc7_DRyqyKciDwUnYX1k95tPZqFyuQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
