If the agent is running something there is traffic, so it si not related to keep alive settings, I wonder if after the issue happens there is a remoting process running on the agent. Did you check the SSHD service logs on your agent? https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md#enable-verbose-ssh-server-log-output Did you check the remoting logs? https://github.com/jenkinsci/remoting/blob/master/docs/logging.md Did you check for hs_err_pid error files in the root fs of the agent? http://www.oracle.com/technetwork/java/javase/felog-138657.html#gbwcy How much memory you set for your remoting process? How much memory your maven build needs?
El miércoles, 30 de diciembre de 2020 a las 23:40:38 UTC+1, [email protected] escribió: > Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it > runs into " Resource temporarily unavailable". > I have > > - verified TCP parameters > i.e. tcp_keepalive_time, tcp_keepalive_intvl, tcp_keepalive_probes > & tcp_fin_timeout ; All of them have higher values then what cloudbees > recommended at > > https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected > - set heap space to 2-4Gb > - disabled ping thread from Jenkins agent by setting > "-Dhudson.remoting.Launcher.pingIntervalSec=-1" > > Please let me know if there is anything else I should consider. > On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [email protected] > wrote: > >> The most common issue is related to disconnections because there is no >> traffic between the Jenkins instance and the agent, for that, you have to >> tune the TCP stack of your OS (see >> https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), >> >> or enable the keepalive option in the SSH protocol this can be configured >> by setting ClientAliveInterval or TCPKeepAlive on the SSH server >> (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive >> options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config) >> >> https://www.freebsd.org/cgi/man.cgi?sshd_config(5) >> https://www.freebsd.org/cgi/man.cgi?ssh_config(5) >> >> Also, check you follow the best practices to configure your SSH agents >> and enable SSH verbose log output in your service (see >> https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md >> ) >> >> El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, >> [email protected] escribió: >> >>> Hello Jenkins community, >>> >>> I have setup Jenkins agents over SSH and this agent frequently goes >>> offline with Resource temporarily unavailable error. I had to configure my >>> Jenkins agents to communicate over JNLP. Any advise on how to fix this >>> issue? >>> >>> Jenkins v2.249.1 >>> >>> [12/04/20 15:23:00] [SSH] Checking java version of java >>> [12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202. >>> [12/04/20 15:23:01] [SSH] Starting sftp client. >>> [12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does >>> not exist. Will try to create it... >>> [12/04/20 15:23:03] [SSH] Copying latest remoting.jar... >>> [12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes. >>> Expanded the channel window size to 4MB >>> [12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" >>> && java -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache >>> $JENKINS_SSH_DATA/remoting/jarCache >>> Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager >>> initializeWorkDir >>> INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory >>> Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager >>> setupLogging >>> INFO: Both error and output logs will be printed to >>> $JENKINS_SSH_DATA/remoting >>> <===[JENKINS REMOTING CAPACITY]===>channel started >>> Remoting version: 4.5 >>> This is a Unix agent >>> Evacuated stdout >>> Agent successfully connected and online >>> The Agent is connected, disconnect it before to try to connect it again. >>> Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig >>> WARNING: Exception caught during execution of command '[git, config, >>> --system, --edit]' in '$GIT_PATH/bin', return code '128', error message >>> 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory >>> ' >>> Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes >>> saveToConfig >>> WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] >>> failed after 5 retries >>> Dec 04, 2020 3:33:59 PM >>> org.jenkinsci.remoting.util.AnonymousClassWarnings warn >>> WARNING: Attempt to (de-)serialize anonymous class >>> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; >>> >>> see: https://jenkins.io/redirect/serialization-of-anonymous-classes/ >>> Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run >>> WARNING: Failed to send back a reply to the request >>> hudson.remoting.Request$2@493c5a4e >>> java.io.IOException: Resource temporarily unavailable >>> at java.io.FileOutputStream.writeBytes(Native Method) >>> at java.io.FileOutputStream.write(FileOutputStream.java:313) >>> at >>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83) >>> at >>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89) >>> at >>> hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62) >>> at >>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46) >>> at >>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) >>> at hudson.remoting.Channel.send(Channel.java:766) >>> at hudson.remoting.Request$2.run(Request.java:388) >>> at >>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> >>> Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run >>> WARNING: Failed to send back a reply to the request >>> hudson.remoting.Request$2@2511e2d4 >>> java.io.IOException: Resource temporarily unavailable >>> at java.io.FileOutputStream.writeBytes(Native Method) >>> at java.io.FileOutputStream.write(FileOutputStream.java:326) >>> at >>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88) >>> at >>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90) >>> at >>> hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85) >>> at >>> hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54) >>> at java.io.OutputStream.write(OutputStream.java:75) >>> at >>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45) >>> at >>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) >>> at hudson.remoting.Channel.send(Channel.java:766) >>> at hudson.remoting.Request$2.run(Request.java:388) >>> at >>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> >>> ERROR: Connection terminated >>> java.io.StreamCorruptedException: invalid stream header: 00025B42 >>> at >>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866) >>> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358) >>> at >>> hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) >>> at hudson.remoting.Command.readFrom(Command.java:142) >>> at hudson.remoting.Command.readFrom(Command.java:128) >>> at >>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) >>> at >>> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) >>> Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead >>> INFO: Ping failed. Terminating the channel channel. >>> java.util.concurrent.TimeoutException: Ping started at 1607120910589 >>> hasn't completed by 1607121150590 >>> at hudson.remoting.PingThread.ping(PingThread.java:134) >>> at hudson.remoting.PingThread.run(PingThread.java:90) >>> >>> Thanks, >>> Jigar R >>> >>> -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2b23ef01-3890-4fbd-962b-970a5efec3d8n%40googlegroups.com.
