Weird, Could you share a screencapture of what you configurate? Also the whole exception those lines alone mean nothing. Know the version of Jenkins and the version of the SSH build agents plugins you use can help, the memory you have in your agents, if they are bare metal or cloud. In overall if you want help please provide more context.
El mar, 2 feb 2021 a las 22:52, Jigar R (<jigarrathod...@gmail.com>) escribió: > > > On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 kuisat...@gmail.com > wrote: > >> >I have 2 different kind of jenkins agents. >> >1. java web start >> >2. ssh >> >If the memory was issue then wouldn't it fail on both cases? >> >> Not necessarily, starting by the point that are different ways to >> establish the connection, the JNLP agents could not update the remoting jar >> file (depends on your configurations), so you can be running different >> versions of remoting. I agree with Jeff looks like an OOM issue, review my >> comments at >> https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ >> my recommendation is to fix the mem for the remoting process to 1024M >> (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, >> you would have to adjust the remoting process memory to the right one >> between 256M-1024M, to use 512M usually is safe a not too much (but depends >> on your agents' memory we do not know how much they have) >> > > I created SSH agent with -Xmx1024m -Xms1024m. > I do see bunch of Warnings about "Ping failed. Terminating the channel > channel". > I got following error: > " > WARNING: Failed to send back a reply to te rquest > hudson.remoting.Request$.... > java.io.IOException: Resource temporarily unavailable > at java.io.FileOutputStream.writeBytes(Native method) > ..... > " > > >> >> El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, jigarra...@gmail.com >> escribió: >> >>> On Monday, January 4, 2021 at 12:37:52 PM UTC-5 jtho...@cloudbees.com >>> wrote: >>> >>>> When I saw something like this in the past, it was because the process >>>> was running out of resources. Specifically when running SpotBugs, I got >>>> out-of-memory errors. I had to modify the pom to allocate more memory. >>>> >>> I have 2 different kind of jenkins agents. >>> 1. java web start >>> 2. ssh >>> If the memory was issue then wouldn't it fail on both cases? >>> >>> I do see "Agent went offline during build Connection was broker: >>> java.io.StreamCorruptedException: invalid stream header:". >>> >>>> I recommend ensuring your build runs normally on the agent without the >>>> additional complexities of the Jenkins environment. This might show where >>>> additional resources are needed. If that all passes, then continue on to >>>> the additional troubleshooting steps involving the Jenkins controller and >>>> agent. >>>> >>>> Jeff Thompson >>>> On 12/30/20 3:40 PM, jiga...@gmail.com wrote: >>>> >>>> Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, >>>> it runs into " Resource temporarily unavailable". >>>> I have >>>> >>>> - verified TCP parameters >>>> i.e. tcp_keepalive_time, tcp_keepalive_intvl, tcp_keepalive_probes >>>> & tcp_fin_timeout ; All of them have higher values then what cloudbees >>>> recommended at >>>> >>>> https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected >>>> - set heap space to 2-4Gb >>>> - disabled ping thread from Jenkins agent by setting >>>> "-Dhudson.remoting.Launcher.pingIntervalSec=-1" >>>> >>>> Please let me know if there is anything else I should consider. >>>> On Friday, December 11, 2020 at 12:11:39 PM UTC-5 kuisat...@gmail.com >>>> wrote: >>>> >>>>> The most common issue is related to disconnections because there is no >>>>> traffic between the Jenkins instance and the agent, for that, you have to >>>>> tune the TCP stack of your OS (see >>>>> https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), >>>>> or enable the keepalive option in the SSH protocol this can be configured >>>>> by setting ClientAliveInterval or TCPKeepAlive on the SSH server >>>>> (/etc/ssh/sshd_config), also by setting ServerAliveInterval or >>>>> TCPKeepAlive >>>>> options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config) >>>>> >>>>> https://www.freebsd.org/cgi/man.cgi?sshd_config(5) >>>>> https://www.freebsd.org/cgi/man.cgi?ssh_config(5) >>>>> >>>>> Also, check you follow the best practices to configure your SSH agents >>>>> and enable SSH verbose log output in your service (see >>>>> https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md >>>>> ) >>>>> >>>>> El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, >>>>> jiga...@gmail.com escribió: >>>>> >>>>>> Hello Jenkins community, >>>>>> >>>>>> I have setup Jenkins agents over SSH and this agent frequently goes >>>>>> offline with Resource temporarily unavailable error. I had to configure >>>>>> my >>>>>> Jenkins agents to communicate over JNLP. Any advise on how to fix this >>>>>> issue? >>>>>> >>>>>> Jenkins v2.249.1 >>>>>> >>>>>> [12/04/20 15:23:00] [SSH] Checking java version of java >>>>>> [12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202. >>>>>> [12/04/20 15:23:01] [SSH] Starting sftp client. >>>>>> [12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA >>>>>> does not exist. Will try to create it... >>>>>> [12/04/20 15:23:03] [SSH] Copying latest remoting.jar... >>>>>> [12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes. >>>>>> Expanded the channel window size to 4MB >>>>>> [12/04/20 15:23:03] [SSH] Starting agent process: cd >>>>>> "$JENKINS_SSH_DATA" && java -jar remoting.jar -workDir $JENKINS_SSH_DATA >>>>>> -jar-cache $JENKINS_SSH_DATA/remoting/jarCache >>>>>> Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager >>>>>> initializeWorkDir >>>>>> INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory >>>>>> Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager >>>>>> setupLogging >>>>>> INFO: Both error and output logs will be printed to >>>>>> $JENKINS_SSH_DATA/remoting >>>>>> <===[JENKINS REMOTING CAPACITY]===>channel started >>>>>> Remoting version: 4.5 >>>>>> This is a Unix agent >>>>>> Evacuated stdout >>>>>> Agent successfully connected and online >>>>>> The Agent is connected, disconnect it before to try to connect it >>>>>> again. >>>>>> Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS >>>>>> discoverGitSystemConfig >>>>>> WARNING: Exception caught during execution of command '[git, config, >>>>>> --system, --edit]' in '$GIT_PATH/bin', return code '128', error message >>>>>> 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory >>>>>> ' >>>>>> Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes >>>>>> saveToConfig >>>>>> WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] >>>>>> failed after 5 retries >>>>>> Dec 04, 2020 3:33:59 PM >>>>>> org.jenkinsci.remoting.util.AnonymousClassWarnings warn >>>>>> WARNING: Attempt to (de-)serialize anonymous class >>>>>> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; >>>>>> see: https://jenkins.io/redirect/serialization-of-anonymous-classes/ >>>>>> Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run >>>>>> WARNING: Failed to send back a reply to the request >>>>>> hudson.remoting.Request$2@493c5a4e >>>>>> java.io.IOException: Resource temporarily unavailable >>>>>> at java.io.FileOutputStream.writeBytes(Native Method) >>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:313) >>>>>> at >>>>>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83) >>>>>> at >>>>>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89) >>>>>> at >>>>>> hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62) >>>>>> at >>>>>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46) >>>>>> at >>>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) >>>>>> at hudson.remoting.Channel.send(Channel.java:766) >>>>>> at hudson.remoting.Request$2.run(Request.java:388) >>>>>> at >>>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73) >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>> >>>>>> Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run >>>>>> WARNING: Failed to send back a reply to the request >>>>>> hudson.remoting.Request$2@2511e2d4 >>>>>> java.io.IOException: Resource temporarily unavailable >>>>>> at java.io.FileOutputStream.writeBytes(Native Method) >>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:326) >>>>>> at >>>>>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88) >>>>>> at >>>>>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90) >>>>>> at >>>>>> hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85) >>>>>> at >>>>>> hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54) >>>>>> at java.io.OutputStream.write(OutputStream.java:75) >>>>>> at >>>>>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45) >>>>>> at >>>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) >>>>>> at hudson.remoting.Channel.send(Channel.java:766) >>>>>> at hudson.remoting.Request$2.run(Request.java:388) >>>>>> at >>>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73) >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>> >>>>>> ERROR: Connection terminated >>>>>> java.io.StreamCorruptedException: invalid stream header: 00025B42 >>>>>> at >>>>>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866) >>>>>> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358) >>>>>> at >>>>>> hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) >>>>>> at hudson.remoting.Command.readFrom(Command.java:142) >>>>>> at hudson.remoting.Command.readFrom(Command.java:128) >>>>>> at >>>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) >>>>>> at >>>>>> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) >>>>>> Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead >>>>>> INFO: Ping failed. Terminating the channel channel. >>>>>> java.util.concurrent.TimeoutException: Ping started at 1607120910589 >>>>>> hasn't completed by 1607121150590 >>>>>> at hudson.remoting.PingThread.ping(PingThread.java:134) >>>>>> at hudson.remoting.PingThread.run(PingThread.java:90) >>>>>> >>>>>> Thanks, >>>>>> Jigar R >>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to jenkinsci-use...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> -- > You received this message because you are subscribed to a topic in the > Google Groups "Jenkins Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > jenkinsci-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Un Saludo Iván Fernández Calvo https://www.linkedin.com/in/iv%C3%A1n-fern%C3%A1ndez-calvo-21425033 -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAKo5QrpE%2Bxys4CwNGe6Mx7NRozmeFxaJO0uZ77Rz3NUK057WEA%40mail.gmail.com.
