[JIRA] [ssh-slaves] (JENKINS-22320) High CPU consumption because of SSH communication

Mon, 24 Mar 2014 01:42:31 -0700

Issue Type: Bug Bug
Assignee: Kohsuke Kawaguchi
Attachments: Selection_145.png, Selection_146.png
Components: ssh-slaves
Created: 24/Mar/14 8:41 AM
Description:

Our Jenkins instance runs 1.553 (since two weeks, was an three year old version before) on Linux with 10 slaves: 6 Linux connected through SSH and 4 windows with WebStart agent started as services.

When no job is running, CPU is almost 0. Correct.

When only two or three jobs are running, Jenkins CPU raises between 70% to 160%. atop declares 96% of CPU time spent in IRQ (even with "no" disk access), most CPU consumption is considered as system time. On average since boot, the master node consumes 100% of one CPU. Even if it has 4 CPUs, job execution time is between 2x and 3x compared to older version.

I configured JMX and did a quick CPU profiling. Top consumer threads are unnamed and are all related to SSH communication

"Thread-13" - Thread t@90
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41)
	at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52)
	at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79)
	at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
	at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232)
	at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:682)
	at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:480)
	at java.lang.Thread.run(Thread.java:744)



So there is chance the "SSH agent plugin" is concerned.



I am ready to do deeper analysis on my system if required, and of course to test patches.
Project: Jenkins
Labels: performance
Priority: Critical Critical
Reporter: Yves Martin
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Reply via email to