Anyone ? Can someone please take a look on this issue ? This is a very serious problem.
On Wednesday 01 September 2010 19:01:55 Doron Fediuck wrote: > Hi, > I have an implementation of SSH client using Mina SSHD. > I've noticed that working with it on several hosts will take the CPU to 100% > for every SSH > client session I have. For me this is a severe state, so I ran some debug / > profiling stuff. > > First I used jstack to see what's running. > I saw- > "ClientInputStreamPump" daemon prio=6 tid=0x34db8400 nid=0xec8 runnable > [0x478df000] > java.lang.Thread.State: RUNNABLE > at > org.apache.sshd.client.channel.ChannelSession.securedRead(ChannelSession.java:107) > at > org.apache.sshd.client.channel.ChannelSession.pumpInputStream(ChannelSession.java:81) > at > org.apache.sshd.client.channel.ChannelSession$1.run(ChannelSession.java:53) > > Locked ownable synchronizers: > - None > > The, I wrote an application that connects to the other VM and list the thread > utilization... > Here's what I got- > > T ID: 2637 Name: JMX server connection timeout 2637 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 2636 Name: RMI TCP Connection(310)-10.35.17.171 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 2611 Name: RMI TCP Accept-0 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 2528 Name: WorkerThread#0[127.0.0.1:2566] > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 2519 Name: ClientInputStreamPump > endCpuTime : 17624 sec's > endUserTime : 17619 sec's > T ID: 1101 Name: http-0.0.0.0-8080-27 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1100 Name: http-0.0.0.0-8080-26 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1099 Name: http-0.0.0.0-8080-25 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1095 Name: http-0.0.0.0-8080-24 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1093 Name: http-0.0.0.0-8080-23 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1092 Name: http-0.0.0.0-8080-22 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1091 Name: http-0.0.0.0-8080-21 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1090 Name: http-0.0.0.0-8080-20 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1089 Name: http-0.0.0.0-8080-19 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1088 Name: http-0.0.0.0-8080-18 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1087 Name: http-0.0.0.0-8080-17 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1086 Name: http-0.0.0.0-8080-16 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1085 Name: http-0.0.0.0-8080-15 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1084 Name: http-0.0.0.0-8080-14 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1083 Name: http-0.0.0.0-8080-13 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1082 Name: http-0.0.0.0-8080-12 > endCpuTime : 114 sec's > endUserTime : 103 sec's > T ID: 1081 Name: http-0.0.0.0-8080-11 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1080 Name: http-0.0.0.0-8080-10 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1079 Name: http-0.0.0.0-8080-9 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1078 Name: http-0.0.0.0-8080-8 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1077 Name: http-0.0.0.0-8080-7 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1076 Name: http-0.0.0.0-8080-6 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1075 Name: http-0.0.0.0-8080-5 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1074 Name: http-0.0.0.0-8080-4 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 1073 Name: pool-22-thread-6 > endCpuTime : 544 sec's > endUserTime : 534 sec's > T ID: 274 Name: http-0.0.0.0-8080-3 > endCpuTime : 13 sec's > endUserTime : 13 sec's > T ID: 201 Name: MultiThreadedHttpConnectionManager cleanup > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 181 Name: http-0.0.0.0-8080-2 > endCpuTime : 497 sec's > endUserTime : 444 sec's > T ID: 173 Name: http-0.0.0.0-8080-1 > endCpuTime : 67 sec's > endUserTime : 60 sec's > T ID: 172 Name: ajp-0.0.0.0-8009-Acceptor-0 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 171 Name: http-0.0.0.0-8080-Acceptor-0 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 170 Name: JBossLifeThread > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 161 Name: IdleRemover > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 154 Name: Thread-17 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 153 Name: JCA PoolFiller > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 152 Name: HSQLDB Timer @1dc8b22 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 151 Name: QuartzScheduler_QuartzSchedulerThread > endCpuTime : 23 sec's > endUserTime : 22 sec's > T ID: 150 Name: QuartzScheduler_Worker-99 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 149 Name: QuartzScheduler_Worker-98 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 148 Name: QuartzScheduler_Worker-97 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 147 Name: QuartzScheduler_Worker-96 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 146 Name: QuartzScheduler_Worker-95 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 145 Name: QuartzScheduler_Worker-94 > endCpuTime : 11 sec's > endUserTime : 10 sec's > T ID: 144 Name: QuartzScheduler_Worker-93 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 143 Name: QuartzScheduler_Worker-92 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 142 Name: QuartzScheduler_Worker-91 > endCpuTime : 12 sec's > endUserTime : 11 sec's > T ID: 141 Name: QuartzScheduler_Worker-90 > endCpuTime : 13 sec's > endUserTime : 10 sec's > T ID: 140 Name: QuartzScheduler_Worker-89 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 139 Name: QuartzScheduler_Worker-88 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 138 Name: QuartzScheduler_Worker-87 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 137 Name: QuartzScheduler_Worker-86 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 136 Name: QuartzScheduler_Worker-85 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 135 Name: QuartzScheduler_Worker-84 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 134 Name: QuartzScheduler_Worker-83 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 133 Name: QuartzScheduler_Worker-82 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 132 Name: QuartzScheduler_Worker-81 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 131 Name: QuartzScheduler_Worker-80 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 130 Name: QuartzScheduler_Worker-79 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 129 Name: QuartzScheduler_Worker-78 > endCpuTime : 12 sec's > endUserTime : 11 sec's > T ID: 128 Name: QuartzScheduler_Worker-77 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 127 Name: QuartzScheduler_Worker-76 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 126 Name: QuartzScheduler_Worker-75 > endCpuTime : 11 sec's > endUserTime : 10 sec's > T ID: 125 Name: QuartzScheduler_Worker-74 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 124 Name: QuartzScheduler_Worker-73 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 123 Name: QuartzScheduler_Worker-72 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 122 Name: QuartzScheduler_Worker-71 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 121 Name: QuartzScheduler_Worker-70 > endCpuTime : 11 sec's > endUserTime : 9 sec's > T ID: 120 Name: QuartzScheduler_Worker-69 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 119 Name: QuartzScheduler_Worker-68 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 118 Name: QuartzScheduler_Worker-67 > endCpuTime : 11 sec's > endUserTime : 9 sec's > T ID: 117 Name: QuartzScheduler_Worker-66 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 116 Name: QuartzScheduler_Worker-65 > endCpuTime : 12 sec's > endUserTime : 11 sec's > T ID: 115 Name: QuartzScheduler_Worker-64 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 114 Name: QuartzScheduler_Worker-63 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 113 Name: QuartzScheduler_Worker-62 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 112 Name: QuartzScheduler_Worker-61 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 111 Name: QuartzScheduler_Worker-60 > endCpuTime : 12 sec's > endUserTime : 11 sec's > T ID: 110 Name: QuartzScheduler_Worker-59 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 109 Name: QuartzScheduler_Worker-58 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 108 Name: QuartzScheduler_Worker-57 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 107 Name: QuartzScheduler_Worker-56 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 106 Name: QuartzScheduler_Worker-55 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 105 Name: QuartzScheduler_Worker-54 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 104 Name: QuartzScheduler_Worker-53 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 103 Name: QuartzScheduler_Worker-52 > endCpuTime : 11 sec's > endUserTime : 10 sec's > T ID: 102 Name: QuartzScheduler_Worker-51 > endCpuTime : 11 sec's > endUserTime : 10 sec's > T ID: 101 Name: QuartzScheduler_Worker-50 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 100 Name: QuartzScheduler_Worker-49 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 99 Name: QuartzScheduler_Worker-48 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 98 Name: QuartzScheduler_Worker-47 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 97 Name: QuartzScheduler_Worker-46 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 96 Name: QuartzScheduler_Worker-45 > endCpuTime : 13 sec's > endUserTime : 10 sec's > T ID: 95 Name: QuartzScheduler_Worker-44 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 94 Name: QuartzScheduler_Worker-43 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 93 Name: QuartzScheduler_Worker-42 > endCpuTime : 12 sec's > endUserTime : 11 sec's > T ID: 92 Name: QuartzScheduler_Worker-41 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 91 Name: QuartzScheduler_Worker-40 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 90 Name: QuartzScheduler_Worker-39 > endCpuTime : 12 sec's > endUserTime : 11 sec's > T ID: 89 Name: QuartzScheduler_Worker-38 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 88 Name: QuartzScheduler_Worker-37 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 87 Name: QuartzScheduler_Worker-36 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 86 Name: QuartzScheduler_Worker-35 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 85 Name: QuartzScheduler_Worker-34 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 84 Name: QuartzScheduler_Worker-33 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 83 Name: QuartzScheduler_Worker-32 > endCpuTime : 12 sec's > endUserTime : 11 sec's > T ID: 82 Name: QuartzScheduler_Worker-31 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 81 Name: QuartzScheduler_Worker-30 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 80 Name: QuartzScheduler_Worker-29 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 79 Name: QuartzScheduler_Worker-28 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 78 Name: QuartzScheduler_Worker-27 > endCpuTime : 12 sec's > endUserTime : 11 sec's > T ID: 77 Name: QuartzScheduler_Worker-26 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 76 Name: QuartzScheduler_Worker-25 > endCpuTime : 11 sec's > endUserTime : 9 sec's > T ID: 75 Name: QuartzScheduler_Worker-24 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 74 Name: QuartzScheduler_Worker-23 > endCpuTime : 12 sec's > endUserTime : 11 sec's > T ID: 73 Name: QuartzScheduler_Worker-22 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 72 Name: QuartzScheduler_Worker-21 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 71 Name: QuartzScheduler_Worker-20 > endCpuTime : 11 sec's > endUserTime : 9 sec's > T ID: 70 Name: QuartzScheduler_Worker-19 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 69 Name: QuartzScheduler_Worker-18 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 68 Name: QuartzScheduler_Worker-17 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 67 Name: QuartzScheduler_Worker-16 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 66 Name: QuartzScheduler_Worker-15 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 65 Name: QuartzScheduler_Worker-14 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 64 Name: QuartzScheduler_Worker-13 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 63 Name: QuartzScheduler_Worker-12 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 62 Name: QuartzScheduler_Worker-11 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 61 Name: QuartzScheduler_Worker-10 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 60 Name: QuartzScheduler_Worker-9 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 59 Name: QuartzScheduler_Worker-8 > endCpuTime : 11 sec's > endUserTime : 10 sec's > T ID: 58 Name: QuartzScheduler_Worker-7 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 57 Name: QuartzScheduler_Worker-6 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 56 Name: QuartzScheduler_Worker-5 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 55 Name: QuartzScheduler_Worker-4 > endCpuTime : 12 sec's > endUserTime : 10 sec's > T ID: 54 Name: QuartzScheduler_Worker-3 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 53 Name: QuartzScheduler_Worker-2 > endCpuTime : 13 sec's > endUserTime : 11 sec's > T ID: 52 Name: QuartzScheduler_Worker-1 > endCpuTime : 11 sec's > endUserTime : 9 sec's > T ID: 51 Name: QuartzScheduler_Worker-0 > endCpuTime : 14 sec's > endUserTime : 11 sec's > T ID: 50 Name: WorkManager(2)-1 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 49 Name: AOPListner > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 48 Name: ContainerBackgroundProcessor[StandardEngine[jboss.web]] > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 47 Name: RMI TCP Accept-4444 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 46 Name: PooledInvokerAcceptor#0-4445 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 43 Name: Thread-14 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 44 Name: Listener:4712 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 42 Name: Thread-13 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 41 Name: Thread-12 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 40 Name: Thread-11 > endCpuTime : 5 sec's > endUserTime : 5 sec's > T ID: 38 Name: Listener:4713 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 37 Name: > AcceptorThread[ServerSocket[addr=/0.0.0.0,port=0,localport=4446]] > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 36 Name: ServerSocketRefresh > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 35 Name: HDScanner > endCpuTime : 319 sec's > endUserTime : 136 sec's > T ID: 34 Name: > AcceptorThread[ServerSocket[addr=/0.0.0.0,port=0,localport=3873]] > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 33 Name: ServerSocketRefresh > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 32 Name: secondaryServerSocketThread[0] > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 31 Name: > AcceptorThread[ServerSocket[addr=/0.0.0.0,port=0,localport=4457]] > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 30 Name: ServerSocketRefresh > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 28 Name: RMI RenewClean-[qumranetyk:1175] > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 27 Name: RMI Scheduler(0) > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 25 Name: RMI TCP Accept-0 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 24 Name: RMI TCP Accept-1090 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 23 Name: > AcceptorThread[ServerSocket[addr=/0.0.0.0,port=0,localport=5400]] > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 22 Name: ServerSocketRefresh > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 21 Name: JBoss System Threads(1)-2 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 20 Name: GC Daemon > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 19 Name: RMI Reaper > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 18 Name: RMI TCP Accept-1098 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 17 Name: JBoss System Threads(1)-1 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 16 Name: Timer-Log4jService > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 15 Name: ZipFile Lock Reaper > endCpuTime : 1 sec's > endUserTime : 1 sec's > T ID: 14 Name: Timer-0 > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 11 Name: DestroyJavaVM > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 5 Name: Attach Listener > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 4 Name: Signal Dispatcher > endCpuTime : 0 sec's > endUserTime : 0 sec's > T ID: 3 Name: Finalizer > endCpuTime : 8 sec's > endUserTime : 7 sec's > T ID: 2 Name: Reference Handler > endCpuTime : 2 sec's > endUserTime : 1 sec's > > Just in case you missed it- > T ID: 2519 Name: ClientInputStreamPump > endCpuTime : 17624 sec's > endUserTime : 17619 sec's > > It looks like ClientInputStreamPump is pumping my CPU :( > Any idea how to resolve it ? > > -- /d
