Thanks for the dumps. Below are the stack traces responsible for the high CPU usages in CG and AS respectively. First stack trace is from the in build Thrift server which keeps the request messages. The second stack trace is from the polling transport which polls the thrift server buffers on a busy loop. CG Agent has to keep on polling in order to see if there is any request messages and there is no way the server can send a notification to the agent( because of the firewall problem). This was implemented this way to work around the firewall problem and to achieve maximum performance. See slide 10 at, https://svn.wso2.org/repos/wso2/carbon/platform/branches/4.0.0/products/cg/1.0.0/docs/paper/2012-05-25-CSG-CloudFlow2012.pdf, for more information.
Any way I will look for any optimization that can be done. Thanks for your input. I appreciate that. Rajika 378 "pool-3-thread-1" prio=10 tid=0x00007f755c002000 nid=0x24b6 runnable [0x00007f7540dcc000] 379 java.lang.Thread.State: RUNNABLE 380 at java.net.SocketOutputStream.socketWrite0(Native Method) 381 at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) 382 at java.net.SocketOutputStream.write(SocketOutputStream.java:136) 383 at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:297) 384 at com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:286) 385 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:748) 386 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:736) 387 at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:88) 388 - locked <0x00000000c4ead618> (a com.sun.net.ssl.internal.ssl.AppOutputStream) 389 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) 390 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) 391 - locked <0x00000000c4ead600> (a java.io.BufferedOutputStream) 392 at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) 393 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36) 394 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) 395 at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) 396 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) 397 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) 398 at java.lang.Thread.run(Thread.java:662) "CGPollingTransportReceiver-worker-3" prio=10 tid=0x00007f47b8032000 nid=0x24bb runnable [0x00007f47b4cc2000] 162 java.lang.Thread.State: RUNNABLE 163 at java.net.SocketInputStream.socketRead0(Native Method) 164 at java.net.SocketInputStream.read(SocketInputStream.java:129) 165 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293) 166 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331) 167 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:830) 168 - locked <0x00000000c5d2b770> (a java.lang.Object) 169 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:787) 170 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75) 171 - locked <0x00000000c5d34fe0> (a com.sun.net.ssl.internal.ssl.AppInputStream) 172 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) 173 at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) 174 at java.io.BufferedInputStream.read(BufferedInputStream.java:317) 175 - locked <0x00000000c5e062e8> (a java.io.BufferedInputStream) 176 at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) 177 at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) 178 at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) 179 at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) 180 at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) 181 at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) 182 at org.wso2.carbon.cloud.gateway.common.thrift.gen.CloudGatewayService$Client.recv_exchange(CloudGatewayService.java:114) 183 at org.wso2.carbon.cloud.gateway.common.thrift.gen.CloudGatewayService$Client.exchange(CloudGatewayService.java:99) 184 at org.wso2.carbon.cloud.gateway.common.thrift.CGThriftClient.exchange(CGThriftClient.java:63) 185 at org.wso2.carbon.cloud.gateway.agent.transport.CGPollingTransportTaskManager$MessageExchangeTask.run(CGPollingTransportTaskManager.java:294) 186 at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172) 187 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) 188 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) 189 at java.lang.Thread.run(Thread.java:662) On Thu, Sep 13, 2012 at 3:02 PM, Kasun Weranga <[email protected]> wrote: > Hi Rajika, > > Here I have attached the thread dumps and output of your command. > > > PID 9109 is CG > PID 9163 is AS > > %CPU CPU NI S TIME PID TID > 0.0 - 0 S 00:00:00 9109 9109 > 0.0 - 0 S 00:00:00 9109 9110 > 0.1 - 0 S 00:00:04 9109 9111 > 0.1 - 0 S 00:00:04 9109 9112 > 0.1 - 0 S 00:00:04 9109 9113 > 0.1 - 0 S 00:00:04 9109 9114 > 0.3 - 0 S 00:00:13 9109 9115 > 0.0 - 0 S 00:00:00 9109 9116 > 0.0 - 0 S 00:00:00 9109 9117 > 0.0 - 0 S 00:00:00 9109 9118 > 0.2 - 0 S 00:00:09 9109 9119 > 0.2 - 0 S 00:00:08 9109 9120 > 0.0 - 0 S 00:00:00 9109 9121 > 0.0 - 0 S 00:00:00 9109 9122 > 0.0 - 0 S 00:00:01 9109 9123 > 0.0 - 0 S 00:00:00 9109 9126 > 0.1 - 0 S 00:00:04 9109 9128 > 0.1 - 0 S 00:00:04 9109 9137 > 0.0 - 0 S 00:00:00 9109 9140 > 0.0 - 0 S 00:00:00 9109 9185 > 0.0 - 0 S 00:00:00 9109 9187 > 0.0 - 0 S 00:00:00 9109 9188 > 0.0 - 0 S 00:00:00 9109 9190 > 0.0 - 0 S 00:00:00 9109 9195 > 0.0 - 0 S 00:00:00 9109 9205 > 0.0 - 0 S 00:00:00 9109 9207 > 0.0 - 0 S 00:00:00 9109 9208 > 0.0 - 0 S 00:00:00 9109 9209 > 0.0 - 0 S 00:00:00 9109 9210 > 0.0 - 0 S 00:00:00 9109 9211 > 0.0 - 0 S 00:00:00 9109 9212 > 0.0 - 0 S 00:00:00 9109 9214 > 0.0 - 0 S 00:00:00 9109 9215 > 0.0 - 0 S 00:00:01 9109 9216 > 0.0 - 0 S 00:00:00 9109 9217 > 0.0 - 0 S 00:00:00 9109 9218 > 0.0 - 0 S 00:00:01 9109 9219 > 0.0 - 0 S 00:00:00 9109 9220 > 0.0 - 0 S 00:00:00 9109 9221 > 0.0 - 0 S 00:00:00 9109 9222 > 0.0 - 0 S 00:00:00 9109 9223 > 0.0 - 0 S 00:00:00 9109 9224 > 0.0 - 0 S 00:00:00 9109 9225 > 0.0 - 0 S 00:00:00 9109 9226 > 0.0 - 0 S 00:00:00 9109 9227 > 0.0 - 0 S 00:00:00 9109 9228 > 0.0 - 0 S 00:00:00 9109 9229 > 0.0 - 0 S 00:00:00 9109 9230 > 0.0 - 0 S 00:00:00 9109 9231 > 0.0 - 0 S 00:00:00 9109 9232 > 0.0 - 0 S 00:00:00 9109 9233 > 0.0 - 0 S 00:00:00 9109 9234 > 0.0 - 0 S 00:00:00 9109 9237 > 0.0 - 0 S 00:00:00 9109 9238 > 0.0 - 0 S 00:00:00 9109 9239 > 0.0 - 0 S 00:00:00 9109 9240 > 0.0 - 0 S 00:00:00 9109 9253 > 0.0 - 0 S 00:00:00 9109 9254 > 0.0 - 0 S 00:00:01 9109 9255 > 0.0 - 0 S 00:00:00 9109 9256 > 0.0 - 0 S 00:00:00 9109 9257 > 0.0 - 0 S 00:00:00 9109 9258 > 0.0 - 0 S 00:00:00 9109 9259 > 0.0 - 0 S 00:00:00 9109 9260 > 0.0 - 0 S 00:00:01 9109 9261 > 0.0 - 0 S 00:00:00 9109 9262 > 0.0 - 0 S 00:00:00 9109 9263 > 0.0 - 0 S 00:00:00 9109 9264 > 0.0 - 0 S 00:00:00 9109 9265 > 0.0 - 0 S 00:00:00 9109 9266 > 0.0 - 0 S 00:00:00 9109 9267 > 0.0 - 0 S 00:00:00 9109 9268 > 0.0 - 0 S 00:00:00 9109 9269 > 0.0 - 0 S 00:00:00 9109 9270 > 0.0 - 0 S 00:00:00 9109 9271 > 0.0 - 0 S 00:00:00 9109 9272 > 0.0 - 0 S 00:00:00 9109 9273 > 0.0 - 0 S 00:00:00 9109 9274 > 0.0 - 0 S 00:00:00 9109 9275 > 0.0 - 0 S 00:00:00 9109 9276 > 0.0 - 0 S 00:00:00 9109 9277 > 0.0 - 0 S 00:00:00 9109 9278 > 0.0 - 0 S 00:00:00 9109 9280 > 0.0 - 0 S 00:00:00 9109 9281 > 0.0 - 0 S 00:00:00 9109 9369 > 0.0 - 0 S 00:00:00 9109 9370 > 0.0 - 0 S 00:00:00 9109 9371 > 0.0 - 0 S 00:00:00 9109 9372 > 0.0 - 0 S 00:00:00 9109 9373 > 0.0 - 0 S 00:00:00 9109 9374 > 0.0 - 0 S 00:00:00 9109 9375 > 0.0 - 0 S 00:00:00 9109 9376 > 0.0 - 0 S 00:00:00 9109 9377 > 0.0 - 0 S 00:00:00 9109 9378 > 0.0 - 0 S 00:00:00 9109 9379 > 0.0 - 0 S 00:00:00 9109 9380 > 0.0 - 0 S 00:00:00 9109 9381 > 0.0 - 0 S 00:00:00 9109 9382 > 0.0 - 0 S 00:00:00 9109 9383 > 0.0 - 0 S 00:00:00 9109 9384 > 0.0 - 0 S 00:00:00 9109 9385 > 0.0 - 0 S 00:00:00 9109 9386 > 0.0 - 0 S 00:00:00 9109 9387 > 0.0 - 0 S 00:00:00 9109 9388 > 0.0 - 0 S 00:00:00 9109 9389 > 0.0 - 0 S 00:00:00 9109 9390 > 0.0 - 0 S 00:00:00 9109 9391 > 0.0 - 0 S 00:00:00 9109 9392 > 0.0 - 0 S 00:00:00 9109 9393 > 0.0 - 0 S 00:00:00 9109 9394 > 0.0 - 0 S 00:00:00 9109 9395 > 0.0 - 0 S 00:00:00 9109 9396 > 0.0 - 0 S 00:00:00 9109 9397 > 47.2 - 0 R 00:27:29 9109 9398 > 0.0 - 0 S 00:00:00 9109 9399 > 47.1 - 0 S 00:27:24 9109 9402 > 0.0 - 0 S 00:00:00 9109 9405 > 0.0 - 0 S 00:00:00 9109 9406 > 0.0 - 0 S 00:00:00 9109 9407 > 0.0 - 0 S 00:00:00 9109 9408 > 0.0 - 0 S 00:00:00 9109 9409 > 0.0 - 0 S 00:00:00 9109 9410 > 0.0 - 0 S 00:00:00 9109 9411 > 0.0 - 0 S 00:00:00 9109 9412 > 0.0 - 0 S 00:00:00 9109 9413 > 0.0 - 0 S 00:00:00 9109 9414 > 0.0 - 0 S 00:00:00 9109 9415 > 0.0 - 0 S 00:00:00 9109 9416 > 0.0 - 0 S 00:00:00 9109 9417 > 0.0 - 0 S 00:00:00 9109 9418 > 0.0 - 0 S 00:00:00 9109 9419 > 0.0 - 0 S 00:00:00 9109 9420 > 0.0 - 0 S 00:00:00 9109 9421 > 0.0 - 0 S 00:00:00 9109 9422 > 0.0 - 0 S 00:00:00 9109 9423 > 0.0 - 0 S 00:00:00 9109 9424 > 0.0 - 0 S 00:00:00 9109 9425 > 0.0 - 0 S 00:00:00 9109 9426 > 47.1 - 0 R 00:27:25 9109 9427 > 0.0 - 0 S 00:00:00 9109 9428 > 0.0 - 0 S 00:00:00 9109 9436 > 0.0 - 0 S 00:00:00 9109 11491 > 0.0 - 0 S 00:00:00 9163 9163 > 0.0 - 0 S 00:00:01 9163 9164 > 0.1 - 0 S 00:00:04 9163 9165 > 0.1 - 0 S 00:00:04 9163 9166 > 0.1 - 0 S 00:00:04 9163 9167 > 0.1 - 0 S 00:00:04 9163 9168 > 0.0 - 0 S 00:00:01 9163 9169 > 0.0 - 0 S 00:00:00 9163 9170 > 0.0 - 0 S 00:00:00 9163 9171 > 0.0 - 0 S 00:00:00 9163 9172 > 0.2 - 0 S 00:00:10 9163 9173 > 0.3 - 0 S 00:00:11 9163 9174 > 0.0 - 0 S 00:00:00 9163 9175 > 0.0 - 0 S 00:00:00 9163 9176 > 0.0 - 0 S 00:00:01 9163 9177 > 0.0 - 0 S 00:00:00 9163 9180 > 0.2 - 0 S 00:00:07 9163 9182 > 0.1 - 0 S 00:00:05 9163 9184 > 0.0 - 0 S 00:00:00 9163 9189 > 0.0 - 0 S 00:00:00 9163 9250 > 0.0 - 0 S 00:00:00 9163 9252 > 0.0 - 0 S 00:00:01 9163 9283 > 0.0 - 0 S 00:00:00 9163 9285 > 0.0 - 0 S 00:00:00 9163 9294 > 0.0 - 0 S 00:00:00 9163 9295 > 0.0 - 0 S 00:00:00 9163 9298 > 0.0 - 0 S 00:00:00 9163 9302 > 0.0 - 0 S 00:00:00 9163 9303 > 0.0 - 0 S 00:00:00 9163 9304 > 0.0 - 0 S 00:00:00 9163 9306 > 0.0 - 0 S 00:00:00 9163 9307 > 0.0 - 0 S 00:00:00 9163 9308 > 0.0 - 0 S 00:00:00 9163 9309 > 0.0 - 0 S 00:00:00 9163 9311 > 0.0 - 0 S 00:00:00 9163 9312 > 0.0 - 0 S 00:00:00 9163 9322 > 0.3 - 0 S 00:00:13 9163 9323 > 0.0 - 0 S 00:00:00 9163 9324 > 0.0 - 0 S 00:00:00 9163 9325 > 0.0 - 0 S 00:00:00 9163 9326 > 0.0 - 0 S 00:00:00 9163 9327 > 0.0 - 0 S 00:00:00 9163 9330 > 0.0 - 0 S 00:00:00 9163 9331 > 0.0 - 0 S 00:00:00 9163 9332 > 0.0 - 0 S 00:00:00 9163 9333 > 0.0 - 0 S 00:00:00 9163 9334 > 0.0 - 0 S 00:00:00 9163 9335 > 0.0 - 0 S 00:00:00 9163 9336 > 0.0 - 0 S 00:00:00 9163 9337 > 0.0 - 0 S 00:00:00 9163 9338 > 0.0 - 0 S 00:00:00 9163 9339 > 0.0 - 0 S 00:00:00 9163 9340 > 0.0 - 0 S 00:00:00 9163 9341 > 0.0 - 0 S 00:00:00 9163 9342 > 0.0 - 0 S 00:00:00 9163 9343 > 0.0 - 0 S 00:00:00 9163 9344 > 0.0 - 0 S 00:00:00 9163 9345 > 0.0 - 0 S 00:00:00 9163 9346 > 0.0 - 0 S 00:00:00 9163 9347 > 0.0 - 0 S 00:00:00 9163 9348 > 0.0 - 0 S 00:00:00 9163 9349 > 0.0 - 0 S 00:00:00 9163 9350 > 0.0 - 0 S 00:00:00 9163 9351 > 0.0 - 0 S 00:00:00 9163 9352 > 0.0 - 0 S 00:00:00 9163 9353 > 0.0 - 0 S 00:00:00 9163 9355 > 0.0 - 0 S 00:00:00 9163 9356 > 0.0 - 0 S 00:00:00 9163 9367 > 0.0 - 0 S 00:00:00 9163 9368 > 0.0 - 0 S 00:00:00 9163 9400 > 0.0 - 0 S 00:00:00 9163 9401 > 50.3 - 0 S 00:29:18 9163 9403 > 0.0 - 0 S 00:00:00 9163 9404 > 0.0 - 0 S 00:00:00 9163 9429 > 0.0 - 0 S 00:00:00 9163 9430 > 50.3 - 0 R 00:29:16 9163 9431 > 0.0 - 0 S 00:00:00 9163 9432 > 0.0 - 0 S 00:00:00 9163 9433 > 50.1 - 0 R 00:29:11 9163 9434 > 0.0 - 0 S 00:00:00 9163 9435 > > > Thanks, > KasunW > > On Thu, Sep 13, 2012 at 12:54 PM, Rajika Kumarasiri <[email protected]>wrote: > >> Thanks for your feedback. See explanations below. >> >> When you published a service using Cloud Gateway Agent it enables the CG >> Polling transport in order to receive messages from CG server. When the >> transport is enabled for a one service it will be enabled for all deployed >> service. This is a limitation in Axis2 transport and need to fix >> https://wso2.org/jira/browse/CARBON-13827. >> >> CGPolling transport runs on a busy loop so there can be some overhead but >> CG server should not kill the CPU. Can you send couple of threads dumps in >> CG server and AS server with the out put of the following command for the >> two processes ? I will check this on my side as well. >> >> ps -C java -L -o pcpu,cpu,nice,state,cputime,pid,tid >> >> Rajika >> >> On Thu, Sep 13, 2012 at 12:26 PM, Kasun Weranga <[email protected]> wrote: >> >>> Hi, >>> >>> >>> I face some issue, after publishing some services in AS to Cloud >>> gateway. After sometime I observed cpu usage on my machine was fully >>> occupied by both AS and CG. I have only published 2 services to cloud >>> gateway and it does nothing. Also in my AS I can see some faulty services, >>> it has mentioned the reason for this issue as "Unable to configure the >>> service HelloService for the CGPOLLING transport: The secure token is not >>> set for service 'HelloService'. This service is being marked as faulty and >>> will not be available over the CGPOLLING transport." >>> I suspect there is some issue in the Cloud Gateway Agent. Please have a >>> look. >>> >>> >>> This is the result of the top command >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>> >>> 5012 kasunw 20 0 3539m 382m 17m S 157 5.0 10:37.44 java >>> >>> 4652 kasunw 20 0 3519m 334m 13m S 142 4.3 9:42.11 java >>> >>> >>> >>> Thanks, >>> KasunW >>> >>> >>> >> > > >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
