After increasing memory, Most of the drpc request fail. I get the following error,
=== 2016-08-12T23:50:12.461+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 10 start at 1471045208 2016-08-12T23:55:17.471+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 11 start at 1471045515 2016-08-12T23:55:32.473+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 12 start at 1471045530 2016-08-12T23:55:47.473+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 13 start at 1471045545 2016-08-12T23:56:27.475+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 14 start at 1471045586 2016-08-12T23:59:37.482+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 15 start at 1471045775 2016-08-13T00:01:12.486+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 16 start at 1471045867 2016-08-13T00:09:42.501+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 17 start at 1471046381 === Is there anyway to debug drpc based issues ? Sometime restarting drpc server resolves the issue, sometimes, it doesnt. Thanks, ./Rahul On Wed, Aug 10, 2016 at 11:17 PM, Rahul R <[email protected]> wrote: > Sure , thats one of the solutions. But I am not sure, whats causing DRPC > to consume soo much of memory. Any suggestions of how to debug this ? > > Thanks, > ./Rahul > > On Wed, Aug 10, 2016 at 6:24 PM, darion.yaphet <[email protected]> wrote: > >> Maybe you should check the JVM memory parameter -Xms and -Xmx to found >> memory usage. >> >> >> >> At 2016-08-11 08:00:54, "Rahul R" <[email protected]> wrote: >> >> I have a drpc server running in nimbus server. After every 'X' minutes, >> I see the following in the >> drpc log >> >> === >> java.lang.OutOfMemoryError: Java heap space >> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_91] >> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_91] >> at org.apache.thrift7.server.TNonblockingServer$FrameBuffer. >> read(TNonblockingServer.java:491) ~[storm-core-0.9.6.jar:0.9.6] >> at org.apache.thrift7.server.TNonblockingServer$SelectThread. >> handleRead(TNonblockingServer.java:359) ~[storm-core-0.9.6.jar:0.9.6] >> at org.apache.thrift7.server.TNonblockingServer$SelectThread. >> select(TNonblockingServer.java:304) ~[storm-core-0.9.6.jar:0.9.6] >> at org.apache.thrift7.server.TNonblockingServer$SelectThread. >> run(TNonblockingServer.java:243) ~[storm-core-0.9.6.jar:0.9.6] >> ==== >> >> >> There is enough memory available in the system . >> >> ==== >> storm@alerts:~/apache-storm-0.9.6/logs$ free -m >> total used free shared buffers cached >> Mem: 7985 2217 5767 0 81 1203 >> -/+ buffers/cache: 932 7052 >> Swap: 0 0 0 >> ==== >> >> >> >> My DRPC Spout throws the following error >> >> ===== >> java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: >> java.net.ConnectException: Connection refused at >> backtype.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40) >> at backtype.storm.drpc.DRPCSpout.open(DRPCSpout.java:87) at >> backtype.storm.daemon.executor$fn*3371$fn*3386.invoke(executor.clj:529) >> at backtype.storm.util$async_loop$fn__460.invoke(util.clj:461) at >> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745) >> Caused by: org.apache.thrift7.transport.TTransportException: >> java.net.ConnectException: >> Connection refused at >> org.apache.thrift7.transport.TSocket.open(TSocket.java:183) >> at >> org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81) >> at >> backtype.storm.drpc.DRPCInvocationsClient.connect(DRPCInvocationsClient.java:47) >> at >> backtype.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:38) >> ... 5 more Caused by: java.net.ConnectException: Connection refused at >> java.net.PlainSocketImpl.socketConnect(Native Method) at java.net >> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) >> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) >> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net >> .Socket.connect(Socket.java:589) at >> org.apache.thrift7.transport.TSocket.open(TSocket.java:178) >> ... 8 more >> ======= >> >> >> Restarting the DRPC Server solves the problem for the moment. Is there >> any other better solution ? >> >> >> >> Thanks, >> ./Rahul >> >> >> >> >> > >
