Hello,
We've been using Thrift for over a year now and have been very happy
with how it makes it easy to implement RPC across languages. However,
we were facing a perf bottleneck since each call was taking about
50-75ms which means we can't make more than 50k calls in an hour. We
are using this for a message broker and want to be able to send out
millions of messages per hour. This was on a single thread. On trying
out the most obvious multi-threaded implementation where we pump out
the calls on multiple-threads, I get the exceptions below.
What I am not sure about is why thrift has an out of memory problem on
the server (thrift-handler) end when I am pumping in messages on
multiple-threads from the client. I have a TThreadPoolServer on the
server end.
Is there something I am missing?
Regards
Krishna.
Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space
at com.facebook.thrift.protocol.TBinaryProtocol.readStringBody(Unknown
Source)
at com.facebook.thrift.protocol.TBinaryProtocol.readString(Unknown
Source)
at com.facebook.thrift.protocol.TBinaryProtocol.readMessageBegin(Unknown
Source)
at
in.capillary.ifaces.nsadmin.NSAdminService$Client.recv_processMessageFromClient(NSAdminService.java:154)
at
in.capillary.ifaces.nsadmin.NSAdminService$Client.processMessageFromClient(NSAdminService.java:139)
at
in.capillary.msging.system.NSAdminClientAdapter.send(NSAdminClientAdapter.java:108)
at
in.capillary.msging.system.NotificationDeliveryClient.send(NotificationDeliveryClient.java:42)
at
in.capillary.msging.system.ThreadWorker.retrieveAndSendSingleMessage(ThreadWorker.java:64)
at
in.capillary.msging.system.ThreadWorker.makeDeliveriesTillStop(ThreadWorker.java:82)
at
in.capillary.msging.test.TestShopbookPerf$3.run(TestShopbookPerf.java:108)
at java.lang.Thread.run(Thread.java:619)
00:24:21,092 INFO [Thread-0] NSAdminClientAdapter:80 - Registering with NSAdmin
00:24:21,094 ERROR [Thread-0] NSAdminClientAdapter:24 - Exception
thrown: com.facebook.thrift.TApplicationException:
processMessageFromClient failed: unknown result
At:
NSAdminService.java:in.capillary.ifaces.nsadmin.NSAdminService$Client:recv_processMessageFromClient:169
NSAdminService.java:in.capillary.ifaces.nsadmin.NSAdminService$Client:processMessageFromClient:139
NSAdminClientAdapter.java:in.capillary.msging.system.NSAdminClientAdapter:send:108
NotificationDeliveryClient.java:in.capillary.msging.system.NotificationDeliveryClient:send:42
ThreadWorker.java:in.capillary.msging.system.ThreadWorker:retrieveAndSendSingleMessage:64
ThreadWorker.java:in.capillary.msging.system.ThreadWorker:makeDeliveriesTillStop:82
TestShopbookPerf.java:in.capillary.msging.test.TestShopbookPerf$1:run:86
Thread.java:java.lang.Thread:run:619
com.facebook.thrift.TApplicationException: processMessageFromClient
failed: unknown result
at
in.capillary.msging.system.NSAdminClientAdapter.send(NSAdminClientAdapter.java:111)
at
in.capillary.msging.system.NotificationDeliveryClient.send(NotificationDeliveryClient.java:42)
at
in.capillary.msging.system.ThreadWorker.retrieveAndSendSingleMessage(ThreadWorker.java:64)
at
in.capillary.msging.system.ThreadWorker.makeDeliveriesTillStop(ThreadWorker.java:82)
at
in.capillary.msging.test.TestShopbookPerf$1.run(TestShopbookPerf.java:86)
at java.lang.Thread.run(Thread.java:619)
--
http://www.linkedin.com/in/kpowerinfinity
Note: this is the original facebook implementation of thrift.
