Just a perspective on this We looked at DRPC, and decided *NOT* to go that route. For small computations (< 2ms), the network hop itself can add an order of magnitude to response times. (also much simpler to keep the computation within a single JVM if you can)
Instead, we are looking to embed Storm directly within our web services. So far, we¹ve embedded Trident State. We may look at topologies next. More thoughts on the topic: http://brianoneill.blogspot.com/2015/03/delta-architectures-unifying-lambda. html (and I apologize for NOT answering your question =) -brian --- Brian O'Neill Chief Technology Officer Health Market Science, a LexisNexis Company 215.588.6024 Mobile @boneill42 <http://www.twitter.com/boneill42> This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited. From: Kashyap Mhaisekar <[email protected]> Reply-To: <[email protected]> Date: Tuesday, March 24, 2015 at 11:58 AM To: "[email protected]" <[email protected]>, <[email protected]> Subject: Storm DRPC Performance Hi, Am doing a feasibility as to using the storm DRPC to power our backend APIs and have run into some problems. The core functionality that needs to be executed takes around 2 ms to execute but in storm takes close to 400 msec. Requesting an approach for them - 1. The LinearDRPCTopologyBuilder uses DRPCSpout by default. Is there a way to increase the no. of instances on them? I see that spout always has executors at 1. The problem is that when more than one request comes in at the same time, the times go up drastically. 2. The bolts are defined as bolt0, bolt1, bolt2 etc. I used only 3 bolts but there are 5 bolts shown (bolt0 to bolt4). Is there a way I can change the names of the bolts shown in storm UI to more meaningful way the way it happens in regular storm? 3. For some of the bolts, the Process Latency is very very high compared to execute latency. I am trying to figure out the reason but am unsuccessful. 4. The Complete Latency on the spout is very high at 500+ msc. I am trying to figure out the reason but am unsuccessful. Configurations used as below: : config.put(Config.DRPC_WORKER_THREADS,64); config.setNumWorkers(50); config.setMaxTaskParallelism(50); config.setMaxSpoutPending(5000); config.put(Config.TOPOLOGY_ACKER_EXECUTORS, 50); config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 16384); config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 16384); Supervisor slot ports: [6700 6701 6702 6703 6704 6705 6706 6707 6708 6709 6710 6711] I have cluster of 5 servers with 8 GB and 4 cores. Of the 5, 3 servers have DRPC running. Storm UI Details: Please do help me out. Regards, Kashyap
