Brian, Thanks for the suggestion. The 2 ms operation will need to be iterated over a million times :) and thats where we wanted to parallelize things and hence evaluating storm.
Will look into this option too. Regards, Kashyap On Tue, Mar 24, 2015 at 11:06 AM, Brian O'Neill <[email protected]> wrote: > > Just a perspective on this… > > We looked at DRPC, and decided *NOT* to go that route. For small > computations (< 2ms), the network hop itself can add an order of magnitude > to response times. (also much simpler to keep the computation within a > single JVM — if you can) > > Instead, we are looking to embed Storm directly within our web services. > So far, we’ve embedded Trident State. We may look at topologies next. > > More thoughts on the topic: > > http://brianoneill.blogspot.com/2015/03/delta-architectures-unifying-lambda.html > > (and I apologize for NOT answering your question =) > > -brian > > --- > > *Brian O'Neill * > > Chief Technology Officer > > Health Market Science, a LexisNexis Company > > 215.588.6024 Mobile • @boneill42 <http://www.twitter.com/boneill42> > > > This information transmitted in this email message is for the intended > recipient only and may contain confidential and/or privileged material. If > you received this email in error and are not the intended recipient, or the > person responsible to deliver it to the intended recipient, please contact > the sender at the email above and delete this email and any attachments and > destroy any copies thereof. Any review, retransmission, dissemination, > copying or other use of, or taking any action in reliance upon, this > information by persons or entities other than the intended recipient is > strictly prohibited. > > > > > From: Kashyap Mhaisekar <[email protected]> > Reply-To: <[email protected]> > Date: Tuesday, March 24, 2015 at 11:58 AM > To: "[email protected]" <[email protected]>, < > [email protected]> > Subject: Storm DRPC Performance > > Hi, > Am doing a feasibility as to using the storm DRPC to power our backend > APIs and have run into some problems. The core functionality that needs to > be executed takes around 2 ms to execute but in storm takes close to 400 > msec. Requesting an approach for them - > > 1. The LinearDRPCTopologyBuilder uses DRPCSpout by default. Is there a way > to increase the no. of instances on them? I see that spout always has > executors at 1. The problem is that when more than one request comes in at > the same time, the times go up drastically. > 2. The bolts are defined as bolt0, bolt1, bolt2 etc. I used only 3 bolts > but there are 5 bolts shown (bolt0 to bolt4). Is there a way I can change > the names of the bolts shown in storm UI to more meaningful way the way it > happens in regular storm? > 3. For some of the bolts, the Process Latency is very very high compared > to execute latency. I am trying to figure out the reason but am > unsuccessful. > 4. The Complete Latency on the spout is very high at 500+ msc. I am > trying to figure out the reason but am unsuccessful. > > Configurations used as below: : > > config.put(Config.DRPC_WORKER_THREADS,64); > config.setNumWorkers(50); > config.setMaxTaskParallelism(50); > config.setMaxSpoutPending(5000); > config.put(Config.TOPOLOGY_ACKER_EXECUTORS, 50); > config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 16384); > config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 16384); > Supervisor slot ports: [6700 6701 6702 6703 6704 6705 6706 6707 6708 6709 > 6710 6711] > I have cluster of 5 servers with 8 GB and 4 cores. Of the 5, 3 servers > have DRPC running. > > > *Storm UI Details:* > > [image: Inline image 1] > > Please do help me out. > > Regards, > Kashyap >
