Jia, Because the BatchBolt is not supported by TopologyBuilder and we need to use LinearDRPCTopologyBuilder. I believe this was given as a comment by Nathan Marz in some mail... Thanks Kashyap
On Tue, Mar 24, 2015 at 12:16 PM, <[email protected]> wrote: > I am no expert on this, but for your first question, if you want to have > more spouts for the drpc, why not use a regular TopologyBuilder? > > DRPCSpout drpcSpout; > if (mode.equals("LocalMode")) > { > LocalDRPC drpc = new LocalDRPC(); > drpcSpout = new DRPCSpout(“name", drpc); > } else > { > drpcSpout = new DRPCSpout(“name"); > topoConf.put(Config.DRPC_SERVERS, Arrays.asList(“drpcServer")); > } > > TopologyBuilder builder = new TopologyBuilder(); > > builder.setSpout("drpc", drpcSpout, 10); > > > -Jia > > > On Tue, Mar 24, 2015 at 9:45 AM, Kashyap Mhaisekar <[email protected]> > wrote: > >> Brian, >> Thanks for the suggestion. The 2 ms operation will need to be iterated >> over a million times :) and thats where we wanted to parallelize things and >> hence evaluating storm. >> >> Will look into this option too. >> >> Regards, >> Kashyap >> >> On Tue, Mar 24, 2015 at 11:06 AM, Brian O'Neill <[email protected]> >> wrote: >> >>> >>> Just a perspective on this… >>> >>> We looked at DRPC, and decided *NOT* to go that route. For small >>> computations (< 2ms), the network hop itself can add an order of magnitude >>> to response times. (also much simpler to keep the computation within a >>> single JVM — if you can) >>> >>> Instead, we are looking to embed Storm directly within our web services. >>> So far, we’ve embedded Trident State. We may look at topologies next. >>> >>> More thoughts on the topic: >>> >>> http://brianoneill.blogspot.com/2015/03/delta-architectures-unifying-lambda.html >>> >>> (and I apologize for NOT answering your question =) >>> >>> -brian >>> >>> --- >>> >>> *Brian O'Neill * >>> >>> Chief Technology Officer >>> >>> Health Market Science, a LexisNexis Company >>> >>> 215.588.6024 Mobile • @boneill42 <http://www.twitter.com/boneill42> >>> >>> >>> This information transmitted in this email message is for the intended >>> recipient only and may contain confidential and/or privileged material. If >>> you received this email in error and are not the intended recipient, or the >>> person responsible to deliver it to the intended recipient, please contact >>> the sender at the email above and delete this email and any attachments and >>> destroy any copies thereof. Any review, retransmission, dissemination, >>> copying or other use of, or taking any action in reliance upon, this >>> information by persons or entities other than the intended recipient is >>> strictly prohibited. >>> >>> >>> >>> >>> From: Kashyap Mhaisekar <[email protected]> >>> Reply-To: <[email protected]> >>> Date: Tuesday, March 24, 2015 at 11:58 AM >>> To: "[email protected]" <[email protected]>, >>> <[email protected]> >>> Subject: Storm DRPC Performance >>> >>> Hi, >>> Am doing a feasibility as to using the storm DRPC to power our backend >>> APIs and have run into some problems. The core functionality that needs to >>> be executed takes around 2 ms to execute but in storm takes close to 400 >>> msec. Requesting an approach for them - >>> >>> 1. The LinearDRPCTopologyBuilder uses DRPCSpout by default. Is there a >>> way to increase the no. of instances on them? I see that spout always has >>> executors at 1. The problem is that when more than one request comes in at >>> the same time, the times go up drastically. >>> 2. The bolts are defined as bolt0, bolt1, bolt2 etc. I used only 3 bolts >>> but there are 5 bolts shown (bolt0 to bolt4). Is there a way I can change >>> the names of the bolts shown in storm UI to more meaningful way the way it >>> happens in regular storm? >>> 3. For some of the bolts, the Process Latency is very very high compared >>> to execute latency. I am trying to figure out the reason but am >>> unsuccessful. >>> 4. The Complete Latency on the spout is very high at 500+ msc. I am >>> trying to figure out the reason but am unsuccessful. >>> >>> Configurations used as below: : >>> >>> config.put(Config.DRPC_WORKER_THREADS,64); >>> config.setNumWorkers(50); >>> config.setMaxTaskParallelism(50); >>> config.setMaxSpoutPending(5000); >>> config.put(Config.TOPOLOGY_ACKER_EXECUTORS, 50); >>> config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 16384); >>> config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 16384); >>> Supervisor slot ports: [6700 6701 6702 6703 6704 6705 6706 6707 6708 >>> 6709 6710 6711] >>> I have cluster of 5 servers with 8 GB and 4 cores. Of the 5, 3 servers >>> have DRPC running. >>> >>> >>> *Storm UI Details:* >>> >>> <image.png> >>> >>> >>> Please do help me out. >>> >>> Regards, >>> Kashyap >>> >> >> >
