Guys Help!!! On Tue, Mar 24, 2015 at 1:18 PM, Kashyap Mhaisekar <[email protected]> wrote:
> Jia, > Because the BatchBolt is not supported by TopologyBuilder and we need to > use LinearDRPCTopologyBuilder. I believe this was given as a comment by > Nathan Marz in some mail... > Thanks > Kashyap > > On Tue, Mar 24, 2015 at 12:16 PM, <[email protected]> wrote: > >> I am no expert on this, but for your first question, if you want to have >> more spouts for the drpc, why not use a regular TopologyBuilder? >> >> DRPCSpout drpcSpout; >> if (mode.equals("LocalMode")) >> { >> LocalDRPC drpc = new LocalDRPC(); >> drpcSpout = new DRPCSpout(“name", drpc); >> } else >> { >> drpcSpout = new DRPCSpout(“name"); >> topoConf.put(Config.DRPC_SERVERS, Arrays.asList(“drpcServer")); >> } >> >> TopologyBuilder builder = new TopologyBuilder(); >> >> builder.setSpout("drpc", drpcSpout, 10); >> >> >> -Jia >> >> >> On Tue, Mar 24, 2015 at 9:45 AM, Kashyap Mhaisekar <[email protected]> >> wrote: >> >>> Brian, >>> Thanks for the suggestion. The 2 ms operation will need to be iterated >>> over a million times :) and thats where we wanted to parallelize things and >>> hence evaluating storm. >>> >>> Will look into this option too. >>> >>> Regards, >>> Kashyap >>> >>> On Tue, Mar 24, 2015 at 11:06 AM, Brian O'Neill <[email protected]> >>> wrote: >>> >>>> >>>> Just a perspective on this… >>>> >>>> We looked at DRPC, and decided *NOT* to go that route. For small >>>> computations (< 2ms), the network hop itself can add an order of magnitude >>>> to response times. (also much simpler to keep the computation within a >>>> single JVM — if you can) >>>> >>>> Instead, we are looking to embed Storm directly within our web services. >>>> So far, we’ve embedded Trident State. We may look at topologies next. >>>> >>>> More thoughts on the topic: >>>> >>>> http://brianoneill.blogspot.com/2015/03/delta-architectures-unifying-lambda.html >>>> >>>> (and I apologize for NOT answering your question =) >>>> >>>> -brian >>>> >>>> --- >>>> >>>> *Brian O'Neill * >>>> >>>> Chief Technology Officer >>>> >>>> Health Market Science, a LexisNexis Company >>>> >>>> 215.588.6024 Mobile • @boneill42 <http://www.twitter.com/boneill42> >>>> >>>> >>>> This information transmitted in this email message is for the >>>> intended recipient only and may contain confidential and/or privileged >>>> material. If you received this email in error and are not the intended >>>> recipient, or the person responsible to deliver it to the intended >>>> recipient, please contact the sender at the email above and delete this >>>> email and any attachments and destroy any copies thereof. Any review, >>>> retransmission, dissemination, copying or other use of, or taking any >>>> action in reliance upon, this information by persons or entities other than >>>> the intended recipient is strictly prohibited. >>>> >>>> >>>> >>>> >>>> From: Kashyap Mhaisekar <[email protected]> >>>> Reply-To: <[email protected]> >>>> Date: Tuesday, March 24, 2015 at 11:58 AM >>>> To: "[email protected]" <[email protected]>, >>>> <[email protected]> >>>> Subject: Storm DRPC Performance >>>> >>>> Hi, >>>> Am doing a feasibility as to using the storm DRPC to power our backend >>>> APIs and have run into some problems. The core functionality that needs to >>>> be executed takes around 2 ms to execute but in storm takes close to 400 >>>> msec. Requesting an approach for them - >>>> >>>> 1. The LinearDRPCTopologyBuilder uses DRPCSpout by default. Is there a >>>> way to increase the no. of instances on them? I see that spout always has >>>> executors at 1. The problem is that when more than one request comes in at >>>> the same time, the times go up drastically. >>>> 2. The bolts are defined as bolt0, bolt1, bolt2 etc. I used only 3 >>>> bolts but there are 5 bolts shown (bolt0 to bolt4). Is there a way I can >>>> change the names of the bolts shown in storm UI to more meaningful way the >>>> way it happens in regular storm? >>>> 3. For some of the bolts, the Process Latency is very very high >>>> compared to execute latency. I am trying to figure out the reason but am >>>> unsuccessful. >>>> 4. The Complete Latency on the spout is very high at 500+ msc. I am >>>> trying to figure out the reason but am unsuccessful. >>>> >>>> Configurations used as below: : >>>> >>>> config.put(Config.DRPC_WORKER_THREADS,64); >>>> config.setNumWorkers(50); >>>> config.setMaxTaskParallelism(50); >>>> config.setMaxSpoutPending(5000); >>>> config.put(Config.TOPOLOGY_ACKER_EXECUTORS, 50); >>>> config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 16384); >>>> config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 16384); >>>> Supervisor slot ports: [6700 6701 6702 6703 6704 6705 6706 6707 6708 >>>> 6709 6710 6711] >>>> I have cluster of 5 servers with 8 GB and 4 cores. Of the 5, 3 servers >>>> have DRPC running. >>>> >>>> >>>> *Storm UI Details:* >>>> >>>> <image.png> >>>> >>>> >>>> Please do help me out. >>>> >>>> Regards, >>>> Kashyap >>>> >>> >>> >> >
