I am no expert on this, but for your first question, if you want to have more
spouts for the drpc, why not use a regular TopologyBuilder?
DRPCSpout drpcSpout;
if (mode.equals("LocalMode"))
{
LocalDRPC drpc = new LocalDRPC();
drpcSpout = new DRPCSpout(“name", drpc);
} else
{
drpcSpout = new DRPCSpout(“name");
topoConf.put(Config.DRPC_SERVERS, Arrays.asList(“drpcServer"));
}
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("drpc", drpcSpout, 10);
-Jia
On Tue, Mar 24, 2015 at 9:45 AM, Kashyap Mhaisekar <[email protected]>
wrote:
> Brian,
> Thanks for the suggestion. The 2 ms operation will need to be iterated
> over a million times :) and thats where we wanted to parallelize things and
> hence evaluating storm.
> Will look into this option too.
> Regards,
> Kashyap
> On Tue, Mar 24, 2015 at 11:06 AM, Brian O'Neill <[email protected]>
> wrote:
>>
>> Just a perspective on this…
>>
>> We looked at DRPC, and decided *NOT* to go that route. For small
>> computations (< 2ms), the network hop itself can add an order of magnitude
>> to response times. (also much simpler to keep the computation within a
>> single JVM — if you can)
>>
>> Instead, we are looking to embed Storm directly within our web services.
>> So far, we’ve embedded Trident State. We may look at topologies next.
>>
>> More thoughts on the topic:
>>
>> http://brianoneill.blogspot.com/2015/03/delta-architectures-unifying-lambda.html
>>
>> (and I apologize for NOT answering your question =)
>>
>> -brian
>>
>> ---
>>
>> *Brian O'Neill *
>>
>> Chief Technology Officer
>>
>> Health Market Science, a LexisNexis Company
>>
>> 215.588.6024 Mobile • @boneill42 <http://www.twitter.com/boneill42>
>>
>>
>> This information transmitted in this email message is for the intended
>> recipient only and may contain confidential and/or privileged material. If
>> you received this email in error and are not the intended recipient, or the
>> person responsible to deliver it to the intended recipient, please contact
>> the sender at the email above and delete this email and any attachments and
>> destroy any copies thereof. Any review, retransmission, dissemination,
>> copying or other use of, or taking any action in reliance upon, this
>> information by persons or entities other than the intended recipient is
>> strictly prohibited.
>>
>>
>>
>>
>> From: Kashyap Mhaisekar <[email protected]>
>> Reply-To: <[email protected]>
>> Date: Tuesday, March 24, 2015 at 11:58 AM
>> To: "[email protected]" <[email protected]>, <
>> [email protected]>
>> Subject: Storm DRPC Performance
>>
>> Hi,
>> Am doing a feasibility as to using the storm DRPC to power our backend
>> APIs and have run into some problems. The core functionality that needs to
>> be executed takes around 2 ms to execute but in storm takes close to 400
>> msec. Requesting an approach for them -
>>
>> 1. The LinearDRPCTopologyBuilder uses DRPCSpout by default. Is there a way
>> to increase the no. of instances on them? I see that spout always has
>> executors at 1. The problem is that when more than one request comes in at
>> the same time, the times go up drastically.
>> 2. The bolts are defined as bolt0, bolt1, bolt2 etc. I used only 3 bolts
>> but there are 5 bolts shown (bolt0 to bolt4). Is there a way I can change
>> the names of the bolts shown in storm UI to more meaningful way the way it
>> happens in regular storm?
>> 3. For some of the bolts, the Process Latency is very very high compared
>> to execute latency. I am trying to figure out the reason but am
>> unsuccessful.
>> 4. The Complete Latency on the spout is very high at 500+ msc. I am
>> trying to figure out the reason but am unsuccessful.
>>
>> Configurations used as below: :
>>
>> config.put(Config.DRPC_WORKER_THREADS,64);
>> config.setNumWorkers(50);
>> config.setMaxTaskParallelism(50);
>> config.setMaxSpoutPending(5000);
>> config.put(Config.TOPOLOGY_ACKER_EXECUTORS, 50);
>> config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 16384);
>> config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 16384);
>> Supervisor slot ports: [6700 6701 6702 6703 6704 6705 6706 6707 6708 6709
>> 6710 6711]
>> I have cluster of 5 servers with 8 GB and 4 cores. Of the 5, 3 servers
>> have DRPC running.
>>
>>
>> *Storm UI Details:*
>>
>> [image: Inline image 1]
>>
>> Please do help me out.
>>
>> Regards,
>> Kashyap
>>
