typo in previous email The emit method in the query bolt takes about 200(instead of 20) seconds..
On Fri, Jul 11, 2014 at 11:58 AM, Chen Wang <[email protected]> wrote: > Hi, Guys, > I have a storm topology, with a single thread bolt querying large amount > of data (From elasticsearch), and emit to a HBase bolt(10 threads), doing > some filtering, then emit to Arvo bolt.(10threads) The arvo bolt simply > emit the tuple to arvo client, which will be received by two flume node and > then sink into hdfs. I am testing in local mode. > > In the query bolt, i am getting around 15000 entries in a batch, the > query itself takes about 4second, however, he emit method in the query bolt > takes about 20 seconds. Does it mean that > the downstream bolt(HBaseBolt and Avro bolt) cannot catch up with the > query bolt? > > How can I tune my topology to make this process as fast as possible? I > tried to increase the HBase thread to 20 but it does not seem to help. > > I use shuffleGrouping from query bolt to hbase bolt, and from hbase bolt > to avro. > > Thanks for any advice. > Chen >
