Here is my storm config.
storm.config.setMaxTaskParallelism=4 storm.config.setNumWorkers=20 storm.config.setMaxSpoutPending=5000 storm.config.numAckers=1000 I am guessing I need to increase the maxTaskParallelism more. IF that is the case how much would you suggest? Any help will be highly appreciated. Thanks. -- Kushan Maskey 817.403.7500 On Mon, Sep 22, 2014 at 9:20 PM, Michael Rose <mich...@fullcontact.com> wrote: > Storm is not your bottleneck. Check your Storm code to 1) ensure you're > parallelizing your writes and 2) you're batching writes to your external > resources if possible. Some quick napkin math shows you only doing 110 > writes/s, which seems awfully low. > > Michael Rose (@Xorlev <https://twitter.com/xorlev>) > Senior Platform Engineer, FullContact <http://www.fullcontact.com/> > mich...@fullcontact.com > > On Mon, Sep 22, 2014 at 8:05 PM, Kushan Maskey < > kushan.mas...@mmillerassociates.com> wrote: > >> I am trying to load 20 M records into Cassandra database through >> Kafka-Storm. I am able to post all the data in 5 mins into Kafka. But >> reading it from storm and inserting into Cassandra, Couch and Solr is kind >> of very slow. It has been running for past 5 hours and so far only 2 >> Million records. >> >> How do I make the storm perform faster? Coz in this pace it will take >> couple of days to load all the data. >> >> -- >> Kushan Maskey >> >> >
