Thanks Harsha, I had looked into that but I just didnt know if its a good idea to add another database query to the process. But I actually ended up doing that which seem to be pretty good. Thanks.
Kushan -- Kushan Maskey 817.403.7500 On Fri, Aug 29, 2014 at 5:14 PM, Harsha <[email protected]> wrote: > Kushan, > Why not use cassandra counter to implement this > http://www.datastax.com/documentation/cql/3.0/cql/cql_using/use_counter_t.html > . > you can create a counter field in a table in cassandra and let the storm > bolts update it. > I don't have much knowledge on internal representation of cassandra > counters and how accurate they will be. > -Harsha > > > On Fri, Aug 29, 2014, at 12:15 PM, Kushan Maskey wrote: > > I have a batch process that runs more than 100K records of data and loads > into Cassandra. I am having hard time validating the exact number of data > that gets stored into C*. Now C* has more than 20Million records and when I > do Select COUNT(1) FROM TABLE, I get *Request did not complete within > rpc_timeout. *I tried to increate teh rpc_timeout but didnt help. The > load process completes successfully without having any errors in the log. > So I assumed that storm and kafka is set up correctly. > > I have 5 bolts and now I am at a point to add counter feature to the bolts > to count how many messages successfully inserted. I tried to add a static > counter field. But this will not work as its in clustered environment and > static fields are not good. Can anyone suggest me a better way to validate > the number of records that gets inserted into C*? This is one of the > initail requirements to make sure that x number of records we processed > thru batch and same number of records got inserted into C*. > > I also tried to set StormConfig with new property like > > stormConfig.put("Records_add_counter", 0); > Then I wanted to increment the counter by one everytie the message comes a > particularbolt but I get UnsupportedOperationException. I am thinking you > cannot update the the value of a property at t his point. > > Any help will be appreciated. Thanks. > > -- > Kushan Maskey > 817.403.7500 > > >
