Kushan,
Why not use cassandra counter to implement
this [1]http://www.datastax.com/documentation/cql/3.0/cql/cql_u
sing/use_counter_t.html.
you can create a counter field in a table in cassandra and let
the storm bolts update it.
I don't have much knowledge on internal representation of
cassandra counters and how accurate they will be.
-Harsha
On Fri, Aug 29, 2014, at 12:15 PM, Kushan Maskey wrote:
I have a batch process that runs more than 100K records of data
and loads into Cassandra. I am having hard time validating the
exact number of data that gets stored into C*. Now C* has more
than 20Million records and when I do Select COUNT(1) FROM
TABLE, I get Request did not complete within rpc_timeout. I
tried to increate teh rpc_timeout but didnt help. The load
process completes successfully without having any errors in the
log. So I assumed that storm and kafka is set up correctly.
I have 5 bolts and now I am at a point to add counter feature
to the bolts to count how many messages successfully inserted.
I tried to add a static counter field. But this will not work
as its in clustered environment and static fields are not good.
Can anyone suggest me a better way to validate the number of
records that gets inserted into C*? This is one of the initail
requirements to make sure that x number of records we processed
thru batch and same number of records got inserted into C*.
I also tried to set StormConfig with new property like
stormConfig.put("Records_add_counter", 0);
Then I wanted to increment the counter by one everytie the
message comes a particularbolt but I get
UnsupportedOperationException. I am thinking you cannot update
the the value of a property at t his point.
Any help will be appreciated. Thanks.
--
Kushan Maskey
817.403.7500
References
1.
http://www.datastax.com/documentation/cql/3.0/cql/cql_using/use_counter_t.html
