You can totally avoid the impact of tombstones by rotating your partition key in the exact counts table, and only deleting whole partitions once you've counted them. Once you've counted them you never have cause to read that partition key again.
You can totally store the final counts in Cassandra as a standard (non-counter) column, and you can even use counters to keep track of the time slices which haven't been formally counted yet so that you can get reasonably accurate information about time slices that haven't been trued up yet. This is basically what's called a Lambda architecture - use efficient real time processing to get pretty close to accurate values when real time performance matters, then use a cleanup process to get perfectly accurate values when you can afford non-real-time processing times, and store that final computation so that you can continue to access it quickly. > is there any technical reason behind it (just out of curiosity)? Distributed counting is a fundamentally hard problem if you wish to do so in a manner that avoids bottlenecks (i.e. not distributed) and also provides for perfect accuracy. There's plenty of research in this area, and there isn't a single algorithm that provides for all the properties we would hope for. Instead there are different algorithms that make different tradeoffs. The way that Cassandra's counters can fail is that most operations in Cassandra are idempotent - if we're not sure whether an update has been applied correctly or not, we can simply apply it again, because it's safe to do twice. Counters are not idempotent. If you try to increment a counter, and you're not certain whether the increment was successful or not, it is *not* safe to try again (if it was successful the previous time, you've now incremented twice when it should have been once). Most of the time counters are reasonable and accurate, but in failure scenarios you may get some changes applied more than once, or not at all. With that in mind, you might find that being perfectly accurate most of the time, and being within a fraction of a percent the other times is acceptable. If so, counters are your friend, and if not, a more complex lambda style approach as we've been advocating here is best. On Tue, Dec 30, 2014 at 10:54 PM, Ajay <ajay.ga...@gmail.com> wrote: > Thanks Janne and Rob. > > The idea is like this : To store the User clicks on Cassandra and a > scheduler to count/aggregate the clicks per link or ad > hourly/daily/monthly and store in My SQL (or may be in Cassandra itself). > Since tombstones will be deleted only after some days (as per > configuration), could the subsequent queries to count the rows get affected > (I mean say thousands of tombstones will affect the performance of the > query) ? > > Secondly as I understand from this mail thread, the counter is not correct > for this use case, is there any technical reason behind it (just out of > curiosity)? > > Thanks > Ajay > > On Tue, Dec 30, 2014 at 10:37 PM, Janne Jalkanen <janne.jalka...@ecyrd.com > > wrote: > >> >> Hi! >> >> Yes, since all the writes for a partition (or row if you speak Thrift) >> always go to the same replicas, you will need to design to avoid hotspots - >> a pure day row will cause all the writes for a single day to go to the same >> replicas, so those nodes will have to work really hard for a day, and then >> the next day it’s again hard work for some other nodes. If you have an >> user id there in front, then it would distribute better. >> >> For tombstone purposes think of your access patterns; if you have a >> date-based system, it probably does not matter since you will scan those >> UUIDs once, and then they will be tombstoned away. It’s cleaner if you can >> delete the entire row with a single command, but as long as you never read >> it again, I don’t think this matters much. >> >> The real problems with wide rows come with compaction, and you shouldn’t >> have much problems with compaction because this is an append-only row, so >> it should be fine as a fairly wide row. Make some back-of-the-envelope >> calculations and if it looks like you’re going to be hitting tens of >> millions of columns per day, then store per hour. >> >> One important thing: in order not to lose clicks, always use timeuuids >> instead of timestamps (or else two clicks coming in for the same id would >> overwrite itself and count as one). >> >> /Janne >> >> On 30 Dec 2014, at 06:28, Ajay <ajay.ga...@gmail.com> wrote: >> >> Thanks Janne, Alain and Eric. >> >> Now say I go with counters (hourly, daily, monthly) and also store UUID >> as below: >> >> user Id : yyyy/mm/dd as row key and dynamic columns for each click with >> column key as timestamp and value as empty. Periodically count the columns >> and rows and correct the counters. Now in this case, there will be one row >> per day but as many columns as user click. >> >> Other way is to store row per hour >> user id : yyyy/mm/dd/hh as row key and dynamic columns for each click >> with column key as timestamp and value as empty. >> >> Is there any difference (in performance or any known issues) between more >> rows Vs more columns as Cassandra deletes them through tombstones (say by >> default 20 days). >> >> Thanks >> Ajay >> >> On Mon, Dec 29, 2014 at 7:47 PM, Eric Stevens <migh...@gmail.com> wrote: >> >>> > If the counters get incorrect, it could't be corrected >>> >>> You'd have to store something that allowed you to correct it. For >>> example, the TimeUUID approach to keep true counts, which are slow to read >>> but accurate, and a background process that trues up your counter columns >>> periodically. >>> >>> On Mon, Dec 29, 2014 at 7:05 AM, Ajay <ajay.ga...@gmail.com> wrote: >>> >>>> Thanks for the clarification. >>>> >>>> In my case, Cassandra is the only storage. If the counters get >>>> incorrect, it could't be corrected. For that if we store raw data, we can >>>> as well go that approach. But the granularity has to be as seconds level as >>>> more than one user can click the same link. So the data will be huge with >>>> more writes and more rows to count for reads right? >>>> >>>> Thanks >>>> Ajay >>>> >>>> >>>> On Mon, Dec 29, 2014 at 7:10 PM, Alain RODRIGUEZ <arodr...@gmail.com> >>>> wrote: >>>> >>>>> Hi Ajay, >>>>> >>>>> Here is a good explanation you might want to read. >>>>> >>>>> >>>>> http://www.datastax.com/dev/blog/whats-new-in-cassandra-2-1-a-better-implementation-of-counters >>>>> >>>>> Though we use counters for 3 years now, we used them from start C* 0.8 >>>>> and we are happy with them. Limits I can see in both ways are: >>>>> >>>>> Counters: >>>>> >>>>> - accuracy indeed (Tend to be small in our use case < 5% - when the >>>>> business allow 10%, so fair enough for us) + we recount them through a >>>>> batch processing tool (spark / hadoop - Kind of lambda architecture). So >>>>> our real-time stats are inaccurate and after a few minutes or hours we >>>>> have >>>>> the real value. >>>>> - Read-Before-Write model, which is an anti-pattern. Makes you use >>>>> more machine due to the pressure involved, affordable for us too. >>>>> >>>>> Raw data (counted) >>>>> >>>>> - Space used (can become quite impressive very fast, depending on your >>>>> business) ! >>>>> - Time to answer a request (we expose the data to customer, they don't >>>>> want to wait 10 sec for Cassandra to read 1 000 000 + columns) >>>>> - Performances in o(n) (linear) instead of o(1) (constant). Customer >>>>> won't always understand that for you it is harder to read 1 than 1 000 >>>>> 000, >>>>> since it should be reading 1 number in both case, and your interface will >>>>> have very unstable read time. >>>>> >>>>> Pick the best solution (or combination) for your use case. Those >>>>> disadvantages lists are not exhaustive, just things that came to my mind >>>>> right now. >>>>> >>>>> C*heers >>>>> >>>>> Alain >>>>> >>>>> 2014-12-29 13:33 GMT+01:00 Ajay <ajay.ga...@gmail.com>: >>>>> >>>>>> Hi, >>>>>> >>>>>> So you mean to say counters are not accurate? (It is highly likely >>>>>> that multiple parallel threads trying to increment the counter as users >>>>>> click the links). >>>>>> >>>>>> Thanks >>>>>> Ajay >>>>>> >>>>>> >>>>>> On Mon, Dec 29, 2014 at 4:49 PM, Janne Jalkanen < >>>>>> janne.jalka...@ecyrd.com> wrote: >>>>>> >>>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> It’s really a tradeoff between accurate and fast and your read >>>>>>> access patterns; if you need it to be fairly fast, use counters by all >>>>>>> means, but accept the fact that they will (especially in older versions >>>>>>> of >>>>>>> cassandra or adverse network conditions) drift off from the true click >>>>>>> count. If you need accurate, use a timeuuid and count the rows (this is >>>>>>> fairly safe for replays too). However, if using timeuuids your storage >>>>>>> will need lots of space; and your reads will be slow if the click counts >>>>>>> are huge (because Cassandra will need to read every item). Using >>>>>>> counters >>>>>>> makes it easy to just grab a slice of the time series data and shove it >>>>>>> to >>>>>>> a client for visualization. >>>>>>> >>>>>>> You could of course do a hybrid system; use timeuuids and then >>>>>>> periodically count and add the result to a regular column, and then >>>>>>> remove >>>>>>> the columns. Note that you might want to optimize this so that you >>>>>>> don’t >>>>>>> end up with a lot of tombstones, e.g. by bucketing the writes so that >>>>>>> you >>>>>>> can delete everything with just a single partition delete. >>>>>>> >>>>>>> At Thinglink some of the more important counters that we use are >>>>>>> backed up by the actual data. So for speed purposes we use always >>>>>>> counters >>>>>>> for reads, but there’s a repair process that fixes the counter value if >>>>>>> we >>>>>>> suspect it starts drifting off the real data too much. (You might be >>>>>>> able >>>>>>> to tell that we’ve been using counters for quite some time :-P) >>>>>>> >>>>>>> /Janne >>>>>>> >>>>>>> On 29 Dec 2014, at 13:00, Ajay <ajay.ga...@gmail.com> wrote: >>>>>>> >>>>>>> > Hi, >>>>>>> > >>>>>>> > Is it better to use Counter to User click count than maintaining >>>>>>> creating new row as user id : timestamp and count it. >>>>>>> > >>>>>>> > Basically we want to track the user clicks and use the same for >>>>>>> hourly/daily/monthly report. >>>>>>> > >>>>>>> > Thanks >>>>>>> > Ajay >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> >
