Oleg, The system keyspace is not replicated it is local to the node. You should check your logs to see if there are Timeouts from streaming hints, i believe the default value to stream hints it 10 seconds. When i ran into this problem i truncated hints to clear out the space and then ran a repair so ensure that all the data was consistant across all nodes, even if there was a failure.
-rm On Tue, Nov 5, 2013 at 6:29 PM, Oleg Dulin <oleg.du...@gmail.com> wrote: > What happens if they are not being successfully delivered ? Will they > eventually TTL-out ? > > > Also, do I need to truncate hints on every node or is it replicated ? > > > Oleg > > > On 2013-11-04 21:34:55 +0000, Robert Coli said: > > > On Mon, Nov 4, 2013 at 11:34 AM, Oleg Dulin <oleg.du...@gmail.com> wrote: > > I have a dual DC setup, 4 nodes, RF=4 in each. > > > The one that is used as primary has its system keyspace fill up with 200 > gigs of data, majority of which is hints. > > > Why does this happen ? > > > How can I clean it up ? > > > If you have this many hints, you probably have flapping / frequent network > partition, or very overloaded nodes. If you compare the number of hints to > the number of dropped messages, that would be informative. If you're > hinting because you're dropping, increase capacity. If you're hinting > because of partition, figure out why there's so much partition. > > > WRT cleaning up hints, they will automatically be cleaned up eventually, > as long as they are successfully being delivered. If you need to manually > clean them up you can truncate system.hints keyspace. > > > =Rob > > > > > > > -- > > Regards, > > Oleg Dulin > > http://www.olegdulin.com >
