Re: Hinted handoff not working
Nope. I added millions of records and several GB to the cluster while one node was down, and then ran nodetool flush system hints on a couple of nodes that were up, and system/hints has less than 200K in it. Here’s the relevant part of nodetool cfstats system.hints: Keyspace: system Read Count: 28572 Read Latency: 0.01806502869942601 ms. Write Count: 351 Write Latency: 0.04547008547008547 ms. Pending Tasks: 0 Table: hints SSTable count: 1 Space used (live), bytes: 7446 Space used (total), bytes: 80062 SSTable Compression Ratio: 0.2651441528992549 Number of keys (estimate): 128 Memtable cell count: 1 Memtable data size, bytes: 1740 The hints are definitely not being stored. Robert On Dec 14, 2014, at 11:44 PM, Jens Rantil jens.ran...@tink.semailto:jens.ran...@tink.se wrote: Hi Robert , Maybe you need to flush your memtables to actually see the disk usage increase? This applies to both hosts. Cheers, Jens On Sun, Dec 14, 2014 at 3:52 PM, Robert Wille rwi...@fold3.commailto:rwi...@fold3.com wrote: I have a cluster with RF=3. If I shut down one node, add a bunch of data to the cluster, I don’t see a bunch of records added to system.hints. Also, du of /var/lib/cassandra/data/system/hints of the nodes that are up shows that hints aren’t being stored. When I start the down node, its data doesn’t grow until I run repair, which then takes a really long time because it is significantly out of date. Is there some magic setting I cannot find in the documentation to enable hinted handoff? I’m running 2.0.11. Any insights would be greatly appreciated. Thanks Robert
Hinted handoff not working
I have a cluster with RF=3. If I shut down one node, add a bunch of data to the cluster, I don’t see a bunch of records added to system.hints. Also, du of /var/lib/cassandra/data/system/hints of the nodes that are up shows that hints aren’t being stored. When I start the down node, its data doesn’t grow until I run repair, which then takes a really long time because it is significantly out of date. Is there some magic setting I cannot find in the documentation to enable hinted handoff? I’m running 2.0.11. Any insights would be greatly appreciated. Thanks Robert
Re: Hinted handoff not working
http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html?scroll=reference_ds_qfg_n1r_1k__hinted_handoff_enabled Rahul On Dec 14, 2014, at 9:46 AM, Robert Wille rwi...@fold3.com wrote: I have a cluster with RF=3. If I shut down one node, add a bunch of data to the cluster, I don’t see a bunch of records added to system.hints. Also, du of /var/lib/cassandra/data/system/hints of the nodes that are up shows that hints aren’t being stored. When I start the down node, its data doesn’t grow until I run repair, which then takes a really long time because it is significantly out of date. Is there some magic setting I cannot find in the documentation to enable hinted handoff? I’m running 2.0.11. Any insights would be greatly appreciated. Thanks Robert
Re: Hinted handoff not working
I’ve got hinted_handoff_enabled: true in cassandra.yaml. My settings are all default except for the DC, listen addresses and snitch. I should have mentioned this in my original post. On Dec 14, 2014, at 8:02 AM, Rahul Neelakantan ra...@rahul.be wrote: http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html?scroll=reference_ds_qfg_n1r_1k__hinted_handoff_enabled Rahul On Dec 14, 2014, at 9:46 AM, Robert Wille rwi...@fold3.com wrote: I have a cluster with RF=3. If I shut down one node, add a bunch of data to the cluster, I don’t see a bunch of records added to system.hints. Also, du of /var/lib/cassandra/data/system/hints of the nodes that are up shows that hints aren’t being stored. When I start the down node, its data doesn’t grow until I run repair, which then takes a really long time because it is significantly out of date. Is there some magic setting I cannot find in the documentation to enable hinted handoff? I’m running 2.0.11. Any insights would be greatly appreciated. Thanks Robert
Re: Hinted handoff not working
Hi Robert , Maybe you need to flush your memtables to actually see the disk usage increase? This applies to both hosts. Cheers, Jens On Sun, Dec 14, 2014 at 3:52 PM, Robert Wille rwi...@fold3.com wrote: I have a cluster with RF=3. If I shut down one node, add a bunch of data to the cluster, I don’t see a bunch of records added to system.hints. Also, du of /var/lib/cassandra/data/system/hints of the nodes that are up shows that hints aren’t being stored. When I start the down node, its data doesn’t grow until I run repair, which then takes a really long time because it is significantly out of date. Is there some magic setting I cannot find in the documentation to enable hinted handoff? I’m running 2.0.11. Any insights would be greatly appreciated. Thanks Robert
