Re: Hinted handoff not working
Nope. I added millions of records and several GB to the cluster while one node was down, and then ran "nodetool flush system hints" on a couple of nodes that were up, and system/hints has less than 200K in it. Here’s the relevant part of "nodetool cfstats system.hints": Keyspace: system Read Count: 28572 Read Latency: 0.01806502869942601 ms. Write Count: 351 Write Latency: 0.04547008547008547 ms. Pending Tasks: 0 Table: hints SSTable count: 1 Space used (live), bytes: 7446 Space used (total), bytes: 80062 SSTable Compression Ratio: 0.2651441528992549 Number of keys (estimate): 128 Memtable cell count: 1 Memtable data size, bytes: 1740 The hints are definitely not being stored. Robert On Dec 14, 2014, at 11:44 PM, Jens Rantil mailto:jens.ran...@tink.se>> wrote: Hi Robert , Maybe you need to flush your memtables to actually see the disk usage increase? This applies to both hosts. Cheers, Jens On Sun, Dec 14, 2014 at 3:52 PM, Robert Wille mailto:rwi...@fold3.com>> wrote: I have a cluster with RF=3. If I shut down one node, add a bunch of data to the cluster, I don’t see a bunch of records added to system.hints. Also, du of /var/lib/cassandra/data/system/hints of the nodes that are up shows that hints aren’t being stored. When I start the down node, its data doesn’t grow until I run repair, which then takes a really long time because it is significantly out of date. Is there some magic setting I cannot find in the documentation to enable hinted handoff? I’m running 2.0.11. Any insights would be greatly appreciated. Thanks Robert
Re: Hinted handoff not working
Hi Robert , Maybe you need to flush your memtables to actually see the disk usage increase? This applies to both hosts. Cheers, Jens On Sun, Dec 14, 2014 at 3:52 PM, Robert Wille wrote: > I have a cluster with RF=3. If I shut down one node, add a bunch of data to > the cluster, I don’t see a bunch of records added to system.hints. Also, du > of /var/lib/cassandra/data/system/hints of the nodes that are up shows that > hints aren’t being stored. When I start the down node, its data doesn’t grow > until I run repair, which then takes a really long time because it is > significantly out of date. Is there some magic setting I cannot find in the > documentation to enable hinted handoff? I’m running 2.0.11. Any insights > would be greatly appreciated. > Thanks > Robert
Re: Hinted handoff not working
I’ve got "hinted_handoff_enabled: true" in cassandra.yaml. My settings are all default except for the DC, listen addresses and snitch. I should have mentioned this in my original post. On Dec 14, 2014, at 8:02 AM, Rahul Neelakantan wrote: > http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html?scroll=reference_ds_qfg_n1r_1k__hinted_handoff_enabled > > Rahul > >> On Dec 14, 2014, at 9:46 AM, Robert Wille wrote: >> >> I have a cluster with RF=3. If I shut down one node, add a bunch of data to >> the cluster, I don’t see a bunch of records added to system.hints. Also, du >> of /var/lib/cassandra/data/system/hints of the nodes that are up shows that >> hints aren’t being stored. When I start the down node, its data doesn’t grow >> until I run repair, which then takes a really long time because it is >> significantly out of date. Is there some magic setting I cannot find in the >> documentation to enable hinted handoff? I’m running 2.0.11. Any insights >> would be greatly appreciated. >> >> Thanks >> >> Robert >>
Re: Hinted handoff not working
http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html?scroll=reference_ds_qfg_n1r_1k__hinted_handoff_enabled Rahul > On Dec 14, 2014, at 9:46 AM, Robert Wille wrote: > > I have a cluster with RF=3. If I shut down one node, add a bunch of data to > the cluster, I don’t see a bunch of records added to system.hints. Also, du > of /var/lib/cassandra/data/system/hints of the nodes that are up shows that > hints aren’t being stored. When I start the down node, its data doesn’t grow > until I run repair, which then takes a really long time because it is > significantly out of date. Is there some magic setting I cannot find in the > documentation to enable hinted handoff? I’m running 2.0.11. Any insights > would be greatly appreciated. > > Thanks > > Robert >
Hinted handoff not working
I have a cluster with RF=3. If I shut down one node, add a bunch of data to the cluster, I don’t see a bunch of records added to system.hints. Also, du of /var/lib/cassandra/data/system/hints of the nodes that are up shows that hints aren’t being stored. When I start the down node, its data doesn’t grow until I run repair, which then takes a really long time because it is significantly out of date. Is there some magic setting I cannot find in the documentation to enable hinted handoff? I’m running 2.0.11. Any insights would be greatly appreciated. Thanks Robert
