Re: time tracking for down node for nodetool repair
Valid suggestion. Stick to the plan, avoid downtime of a node more than hinted handoff window. OR increase window to a larger value, if you know it is going to take longer than current setting Regards, Nitan Cell: 510 449 9629 > On Apr 8, 2019, at 8:43 PM, Soumya Jena wrote: > > Cassandra tracks it and no new hints will be created once the default 3 hours > window is passed . However , cassandra will not automatically trigger a > repair if your node is down for more than 3 hours .Default settings of 3 > hours for hints is defined in cassandra.yaml file . Look for > "max_hint_window_in_ms" in the cassandra.yaml file. Its configurable . Apart > from the periodic repair you should start a repair when you bring up a node > which has missed some writes . > > One more thing is if node is down for long time and missed a lot of writes > sometimes it may be better to add that as a new fresh node rather than adding > it and then doing repair . > >> On Mon, Apr 8, 2019 at 4:49 PM Stefan Miklosovic >> wrote: >> Ah I see it is the default for hinted handoffs. I was somehow thinking >> its bigger figure I do not know why :) >> >> I would say you should run repairs continuously / periodically so you >> would not even have to do some thinking about that and it should run >> in the background in a scheduled manner if possible. >> >> Regards >> >> On Tue, 9 Apr 2019 at 04:19, Kunal wrote: >> > >> > Hello everyone.. >> > >> > >> > >> > I have a 6 node Cassandra datacenter, 3 nodes on each datacenter. If one >> > of the node goes down and remain down for more than 3 hr, I have to run >> > nodetool repair. Just wanted to ask if Cassandra automatically tracks the >> > time when one of the Cassandra node goes down or do I need to write code >> > to track the time and run repair when node comes back online after 3 hrs. >> > >> > >> > Thanks in anticipation. >> > >> > Regards, >> > Kunal Vaid >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >>
Re: time tracking for down node for nodetool repair
Cassandra tracks it and no new hints will be created once the default 3 hours window is passed . However , cassandra will not automatically trigger a repair if your node is down for more than 3 hours .Default settings of 3 hours for hints is defined in cassandra.yaml file . Look for "max_hint_window_in_ms" in the cassandra.yaml file. Its configurable . Apart from the periodic repair you should start a repair when you bring up a node which has missed some writes . One more thing is if node is down for long time and missed a lot of writes sometimes it may be better to add that as a new fresh node rather than adding it and then doing repair . On Mon, Apr 8, 2019 at 4:49 PM Stefan Miklosovic < stefan.mikloso...@instaclustr.com> wrote: > Ah I see it is the default for hinted handoffs. I was somehow thinking > its bigger figure I do not know why :) > > I would say you should run repairs continuously / periodically so you > would not even have to do some thinking about that and it should run > in the background in a scheduled manner if possible. > > Regards > > On Tue, 9 Apr 2019 at 04:19, Kunal wrote: > > > > Hello everyone.. > > > > > > > > I have a 6 node Cassandra datacenter, 3 nodes on each datacenter. If one > of the node goes down and remain down for more than 3 hr, I have to run > nodetool repair. Just wanted to ask if Cassandra automatically tracks the > time when one of the Cassandra node goes down or do I need to write code to > track the time and run repair when node comes back online after 3 hrs. > > > > > > Thanks in anticipation. > > > > Regards, > > Kunal Vaid > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Re: time tracking for down node for nodetool repair
Ah I see it is the default for hinted handoffs. I was somehow thinking its bigger figure I do not know why :) I would say you should run repairs continuously / periodically so you would not even have to do some thinking about that and it should run in the background in a scheduled manner if possible. Regards On Tue, 9 Apr 2019 at 04:19, Kunal wrote: > > Hello everyone.. > > > > I have a 6 node Cassandra datacenter, 3 nodes on each datacenter. If one of > the node goes down and remain down for more than 3 hr, I have to run nodetool > repair. Just wanted to ask if Cassandra automatically tracks the time when > one of the Cassandra node goes down or do I need to write code to track the > time and run repair when node comes back online after 3 hrs. > > > Thanks in anticipation. > > Regards, > Kunal Vaid - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: time tracking for down node for nodetool repair
Hi Kunal, where do you have that "more than 3 hours" from? Regards On Tue, 9 Apr 2019 at 04:19, Kunal wrote: > > Hello everyone.. > > > > I have a 6 node Cassandra datacenter, 3 nodes on each datacenter. If one of > the node goes down and remain down for more than 3 hr, I have to run nodetool > repair. Just wanted to ask if Cassandra automatically tracks the time when > one of the Cassandra node goes down or do I need to write code to track the > time and run repair when node comes back online after 3 hrs. > > > Thanks in anticipation. > > Regards, > Kunal Vaid - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: How to monitor datastax driver compression performance?
If it were me, I'd look at raw request rates (in terms of requests / second as well as request latency), network throughput and then some flame graphs of both the server and your application: https://github.com/jvm-profiling-tools/async-profiler. I've created an issue in tlp-stress to add compression options for the driver: https://github.com/thelastpickle/tlp-stress/issues/67. If you're interested in contributing the feature I think tlp-stress will more or less solve the remainder of the problem for you (the load part, not the os numbers). Jon On Mon, Apr 8, 2019 at 7:26 AM Gabriel Giussi wrote: > > Hi, I'm trying to test if adding driver compression will bring me any benefit. > I understand that the trade-off is less bandwidth but increased CPU usage in > both cassandra nodes (compression) and client nodes (decompression) but I > want to know what are the key metrics and how to monitor them to probe > compression is giving good results? > I guess I should look at latency percentiles reported by > com.datastax.driver.core.Metrics and CPU usage, but what about bandwith usage > and compression ratio? > Should I use tcpdump to capture packets length coming from cassandra nodes? > Something like tcpdump -n "src port 9042 and tcp[13] & 8 != 0" | sed -n > "s/^.*length \(.*\).*$/\1/p" would be enough? > > Thanks - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
time tracking for down node for nodetool repair
Hello everyone.. I have a 6 node Cassandra datacenter, 3 nodes on each datacenter. If one of the node goes down and remain down for more than 3 hr, I have to run nodetool repair. Just wanted to ask if Cassandra automatically tracks the time when one of the Cassandra node goes down or do I need to write code to track the time and run repair when node comes back online after 3 hrs. Thanks in anticipation. Regards, Kunal Vaid
How to monitor datastax driver compression performance?
Hi, I'm trying to test if adding driver compression will bring me any benefit. I understand that the trade-off is less bandwidth but increased CPU usage in both cassandra nodes (compression) and client nodes (decompression) but I want to know what are the key metrics and how to monitor them to probe compression is giving good results? I guess I should look at latency percentiles reported by com.datastax.driver.core.Metrics and CPU usage, but what about bandwith usage and compression ratio? Should I use tcpdump to capture packets length coming from cassandra nodes? Something like* tcpdump -n "src port 9042 and tcp[13] & 8 != 0" | sed -n "s/^.*length \(.*\).*$/\1/p"* would be enough? Thanks
