Re: hinted handoff disabling trade-offs
Thanks Aaron, appreciate the advice. On Tue, Mar 19, 2013 at 3:14 AM, aaron morton aa...@thelastpickle.com wrote: I think I understand what it means for application-level data, but the part I'm not entirely sure about is what it could mean for Cassandra internals. Internally it means the write will not be retries to nodes that were either down or did not ack before rpc_timeout. That's all. If you are doing thing with read_repair_chance == 0 and CL ONE you are in a very eventually consistent world. The only thing that will guarantee consistency for you now is running nodetool repair. My cluster is under heavy write load. I'm considering disabling Hinted Handoffs so the nodes recover quicker in case compactions begin to back up. If the node cluster is approaching capacity, then ultimately the thing to do is add more nodes. The only things to do are disable the commit log and use a lower CL. If it's approaching capacity you will start to see pending mutations back up, maybe some dropped mutations and the maybe an increase in the difference between the latency reported in the proxyhistograms and the cfhistograms or cfstats. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 16/03/2013, at 4:50 PM, Matt Kap matvey1...@gmail.com wrote: Thanks Aaron. I am using CL=ONE. read_repair_chance=0. The part which I'm wondering about is what happens to the internal Cassandra writes if Hinted Handoffs are disabled. I think I understand what it means for application-level data, but the part I'm not entirely sure about is what it could mean for Cassandra internals. My cluster is under heavy write load. I'm considering disabling Hinted Handoffs so the nodes recover quicker in case compactions begin to back up. On Wed, Mar 6, 2013 at 2:06 AM, aaron morton aa...@thelastpickle.com wrote: The advantage of HH is that it reduces the probability of a DigestMismatch when using a CL ONE. A DigestMismatch means the read has to run a second time before returning to the client. - No risk of hinted-handoffs building up - No risk of hinted-handoffs flooding a node that just came up See the yaml config settings for the max hint window and the throttling. Can anyone suggest any other factors that I'm missing here. Specifically reasons not to do this. If you are doing this for performance first make sure your data model is efficient, that you are doing the most efficient reads (see my presentation here http://www.datastax.com/events/cassandrasummit2012/presentations), and your caching is bang on. Then consider if you can tune the CL, and if your client is token aware so it directs traffic to a node that has it. Cheers - Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 4/03/2013, at 9:19 PM, Michael Kjellman mkjell...@barracuda.com wrote: Also, if you have enough hints being created that its significantly impacting your heap I have a feeling things are going to get out of sync very quickly. On Mar 4, 2013, at 9:17 PM, Wz1975 wz1...@yahoo.com wrote: Why do you think disabling hinted handoff will improve memory usage? Thanks. -Wei Sent from my Samsung smartphone on ATT Original message Subject: Re: hinted handoff disabling trade-offs From: Michael Kjellman mkjell...@barracuda.com To: user@cassandra.apache.org user@cassandra.apache.org CC: Repair is slow. On Mar 4, 2013, at 8:07 PM, Matt Kap matvey1...@gmail.com wrote: I am looking to get a second opinion about disabling hinted-handoffs. I have an application that can tolerate a fair amount of inconsistency (advertising domain), and so I'm weighting the pros and cons of hinted handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon. Pros of disabling hinted handoffs: - Reduces heap - Improves GC performance - No risk of hinted-handoffs building up - No risk of hinted-handoffs flooding a node that just came up Cons - Some writes can be lost, at least until repair runs Can anyone suggest any other factors that I'm missing here. Specifically reasons not to do this. Cheers! -Matt Copy, by Barracuda, helps you store, protect, and share all your amazing things. Start today: www.copy.com. -- Copy, by Barracuda, helps you store, protect, and share all your amazing things. Start today: www.copy.com. -- www.calcmachine.com - easy online calculator. -- www.calcmachine.com - easy online calculator.
Re: hinted handoff disabling trade-offs
Thanks Aaron. I am using CL=ONE. read_repair_chance=0. The part which I'm wondering about is what happens to the internal Cassandra writes if Hinted Handoffs are disabled. I think I understand what it means for application-level data, but the part I'm not entirely sure about is what it could mean for Cassandra internals. My cluster is under heavy write load. I'm considering disabling Hinted Handoffs so the nodes recover quicker in case compactions begin to back up. On Wed, Mar 6, 2013 at 2:06 AM, aaron morton aa...@thelastpickle.com wrote: The advantage of HH is that it reduces the probability of a DigestMismatch when using a CL ONE. A DigestMismatch means the read has to run a second time before returning to the client. - No risk of hinted-handoffs building up - No risk of hinted-handoffs flooding a node that just came up See the yaml config settings for the max hint window and the throttling. Can anyone suggest any other factors that I'm missing here. Specifically reasons not to do this. If you are doing this for performance first make sure your data model is efficient, that you are doing the most efficient reads (see my presentation here http://www.datastax.com/events/cassandrasummit2012/presentations), and your caching is bang on. Then consider if you can tune the CL, and if your client is token aware so it directs traffic to a node that has it. Cheers - Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 4/03/2013, at 9:19 PM, Michael Kjellman mkjell...@barracuda.com wrote: Also, if you have enough hints being created that its significantly impacting your heap I have a feeling things are going to get out of sync very quickly. On Mar 4, 2013, at 9:17 PM, Wz1975 wz1...@yahoo.com wrote: Why do you think disabling hinted handoff will improve memory usage? Thanks. -Wei Sent from my Samsung smartphone on ATT Original message Subject: Re: hinted handoff disabling trade-offs From: Michael Kjellman mkjell...@barracuda.com To: user@cassandra.apache.org user@cassandra.apache.org CC: Repair is slow. On Mar 4, 2013, at 8:07 PM, Matt Kap matvey1...@gmail.com wrote: I am looking to get a second opinion about disabling hinted-handoffs. I have an application that can tolerate a fair amount of inconsistency (advertising domain), and so I'm weighting the pros and cons of hinted handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon. Pros of disabling hinted handoffs: - Reduces heap - Improves GC performance - No risk of hinted-handoffs building up - No risk of hinted-handoffs flooding a node that just came up Cons - Some writes can be lost, at least until repair runs Can anyone suggest any other factors that I'm missing here. Specifically reasons not to do this. Cheers! -Matt Copy, by Barracuda, helps you store, protect, and share all your amazing things. Start today: www.copy.com. -- Copy, by Barracuda, helps you store, protect, and share all your amazing things. Start today: www.copy.com. -- www.calcmachine.com - easy online calculator.
Re: hinted handoff disabling trade-offs
The advantage of HH is that it reduces the probability of a DigestMismatch when using a CL ONE. A DigestMismatch means the read has to run a second time before returning to the client. - No risk of hinted-handoffs building up - No risk of hinted-handoffs flooding a node that just came up See the yaml config settings for the max hint window and the throttling. Can anyone suggest any other factors that I'm missing here. Specifically reasons not to do this. If you are doing this for performance first make sure your data model is efficient, that you are doing the most efficient reads (see my presentation here http://www.datastax.com/events/cassandrasummit2012/presentations), and your caching is bang on. Then consider if you can tune the CL, and if your client is token aware so it directs traffic to a node that has it. Cheers - Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 4/03/2013, at 9:19 PM, Michael Kjellman mkjell...@barracuda.com wrote: Also, if you have enough hints being created that its significantly impacting your heap I have a feeling things are going to get out of sync very quickly. On Mar 4, 2013, at 9:17 PM, Wz1975 wz1...@yahoo.com wrote: Why do you think disabling hinted handoff will improve memory usage? Thanks. -Wei Sent from my Samsung smartphone on ATT Original message Subject: Re: hinted handoff disabling trade-offs From: Michael Kjellman mkjell...@barracuda.com To: user@cassandra.apache.org user@cassandra.apache.org CC: Repair is slow. On Mar 4, 2013, at 8:07 PM, Matt Kap matvey1...@gmail.com wrote: I am looking to get a second opinion about disabling hinted-handoffs. I have an application that can tolerate a fair amount of inconsistency (advertising domain), and so I'm weighting the pros and cons of hinted handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon. Pros of disabling hinted handoffs: - Reduces heap - Improves GC performance - No risk of hinted-handoffs building up - No risk of hinted-handoffs flooding a node that just came up Cons - Some writes can be lost, at least until repair runs Can anyone suggest any other factors that I'm missing here. Specifically reasons not to do this. Cheers! -Matt Copy, by Barracuda, helps you store, protect, and share all your amazing things. Start today: www.copy.com. -- Copy, by Barracuda, helps you store, protect, and share all your amazing things. Start today: www.copy.com.
hinted handoff disabling trade-offs
I am looking to get a second opinion about disabling hinted-handoffs. I have an application that can tolerate a fair amount of inconsistency (advertising domain), and so I'm weighting the pros and cons of hinted handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon. Pros of disabling hinted handoffs: - Reduces heap - Improves GC performance - No risk of hinted-handoffs building up - No risk of hinted-handoffs flooding a node that just came up Cons - Some writes can be lost, at least until repair runs Can anyone suggest any other factors that I'm missing here. Specifically reasons not to do this. Cheers! -Matt
Re: hinted handoff disabling trade-offs
Repair is slow. On Mar 4, 2013, at 8:07 PM, Matt Kap matvey1...@gmail.com wrote: I am looking to get a second opinion about disabling hinted-handoffs. I have an application that can tolerate a fair amount of inconsistency (advertising domain), and so I'm weighting the pros and cons of hinted handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon. Pros of disabling hinted handoffs: - Reduces heap - Improves GC performance - No risk of hinted-handoffs building up - No risk of hinted-handoffs flooding a node that just came up Cons - Some writes can be lost, at least until repair runs Can anyone suggest any other factors that I'm missing here. Specifically reasons not to do this. Cheers! -Matt Copy, by Barracuda, helps you store, protect, and share all your amazing things. Start today: www.copy.com.
Re: hinted handoff disabling trade-offs
Why do you think disabling hinted handoff will improve memory usage? Thanks. -Wei Sent from my Samsung smartphone on ATT Original message Subject: Re: hinted handoff disabling trade-offs From: Michael Kjellman mkjell...@barracuda.com To: user@cassandra.apache.org user@cassandra.apache.org CC: Repair is slow. On Mar 4, 2013, at 8:07 PM, Matt Kap matvey1...@gmail.com wrote: I am looking to get a second opinion about disabling hinted-handoffs. I have an application that can tolerate a fair amount of inconsistency (advertising domain), and so I'm weighting the pros and cons of hinted handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon. Pros of disabling hinted handoffs: - Reduces heap - Improves GC performance - No risk of hinted-handoffs building up - No risk of hinted-handoffs flooding a node that just came up Cons - Some writes can be lost, at least until repair runs Can anyone suggest any other factors that I'm missing here. Specifically reasons not to do this. Cheers! -Matt Copy, by Barracuda, helps you store, protect, and share all your amazing things. Start today: www.copy.com.
Re: hinted handoff disabling trade-offs
Also, if you have enough hints being created that its significantly impacting your heap I have a feeling things are going to get out of sync very quickly. On Mar 4, 2013, at 9:17 PM, Wz1975 wz1...@yahoo.commailto:wz1...@yahoo.com wrote: Why do you think disabling hinted handoff will improve memory usage? Thanks. -Wei Sent from my Samsung smartphone on ATT Original message Subject: Re: hinted handoff disabling trade-offs From: Michael Kjellman mkjell...@barracuda.commailto:mkjell...@barracuda.com To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org CC: Repair is slow. On Mar 4, 2013, at 8:07 PM, Matt Kap matvey1...@gmail.commailto:matvey1...@gmail.com wrote: I am looking to get a second opinion about disabling hinted-handoffs. I have an application that can tolerate a fair amount of inconsistency (advertising domain), and so I'm weighting the pros and cons of hinted handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon. Pros of disabling hinted handoffs: - Reduces heap - Improves GC performance - No risk of hinted-handoffs building up - No risk of hinted-handoffs flooding a node that just came up Cons - Some writes can be lost, at least until repair runs Can anyone suggest any other factors that I'm missing here. Specifically reasons not to do this. Cheers! -Matt Copy, by Barracuda, helps you store, protect, and share all your amazing things. Start today: www.copy.comhttp://www.copy.com. Copy, by Barracuda, helps you store, protect, and share all your amazing things. Start today: www.copy.com.
