RE: [EXTERNAL] Re: A Question About Hints
Versions 2.0 and 2.1 were generally very stable, so I can understand a reticence to move when there are so many other things competing for time and attention. Sean Durity From: shalom sagges Sent: Monday, March 04, 2019 4:21 PM To: user@cassandra.apache.org Subject: [EXTERNAL] Re: A Question About Hints Everyone really should move off of the 2.x versions just like you are doing. Tell me about it... But since there are a lot of groups involved, these things take time unfortunately. Thanks for your assistance Kenneth On Mon, Mar 4, 2019 at 11:04 PM Kenneth Brotman mailto:kenbrot...@yahoo.com.invalid>> wrote: Since you are in the process of upgrading, I’d do nothing on the settings right now. But if you wanted to do something on the settings in the meantime, based on my read of the information available, I’d maybe double the default settings. The upgrade will help a lot of things as you know. Everyone really should move off of the 2.x versions just like you are doing. From: shalom sagges [mailto:shalomsag...@gmail.com<mailto:shalomsag...@gmail.com>] Sent: Monday, March 04, 2019 12:34 PM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: A Question About Hints See my comments inline. Do the 8 nodes clusters have the problem too? Yes To the same extent? It depends on the throughput, but basically the smaller clusters get low throughput, so the problem is naturally smaller. Is it any cluster across multi-DC’s? Yes Do all the clusters use nodes with similar specs? All nodes have similar specs within a cluster but different specs on different clusters. The version of Cassandra you are on can make a difference. What version are you on? Currently I'm on various versions, 2.0.14, 2.1.15 and 3.0.12. In the process of upgrading to 3.11.4 Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video at: https://www.youtube.com/watch?v=uN4FtAjYmLU<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DuN4FtAjYmLU=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=rADb6b8CTim6Qjxi48jQvnRdL_GqKB5hM33yBEKS0Y4=oeJjPZZkhxRhyg5r3eApRMk-_tk6qTUrmtF_9eDY5fU=> where he briefly mentions you can get into trouble if you go to fast or two slow? I guess you can say it about almost any parameter you change :) BTW, I thought the comments at the end of the article you mentioned were really good. The entire article is very good, but I wonder if it's still valid since it was created around 4 years ago. Thanks! On Mon, Mar 4, 2019 at 9:37 PM Kenneth Brotman mailto:kenbrotman@yahoocom.invalid>> wrote: Makes sense If you have time and don’t mind, could you answer the following: Do the 8 nodes clusters have the problem too? To the same extent? Is it just the clusters with the large node count? Is it any cluster across multi-DC’s? Do all the clusters use nodes with similar specs? The version of Cassandra you are on can make a difference. What version are you on? Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video at: https://www.youtube.com/watch?v=uN4FtAjYmLU<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DuN4FtAjYmLU=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=rADb6b8CTim6Qjxi48jQvnRdL_GqKB5hM33yBEKS0Y4=oeJjPZZkhxRhyg5r3eApRMk-_tk6qTUrmtF_9eDY5fU=> where he briefly mentions you can get into trouble if you go to fast or two slow? BTW, I thought the comments at the end of the article you mentioned were really good. From: shalom sagges [mailto:shalomsag...@gmail.com<mailto:shalomsag...@gmail.com>] Sent: Monday, March 04, 2019 11:04 AM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: A Question About Hints It varies... Some clusters have 48 nodes, others 24 nodes and some 8 nodes. Both settings are on default. I’d try making a single conservative change to one or the other, measure and reassess. Then do same to other setting. That's the plan, but I thought I might first get some valuable information from someone in the community that has already experienced in this type of change. Thanks! On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman mailto:kenbrot...@yahoo.com.invalid>> wrote: It sounds like your use case might be appropriate for tuning those two settings some. How many nodes are in the cluster? Are both settings definitely on the default values currently? I’d try making a single conservative change to one or the other, measure and reassess. Then do same to other setting. Then of course share your results with us. From: shalom sagges [mailto:shalomsag...@gmail.com<mailto:shalomsag...@gmail.com>] Sent: Monday, March 04, 2019 9:54 AM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: A Question About Hints Hi Kenneth, The concern is that in some cases, hints accumulate on nodes,
Re: A Question About Hints
Everyone really should move off of the 2.x versions just like you are doing. Tell me about it... But since there are a lot of groups involved, these things take time unfortunately. Thanks for your assistance Kenneth On Mon, Mar 4, 2019 at 11:04 PM Kenneth Brotman wrote: > Since you are in the process of upgrading, I’d do nothing on the settings > right now. But if you wanted to do something on the settings in the > meantime, based on my read of the information available, I’d maybe double > the default settings. The upgrade will help a lot of things as you know. > > > > Everyone really should move off of the 2.x versions just like you are > doing. > > > > *From:* shalom sagges [mailto:shalomsag...@gmail.com] > *Sent:* Monday, March 04, 2019 12:34 PM > *To:* user@cassandra.apache.org > *Subject:* Re: A Question About Hints > > > > See my comments inline. > > > > Do the 8 nodes clusters have the problem too? > > Yes > > > > To the same extent? > > It depends on the throughput, but basically the smaller clusters get low > throughput, so the problem is naturally smaller. > > > > Is it any cluster across multi-DC’s? > > Yes > > > > Do all the clusters use nodes with similar specs? > > All nodes have similar specs within a cluster but different specs on > different clusters. > > > > The version of Cassandra you are on can make a difference. What version > are you on? > > Currently I'm on various versions, 2.0.14, 2.1.15 and 3.0.12. In the > process of upgrading to 3.11.4 > > > > Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video > at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions > you can get into trouble if you go to fast or two slow? > > I guess you can say it about almost any parameter you change :) > > > > BTW, I thought the comments at the end of the article you mentioned were > really good. > > The entire article is very good, but I wonder if it's still valid since it > was created around 4 years ago. > > > > Thanks! > > > > > > > > > > On Mon, Mar 4, 2019 at 9:37 PM Kenneth Brotman > wrote: > > Makes sense If you have time and don’t mind, could you answer the > following: > > Do the 8 nodes clusters have the problem too? > > To the same extent? > > Is it just the clusters with the large node count? > > Is it any cluster across multi-DC’s? > > Do all the clusters use nodes with similar specs? > > > > The version of Cassandra you are on can make a difference. What version > are you on? > > > > Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video > at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions > you can get into trouble if you go to fast or two slow? > > BTW, I thought the comments at the end of the article you mentioned were > really good. > > > > > > > > *From:* shalom sagges [mailto:shalomsag...@gmail.com] > *Sent:* Monday, March 04, 2019 11:04 AM > *To:* user@cassandra.apache.org > *Subject:* Re: A Question About Hints > > > > It varies... > > Some clusters have 48 nodes, others 24 nodes and some 8 nodes. > > Both settings are on default. > > > > I’d try making a single conservative change to one or the other, measure > and reassess. Then do same to other setting. > > That's the plan, but I thought I might first get some valuable information > from someone in the community that has already experienced in this type of > change. > > > > Thanks! > > > > On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman > wrote: > > It sounds like your use case might be appropriate for tuning those two > settings some. > > > > How many nodes are in the cluster? > > Are both settings definitely on the default values currently? > > > > I’d try making a single conservative change to one or the other, measure > and reassess. Then do same to other setting. > > > > Then of course share your results with us. > > > > *From:* shalom sagges [mailto:shalomsag...@gmail.com] > *Sent:* Monday, March 04, 2019 9:54 AM > *To:* user@cassandra.apache.org > *Subject:* Re: A Question About Hints > > > > Hi Kenneth, > > > > The concern is that in some cases, hints accumulate on nodes, and it takes > a while until they are delivered (multi DCs). > > I see that whenever there are a lot of hints in play,like after a rolling > restart, the cluster works harder. That's why I want to decrease the hints > delivery time. > > I didn't want to change the configuration blindly and thought th
RE: A Question About Hints
Since you are in the process of upgrading, I’d do nothing on the settings right now. But if you wanted to do something on the settings in the meantime, based on my read of the information available, I’d maybe double the default settings. The upgrade will help a lot of things as you know. Everyone really should move off of the 2.x versions just like you are doing. From: shalom sagges [mailto:shalomsag...@gmail.com] Sent: Monday, March 04, 2019 12:34 PM To: user@cassandra.apache.org Subject: Re: A Question About Hints See my comments inline. Do the 8 nodes clusters have the problem too? Yes To the same extent? It depends on the throughput, but basically the smaller clusters get low throughput, so the problem is naturally smaller. Is it any cluster across multi-DC’s? Yes Do all the clusters use nodes with similar specs? All nodes have similar specs within a cluster but different specs on different clusters. The version of Cassandra you are on can make a difference. What version are you on? Currently I'm on various versions, 2.0.14, 2.1.15 and 3.0.12. In the process of upgrading to 3.11.4 Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions you can get into trouble if you go to fast or two slow? I guess you can say it about almost any parameter you change :) BTW, I thought the comments at the end of the article you mentioned were really good. The entire article is very good, but I wonder if it's still valid since it was created around 4 years ago. Thanks! On Mon, Mar 4, 2019 at 9:37 PM Kenneth Brotman wrote: Makes sense If you have time and don’t mind, could you answer the following: Do the 8 nodes clusters have the problem too? To the same extent? Is it just the clusters with the large node count? Is it any cluster across multi-DC’s? Do all the clusters use nodes with similar specs? The version of Cassandra you are on can make a difference. What version are you on? Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions you can get into trouble if you go to fast or two slow? BTW, I thought the comments at the end of the article you mentioned were really good. From: shalom sagges [mailto:shalomsag...@gmail.com] Sent: Monday, March 04, 2019 11:04 AM To: user@cassandra.apache.org Subject: Re: A Question About Hints It varies... Some clusters have 48 nodes, others 24 nodes and some 8 nodes. Both settings are on default. I’d try making a single conservative change to one or the other, measure and reassess. Then do same to other setting. That's the plan, but I thought I might first get some valuable information from someone in the community that has already experienced in this type of change. Thanks! On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman wrote: It sounds like your use case might be appropriate for tuning those two settings some. How many nodes are in the cluster? Are both settings definitely on the default values currently? I’d try making a single conservative change to one or the other, measure and reassess. Then do same to other setting. Then of course share your results with us. From: shalom sagges [mailto:shalomsag...@gmail.com] Sent: Monday, March 04, 2019 9:54 AM To: user@cassandra.apache.org Subject: Re: A Question About Hints Hi Kenneth, The concern is that in some cases, hints accumulate on nodes, and it takes a while until they are delivered (multi DCs). I see that whenever there are a lot of hints in play,like after a rolling restart, the cluster works harder. That's why I want to decrease the hints delivery time. I didn't want to change the configuration blindly and thought the community might have some experience on this subject. I went over the cassandra.yaml file but didn't find any information on optimizing these attributes, just that the max_throttle is divided between nodes in the cluster and that I should increase the max_hints_delivery_threads because I have multi-dc deployments. # Maximum throttle in KBs per second, per delivery thread. This will be # reduced proportionally to the number of nodes in the cluster. (If there # are two nodes in the cluster, each delivery thread will use the maximum # rate; if there are three, each will throttle to half of the maximum, # since we expect two nodes to be delivering hints simultaneously) hinted_handoff_throttle_in_kb: 1024 # Number of threads with which to deliver hints; # Consider increasing this number when you have multi-dc deployments, since # cross-dc handoff tends to be slower max_hints_delivery_threads: 2 Thanks for your help! On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman wrote: What is the concern? Why are you looking
Re: A Question About Hints
See my comments inline. Do the 8 nodes clusters have the problem too? Yes To the same extent? It depends on the throughput, but basically the smaller clusters get low throughput, so the problem is naturally smaller. Is it any cluster across multi-DC’s? Yes Do all the clusters use nodes with similar specs? All nodes have similar specs within a cluster but different specs on different clusters. The version of Cassandra you are on can make a difference. What version are you on? Currently I'm on various versions, 2.0.14, 2.1.15 and 3.0.12. In the process of upgrading to 3.11.4 Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions you can get into trouble if you go to fast or two slow? I guess you can say it about almost any parameter you change :) BTW, I thought the comments at the end of the article you mentioned were really good. The entire article is very good, but I wonder if it's still valid since it was created around 4 years ago. Thanks! On Mon, Mar 4, 2019 at 9:37 PM Kenneth Brotman wrote: > Makes sense. If you have time and don’t mind, could you answer the > following: > > Do the 8 nodes clusters have the problem too? > > To the same extent? > > Is it just the clusters with the large node count? > > Is it any cluster across multi-DC’s? > > Do all the clusters use nodes with similar specs? > > > > The version of Cassandra you are on can make a difference. What version > are you on? > > > > Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video > at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions > you can get into trouble if you go to fast or two slow? > > BTW, I thought the comments at the end of the article you mentioned were > really good. > > > > > > > > *From:* shalom sagges [mailto:shalomsag...@gmail.com] > *Sent:* Monday, March 04, 2019 11:04 AM > *To:* user@cassandra.apache.org > *Subject:* Re: A Question About Hints > > > > It varies... > > Some clusters have 48 nodes, others 24 nodes and some 8 nodes. > > Both settings are on default. > > > > I’d try making a single conservative change to one or the other, measure > and reassess. Then do same to other setting. > > That's the plan, but I thought I might first get some valuable information > from someone in the community that has already experienced in this type of > change. > > > > Thanks! > > > > On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman > wrote: > > It sounds like your use case might be appropriate for tuning those two > settings some. > > > > How many nodes are in the cluster? > > Are both settings definitely on the default values currently? > > > > I’d try making a single conservative change to one or the other, measure > and reassess. Then do same to other setting. > > > > Then of course share your results with us. > > > > *From:* shalom sagges [mailto:shalomsag...@gmail.com] > *Sent:* Monday, March 04, 2019 9:54 AM > *To:* user@cassandra.apache.org > *Subject:* Re: A Question About Hints > > > > Hi Kenneth, > > > > The concern is that in some cases, hints accumulate on nodes, and it takes > a while until they are delivered (multi DCs). > > I see that whenever there are a lot of hints in play,like after a rolling > restart, the cluster works harder. That's why I want to decrease the hints > delivery time. > > I didn't want to change the configuration blindly and thought the > community might have some experience on this subject. > > > > I went over the cassandra.yaml file but didn't find any information on > optimizing these attributes, just that the max_throttle is divided between > nodes in the cluster and that I should increase the > max_hints_delivery_threads because I have multi-dc deployments. > > > > # Maximum throttle in KBs per second, per delivery thread. This will be > # reduced proportionally to the number of nodes in the cluster. (If there > # are two nodes in the cluster, each delivery thread will use the maximum > # rate; if there are three, each will throttle to half of the maximum, > # since we expect two nodes to be delivering hints simultaneously) > hinted_handoff_throttle_in_kb: 1024 > > # Number of threads with which to deliver hints; > # Consider increasing this number when you have multi-dc deployments, since > # cross-dc handoff tends to be slower > max_hints_delivery_threads: 2 > > > > > > Thanks for your help! > > > > > > On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman > wrote: > > What is the concern
RE: A Question About Hints
Makes sense. If you have time and don’t mind, could you answer the following: Do the 8 nodes clusters have the problem too? To the same extent? Is it just the clusters with the large node count? Is it any cluster across multi-DC’s? Do all the clusters use nodes with similar specs? The version of Cassandra you are on can make a difference. What version are you on? Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions you can get into trouble if you go to fast or two slow? BTW, I thought the comments at the end of the article you mentioned were really good. From: shalom sagges [mailto:shalomsag...@gmail.com] Sent: Monday, March 04, 2019 11:04 AM To: user@cassandra.apache.org Subject: Re: A Question About Hints It varies... Some clusters have 48 nodes, others 24 nodes and some 8 nodes. Both settings are on default. I’d try making a single conservative change to one or the other, measure and reassess. Then do same to other setting. That's the plan, but I thought I might first get some valuable information from someone in the community that has already experienced in this type of change. Thanks! On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman wrote: It sounds like your use case might be appropriate for tuning those two settings some. How many nodes are in the cluster? Are both settings definitely on the default values currently? I’d try making a single conservative change to one or the other, measure and reassess. Then do same to other setting. Then of course share your results with us. From: shalom sagges [mailto:shalomsag...@gmail.com] Sent: Monday, March 04, 2019 9:54 AM To: user@cassandra.apache.org Subject: Re: A Question About Hints Hi Kenneth, The concern is that in some cases, hints accumulate on nodes, and it takes a while until they are delivered (multi DCs). I see that whenever there are a lot of hints in play,like after a rolling restart, the cluster works harder. That's why I want to decrease the hints delivery time. I didn't want to change the configuration blindly and thought the community might have some experience on this subject. I went over the cassandra.yaml file but didn't find any information on optimizing these attributes, just that the max_throttle is divided between nodes in the cluster and that I should increase the max_hints_delivery_threads because I have multi-dc deployments. # Maximum throttle in KBs per second, per delivery thread. This will be # reduced proportionally to the number of nodes in the cluster. (If there # are two nodes in the cluster, each delivery thread will use the maximum # rate; if there are three, each will throttle to half of the maximum, # since we expect two nodes to be delivering hints simultaneously) hinted_handoff_throttle_in_kb: 1024 # Number of threads with which to deliver hints; # Consider increasing this number when you have multi-dc deployments, since # cross-dc handoff tends to be slower max_hints_delivery_threads: 2 Thanks for your help! On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman wrote: What is the concern? Why are you looking there? The casssandra.yml file has some notes about it. Did you read them? From: shalom sagges [mailto:shalomsag...@gmail.com] Sent: Monday, March 04, 2019 7:22 AM To: user@cassandra.apache.org Subject: A Question About Hints Hi All, Does anyone know what is the most optimal hints configuration (multiple DCs) in terms of max_hints_delivery_threads and hinted_handoff_throttle_in_kb? If it's different for various use cases, is there a rule of thumb I can work with? I found this post but it's quite old: http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/ Thanks!
Re: A Question About Hints
It varies... Some clusters have 48 nodes, others 24 nodes and some 8 nodes. Both settings are on default. I’d try making a single conservative change to one or the other, measure and reassess. Then do same to other setting. That's the plan, but I thought I might first get some valuable information from someone in the community that has already experienced in this type of change. Thanks! On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman wrote: > It sounds like your use case might be appropriate for tuning those two > settings some. > > > > How many nodes are in the cluster? > > Are both settings definitely on the default values currently? > > > > I’d try making a single conservative change to one or the other, measure > and reassess. Then do same to other setting. > > > > Then of course share your results with us. > > > > *From:* shalom sagges [mailto:shalomsag...@gmail.com] > *Sent:* Monday, March 04, 2019 9:54 AM > *To:* user@cassandra.apache.org > *Subject:* Re: A Question About Hints > > > > Hi Kenneth, > > > > The concern is that in some cases, hints accumulate on nodes, and it takes > a while until they are delivered (multi DCs). > > I see that whenever there are a lot of hints in play,like after a rolling > restart, the cluster works harder. That's why I want to decrease the hints > delivery time. > > I didn't want to change the configuration blindly and thought the > community might have some experience on this subject. > > > > I went over the cassandra.yaml file but didn't find any information on > optimizing these attributes, just that the max_throttle is divided between > nodes in the cluster and that I should increase the > max_hints_delivery_threads because I have multi-dc deployments. > > > > # Maximum throttle in KBs per second, per delivery thread. This will be > # reduced proportionally to the number of nodes in the cluster. (If there > # are two nodes in the cluster, each delivery thread will use the maximum > # rate; if there are three, each will throttle to half of the maximum, > # since we expect two nodes to be delivering hints simultaneously.) > hinted_handoff_throttle_in_kb: 1024 > > # Number of threads with which to deliver hints; > # Consider increasing this number when you have multi-dc deployments, since > # cross-dc handoff tends to be slower > max_hints_delivery_threads: 2 > > > > > > Thanks for your help! > > > > > > On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman > wrote: > > What is the concern? Why are you looking there? The casssandra.yml file > has some notes about it. Did you read them? > > > > *From:* shalom sagges [mailto:shalomsag...@gmail.com] > *Sent:* Monday, March 04, 2019 7:22 AM > *To:* user@cassandra.apache.org > *Subject:* A Question About Hints > > > > Hi All, > > > > Does anyone know what is the most optimal hints configuration (multiple > DCs) in terms of > > max_hints_delivery_threads and hinted_handoff_throttle_in_kb? > > If it's different for various use cases, is there a rule of thumb I can > work with? > > > > I found this post but it's quite old: > > http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/ > > > > Thanks! > >
RE: A Question About Hints
It sounds like your use case might be appropriate for tuning those two settings some. How many nodes are in the cluster? Are both settings definitely on the default values currently? I’d try making a single conservative change to one or the other, measure and reassess. Then do same to other setting. Then of course share your results with us. From: shalom sagges [mailto:shalomsag...@gmail.com] Sent: Monday, March 04, 2019 9:54 AM To: user@cassandra.apache.org Subject: Re: A Question About Hints Hi Kenneth, The concern is that in some cases, hints accumulate on nodes, and it takes a while until they are delivered (multi DCs). I see that whenever there are a lot of hints in play,like after a rolling restart, the cluster works harder. That's why I want to decrease the hints delivery time. I didn't want to change the configuration blindly and thought the community might have some experience on this subject. I went over the cassandra.yaml file but didn't find any information on optimizing these attributes, just that the max_throttle is divided between nodes in the cluster and that I should increase the max_hints_delivery_threads because I have multi-dc deployments. # Maximum throttle in KBs per second, per delivery thread. This will be # reduced proportionally to the number of nodes in the cluster. (If there # are two nodes in the cluster, each delivery thread will use the maximum # rate; if there are three, each will throttle to half of the maximum, # since we expect two nodes to be delivering hints simultaneously.) hinted_handoff_throttle_in_kb: 1024 # Number of threads with which to deliver hints; # Consider increasing this number when you have multi-dc deployments, since # cross-dc handoff tends to be slower max_hints_delivery_threads: 2 Thanks for your help! On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman wrote: What is the concern? Why are you looking there? The casssandra.yml file has some notes about it. Did you read them? From: shalom sagges [mailto:shalomsag...@gmail.com] Sent: Monday, March 04, 2019 7:22 AM To: user@cassandra.apache.org Subject: A Question About Hints Hi All, Does anyone know what is the most optimal hints configuration (multiple DCs) in terms of max_hints_delivery_threads and hinted_handoff_throttle_in_kb? If it's different for various use cases, is there a rule of thumb I can work with? I found this post but it's quite old: http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/ Thanks!
Re: A Question About Hints
Hi Kenneth, The concern is that in some cases, hints accumulate on nodes, and it takes a while until they are delivered (multi DCs). I see that whenever there are a lot of hints in play,like after a rolling restart, the cluster works harder. That's why I want to decrease the hints delivery time. I didn't want to change the configuration blindly and thought the community might have some experience on this subject. I went over the cassandra.yaml file but didn't find any information on optimizing these attributes, just that the max_throttle is divided between nodes in the cluster and that I should increase the max_hints_delivery_threads because I have multi-dc deployments. # Maximum throttle in KBs per second, per delivery thread. This will be # reduced proportionally to the number of nodes in the cluster. (If there # are two nodes in the cluster, each delivery thread will use the maximum # rate; if there are three, each will throttle to half of the maximum, # since we expect two nodes to be delivering hints simultaneously.) hinted_handoff_throttle_in_kb: 1024 # Number of threads with which to deliver hints; # Consider increasing this number when you have multi-dc deployments, since # cross-dc handoff tends to be slower max_hints_delivery_threads: 2 Thanks for your help! On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman wrote: > What is the concern? Why are you looking there? The casssandra.yml file > has some notes about it. Did you read them? > > > > *From:* shalom sagges [mailto:shalomsag...@gmail.com] > *Sent:* Monday, March 04, 2019 7:22 AM > *To:* user@cassandra.apache.org > *Subject:* A Question About Hints > > > > Hi All, > > > > Does anyone know what is the most optimal hints configuration (multiple > DCs) in terms of > > max_hints_delivery_threads and hinted_handoff_throttle_in_kb? > > If it's different for various use cases, is there a rule of thumb I can > work with? > > > > I found this post but it's quite old: > > http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/ > > > > Thanks! >
RE: A Question About Hints
What is the concern? Why are you looking there? The casssandra.yml file has some notes about it. Did you read them? From: shalom sagges [mailto:shalomsag...@gmail.com] Sent: Monday, March 04, 2019 7:22 AM To: user@cassandra.apache.org Subject: A Question About Hints Hi All, Does anyone know what is the most optimal hints configuration (multiple DCs) in terms of max_hints_delivery_threads and hinted_handoff_throttle_in_kb? If it's different for various use cases, is there a rule of thumb I can work with? I found this post but it's quite old: http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/ Thanks!
