RE: [EXTERNAL] Re: Question on changing node IP address
I am not making a recommendation for anyone else – just sharing our experience and reasoning. It is why I argued for keeping PropertyFileSnitch in some JIRA that proposed dropping it completely. GPFS is the typical recommendation for production use. Just a hurdle not worth my time at the moment. From: Alexander Dejanovski Sent: Wednesday, February 27, 2019 9:22 AM To: user@cassandra.apache.org Subject: Re: [EXTERNAL] Re: Question on changing node IP address It has to be balanced with the dangers related to the PropertyFileSnitch. I've seen such incidents happen twice in the last few months in different places and both times recovery was difficult and hazardous. I still strongly recommend against it. On Wed, Feb 27, 2019 at 3:11 PM Durity, Sean R mailto:sean_r_dur...@homedepot.com>> wrote: We use the PropertyFileSnitch precisely because it is the same on every node. If each node has to have a different file (for GPFS) – deployment is more complicated. (And for any automated configuration you would have a list of hosts and DC/rack information to compile anyway) I do put UNKNOWN as the default DC so that any missed node easily appears in its own unused DC. Sean Durity From: Alexander Dejanovski mailto:a...@thelastpickle.com>> Sent: Wednesday, February 27, 2019 4:43 AM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: [EXTERNAL] Re: Question on changing node IP address This snitch is easy to misconfigure. It allows some nodes to have a different view of the cluster if they are configured differently, which can result in data loss (or at least data that is very hard to recover). Also it has a nasty feature that allows to set a default DC/Rack. If one node isn't properly declared in all the files throughout the cluster, it will be seen as part of that "default" DC and then again, it's hard to recover. Be aware that while the GossipingPropertyFileSnitch will not allow changing rack of DC for a node that already bootstrapped, the PropertyFileSnitch will allow to change it without any notice. So a little misconfiguration could merge all nodes from DC1 into DC2, abruptly changing token ownership (and it could very be the case that DC1 thinks it's part of DC2 but DC2 still thinks DC1 is DC1). So again, I think this snitch is dangerous and shouldn't be used. The GossipingPropertyFileSnitch is much more secure and easy to use. Cheers, On Wed, Feb 27, 2019 at 10:13 AM shalom sagges mailto:shalomsag...@gmail.com>> wrote: If you're using the PropertyFileSnitch, well... you shouldn't as it's a rather dangerous and tedious snitch to use I inherited Cassandra clusters that use the PropertyFileSnitch. It's been working fine, but you've kinda scared me :-) Why is it dangerous to use? If I decide to change the snitch, is it seamless or is there a specific procedure one must follow? Thanks! On Wed, Feb 27, 2019 at 10:08 AM Alexander Dejanovski mailto:a...@thelastpickle.com>> wrote: I confirm what Oleksandr said. Just stop Cassandra, change the IP, and restart Cassandra. If you're using the GossipingPropertyFileSnitch, the node will redeclare its new IP through Gossip and that's it. If you're using the PropertyFileSnitch, well... you shouldn't as it's a rather dangerous and tedious snitch to use. But if you are, it'll require to change the file containing all the IP addresses across the cluster. I've been changing IPs on a whole cluster back in 2.1 this way and it went through seamlessly. Cheers, On Wed, Feb 27, 2019 at 8:54 AM Oleksandr Shulgin mailto:oleksandr.shul...@zalando.de>> wrote: On Wed, Feb 27, 2019 at 4:15 AM wxn...@zjqunshuo.com<mailto:wxn...@zjqunshuo.com> mailto:wxn...@zjqunshuo.com>> wrote: >After restart with the new address the server will notice it and log a >warning, but it will keep token ownership as long as it keeps the old host id >(meaning it must use the same data directory as before restart). Based on my understanding, token range is binded to host id. As long as host id doesn't change, everything is ok. Besides data directory, any other thing can lead to host id change? And how host id is caculated? For example, if I upgrade Cassandra binary to a new version, after restart, will host id change? I believe host id is calculated once the new node is initialized and never changes afterwards, even through major upgrades. It is stored in system keyspace in data directory, and is stable across restarts. -- Alex -- - Alexander Dejanovski France @alexanderdeja Consultant Apache Cassandra Consulting http://www.thelastpickle.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.thelastpickle.com_=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=xojzDh-fJSOl_ZfDMCYIYi4sckWpwqdnKDG5QMx2nUE=HwAxm8xI-Bmc8IFmwEK0we9hlvNwUVuj7DGpXuNM8r4=> -- - Alexander Dejanovski France @alexanderdeja
Re: [EXTERNAL] Re: Question on changing node IP address
On Wed, Feb 27, 2019 at 3:11 PM Durity, Sean R wrote: > We use the PropertyFileSnitch precisely because it is the same on every > node. If each node has to have a different file (for GPFS) – deployment is > more complicated. (And for any automated configuration you would have a > list of hosts and DC/rack information to compile anyway) > > > > I do put UNKNOWN as the default DC so that any missed node easily appears > in its own unused DC. > Alright, it obviously makes a lot of difference which snitch to use. We are deploying to EC2, so we are using the EC2 snitches at all times. I guess some complexity is hidden from us by these custom implementations. At the same time, we do try to assign IP addresses in a predictable manner when deploying a new cluster, in order to fix the list of seed nodes in advance (we wouldn't care about the rest of nodes). So I think, for the original question: careful when changing IP address of seeds nodes. Probably you want to start with non-seeds and promote some of them to seeds before you start changing IP addresses of the old seeds. -- Alex
Re: [EXTERNAL] Re: Question on changing node IP address
It has to be balanced with the dangers related to the PropertyFileSnitch. I've seen such incidents happen twice in the last few months in different places and both times recovery was difficult and hazardous. I still strongly recommend against it. On Wed, Feb 27, 2019 at 3:11 PM Durity, Sean R wrote: > We use the PropertyFileSnitch precisely because it is the same on every > node. If each node has to have a different file (for GPFS) – deployment is > more complicated. (And for any automated configuration you would have a > list of hosts and DC/rack information to compile anyway) > > > > I do put UNKNOWN as the default DC so that any missed node easily appears > in its own unused DC. > > > > > > Sean Durity > > > > *From:* Alexander Dejanovski > *Sent:* Wednesday, February 27, 2019 4:43 AM > *To:* user@cassandra.apache.org > *Subject:* [EXTERNAL] Re: Question on changing node IP address > > > > This snitch is easy to misconfigure. It allows some nodes to have a > different view of the cluster if they are configured differently, which can > result in data loss (or at least data that is very hard to recover). > > Also it has a nasty feature that allows to set a default DC/Rack. If one > node isn't properly declared in all the files throughout the cluster, it > will be seen as part of that "default" DC and then again, it's hard to > recover. > > Be aware that while the GossipingPropertyFileSnitch will not allow > changing rack of DC for a node that already bootstrapped, the > PropertyFileSnitch will allow to change it without any notice. So a little > misconfiguration could merge all nodes from DC1 into DC2, abruptly changing > token ownership (and it could very be the case that DC1 thinks it's part of > DC2 but DC2 still thinks DC1 is DC1). > > So again, I think this snitch is dangerous and shouldn't be used. The > GossipingPropertyFileSnitch is much more secure and easy to use. > > > > Cheers, > > > > > > On Wed, Feb 27, 2019 at 10:13 AM shalom sagges > wrote: > > If you're using the PropertyFileSnitch, well... you shouldn't as it's a > rather dangerous and tedious snitch to use > > > > I inherited Cassandra clusters that use the PropertyFileSnitch. It's been > working fine, but you've kinda scared me :-) > > Why is it dangerous to use? > > If I decide to change the snitch, is it seamless or is there a specific > procedure one must follow? > > > > Thanks! > > > > > > On Wed, Feb 27, 2019 at 10:08 AM Alexander Dejanovski < > a...@thelastpickle.com> wrote: > > I confirm what Oleksandr said. > > Just stop Cassandra, change the IP, and restart Cassandra. > > If you're using the GossipingPropertyFileSnitch, the node will redeclare > its new IP through Gossip and that's it. > > If you're using the PropertyFileSnitch, well... you shouldn't as it's a > rather dangerous and tedious snitch to use. But if you are, it'll require > to change the file containing all the IP addresses across the cluster. > > > > I've been changing IPs on a whole cluster back in 2.1 this way and it went > through seamlessly. > > > > Cheers, > > > > On Wed, Feb 27, 2019 at 8:54 AM Oleksandr Shulgin < > oleksandr.shul...@zalando.de> wrote: > > On Wed, Feb 27, 2019 at 4:15 AM wxn...@zjqunshuo.com > wrote: > > >After restart with the new address the server will notice it and log a > warning, but it will keep token ownership as long as it keeps the old host > id (meaning it must use the same data directory as before restart). > > > > Based on my understanding, token range is binded to host id. As long as > host id doesn't change, everything is ok. Besides data directory, any other > thing can lead to host id change? And how host id is caculated? For > example, if I upgrade Cassandra binary to a new version, after restart, > will host id change? > > > > I believe host id is calculated once the new node is initialized and never > changes afterwards, even through major upgrades. It is stored in system > keyspace in data directory, and is stable across restarts. > > > > -- > > Alex > > > > -- > > - > > Alexander Dejanovski > > France > > @alexanderdeja > > > > Consultant > > Apache Cassandra Consulting > > http://www.thelastpickle.com > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.thelastpickle.com_=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=xojzDh-fJSOl_ZfDMCYIYi4sckWpwqdnKDG5QMx2nUE=HwAxm8xI-Bmc8IFmwEK0we9hlvNwUVuj7DGpXuNM8r4=> > > -- > > - > > Alexander Dejanovski
RE: [EXTERNAL] Re: Question on changing node IP address
We use the PropertyFileSnitch precisely because it is the same on every node. If each node has to have a different file (for GPFS) – deployment is more complicated. (And for any automated configuration you would have a list of hosts and DC/rack information to compile anyway) I do put UNKNOWN as the default DC so that any missed node easily appears in its own unused DC. Sean Durity From: Alexander Dejanovski Sent: Wednesday, February 27, 2019 4:43 AM To: user@cassandra.apache.org Subject: [EXTERNAL] Re: Question on changing node IP address This snitch is easy to misconfigure. It allows some nodes to have a different view of the cluster if they are configured differently, which can result in data loss (or at least data that is very hard to recover). Also it has a nasty feature that allows to set a default DC/Rack. If one node isn't properly declared in all the files throughout the cluster, it will be seen as part of that "default" DC and then again, it's hard to recover. Be aware that while the GossipingPropertyFileSnitch will not allow changing rack of DC for a node that already bootstrapped, the PropertyFileSnitch will allow to change it without any notice. So a little misconfiguration could merge all nodes from DC1 into DC2, abruptly changing token ownership (and it could very be the case that DC1 thinks it's part of DC2 but DC2 still thinks DC1 is DC1). So again, I think this snitch is dangerous and shouldn't be used. The GossipingPropertyFileSnitch is much more secure and easy to use. Cheers, On Wed, Feb 27, 2019 at 10:13 AM shalom sagges mailto:shalomsag...@gmail.com>> wrote: If you're using the PropertyFileSnitch, well... you shouldn't as it's a rather dangerous and tedious snitch to use I inherited Cassandra clusters that use the PropertyFileSnitch. It's been working fine, but you've kinda scared me :-) Why is it dangerous to use? If I decide to change the snitch, is it seamless or is there a specific procedure one must follow? Thanks! On Wed, Feb 27, 2019 at 10:08 AM Alexander Dejanovski mailto:a...@thelastpickle.com>> wrote: I confirm what Oleksandr said. Just stop Cassandra, change the IP, and restart Cassandra. If you're using the GossipingPropertyFileSnitch, the node will redeclare its new IP through Gossip and that's it. If you're using the PropertyFileSnitch, well... you shouldn't as it's a rather dangerous and tedious snitch to use. But if you are, it'll require to change the file containing all the IP addresses across the cluster. I've been changing IPs on a whole cluster back in 2.1 this way and it went through seamlessly. Cheers, On Wed, Feb 27, 2019 at 8:54 AM Oleksandr Shulgin mailto:oleksandr.shul...@zalando.de>> wrote: On Wed, Feb 27, 2019 at 4:15 AM wxn...@zjqunshuo.com<mailto:wxn...@zjqunshuo.com> mailto:wxn...@zjqunshuo.com>> wrote: >After restart with the new address the server will notice it and log a >warning, but it will keep token ownership as long as it keeps the old host id >(meaning it must use the same data directory as before restart). Based on my understanding, token range is binded to host id. As long as host id doesn't change, everything is ok. Besides data directory, any other thing can lead to host id change? And how host id is caculated? For example, if I upgrade Cassandra binary to a new version, after restart, will host id change? I believe host id is calculated once the new node is initialized and never changes afterwards, even through major upgrades. It is stored in system keyspace in data directory, and is stable across restarts. -- Alex -- - Alexander Dejanovski France @alexanderdeja Consultant Apache Cassandra Consulting http://www.thelastpickle.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.thelastpickle.com_=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=xojzDh-fJSOl_ZfDMCYIYi4sckWpwqdnKDG5QMx2nUE=HwAxm8xI-Bmc8IFmwEK0we9hlvNwUVuj7DGpXuNM8r4=> -- - Alexander Dejanovski France @alexanderdeja Consultant Apache Cassandra Consulting http://www.thelastpickle.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.thelastpickle.com_=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=xojzDh-fJSOl_ZfDMCYIYi4sckWpwqdnKDG5QMx2nUE=HwAxm8xI-Bmc8IFmwEK0we9hlvNwUVuj7DGpXuNM8r4=> The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of busin
Re: [EXTERNAL] Re: Question on changing node IP address
On Tue, Feb 26, 2019 at 3:26 PM Durity, Sean R wrote: > This has not been my experience. Changing IP address is one of the worst > admin tasks for Cassandra. System.peers and other information on each nodes > is stored by ip address. And gossip is really good at sending around the > old information mixed with new… > Hm, on which version was it? I might be biased not having worked with anything but 3.0 since recently. -- Alex
RE: [EXTERNAL] Re: Question on changing node IP address
This has not been my experience. Changing IP address is one of the worst admin tasks for Cassandra. System.peers and other information on each nodes is stored by ip address. And gossip is really good at sending around the old information mixed with new… Sean Durity From: Oleksandr Shulgin Sent: Tuesday, February 26, 2019 5:36 AM To: User Subject: [EXTERNAL] Re: Question on changing node IP address On Tue, Feb 26, 2019 at 9:39 AM wxn...@zjqunshuo.com<mailto:wxn...@zjqunshuo.com> mailto:wxn...@zjqunshuo.com>> wrote: I'm running 2.2.8 with vnodes and I'm planning to change node IP address. My procedure is: Turn down one node, setting auto_bootstrap to false in yaml file, then bring it up with -Dcassandra.replace_address. Repeat the procedure one by one for the other nodes. I care about streaming because the data is very large and if there is streaming, it will take a long time. When the node with new IP be brought up, will it take over the token range it has before? I expect no token range reassignment and no streaming. Am I right? Any thing I need care about when making IP address change? Changing the IP address of a node does not require special considerations. After restart with the new address the server will notice it and log a warning, but it will keep token ownership as long as it keeps the old host id (meaning it must use the same data directory as before restart). At the same time, *do not* use the replace_address option: it assumes empty data directory and will try to stream data from other replicas into the node. -- Alex The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
