Re: understand bootstrapping
The process is almost the same as bootstrapping. The leaving node state transits from NORMAL to LEAVING and finally to LEFT. It waits for the ring delay as part of each state transition in order to propagate the entire cluster. Pending ranges are updated. In the case of leaving, there will be nodes responsible for more ranges (in addition to their current ranges). Those nodes and the leaving node keep on receiving writes until LEFT. - Yifan On Thu, Feb 4, 2021 at 1:39 PM Han wrote: > Thank you Yifan for the details. I have a related question: when we issue > a command to remove a node A from the ring, there could be a time that > some node B thinks node A is removed, but some node C still thinks node A > is in the ring and could reach node A. What happens if node C sends a > write request to node A during this time? Is it possible to happen? > > Thanks > Han > > > On Wed, Jan 27, 2021 at 5:48 PM Yifan Cai wrote: > >> Your thoughts regarding Gossip are correct. There could be a time that >> nodes in the cluster hold different views of the ring locally. >> >> In the case of bootstrapping, >> 1. The joining node updates its status to BOOT before streaming data and >> waits for a certain delay in order to populate the update in the cluster. >> 2. The replica nodes calculate the pending ranges, and the joining node >> can get new writes for the token ranges, meanwhile data streaming is >> on-going. >> 3. When the other nodes learn the joining node has finished bootstrapping >> via gossip (after some delay), the new node starts to serve the read as >> well. >> >> (I was wrong about not getting any client traffic earlier. The joining >> node accepts write, but no read) >> >> - Yifan >> >> On Tue, Jan 26, 2021 at 11:36 AM Han wrote: >> >>> > I'm particularly trying to understand the fault-tolerant part of > updating Token Ring state on every node The new node only joins the ring (updates the rings state) when the data streaming (bootstrapping) is successful. Otherwise, the existing ring remains as is, the joining node remains in JOINING state, and it won't get any client traffic. If I understand the question correctly. >>> >>> Thanks Yifan for your response. >>> >>> The ring state update is propagated via gossip, so it is "eventually >>> consistent". This means there is a time period when some existing node has >>> the new token ring but other existing nodes still have the old ring info, >>> right? Is it true that other operations (e.g. replications) are still >>> going on between nodes, even when their local token rings info are not >>> consistent? >>> >>> From the new node point of view, even when it is successful, at the end >>> of `StorageService::joinTokenRing`, it is possible that some existing nodes >>> have not updated their token ring yet, is this correct? >>> >>> Thanks >>> Han >>> >>> >>> Hopefully, the answers help. - Yifan On Sun, Jan 24, 2021 at 1:00 PM Han wrote: > Hi, > > I wanted to understand how the bootstrapping (add a new node) works. > My understanding is that the first step is Token Allocation and the new > node will get a number of tokens. > > My question is: > > How / when do the existing nodes update their Token Ring state? and > is that different between the seed node and non-seed node? > > I'm particularly trying to understand the fault-tolerant part of > updating Token Ring state on every node, but couldn't find relevant info > by > searching. > > Any info or pointers are appreciated. > > Thanks! > Han > >
Re: understand bootstrapping
Thank you Yifan for the details. I have a related question: when we issue a command to remove a node A from the ring, there could be a time that some node B thinks node A is removed, but some node C still thinks node A is in the ring and could reach node A. What happens if node C sends a write request to node A during this time? Is it possible to happen? Thanks Han On Wed, Jan 27, 2021 at 5:48 PM Yifan Cai wrote: > Your thoughts regarding Gossip are correct. There could be a time that > nodes in the cluster hold different views of the ring locally. > > In the case of bootstrapping, > 1. The joining node updates its status to BOOT before streaming data and > waits for a certain delay in order to populate the update in the cluster. > 2. The replica nodes calculate the pending ranges, and the joining node > can get new writes for the token ranges, meanwhile data streaming is > on-going. > 3. When the other nodes learn the joining node has finished bootstrapping > via gossip (after some delay), the new node starts to serve the read as > well. > > (I was wrong about not getting any client traffic earlier. The joining > node accepts write, but no read) > > - Yifan > > On Tue, Jan 26, 2021 at 11:36 AM Han wrote: > >> I'm particularly trying to understand the fault-tolerant part of updating Token Ring state on every node >>> >>> The new node only joins the ring (updates the rings state) when the data >>> streaming (bootstrapping) is successful. Otherwise, the existing ring >>> remains as is, the joining node remains in JOINING state, and it won't get >>> any client traffic. If I understand the question correctly. >>> >> >> Thanks Yifan for your response. >> >> The ring state update is propagated via gossip, so it is "eventually >> consistent". This means there is a time period when some existing node has >> the new token ring but other existing nodes still have the old ring info, >> right? Is it true that other operations (e.g. replications) are still >> going on between nodes, even when their local token rings info are not >> consistent? >> >> From the new node point of view, even when it is successful, at the end >> of `StorageService::joinTokenRing`, it is possible that some existing nodes >> have not updated their token ring yet, is this correct? >> >> Thanks >> Han >> >> >> >>> >>> Hopefully, the answers help. >>> >>> - Yifan >>> >>> On Sun, Jan 24, 2021 at 1:00 PM Han wrote: >>> Hi, I wanted to understand how the bootstrapping (add a new node) works. My understanding is that the first step is Token Allocation and the new node will get a number of tokens. My question is: How / when do the existing nodes update their Token Ring state? and is that different between the seed node and non-seed node? I'm particularly trying to understand the fault-tolerant part of updating Token Ring state on every node, but couldn't find relevant info by searching. Any info or pointers are appreciated. Thanks! Han
Re: understand bootstrapping
Your thoughts regarding Gossip are correct. There could be a time that nodes in the cluster hold different views of the ring locally. In the case of bootstrapping, 1. The joining node updates its status to BOOT before streaming data and waits for a certain delay in order to populate the update in the cluster. 2. The replica nodes calculate the pending ranges, and the joining node can get new writes for the token ranges, meanwhile data streaming is on-going. 3. When the other nodes learn the joining node has finished bootstrapping via gossip (after some delay), the new node starts to serve the read as well. (I was wrong about not getting any client traffic earlier. The joining node accepts write, but no read) - Yifan On Tue, Jan 26, 2021 at 11:36 AM Han wrote: > >>> I'm particularly trying to understand the fault-tolerant part of >>> updating Token Ring state on every node >> >> The new node only joins the ring (updates the rings state) when the data >> streaming (bootstrapping) is successful. Otherwise, the existing ring >> remains as is, the joining node remains in JOINING state, and it won't get >> any client traffic. If I understand the question correctly. >> > > Thanks Yifan for your response. > > The ring state update is propagated via gossip, so it is "eventually > consistent". This means there is a time period when some existing node has > the new token ring but other existing nodes still have the old ring info, > right? Is it true that other operations (e.g. replications) are still > going on between nodes, even when their local token rings info are not > consistent? > > From the new node point of view, even when it is successful, at the end > of `StorageService::joinTokenRing`, it is possible that some existing nodes > have not updated their token ring yet, is this correct? > > Thanks > Han > > > >> >> Hopefully, the answers help. >> >> - Yifan >> >> On Sun, Jan 24, 2021 at 1:00 PM Han wrote: >> >>> Hi, >>> >>> I wanted to understand how the bootstrapping (add a new node) works. My >>> understanding is that the first step is Token Allocation and the new node >>> will get a number of tokens. >>> >>> My question is: >>> >>> How / when do the existing nodes update their Token Ring state? and is >>> that different between the seed node and non-seed node? >>> >>> I'm particularly trying to understand the fault-tolerant part of >>> updating Token Ring state on every node, but couldn't find relevant info by >>> searching. >>> >>> Any info or pointers are appreciated. >>> >>> Thanks! >>> Han >>> >>>
Re: understand bootstrapping
> > >> I'm particularly trying to understand the fault-tolerant part of updating >> Token Ring state on every node > > The new node only joins the ring (updates the rings state) when the data > streaming (bootstrapping) is successful. Otherwise, the existing ring > remains as is, the joining node remains in JOINING state, and it won't get > any client traffic. If I understand the question correctly. > Thanks Yifan for your response. The ring state update is propagated via gossip, so it is "eventually consistent". This means there is a time period when some existing node has the new token ring but other existing nodes still have the old ring info, right? Is it true that other operations (e.g. replications) are still going on between nodes, even when their local token rings info are not consistent? >From the new node point of view, even when it is successful, at the end of `StorageService::joinTokenRing`, it is possible that some existing nodes have not updated their token ring yet, is this correct? Thanks Han > > Hopefully, the answers help. > > - Yifan > > On Sun, Jan 24, 2021 at 1:00 PM Han wrote: > >> Hi, >> >> I wanted to understand how the bootstrapping (add a new node) works. My >> understanding is that the first step is Token Allocation and the new node >> will get a number of tokens. >> >> My question is: >> >> How / when do the existing nodes update their Token Ring state? and is >> that different between the seed node and non-seed node? >> >> I'm particularly trying to understand the fault-tolerant part of updating >> Token Ring state on every node, but couldn't find relevant info by >> searching. >> >> Any info or pointers are appreciated. >> >> Thanks! >> Han >> >>
Re: understand bootstrapping
Hi Han, How / when do the existing nodes update their Token Ring state? The new joining node sets its tokens and populates to the cluster via gossip after completing data streaming. is that different between the seed node and non-seed node? Data streaming step is skipped if a node is seed. Checkout `StorageService#joinTokenRing()` for the details. I'm particularly trying to understand the fault-tolerant part of updating > Token Ring state on every node The new node only joins the ring (updates the rings state) when the data streaming (bootstrapping) is successful. Otherwise, the existing ring remains as is, the joining node remains in JOINING state, and it won't get any client traffic. If I understand the question correctly. Hopefully, the answers help. - Yifan On Sun, Jan 24, 2021 at 1:00 PM Han wrote: > Hi, > > I wanted to understand how the bootstrapping (add a new node) works. My > understanding is that the first step is Token Allocation and the new node > will get a number of tokens. > > My question is: > > How / when do the existing nodes update their Token Ring state? and is > that different between the seed node and non-seed node? > > I'm particularly trying to understand the fault-tolerant part of updating > Token Ring state on every node, but couldn't find relevant info by > searching. > > Any info or pointers are appreciated. > > Thanks! > Han > >
