Re: [Gluster-devel] Corner-Case during snapshot handshake.

2014-04-10 Thread Avra Sengupta

*Re-formatting*

Hi,

In a cluster of 5 nodes a volumes V1 is present, which hosts 4 bricks on
nodes 1 through 4

  N1   N2  N3   N4   N5
--   --  --   --   --
V1N1   N2  N3   N4   X

Now a Snapshot is taken S1 and snap volfiles will be created on nodes

  N1   N2   N3   N4   N5
--   --   --   --   --
S1   S1VOLS1VOLS1VOLS1VOLS1VOL

At this point N5 goes down, and after that we issue a delete on S1. The
delete will be successful, and will not log any missed entries because
all nodes hosting snap bricks are up and we delete the snap volfiles in
those nodes

  N1   N2   N3   N4   N5
--   --   --   --   --
S1XXXXX

At this point we create another snap with the name S1, but it will be a
different snap with a different snap-uuid

  N1   N2   N3   N4   N5
--   --   --   --   --
S1   S1VOLS1VOLS1VOLS1VOLS1VOL(old)

At this point N5 node comes up, and during handshake it receives S1 snap
volume details from other node, which are for a different snap(have
different snap-id).

The challenge is to make the node aware of the right snap vol, without
resorting to a peer reject. This scenario will occur only with a node,
that is not hosting a snap-brick for the said snapshot because the
missed_snap_list keeps track of the missed creates, deletes and
restores, and will handle if any brick misses any of these ops.

What I propose is for the node to check if the incoming new Snap ID
features in the missed snap-deletes list for the node it's handshaking
with. If so, then its a no-op. If not, it should check, if any
snap-bricks for the said snap-volume are hosted in the peer, that it's
handshaking with. If no, then again it will be a no-op. Only in a case
where the node it is handshaking with, has no missed snap-deletes for
the particular snap-id and is hosting a brick for the same, is when it
will take precendence over the current node(N5). and the S1VOL files
will be overwriten.

So the flowchart for any peer having a name conflict during handshake
will be.

Step 1: Am I hosting any snap-bricks for the conflicted snap. If yes
goto step 5

Step 2: Is the node I am handshaking with, has any missed deletes for
the same snap-id. If yes then goto step 5

Step 3: Is the node I am handshaking with, hosting any snap-bricks for
the conflicted snap, If no, then goto step 5

Step 4: The other node takes precedence, and I should delte my snap-vol
files. Move on to the next snap

Step 5. It's a no-op. Move on to the next snap.

Regards,
Avra

On 04/10/2014 07:52 PM, Avra Sengupta wrote:

Hi,

In a cluster of 5 nodes a volumes V1 is present, which hosts 4 bricks 
on nodes 1 through 4


 N1   N2   N3 N4  N5
  -   --- 
  -

V1 B1   B2   B3 B4   X

Now a Snapshot is taken S1 and snap volfiles will be created on nodes

 N1   N2   N3 N4  N5
  -   --- 
  -

S1   S1VOL  S1VOLS1VOL S1VOL S1VOL

At this point N5 goes down, and after that we issue a delete on S1. 
The delete will be successful, and will not log any missed entries 
because all nodes hosting snap bricks are up and we delete the snap 
volfiles in those nodes


 N1   N2   N3 N4  N5
  -   --- 
  -
S1  X  X X X 
S1VOL


At this point we create another snap with the name S1, but it will be 
a different snap with a different snap-uuid


 N1   N2   N3 N4  N5
  -   --- 
  -

S1   S1VOL  S1VOLS1VOL S1VOL S1VOL(old)

At this point N5 node comes up, and during handshake it receives S1 
snap volume details from other node, which are for a different 
snap(have different snap-id).


The challenge is to make the node aware of the right snap vol, without 
resorting to a peer reject. This scenario will occur only with a node, 
that is not hosting a snap-brick for the said snapshot because the 
missed_snap_list keeps track of the missed creates, deletes and 
restores, and will handle if any brick misses any of these ops.


What I propose is for the node to check if the incoming new Snap ID 
features in the missed snap-deletes list for the node it's handshaking 
with. If so, then its a no-op. If not, it should check, if any 
snap-bricks for the said snap-volume are hosted in the peer, that it's 

Re: [Gluster-devel] Corner-Case during snapshot handshake.

2014-04-10 Thread Avra Sengupta

On 04/10/2014 08:57 PM, Atin Mukherjee wrote:


On 04/10/2014 08:05 PM, Avra Sengupta wrote:

*Re-formatting*

Hi,

In a cluster of 5 nodes a volumes V1 is present, which hosts 4 bricks on
nodes 1 through 4

   N1   N2  N3   N4   N5
 --   --  --   --   --
V1N1   N2  N3   N4   X

Now a Snapshot is taken S1 and snap volfiles will be created on nodes

   N1   N2   N3   N4   N5
 --   --   --   --   --
S1   S1VOLS1VOLS1VOLS1VOLS1VOL

At this point N5 goes down, and after that we issue a delete on S1. The
delete will be successful, and will not log any missed entries because
all nodes hosting snap bricks are up and we delete the snap volfiles in
those nodes

   N1   N2   N3   N4   N5
 --   --   --   --   --
S1XXXX S1VOL(old)


It's a typo, my bad. The S1 Vol files won't be deleted in N5, coz during the 
delete operation
N5 was down. So with the new snap create(of the same name S1), the rest of the 
nodes will have
a new snapshot(and respective volume files), leading to the condition below.


At this point we create another snap with the name S1, but it will be a
different snap with a different snap-uuid

   N1   N2   N3   N4   N5
 --   --   --   --   --
S1   S1VOLS1VOLS1VOLS1VOLS1VOL(old)

I may not have the complete sketch of snapshot design in my mind,
however just wondering why would N5 have an old S1VOL as you have
mentioned that its been already deleted. Am I missing any trick here?
Once I get this clarity, then only the further paragraph makes sense for
reading :)
--Atin

At this point N5 node comes up, and during handshake it receives S1 snap
volume details from other node, which are for a different snap(have
different snap-id).

The challenge is to make the node aware of the right snap vol, without
resorting to a peer reject. This scenario will occur only with a node,
that is not hosting a snap-brick for the said snapshot because the
missed_snap_list keeps track of the missed creates, deletes and
restores, and will handle if any brick misses any of these ops.

What I propose is for the node to check if the incoming new Snap ID
features in the missed snap-deletes list for the node it's handshaking
with. If so, then its a no-op. If not, it should check, if any
snap-bricks for the said snap-volume are hosted in the peer, that it's
handshaking with. If no, then again it will be a no-op. Only in a case
where the node it is handshaking with, has no missed snap-deletes for
the particular snap-id and is hosting a brick for the same, is when it
will take precendence over the current node(N5). and the S1VOL files
will be overwriten.

So the flowchart for any peer having a name conflict during handshake
will be.

Step 1: Am I hosting any snap-bricks for the conflicted snap. If yes
goto step 5

Step 2: Is the node I am handshaking with, has any missed deletes for
the same snap-id. If yes then goto step 5

Step 3: Is the node I am handshaking with, hosting any snap-bricks for
the conflicted snap, If no, then goto step 5

Step 4: The other node takes precedence, and I should delte my snap-vol
files. Move on to the next snap

Step 5. It's a no-op. Move on to the next snap.

Regards,
Avra

On 04/10/2014 07:52 PM, Avra Sengupta wrote:

Hi,

In a cluster of 5 nodes a volumes V1 is present, which hosts 4 bricks
on nodes 1 through 4

  N1   N2   N3 N4  N5
   -   ---
  -
V1 B1   B2   B3 B4   X

Now a Snapshot is taken S1 and snap volfiles will be created on nodes

  N1   N2   N3 N4  N5
   -   ---
  -
S1   S1VOL  S1VOLS1VOL S1VOL S1VOL

At this point N5 goes down, and after that we issue a delete on S1.
The delete will be successful, and will not log any missed entries
because all nodes hosting snap bricks are up and we delete the snap
volfiles in those nodes

  N1   N2   N3 N4  N5
   -   ---
  -
S1  X  X X X
S1VOL

At this point we create another snap with the name S1, but it will be
a different snap with a different snap-uuid

  N1   N2   N3 N4  N5
   -   ---
  -
S1   S1VOL  S1VOLS1VOL S1VOL S1VOL(old)

At this point N5 node comes up, and during handshake it receives S1
snap volume details from other node, which are for a different