Re: [Openais] detecting cpg joiners
On Mon, Apr 13, 2009 at 02:17:00PM -0500, David Teigland wrote: On Mon, Apr 13, 2009 at 12:10:33PM -0700, Steven Dake wrote: On Mon, 2009-04-13 at 13:35 -0500, David Teigland wrote: 0. configure token timeout to some long time that is longer than all the following steps take 1. cluster members are nodeid's: 1,2,3,4 2. cpg foo has the following members: nodeid 1, pid 10 nodeid 2, pid 20 nodeid 3, pid 30 nodeid 4, pid 40 3. nodeid 4: ifdown eth0, kill corosync, kill pid 40 (optionally reboot this node now) 4. nodeid 4: ifup eth0, start corosync 5. members of cpg foo (1:10, 2:20, 3:30) all get a confchg showing that 4:40 is not a member 6. nodeid 4: start process pid 41 that joins cpg foo 7. members of cpg foo (1:10, 2:20, 3:30, 4:41) all get a confchg showing that 4:41 is a member (Steps 6 and 7 should work the same even if the process started in step 6 has pid 40 instead of pid 41.) 100% agree that is how it should work. If it doesn't, we will fix it. The only thing that may be strange is if pid in step 6 is the same pid as 40. Are you certain the test case which fails has a differing pid at step 6? If you fix step 5, then I suspect steps 6,7 will just work. After the test failed at step 5 I didn't pay too much attention to 6,7... but I'm sure that the pid in step 6 was different (I didn't reboot the node). It's not clear what the plan was for this, any recent related changes I should try? Dave ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Wed, May 06, 2009 at 02:10:27PM -0700, Steven Dake wrote: On Wed, 2009-05-06 at 15:04 -0500, David Teigland wrote: On Mon, Apr 13, 2009 at 02:17:00PM -0500, David Teigland wrote: On Mon, Apr 13, 2009 at 12:10:33PM -0700, Steven Dake wrote: On Mon, 2009-04-13 at 13:35 -0500, David Teigland wrote: 0. configure token timeout to some long time that is longer than all the following steps take 1. cluster members are nodeid's: 1,2,3,4 2. cpg foo has the following members: nodeid 1, pid 10 nodeid 2, pid 20 nodeid 3, pid 30 nodeid 4, pid 40 3. nodeid 4: ifdown eth0, kill corosync, kill pid 40 (optionally reboot this node now) 4. nodeid 4: ifup eth0, start corosync 5. members of cpg foo (1:10, 2:20, 3:30) all get a confchg showing that 4:40 is not a member 6. nodeid 4: start process pid 41 that joins cpg foo 7. members of cpg foo (1:10, 2:20, 3:30, 4:41) all get a confchg showing that 4:41 is a member (Steps 6 and 7 should work the same even if the process started in step 6 has pid 40 instead of pid 41.) 100% agree that is how it should work. If it doesn't, we will fix it. The only thing that may be strange is if pid in step 6 is the same pid as 40. Are you certain the test case which fails has a differing pid at step 6? If you fix step 5, then I suspect steps 6,7 will just work. After the test failed at step 5 I didn't pay too much attention to 6,7... but I'm sure that the pid in step 6 was different (I didn't reboot the node). It's not clear what the plan was for this, any recent related changes I should try? Dave I haven't tried corosync with this test case, but it should work now. Did you try latest corosync on this case? If it still fails Jan can address before 1.0. Just tried it, and I get the same behavior as before. Dave ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, Apr 09, 2009 at 09:58:15PM -0700, Joel Becker wrote: On Thu, Apr 09, 2009 at 06:06:13PM -0700, Steven Dake wrote: I'd like to clear up that when Andrew talks about the membership not generating a leave event for totem processes in this scenario (which he integrates directly with), this is true. But cpg should generate a leave event. Even if the pid is the same? That is, if my node reboots very fast, and my daemon comes back. What happens in cpg if a) my daemon has a different pid, b) my daemon has the same pid? I'd like to see a) a leave event for the old nodeid+pid and a join event for the new nodeid+pid, b) a leave and a join event for the nodeid+pid. Steve, I never got a reply for this. I want to clarify cpg behavior before I fix up my daemon's routines. Joel -- Life's Little Instruction Book #30 Never buy a house without a fireplace. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, Apr 09, 2009 at 06:02:38PM -0700, Steven Dake wrote: The issue that Dave is talking about I believe is described in the following bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=489451 No, not at all. IMO you should get a leave event for any process that leaves the process group independent of how totem works underneath. CPG should provide the guarantees you seek, and if it doesn't, it is defective. OK, good. Here's what we expect: 0. configure token timeout to some long time that is longer than all the following steps take 1. cluster members are nodeid's: 1,2,3,4 2. cpg foo has the following members: nodeid 1, pid 10 nodeid 2, pid 20 nodeid 3, pid 30 nodeid 4, pid 40 3. nodeid 4: ifdown eth0, kill corosync, kill pid 40 (optionally reboot this node now) 4. nodeid 4: ifup eth0, start corosync 5. members of cpg foo (1:10, 2:20, 3:30) all get a confchg showing that 4:40 is not a member 6. nodeid 4: start process pid 41 that joins cpg foo 7. members of cpg foo (1:10, 2:20, 3:30, 4:41) all get a confchg showing that 4:41 is a member (Steps 6 and 7 should work the same even if the process started in step 6 has pid 40 instead of pid 41.) Dave ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Mon, 2009-04-13 at 13:35 -0500, David Teigland wrote: On Thu, Apr 09, 2009 at 06:02:38PM -0700, Steven Dake wrote: The issue that Dave is talking about I believe is described in the following bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=489451 No, not at all. IMO you should get a leave event for any process that leaves the process group independent of how totem works underneath. CPG should provide the guarantees you seek, and if it doesn't, it is defective. OK, good. Here's what we expect: 0. configure token timeout to some long time that is longer than all the following steps take 1. cluster members are nodeid's: 1,2,3,4 2. cpg foo has the following members: nodeid 1, pid 10 nodeid 2, pid 20 nodeid 3, pid 30 nodeid 4, pid 40 3. nodeid 4: ifdown eth0, kill corosync, kill pid 40 (optionally reboot this node now) 4. nodeid 4: ifup eth0, start corosync 5. members of cpg foo (1:10, 2:20, 3:30) all get a confchg showing that 4:40 is not a member 6. nodeid 4: start process pid 41 that joins cpg foo 7. members of cpg foo (1:10, 2:20, 3:30, 4:41) all get a confchg showing that 4:41 is a member (Steps 6 and 7 should work the same even if the process started in step 6 has pid 40 instead of pid 41.) Dave 100% agree that is how it should work. If it doesn't, we will fix it. The only thing that may be strange is if pid in step 6 is the same pid as 40. Are you certain the test case which fails has a differing pid at step 6? This points out a weakness in the current cpg protocol which could be addressed by adding a pid start time to the multicast message to uniquely identify node restarts with the same pid startup order. Unfortunately this would have to be done in some backward compatible fashion. Regards -steve ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Mon, Apr 13, 2009 at 12:10:33PM -0700, Steven Dake wrote: On Mon, 2009-04-13 at 13:35 -0500, David Teigland wrote: 0. configure token timeout to some long time that is longer than all the following steps take 1. cluster members are nodeid's: 1,2,3,4 2. cpg foo has the following members: nodeid 1, pid 10 nodeid 2, pid 20 nodeid 3, pid 30 nodeid 4, pid 40 3. nodeid 4: ifdown eth0, kill corosync, kill pid 40 (optionally reboot this node now) 4. nodeid 4: ifup eth0, start corosync 5. members of cpg foo (1:10, 2:20, 3:30) all get a confchg showing that 4:40 is not a member 6. nodeid 4: start process pid 41 that joins cpg foo 7. members of cpg foo (1:10, 2:20, 3:30, 4:41) all get a confchg showing that 4:41 is a member (Steps 6 and 7 should work the same even if the process started in step 6 has pid 40 instead of pid 41.) 100% agree that is how it should work. If it doesn't, we will fix it. The only thing that may be strange is if pid in step 6 is the same pid as 40. Are you certain the test case which fails has a differing pid at step 6? If you fix step 5, then I suspect steps 6,7 will just work. After the test failed at step 5 I didn't pay too much attention to 6,7... but I'm sure that the pid in step 6 was different (I didn't reboot the node). Dave ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Mon, Apr 13, 2009 at 02:17:00PM -0500, David Teigland wrote: On Mon, Apr 13, 2009 at 12:10:33PM -0700, Steven Dake wrote: On Mon, 2009-04-13 at 13:35 -0500, David Teigland wrote: 0. configure token timeout to some long time that is longer than all the following steps take 1. cluster members are nodeid's: 1,2,3,4 2. cpg foo has the following members: nodeid 1, pid 10 nodeid 2, pid 20 nodeid 3, pid 30 nodeid 4, pid 40 3. nodeid 4: ifdown eth0, kill corosync, kill pid 40 (optionally reboot this node now) 4. nodeid 4: ifup eth0, start corosync 5. members of cpg foo (1:10, 2:20, 3:30) all get a confchg showing that 4:40 is not a member 6. nodeid 4: start process pid 41 that joins cpg foo 7. members of cpg foo (1:10, 2:20, 3:30, 4:41) all get a confchg showing that 4:41 is a member (Steps 6 and 7 should work the same even if the process started in step 6 has pid 40 instead of pid 41.) 100% agree that is how it should work. If it doesn't, we will fix it. The only thing that may be strange is if pid in step 6 is the same pid as 40. Are you certain the test case which fails has a differing pid at step 6? If you fix step 5, then I suspect steps 6,7 will just work. After the test failed at step 5 I didn't pay too much attention to 6,7... but I'm sure that the pid in step 6 was different (I didn't reboot the node). Yeah, if we reliably get 4:40 leaves; 4:40 joins, we still have the information we need. We need the event. The pid-wrap concern was based on the assumption that 4:40 leaving and a new process 4:40 joining would be considered as a steady-state and we would get no leave event. Joel -- Anything that is too stupid to be spoken is sung. - Voltaire Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
Joel Becker wrote: Steve, Dave, etc, Someone told me a while back that a node joining a cpg group would be by its lonesome in the join message. That is, when the node gets its first confchg, it will be the only node in the list of joins. I've been using this to detect the first joiner of the group (I joined, and the member count is 1). Dave's since told me that this assumption is not valid (if it ever was). So two or more nodes can join in parallel, and each can see more than node in the list of joins for its first confchg. I'm now trying to figure out an algorithm for first joiner. I have a couple of questions: 1) If I see member_count == join_count, does that mean every member has just joinded, and all the members are receiving the same join message? 2) If member_count == join_count, can leave_count be non-zero? If it is, am I guaranteed that we're looking at all old members left, all new members joined? If these both are true, I can simply isolate a first joiner by checking member_count == join_count and selecting the lowest node number. I don't think you can detect a first-joiner using CPG. cman does it by reading the totem confchg messages. It is quite possible for two nodes to join at the same time ... during the same SYNC phase so you certainly can't rely on that. 1) If member_count == join count, then it's a safe bet that they are all new nodes, and yes , it is true that all nodes should see the same confchg messages 2) if join_count 0 then leave_count will always be zero. That's a consequence of how CPG sends its messages really, join and leave messages are always separate. Don't rely on this behaviour though! Although I can't see any reason to change it, I'd rather not have it burned into the defacto specification. -- Chrissie ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, Apr 9, 2009 at 09:37, Chrissie Caulfield ccaul...@redhat.com wrote: Joel Becker wrote: Steve, Dave, etc, Someone told me a while back that a node joining a cpg group would be by its lonesome in the join message. That is, when the node gets its first confchg, it will be the only node in the list of joins. I've been using this to detect the first joiner of the group (I joined, and the member count is 1). Dave's since told me that this assumption is not valid (if it ever was). So two or more nodes can join in parallel, and each can see more than node in the list of joins for its first confchg. I'm now trying to figure out an algorithm for first joiner. I have a couple of questions: 1) If I see member_count == join_count, does that mean every member has just joinded, and all the members are receiving the same join message? 2) If member_count == join_count, can leave_count be non-zero? If it is, am I guaranteed that we're looking at all old members left, all new members joined? If these both are true, I can simply isolate a first joiner by checking member_count == join_count and selecting the lowest node number. I don't think you can detect a first-joiner using CPG. cman does it by reading the totem confchg messages. It is quite possible for two nodes to join at the same time ... during the same SYNC phase so you certainly can't rely on that. 1) If member_count == join count, then it's a safe bet that they are all new nodes, and yes , it is true that all nodes should see the same confchg messages 2) if join_count 0 then leave_count will always be zero. That's a consequence of how CPG sends its messages really, join and leave messages are always separate. Don't rely on this behaviour though! Although I can't see any reason to change it, I'd rather not have it burned into the defacto specification. For added fun, a node that restarts quickly enough (think a VM) won't even appear to have left (or rejoined) the cluster. At the next totem confchg event, It will simply just be there again with no indication that anything happened. At least this is true for the raw corosync/openais membership data, perhaps CPG can infer this some other way. ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On 4/9/2009 at 5:50 AM, in message 26ef5e70904090450s40e92dcfgea0fc34826360...@mail.gmail.com, Andrew Beekhof beek...@gmail.com wrote: On Thu, Apr 9, 2009 at 09:37, Chrissie Caulfield ccaul...@redhat.com wrote: Joel Becker wrote: Steve, Dave, etc, Someone told me a while back that a node joining a cpg group would be by its lonesome in the join message. That is, when the node gets its first confchg, it will be the only node in the list of joins. I've been using this to detect the first joiner of the group (I joined, and the member count is 1). Dave's since told me that this assumption is not valid (if it ever was). So two or more nodes can join in parallel, and each can see more than node in the list of joins for its first confchg. I'm now trying to figure out an algorithm for first joiner. I have a couple of questions: 1) If I see member_count == join_count, does that mean every member has just joinded, and all the members are receiving the same join message? 2) If member_count == join_count, can leave_count be non-zero? If it is, am I guaranteed that we're looking at all old members left, all new members joined? If these both are true, I can simply isolate a first joiner by checking member_count == join_count and selecting the lowest node number. I don't think you can detect a first-joiner using CPG. cman does it by reading the totem confchg messages. It is quite possible for two nodes to join at the same time ... during the same SYNC phase so you certainly can't rely on that. 1) If member_count == join count, then it's a safe bet that they are all new nodes, and yes , it is true that all nodes should see the same confchg messages 2) if join_count 0 then leave_count will always be zero. That's a consequence of how CPG sends its messages really, join and leave messages are always separate. Don't rely on this behaviour though! Although I can't see any reason to change it, I'd rather not have it burned into the defacto specification. For added fun, a node that restarts quickly enough (think a VM) won't even appear to have left (or rejoined) the cluster. At the next totem confchg event, It will simply just be there again with no indication that anything happened. At least this is true for the raw corosync/openais membership data, perhaps CPG can infer this some other way. When a new node joins the group does it also create the group? e.g. http://www.opengroup.org/RI/technologies/cords/gipc.pdf has an epoch number with each join/leave message, the group is created by whoever joined in epoch 0. Hth, Robert ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, Apr 09, 2009 at 01:50:18PM +0200, Andrew Beekhof wrote: For added fun, a node that restarts quickly enough (think a VM) won't even appear to have left (or rejoined) the cluster. At the next totem confchg event, It will simply just be there again with no indication that anything happened. At least this is true for the raw corosync/openais membership data, perhaps CPG can infer this some other way. Cpg should not let a node go away and come back without notice. In practice I'd expect back to back confchg's: one showing it leave and another showing it join. As Chrissie mentioned earlier, cpg shouldn't show the same node both leaving and joining in a single confchg. In theory I think it would be legitimate. Consider a couple examples. m: member list, j: joined list, l: left list 1. nodes A and B join at once A gets confchg: m=A,B j=A,B l= B gets confchg: m=A,B j=A,B l= 2. node C joins A gets confchg: m=A,B,C j=C l= B gets confchg: m=A,B,C j=C l= C gets confchg: m=A,B,C j=C l= 3. node C leaves and quickly rejoins in a single confchg A gets confchg: m=A,B,C j=C l=C B gets confchg: m=A,B,C j=C l=C C gets confchg: m=A,B,C j=C l=C 4. node D joins and quickly leaves (or fails) in a single confchg A gets confchg: m=A,B,C j=D l=D B gets confchg: m=A,B,C j=D l=D C gets confchg: m=A,B,C j=D l=D D gets confchg: m=A,B,C j=D l=D ?* * if D does a quick join+leave it may expect to see this confchg showing it in the joined list, the left list, and not in the member list. Again, the examples in 3 and 4 are, I think, legitimate in theory. In practice it sounds like they won't occur. If a quick leave+join is guaranteed to be visible through cpg, then it must be possible to observe at the lower level from raw corosync data. Dave ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, Apr 09, 2009 at 08:37:00AM +0100, Chrissie Caulfield wrote: 1) If member_count == join count, then it's a safe bet that they are all new nodes, and yes , it is true that all nodes should see the same confchg messages 2) if join_count 0 then leave_count will always be zero. That's a consequence of how CPG sends its messages really, join and leave messages are always separate. Don't rely on this behaviour though! Although I can't see any reason to change it, I'd rather not have it burned into the defacto specification. I agree we shouldn't rely on this. I'm just more concerned that if there is member_count==join_count and leave_count0, we can rely on members == joiners, and thus treat it as a newly created group (all members are in the just joined state). Joel -- War doesn't determine who's right; war determines who's left. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, Apr 09, 2009 at 01:50:18PM +0200, Andrew Beekhof wrote: For added fun, a node that restarts quickly enough (think a VM) won't even appear to have left (or rejoined) the cluster. At the next totem confchg event, It will simply just be there again with no indication that anything happened. This had BETTER not happen. If it does, we can't recover the dead+restarted node, and our filesystems are going to corrupt all the time. Joel -- If you are ever in doubt as to whether or not to kiss a pretty girl, give her the benefit of the doubt -Thomas Carlyle Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, Apr 9, 2009 at 20:49, Joel Becker joel.bec...@oracle.com wrote: On Thu, Apr 09, 2009 at 01:50:18PM +0200, Andrew Beekhof wrote: For added fun, a node that restarts quickly enough (think a VM) won't even appear to have left (or rejoined) the cluster. At the next totem confchg event, It will simply just be there again with no indication that anything happened. This had BETTER not happen. It does, I've seen it enough times that Pacemaker has code to deal with it. If it does, we can't recover the dead+restarted node, and our filesystems are going to corrupt all the time. Joel -- If you are ever in doubt as to whether or not to kiss a pretty girl, give her the benefit of the doubt -Thomas Carlyle Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, Apr 9, 2009 at 19:15, David Teigland teigl...@redhat.com wrote: On Thu, Apr 09, 2009 at 01:50:18PM +0200, Andrew Beekhof wrote: For added fun, a node that restarts quickly enough (think a VM) won't even appear to have left (or rejoined) the cluster. At the next totem confchg event, It will simply just be there again with no indication that anything happened. At least this is true for the raw corosync/openais membership data, perhaps CPG can infer this some other way. Cpg should not let a node go away and come back without notice. In practice I'd expect back to back confchg's: one showing it leave and another showing it join. If you mean the raw confchg's that lcrsos see, then nope. Try this, set token: to longer than your node takes to reboot and reboot a node. For physical nodes this isn't a realistic scenario, but VMs can easily boot in 10 seconds or so. As Chrissie mentioned earlier, cpg shouldn't show the same node both leaving and joining in a single confchg. In theory I think it would be legitimate. Consider a couple examples. m: member list, j: joined list, l: left list 1. nodes A and B join at once A gets confchg: m=A,B j=A,B l= B gets confchg: m=A,B j=A,B l= 2. node C joins A gets confchg: m=A,B,C j=C l= B gets confchg: m=A,B,C j=C l= C gets confchg: m=A,B,C j=C l= 3. node C leaves and quickly rejoins in a single confchg A gets confchg: m=A,B,C j=C l=C B gets confchg: m=A,B,C j=C l=C C gets confchg: m=A,B,C j=C l=C 4. node D joins and quickly leaves (or fails) in a single confchg A gets confchg: m=A,B,C j=D l=D B gets confchg: m=A,B,C j=D l=D C gets confchg: m=A,B,C j=D l=D D gets confchg: m=A,B,C j=D l=D ?* * if D does a quick join+leave it may expect to see this confchg showing it in the joined list, the left list, and not in the member list. Again, the examples in 3 and 4 are, I think, legitimate in theory. In practice it sounds like they won't occur. If a quick leave+join is guaranteed to be visible through cpg, then it must be possible to observe at the lower level from raw corosync data. Dave ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, Apr 09, 2009 at 10:12:43PM +0200, Andrew Beekhof wrote: On Thu, Apr 9, 2009 at 20:49, Joel Becker joel.bec...@oracle.com wrote: On Thu, Apr 09, 2009 at 01:50:18PM +0200, Andrew Beekhof wrote: For added fun, a node that restarts quickly enough (think a VM) won't even appear to have left (or rejoined) the cluster. At the next totem confchg event, It will simply just be there again with no indication that anything happened. ? ? ? ?This had BETTER not happen. It does, I've seen it enough times that Pacemaker has code to deal with it. I'd call that a serious flaw we need to get fixed. I'll see if I can make it happen here. Dave ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, Apr 09, 2009 at 03:17:47PM -0700, Steven Dake wrote: A proper system using this model doesn't care - it synchronizes every time regardless of who left or joined based upon whether it has state to sync that is unique. Dave, If we're going to use cpg for our membership, we need to come up with a scheme to detect these node downs. We probably should do this together, so we don't reinvent it. Joel -- If you are ever in doubt as to whether or not to kiss a pretty girl, give her the benefit of the doubt -Thomas Carlyle Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, Apr 09, 2009 at 10:12:43PM +0200, Andrew Beekhof wrote: On Thu, Apr 9, 2009 at 20:49, Joel Becker joel.bec...@oracle.com wrote: On Thu, Apr 09, 2009 at 01:50:18PM +0200, Andrew Beekhof wrote: For added fun, a node that restarts quickly enough (think a VM) won't even appear to have left (or rejoined) the cluster. At the next totem confchg event, It will simply just be there again with no indication that anything happened. This had BETTER not happen. It does, I've seen it enough times that Pacemaker has code to deal with it. Andrew, I'm mad at you. This is death for filesystems. Next time, please let us know when the system is this bad :-) Joel -- Ninety feet between bases is perhaps as close as man has ever come to perfection. - Red Smith Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, 2009-04-09 at 17:17 -0700, Joel Becker wrote: On Thu, Apr 09, 2009 at 04:09:18PM -0500, David Teigland wrote: On Thu, Apr 09, 2009 at 03:50:08PM -0500, David Teigland wrote: On Thu, Apr 09, 2009 at 10:12:43PM +0200, Andrew Beekhof wrote: On Thu, Apr 9, 2009 at 20:49, Joel Becker joel.bec...@oracle.com wrote: On Thu, Apr 09, 2009 at 01:50:18PM +0200, Andrew Beekhof wrote: For added fun, a node that restarts quickly enough (think a VM) won't even appear to have left (or rejoined) the cluster. At the next totem confchg event, It will simply just be there again with no indication that anything happened. ? ? ? ?This had BETTER not happen. It does, I've seen it enough times that Pacemaker has code to deal with it. I'd call that a serious flaw we need to get fixed. I'll see if I can make it happen here. That was pretty simple. - set token to 5 minutes - nodes 1,2,3,4 are cluster members and members of a cpg - on node4: ifdown eth0, kill corosync, ifup eth0, start corosync - nodes 1,2,3 seem completely unaware that 4 ever went away When node 4 joins the cpg after coming back, the cpg on nodes 1,2,3 think that a new fifth process/node is joining the cpg. The cpg on node 4 shows itself being added as a new fourth cpg member. Steve, If node 4's old process went away, shouldn't we be getting a 'leave' for that, rather than it persisting in the member list? Joel I'd like to clear up that when Andrew talks about the membership not generating a leave event for totem processes in this scenario (which he integrates directly with), this is true. But cpg should generate a leave event. ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
guarantees you seek, and if it doesn't, it is defective. The only exception might be if the new process reuses the same PID since the pid/nodeid/group are the uniqifiers and if pid is the same, there is no way to detect the new process (and remove the old one). PID reuse happens more often than you may think. We finally started to use PID/starttime tuple to get unique process identifiers. - Dietmar ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] detecting cpg joiners
On Thu, Apr 09, 2009 at 06:06:13PM -0700, Steven Dake wrote: I'd like to clear up that when Andrew talks about the membership not generating a leave event for totem processes in this scenario (which he integrates directly with), this is true. But cpg should generate a leave event. Even if the pid is the same? That is, if my node reboots very fast, and my daemon comes back. What happens in cpg if a) my daemon has a different pid, b) my daemon has the same pid? I'd like to see a) a leave event for the old nodeid+pid and a join event for the new nodeid+pid, b) a leave and a join event for the nodeid+pid. Joel -- Life's Little Instruction Book #306 Take a nap on Sunday afternoons. Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
