Re: [DRBD-user] Dual-primary to single node
Hello, On 01/13/2012 10:59 AM, Luis M. Carril wrote: Hello, I´m new to DRBD and I think that I have a mess with some concepts and policies. I have setup a two node cluster (of virtual machines) with a shared volume in dual primary mode with ocfs2 as a basic infrastructure for some testings. I need that when one of the two nodes goes down the other continues working normally (we can assume that the other node never will recover again), but when one node fails the other enter in WFConnection state and the volume is disconnected, I have setup the standar set of policies for split brain: after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; Which policy should I use to achieve the desired behaivour (if one node fails, the other continue working alone)? these policies only take affect if the two nodes see each other again after a split-brain and if you loose one node it is correct behaviour that the remaining node has it's DRBD resources in WFConnection state. What do you mean with: volume is disconnected? How do you manage your cluster? Pacemaker? rgmanager? Without any further information on the rest of you setup and what you think is not working correct it's unable to comment further ... Regards, Andreas -- Need help with DRBD? http://www.hastexo.com/now ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Dual-primary to single node
On 01/13/2012 04:59 AM, Luis M. Carril wrote: Hello, I´m new to DRBD and I think that I have a mess with some concepts and policies. Welcome! DRBD is a bit different from many storage concepts, so it takes a bit to wrap your head around. However, careful not to overthink things... It's fundamentally quite straight forward. I have setup a two node cluster (of virtual machines) with a shared volume in dual primary mode with ocfs2 as a basic infrastructure for some testings. Do you have fencing? Dual-primary can not operate safely without a mechanism for ensuring the state of the remote node. I need that when one of the two nodes goes down the other continues working normally (we can assume that the other node never will recover again), but when one node fails The assumption that the other will never return is not a concept that DRBD can assume. This is where fencing comes in... When a node loses contact with it's peer, it has no way of knowing what state the remote node is in. Is it still running, but thinks the local peer is gone? Is the silent node hung, but might return at some point? Is the remote node powered off? The only think you know is what you don't know. Consider; Both nodes, had they simply assumed silence == death, go StandAlone and Primary. During this time, data is written to either node but that data is not replicated. Now you have divergent data and the only mechanism to recover is to invalidate the changes on one of the nodes. Data loss. The solution is fencing and resource management, which is what Andreas meant when he asked about pacemaker vs. rgmanager. the other enter in WFConnection state and the volume is disconnected, I have setup the standar set of policies for split brain: after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; Which policy should I use to achieve the desired behaivour (if one node fails, the other continue working alone)? Regards Again, as Andreas indicated, this controls the policy when comms are lost (be it because of a network error, peer dieing/hanging, whatever). It is by design that a node, after losing it's peer, goes into WFConnection (waiting for connection). In this state, if/when the peer recovers (as it often does with power fencing), the peer can re-establish the connection, sync changes and return to a normal operating state. -- Digimer E-Mail: digi...@alteeve.com Freenode handle: digimer Papers and Projects: http://alteeve.com Node Assassin: http://nodeassassin.org omg my singularity battery is dead again. stupid hawking radiation. - epitron ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Dual-primary to single node
Luis, I have experienced the same problems, what helped was to 'fence' the other node by forcing it into a reboot. I don't quite know why it worked (worrying) but I found that if I fenced the other node, I was not getting any more time-outs on the drbd block device, which I think is what you describe. I am using HP servers with iLO3 and RHEL/CentOS, so I used the fence_ipmi script included with the CentOS 6.1 distribution (do not forget to attach your fence devices to the correct nodes). Hope it helps! Robert On 13-1-2012 10:59, Luis M. Carril wrote: Hello, I´m new to DRBD and I think that I have a mess with some concepts and policies. I have setup a two node cluster (of virtual machines) with a shared volume in dual primary mode with ocfs2 as a basic infrastructure for some testings. I need that when one of the two nodes goes down the other continues working normally (we can assume that the other node never will recover again), but when one node fails the other enter in WFConnection state and the volume is disconnected, I have setup the standar set of policies for split brain: after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; Which policy should I use to achieve the desired behaivour (if one node fails, the other continue working alone)? Regards ___ Help save paper! Do you really need to print this email? Aan de inhoud van dit bericht kunnen alleen rechten ten opzichte van Morpho B.V. worden ontleend, indien zij door rechtsgeldig ondertekende stukken worden ondersteund. De informatie in dit e-mailbericht is van vertrouwelijke aard en alleen bedoeld voor gebruik door geadresseerde. Als u een bericht onbedoeld heeft ontvangen, wordt u verzocht de verzender hiervan in kennis te stellen en het bericht te vernietigen zonder te vermenigvuldigen of andersoortig te gebruiken. The contents of this electronic mail message are only binding upon Morpho B.V. if the contents of the message are accompanied by a lawfully recognized type of signature. The contents of this electronic mail message are privileged and confidential and are intended only for use by the addressee. If you have received this electronic mail message by error, please notify the sender and delete the message without reproducing it and using it in any way. # Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles ou ayant un caractère privé. S'ils ne vous sont pas destinés, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. Si ce message vous a été transmis par erreur, merci d'en informer l'expéditeur et de supprimer immédiatement de votre système informatique ce courriel ainsi que tous les documents qui y sont attachés. ** This e-mail and any attached documents may contain confidential or proprietary information. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system. # ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Dual-primary to single node
On 01/17/2012 11:07 AM, CAMPBELL Robert wrote: Luis, I have experienced the same problems, what helped was to 'fence' the other node by forcing it into a reboot. I don't quite know why it worked (worrying) but I found that if I fenced the other node, I was not getting any more time-outs on the drbd block device, which I think is what you describe. https://alteeve.com/w/2-Node_Red_Hat_KVM_Cluster_Tutorial#Concept.3B_Fencing :) I am using HP servers with iLO3 and RHEL/CentOS, so I used the fence_ipmi script included with the CentOS 6.1 distribution (do not forget to attach your fence devices to the correct nodes). Hope it helps! One thing that Luis hasn't confirmed yet was what, if any, cluster stack s/he was using with DRBD. fence_ipmilan works with both rgmanager (the supported resource manager under EL6) and pacemaker (the future-but-currently-tech-preview resource manager). From DRBD's perspective, what matters is the fence-handler section. I cover all of this in a KVM tutorial, but you can ignore that part and hopefully the rest might help clear up some things. https://alteeve.com/w/2-Node_Red_Hat_KVM_Cluster_Tutorial#Installing_DRBD https://alteeve.com/w/2-Node_Red_Hat_KVM_Cluster_Tutorial#Configuring_DRBD -- Digimer E-Mail: digi...@alteeve.com Freenode handle: digimer Papers and Projects: http://alteeve.com Node Assassin: http://nodeassassin.org omg my singularity battery is dead again. stupid hawking radiation. - epitron ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Dual-primary to single node
On 01/17/2012 12:32 PM, Luis M. Carril wrote: Hello, Ok, the fencing and splitbrain mechanisms only enter to play when both nodes meet again after some failure. So... meanwhile the nodes doesn´t connect their peer they disallow IO to the volume? Regards No, if both nodes go Standalone and Primary, both will allow access to the underlying storage, which results in a split brain. Fencing kills one of the nodes (either the defective one or the slower one) preventing it from changing it's underlying storage. PS - Please reply to the list. These discussions help others later when they are in the archives. :) -- Digimer E-Mail: digi...@alteeve.com Freenode handle: digimer Papers and Projects: http://alteeve.com Node Assassin: http://nodeassassin.org omg my singularity battery is dead again. stupid hawking radiation. - epitron ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Dual-primary to single node
El 17/01/2012 18:56, Digimer escribió:
On 01/17/2012 12:32 PM, Luis M. Carril wrote:
Hello,
Ok, the fencing and splitbrain mechanisms only enter to play when
both nodes meet again after some failure.
So... meanwhile the nodes doesn´t connect their peer they disallow
IO to the volume?
Regards
No, if both nodes go Standalone and Primary, both will allow access to
the underlying storage, which results in a split brain. Fencing kills
one of the nodes (either the defective one or the slower one) preventing
it from changing it's underlying storage.
Umph, but actually I'm testing to drop one node meanwhile it is writing
in the volume, and the volume in the surviving node is stalled
(drbd-overview freezes, but /proc/drbd shows
that it is WTFConnection, Primary and Uptodate), even if I make drbdadm
disconnect manually to make it go StandAlone, IO operations freeze on
the directory.
Maybe is an issue related to OCFS...
Well my configurations are:
In DRBD
global {
usage-count no;
}
common {
protocol C;
meta-disk internal;
startup {
wfc-timeout 300;
degr-wfc-timeout 120;# 2 minutes.
become-primary-on both;
}
syncer {
rate 10M;
}
disk {
on-io-error detresource r0 {
startup {
become-primary-on both;
}
net {
allow-two-primaries;
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
}
on master {
device/dev/drbd1;
disk /dev/xvde;
address 10.0.0.2:7789;
meta-disk internal;
}
on shadow {
device/dev/drbd1;
disk /dev/xvde;
address 10.0.0.3:7789;
meta-disk internal;
}
}ach;
}
}
In OCFS2:
cluster:
node_count = 2
name = ocfs2
node:
ip_port =
ip_address = 10.0.0.2
number = 0
name = master
cluster = ocfs2
node:
ip_port =
ip_address = 10.0.0.3
number = 1
name = shadow
cluster = ocfs2
In debconf:
ocfs2-tools ocfs2-tools/idle_timeout select 3
ocfs2-tools ocfs2-tools/reconnect_delay select 2000
ocfs2-tools ocfs2-tools/init select true
ocfs2-tools ocfs2-tools/clustername select ocfs2
ocfs2-tools ocfs2-tools/heartbeat_threshold select 31
ocfs2-tools ocfs2-tools/keepalive_delay select 2000
PS - Please reply to the list. These discussions help others later when
they are in the archives. :)
Sorry, my fault!
And thanks to all!
Regards
--
Luis M. Carril
Project Technician
Galicia Supercomputing Center (CESGA)
Avda. de Vigo s/n
15706 Santiago de Compostela
SPAIN
Tel: 34-981569810 ext 249
lmcar...@cesga.es
www.cesga.es
==
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Dual-primary to single node
On 01/17/2012 01:09 PM, Luis M. Carril wrote: El 17/01/2012 18:56, Digimer escribió: On 01/17/2012 12:32 PM, Luis M. Carril wrote: Hello, Ok, the fencing and splitbrain mechanisms only enter to play when both nodes meet again after some failure. So... meanwhile the nodes doesn´t connect their peer they disallow IO to the volume? Regards No, if both nodes go Standalone and Primary, both will allow access to the underlying storage, which results in a split brain. Fencing kills one of the nodes (either the defective one or the slower one) preventing it from changing it's underlying storage. Umph, but actually I'm testing to drop one node meanwhile it is writing in the volume, and the volume in the surviving node is stalled (drbd-overview freezes, but /proc/drbd shows that it is WTFConnection, Primary and Uptodate), even if I make drbdadm disconnect manually to make it go StandAlone, IO operations freeze on the directory. Maybe is an issue related to OCFS... Possibly, I use GFS2, not OCFS, so I can't speak to it's behaviour. I can say though that GFS2 will also block when a node it the cluster disappears, and it will remain blocked until it gets confirmation that the lost node was fenced. This is by design, as a hung cluster is better than a corrupted one. :) -- Digimer E-Mail: digi...@alteeve.com Freenode handle: digimer Papers and Projects: http://alteeve.com Node Assassin: http://nodeassassin.org omg my singularity battery is dead again. stupid hawking radiation. - epitron ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
