[ClusterLabs] Antw: HA domain controller fences newly joined node after fence_ipmilan delay even if transition was aborted.
>>> Vitaly Zolotusky schrieb am 17.12.2018 um 21:43 in >>> Nachricht <1782126841.215210.1545079428...@webmail6.networksolutionsemail.com>: > Hello, > I have a 2 node cluster and stonith is configured for SBD and fence_ipmilan. > fence_ipmilan for node 1 is configured for 0 delay and for node 2 for 30 sec > delay so that nodes do not start killing each other during startup. > In some cases (usually right after installation and when node 1 comes up > first and node 2 second) the node that comes up first (node 1) states that > node 2 is unclean, but can't fence it until quorum reached. I'd concentrate on examining why node2 is considered unclean. Of course that doesn't fix the issue, but if fixing it takes some time, you'll have a work-around ;-) > Then as soon as quorum is reached after startup of corosync on node 2 it > sends a fence request for node 2. > Fence_ipmilan gets into 30 sec delay. > Pacemaker gets started on node 2. > While fence_ipmilan is still waiting for the delay node 1 crmd aborts > transition that requested the fence. > Even though the transition was aborted, when delay time expires node 2 gets > fenced. > Excerpts from messages are below. I also attached messages from both nodes > and pe-input files from node 1. > Any suggestions would be appreciated. > Thank you very much for your help! > Vitaly Zolotusky > > Here are excerpts from the messages: > > Node 1 - controller - rhino66-right 172.18.51.81 - came up first > * > > Nov 29 16:47:54 rhino66-right pengine[22183]: warning: Fencing and resource > management disabled due to lack of quorum > Nov 29 16:47:54 rhino66-right pengine[22183]: warning: Node > rhino66-left.lab.archivas.com is unclean! > Nov 29 16:47:54 rhino66-right pengine[22183]: notice: Cannot fence unclean > nodes until quorum is attained (or no-quorum-policy is set to ignore) > . > Nov 29 16:48:38 rhino66-right corosync[6677]: [TOTEM ] A new membership > (172.16.1.81:60) was formed. Members joined: 2 > Nov 29 16:48:38 rhino66-right corosync[6677]: [VOTEQ ] Waiting for all > cluster members. Current votes: 1 expected_votes: 2 > Nov 29 16:48:38 rhino66-right corosync[6677]: [QUORUM] This node is within > the primary component and will provide service. > Nov 29 16:48:38 rhino66-right corosync[6677]: [QUORUM] Members[2]: 1 2 > Nov 29 16:48:38 rhino66-right corosync[6677]: [MAIN ] Completed service > synchronization, ready to provide service. > Nov 29 16:48:38 rhino66-right crmd[22184]: notice: Quorum acquired > Nov 29 16:48:38 rhino66-right pacemakerd[22152]: notice: Quorum acquired > Nov 29 16:48:38 rhino66-right crmd[22184]: notice: Could not obtain a node > name for corosync nodeid 2 > Nov 29 16:48:38 rhino66-right pacemakerd[22152]: notice: Could not obtain > a node name for corosync nodeid 2 > Nov 29 16:48:38 rhino66-right crmd[22184]: notice: Could not obtain a node > name for corosync nodeid 2 > Nov 29 16:48:38 rhino66-right crmd[22184]: notice: Node (null) state is > now member > Nov 29 16:48:38 rhino66-right pacemakerd[22152]: notice: Could not obtain > a node name for corosync nodeid 2 > Nov 29 16:48:38 rhino66-right pacemakerd[22152]: notice: Node (null) state > is now member > Nov 29 16:48:54 rhino66-right crmd[22184]: notice: State transition S_IDLE >-> S_POLICY_ENGINE > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: Watchdog will be > used via SBD if fencing is required > Nov 29 16:48:54 rhino66-right pengine[22183]: warning: Scheduling Node > rhino66-left.lab.archivas.com for STONITH > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * Fence (reboot) > rhino66-left.lab.archivas.com 'node is unclean' > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * Start > fence_sbd ( rhino66-right.lab.archivas.com ) > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * Start > ipmi-82 ( rhino66-right.lab.archivas.com ) > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * Start S_IP >( rhino66-right.lab.archivas.com ) > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * Start > postgres:0( rhino66-right.lab.archivas.com ) > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * Start > ethmonitor:0 ( rhino66-right.lab.archivas.com ) > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * Start > fs_monitor:0 ( rhino66-right.lab.archivas.com ) due to unrunnable > DBMaster running (blocked) > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * Start > tomcat-instance:0 ( rhino66-right.lab.archivas.com ) due to unrunnable > DBMaster running (blocked) > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * Start > ClusterMonitor:0 ( rhino66-right.lab.archivas.com ) due to unrunnable > DBMaster running (blocked) > Nov 29 16:48:54 rhino66-right pengine[22183]: warning: Calculated >
Re: [ClusterLabs] HA domain controller fences newly joined node after fence_ipmilan delay even if transition was aborted.
Ken, Thank you very much for quick response! I do have "two_node: 1" in the corosync.conf. I have attached it to this email (not from the same system as original messages, but they are all the same). Syncing startup of corosync and pacemaker on different nodes would be a problem for us. I suspect that the problem is that corosync assumes quorum is reached as soon as corosync is started on both nodes, but pacemaker does not abort fencing until pacemaker starts on the other node. I will try to work around this issue by moving corosync and pacemaker startups on single node as close to each other as possible. Thanks again! _Vitaly > On December 17, 2018 at 6:01 PM Ken Gaillot wrote: > > > On Mon, 2018-12-17 at 15:43 -0500, Vitaly Zolotusky wrote: > > Hello, > > I have a 2 node cluster and stonith is configured for SBD and > > fence_ipmilan. > > fence_ipmilan for node 1 is configured for 0 delay and for node 2 for > > 30 sec delay so that nodes do not start killing each other during > > startup. > > If you're using corosync 2 or later, you can set "two_node: 1" in > corosync.conf. That implies the wait_for_all option, so that at start- > up, both nodes must be present before quorum can be reached the first > time. (After that point, one node can go away and quorum will be > retained.) > > Another way to avoid this is to start corosync on all nodes, then start > pacemaker on all nodes. > > > In some cases (usually right after installation and when node 1 comes > > up first and node 2 second) the node that comes up first (node 1) > > states that node 2 is unclean, but can't fence it until quorum > > reached. > > Then as soon as quorum is reached after startup of corosync on node 2 > > it sends a fence request for node 2. > > Fence_ipmilan gets into 30 sec delay. > > Pacemaker gets started on node 2. > > While fence_ipmilan is still waiting for the delay node 1 crmd aborts > > transition that requested the fence. > > Even though the transition was aborted, when delay time expires node > > 2 gets fenced. > > Currently, pacemaker has no way of cancelling fencing once it's been > initiated. Technically, it would be possible to cancel an operation > that's in the delay stage (assuming that no other fence device has > already been attempted, if there are more than one), but that hasn't > been implemented. > > > Excerpts from messages are below. I also attached messages from both > > nodes and pe-input files from node 1. > > Any suggestions would be appreciated. > > Thank you very much for your help! > > Vitaly Zolotusky > > > > Here are excerpts from the messages: > > > > Node 1 - controller - rhino66-right 172.18.51.81 - came up > > first * > > > > Nov 29 16:47:54 rhino66-right pengine[22183]: warning: Fencing and > > resource management disabled due to lack of quorum > > Nov 29 16:47:54 rhino66-right pengine[22183]: warning: Node rhino66- > > left.lab.archivas.com is unclean! > > Nov 29 16:47:54 rhino66-right pengine[22183]: notice: Cannot fence > > unclean nodes until quorum is attained (or no-quorum-policy is set to > > ignore) > > . > > Nov 29 16:48:38 rhino66-right corosync[6677]: [TOTEM ] A new > > membership (172.16.1.81:60) was formed. Members joined: 2 > > Nov 29 16:48:38 rhino66-right corosync[6677]: [VOTEQ ] Waiting for > > all cluster members. Current votes: 1 expected_votes: 2 > > Nov 29 16:48:38 rhino66-right corosync[6677]: [QUORUM] This node is > > within the primary component and will provide service. > > Nov 29 16:48:38 rhino66-right corosync[6677]: [QUORUM] Members[2]: > > 1 2 > > Nov 29 16:48:38 rhino66-right corosync[6677]: [MAIN ] Completed > > service synchronization, ready to provide service. > > Nov 29 16:48:38 rhino66-right crmd[22184]: notice: Quorum acquired > > Nov 29 16:48:38 rhino66-right pacemakerd[22152]: notice: Quorum > > acquired > > Nov 29 16:48:38 rhino66-right crmd[22184]: notice: Could not obtain > > a node name for corosync nodeid 2 > > Nov 29 16:48:38 rhino66-right pacemakerd[22152]: notice: Could not > > obtain a node name for corosync nodeid 2 > > Nov 29 16:48:38 rhino66-right crmd[22184]: notice: Could not obtain > > a node name for corosync nodeid 2 > > Nov 29 16:48:38 rhino66-right crmd[22184]: notice: Node (null) > > state is now member > > Nov 29 16:48:38 rhino66-right pacemakerd[22152]: notice: Could not > > obtain a node name for corosync nodeid 2 > > Nov 29 16:48:38 rhino66-right pacemakerd[22152]: notice: Node > > (null) state is now member > > Nov 29 16:48:54 rhino66-right crmd[22184]: notice: State transition > > S_IDLE -> S_POLICY_ENGINE > > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: Watchdog will > > be used via SBD if fencing is required > > Nov 29 16:48:54 rhino66-right pengine[22183]: warning: Scheduling > > Node rhino66-left.lab.archivas.com for STONITH > > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * Fence > > (reboot) rhino66-left.lab.archivas.com 'node
Re: [ClusterLabs] HA domain controller fences newly joined node after fence_ipmilan delay even if transition was aborted.
On Mon, 2018-12-17 at 15:43 -0500, Vitaly Zolotusky wrote: > Hello, > I have a 2 node cluster and stonith is configured for SBD and > fence_ipmilan. > fence_ipmilan for node 1 is configured for 0 delay and for node 2 for > 30 sec delay so that nodes do not start killing each other during > startup. If you're using corosync 2 or later, you can set "two_node: 1" in corosync.conf. That implies the wait_for_all option, so that at start- up, both nodes must be present before quorum can be reached the first time. (After that point, one node can go away and quorum will be retained.) Another way to avoid this is to start corosync on all nodes, then start pacemaker on all nodes. > In some cases (usually right after installation and when node 1 comes > up first and node 2 second) the node that comes up first (node 1) > states that node 2 is unclean, but can't fence it until quorum > reached. > Then as soon as quorum is reached after startup of corosync on node 2 > it sends a fence request for node 2. > Fence_ipmilan gets into 30 sec delay. > Pacemaker gets started on node 2. > While fence_ipmilan is still waiting for the delay node 1 crmd aborts > transition that requested the fence. > Even though the transition was aborted, when delay time expires node > 2 gets fenced. Currently, pacemaker has no way of cancelling fencing once it's been initiated. Technically, it would be possible to cancel an operation that's in the delay stage (assuming that no other fence device has already been attempted, if there are more than one), but that hasn't been implemented. > Excerpts from messages are below. I also attached messages from both > nodes and pe-input files from node 1. > Any suggestions would be appreciated. > Thank you very much for your help! > Vitaly Zolotusky > > Here are excerpts from the messages: > > Node 1 - controller - rhino66-right 172.18.51.81 - came up > first * > > Nov 29 16:47:54 rhino66-right pengine[22183]: warning: Fencing and > resource management disabled due to lack of quorum > Nov 29 16:47:54 rhino66-right pengine[22183]: warning: Node rhino66- > left.lab.archivas.com is unclean! > Nov 29 16:47:54 rhino66-right pengine[22183]: notice: Cannot fence > unclean nodes until quorum is attained (or no-quorum-policy is set to > ignore) > . > Nov 29 16:48:38 rhino66-right corosync[6677]: [TOTEM ] A new > membership (172.16.1.81:60) was formed. Members joined: 2 > Nov 29 16:48:38 rhino66-right corosync[6677]: [VOTEQ ] Waiting for > all cluster members. Current votes: 1 expected_votes: 2 > Nov 29 16:48:38 rhino66-right corosync[6677]: [QUORUM] This node is > within the primary component and will provide service. > Nov 29 16:48:38 rhino66-right corosync[6677]: [QUORUM] Members[2]: > 1 2 > Nov 29 16:48:38 rhino66-right corosync[6677]: [MAIN ] Completed > service synchronization, ready to provide service. > Nov 29 16:48:38 rhino66-right crmd[22184]: notice: Quorum acquired > Nov 29 16:48:38 rhino66-right pacemakerd[22152]: notice: Quorum > acquired > Nov 29 16:48:38 rhino66-right crmd[22184]: notice: Could not obtain > a node name for corosync nodeid 2 > Nov 29 16:48:38 rhino66-right pacemakerd[22152]: notice: Could not > obtain a node name for corosync nodeid 2 > Nov 29 16:48:38 rhino66-right crmd[22184]: notice: Could not obtain > a node name for corosync nodeid 2 > Nov 29 16:48:38 rhino66-right crmd[22184]: notice: Node (null) > state is now member > Nov 29 16:48:38 rhino66-right pacemakerd[22152]: notice: Could not > obtain a node name for corosync nodeid 2 > Nov 29 16:48:38 rhino66-right pacemakerd[22152]: notice: Node > (null) state is now member > Nov 29 16:48:54 rhino66-right crmd[22184]: notice: State transition > S_IDLE -> S_POLICY_ENGINE > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: Watchdog will > be used via SBD if fencing is required > Nov 29 16:48:54 rhino66-right pengine[22183]: warning: Scheduling > Node rhino66-left.lab.archivas.com for STONITH > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * Fence > (reboot) rhino66-left.lab.archivas.com 'node is unclean' > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * > Start fence_sbd ( rhino66-right.lab.archivas.com ) > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * > Start ipmi-82 ( rhino66-right.lab.archivas.com ) > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * > Start S_IP ( rhino66-right.lab.archivas.com ) > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * > Start postgres:0( rhino66-right.lab.archivas.com ) > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * > Start ethmonitor:0 ( rhino66-right.lab.archivas.com ) > Nov 29 16:48:54 rhino66-right pengine[22183]: notice: * > Start fs_monitor:0 ( rhino66-right.lab.archivas.com > ) due to unrunnable DBMaster running (blocked) > Nov 29 16:48:54 rhino66-right
Re: [ClusterLabs] Corosync 3.0.0 is available at corosync.org!
On 17/12/2018 12:14, Jan Pokorný wrote: > On 17/12/18 10:04 +, Christine Caulfield wrote: >> On 17/12/2018 09:34, Ulrich Windl wrote: >>> I wonder: Is there a migration script that can converts corosync.conf files? >>> At least you have a few version components in the config file that will help >>> such tool to know what to do... ;-) >> >> Sadly not - that I know of. The clufter project *may* be looking into it >> but I have inside knowledge on that. > > Yes, it's in the backlog, stay tuned. excellent! thank you :) Chrissie > >> Single ring UDP/UDPU config files should work without change anyway. >> Multi-ring configs will need changing to transport: knet and names >> adding to nodes. > ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Corosync 3.0.0 is available at corosync.org!
On 17/12/18 10:04 +, Christine Caulfield wrote: > On 17/12/2018 09:34, Ulrich Windl wrote: >> I wonder: Is there a migration script that can converts corosync.conf files? >> At least you have a few version components in the config file that will help >> such tool to know what to do... ;-) > > Sadly not - that I know of. The clufter project *may* be looking into it > but I have inside knowledge on that. Yes, it's in the backlog, stay tuned. > Single ring UDP/UDPU config files should work without change anyway. > Multi-ring configs will need changing to transport: knet and names > adding to nodes. -- Jan (Poki) pgpjjTinrqnfe.pgp Description: PGP signature ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Corosync 3.0.0 is available at corosync.org!
On 17/12/2018 09:34, Ulrich Windl wrote: Jan Friesse schrieb am 14.12.2018 um 15:06 in > Nachricht > <991569e4-2430-30f1-1bbc-827be7637...@redhat.com>: > [...] >> ‑ UDP/UDPU transports are still present, but supports only single ring >> (RRP is gone in favor of Knet) and doesn't support encryption > [...] > > I wonder: Is there a migration script that can converts corosync.conf files? > At least you have a few version components in the config file that will help > such tool to know what to do... ;-) > Sadly not - that I know of. The clufter project *may* be looking into it but I have inside knowledge on that. Single ring UDP/UDPU config files should work without change anyway. Multi-ring configs will need changing to transport: knet and names adding to nodes. Chrissie ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Antw: Corosync 3.0.0 is available at corosync.org!
>>> Jan Friesse schrieb am 14.12.2018 um 15:06 in Nachricht <991569e4-2430-30f1-1bbc-827be7637...@redhat.com>: [...] > ‑ UDP/UDPU transports are still present, but supports only single ring > (RRP is gone in favor of Knet) and doesn't support encryption [...] I wonder: Is there a migration script that can converts corosync.conf files? At least you have a few version components in the config file that will help such tool to know what to do... ;-) Regards, Ulrich ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Coming in Pacemaker 2.0.1 / 1.1.20: improved fencing history
On 12/17/2018 09:06 AM, Ulrich Windl wrote: Ken Gaillot schrieb am 11.12.2018 um 21:48 in > Nachricht > <33316cc0570a12255c7de7dd387caee9c5058e37.ca...@redhat.com>: >> Hi all, >> >> I expect to have the first release candidate for Pacemaker 2.0.1 >> available soon! It will mostly be a bug fix release, but one >> interesting new feature is convenient access to fencing history. >> >> Pacemaker has long had the stonith_admin ‑‑history option to show a >> history of past fencing actions that the cluster has carried out. >> However, this list included only events since the node it was run on >> had joined the cluster, and it just wasn't very convenient. >> >> In the upcoming release, the cluster keeps the fence history >> synchronized across all nodes, so you get the same answer no matter >> which node you query. > So I guess it's in the CIB then. Right? Nope. Record-taking of fenced is unchanged in principle - in memory recording with distribution of actions taking place over the cluster. Only thing added is syncing when nodes are joining and a possibility to cleanup. Klaus > >> More importantly, crm_mon will now show fencing failures and pending >> fencing actions by default. This gives a much greater visibility into >> past and current issues in a convenient way. You can control the level >> of detail (or whether it shows up at all) via command‑line options. >> >> Along with that, there is a new stonith_admin ‑‑cleanup option to erase >> the fence history, once you no longer want to see it in the crm_mon >> status display. >> ‑‑ >> Ken Gaillot >> >> ___ >> Users mailing list: Users@clusterlabs.org >> https://lists.clusterlabs.org/mailman/listinfo/users >> >> Project Home: http://www.clusterlabs.org >> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org > > > ___ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Antw: Corosync-qdevice 3.0.0 is available at GitHub!
Hi! Once again you forgot the one-line-summary what is is ;-) I guess a quorum device... Regards, Ulrich >>> Jan Friesse schrieb am 12.12.2018 um 15:20 in Nachricht : > I am pleased to announce the first stable release of Corosync‑Qdevice > 3.0.0 available immediately from GitHub at > https://github.com/corosync/corosync‑qdevice/releases as > corosync‑qdevice‑3.0.0. > > Corosync‑qdevice project is split of Corosync qdevice and qnetd daemons > found in Corosync Needle since 2.4.0 (but no longer in Corosync > Camelback 3). > > qdevice and qnetd daemons found in Corosync Needle will be getting > backports from Corosync‑qdevice project which is now consider primary > home for these daemons (so please use Corosync‑qdevice project for > filling issues/PR). > > This release contains just one small fix for warning found by coverity. > > Complete changelog for final version (compared to RC 1): > > Jan Friesse (1): >certutils: Fix warnings found by coverity > > Thanks/congratulations to all people that contributed to achieve this > great milestone. > ___ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Antw: Coming in Pacemaker 2.0.1 / 1.1.20: improved fencing history
>>> Ken Gaillot schrieb am 11.12.2018 um 21:48 in Nachricht <33316cc0570a12255c7de7dd387caee9c5058e37.ca...@redhat.com>: > Hi all, > > I expect to have the first release candidate for Pacemaker 2.0.1 > available soon! It will mostly be a bug fix release, but one > interesting new feature is convenient access to fencing history. > > Pacemaker has long had the stonith_admin ‑‑history option to show a > history of past fencing actions that the cluster has carried out. > However, this list included only events since the node it was run on > had joined the cluster, and it just wasn't very convenient. > > In the upcoming release, the cluster keeps the fence history > synchronized across all nodes, so you get the same answer no matter > which node you query. So I guess it's in the CIB then. Right? > > More importantly, crm_mon will now show fencing failures and pending > fencing actions by default. This gives a much greater visibility into > past and current issues in a convenient way. You can control the level > of detail (or whether it shows up at all) via command‑line options. > > Along with that, there is a new stonith_admin ‑‑cleanup option to erase > the fence history, once you no longer want to see it in the crm_mon > status display. > ‑‑ > Ken Gaillot > > ___ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
