Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-27 Thread emmanuel segura
William But i would like to know if you have a lvm resource in your pacemaker configuration Remember clvmd it's not for active di vg or lv it's for propagate the lvm meta data on all node of the cluster Il giorno 26 marzo 2012 23:17, William Seligman selig...@nevis.columbia.edu ha scritto:

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes - SOLVED

2012-03-27 Thread emmanuel segura
William :-) So now your cluster it's OK? Il giorno 27 marzo 2012 00:33, William Seligman selig...@nevis.columbia.edu ha scritto: On 3/26/12 5:31 PM, William Seligman wrote: On 3/26/12 5:17 PM, William Seligman wrote: On 3/26/12 4:28 PM, emmanuel segura wrote: and i suggest you to start

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes - SOLVED

2012-03-27 Thread William Seligman
On 3/27/12 4:52 AM, emmanuel segura wrote: So now your cluster it's OK? *Laughs* No! There's another problem I have to solve. But it's completely unrelated to this one. I'll work on it some more, and if I can't solve it I'll start a new thread. Thanks for asking, Emmanuel. (I want to prove I

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-26 Thread William Seligman
On 3/26/12 4:28 PM, emmanuel segura wrote: Sorry Willian i can't post my config now because i'm at home now not in my job I think it's no a problem if clvm start before drbd, because clvm not needed and devices to start This it's the point, i hope to be clear The introduction of

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-26 Thread William Seligman
On 3/26/12 5:17 PM, William Seligman wrote: On 3/26/12 4:28 PM, emmanuel segura wrote: Sorry Willian i can't post my config now because i'm at home now not in my job I think it's no a problem if clvm start before drbd, because clvm not needed and devices to start This it's the point, i

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes - SOLVED

2012-03-26 Thread William Seligman
On 3/26/12 5:31 PM, William Seligman wrote: On 3/26/12 5:17 PM, William Seligman wrote: On 3/26/12 4:28 PM, emmanuel segura wrote: and i suggest you to start clvmd at boot time chkconfig clvmd on I'm afraid this doesn't work. It's as I predicted; when gfs2 starts I get: Mounting GFS2

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-24 Thread emmanuel segura
How do you configure clvmd? with cman or with pacemaker? Il giorno 23 marzo 2012 22:14, William Seligman selig...@nevis.columbia.edu ha scritto: On 3/23/12 5:03 PM, emmanuel segura wrote: Sorry but i would to know if can show me your /etc/cluster/cluster.conf Here it is:

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-24 Thread William Seligman
On 3/24/12 4:47 AM, emmanuel segura wrote: How do you configure clvmd? with cman or with pacemaker? Pacemaker. Here's the output of 'crm configure show': http://pastebin.com/426CdVwN Il giorno 23 marzo 2012 22:14, William Seligman selig...@nevis.columbia.edu ha scritto: On 3/23/12 5:03

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-24 Thread emmanuel segura
I think it's better you use clvmd with cman I don't now why you use the lsb script of clvm On Redhat clvmd need of cman and you try to running with pacemaker, i not sure this is the problem but this type of configuration it's so strange I made it a virtual cluster with kvm and i not foud a

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-23 Thread William Seligman
On 3/22/12 2:43 PM, William Seligman wrote: On 3/20/12 4:55 PM, Lars Ellenberg wrote: On Fri, Mar 16, 2012 at 05:06:04PM -0400, William Seligman wrote: On 3/16/12 12:12 PM, William Seligman wrote: On 3/16/12 7:02 AM, Andreas Kurz wrote: On 03/15/2012 11:50 PM, William Seligman wrote: On

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-23 Thread emmanuel segura
Hello William Sorry but i would to know if can show me your /etc/cluster/cluster.conf Il giorno 23 marzo 2012 21:50, William Seligman selig...@nevis.columbia.edu ha scritto: On 3/22/12 2:43 PM, William Seligman wrote: On 3/20/12 4:55 PM, Lars Ellenberg wrote: On Fri, Mar 16, 2012 at

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-23 Thread William Seligman
On 3/23/12 5:03 PM, emmanuel segura wrote: Sorry but i would to know if can show me your /etc/cluster/cluster.conf Here it is: http://pastebin.com/GUr0CEgZ Il giorno 23 marzo 2012 21:50, William Seligman selig...@nevis.columbia.edu ha scritto: On 3/22/12 2:43 PM, William Seligman wrote:

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-22 Thread William Seligman
On 3/20/12 4:55 PM, Lars Ellenberg wrote: On Fri, Mar 16, 2012 at 05:06:04PM -0400, William Seligman wrote: On 3/16/12 12:12 PM, William Seligman wrote: On 3/16/12 7:02 AM, Andreas Kurz wrote: On 03/15/2012 11:50 PM, William Seligman wrote: On 3/15/12 6:07 PM, William Seligman wrote: On

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-22 Thread David Coulson
On 3/22/12 2:43 PM, William Seligman wrote: I still haven't solved the problem, but this advice has gotten me further than before. First, Lars was correct: I did not have execute permissions set on my fence peer scripts. (D'oh!) I turned them on, but that did not change anything:

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-22 Thread William Seligman
On 3/22/12 2:49 PM, David Coulson wrote: On 3/22/12 2:43 PM, William Seligman wrote: I still haven't solved the problem, but this advice has gotten me further than before. First, Lars was correct: I did not have execute permissions set on my fence peer scripts. (D'oh!) I turned them on,

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-20 Thread Lars Ellenberg
On Fri, Mar 16, 2012 at 05:06:04PM -0400, William Seligman wrote: On 3/16/12 12:12 PM, William Seligman wrote: On 3/16/12 7:02 AM, Andreas Kurz wrote: On 03/15/2012 11:50 PM, William Seligman wrote: On 3/15/12 6:07 PM, William Seligman wrote: On 3/15/12 6:05 PM, William Seligman wrote:

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-16 Thread emmanuel segura
Hello William for the lvm hang you can use this in your /etc/lvm/lvm.conf ignore_suspended_devices = 1 because i seen in the lvm log, === and then it hangs. Comparing the two, it looks like it can't close /dev/drbd0

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-16 Thread Andreas Kurz
On 03/15/2012 11:50 PM, William Seligman wrote: On 3/15/12 6:07 PM, William Seligman wrote: On 3/15/12 6:05 PM, William Seligman wrote: On 3/15/12 4:57 PM, emmanuel segura wrote: we can try to understand what happen when clvm hang edit the /etc/lvm/lvm.conf and change level = 7 in the log

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-16 Thread William Seligman
On 3/16/12 7:02 AM, Andreas Kurz wrote: On 03/15/2012 11:50 PM, William Seligman wrote: On 3/15/12 6:07 PM, William Seligman wrote: On 3/15/12 6:05 PM, William Seligman wrote: On 3/15/12 4:57 PM, emmanuel segura wrote: we can try to understand what happen when clvm hang edit the

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-16 Thread William Seligman
On 3/16/12 4:53 AM, emmanuel segura wrote: for the lvm hang you can use this in your /etc/lvm/lvm.conf ignore_suspended_devices = 1 because i seen in the lvm log, === and then it hangs. Comparing the two, it looks like it can't close

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-16 Thread William Seligman
On 3/16/12 12:12 PM, William Seligman wrote: On 3/16/12 7:02 AM, Andreas Kurz wrote: On 03/15/2012 11:50 PM, William Seligman wrote: On 3/15/12 6:07 PM, William Seligman wrote: On 3/15/12 6:05 PM, William Seligman wrote: On 3/15/12 4:57 PM, emmanuel segura wrote: we can try to understand

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread Vladislav Bogdanov
14.03.2012 00:42, William Seligman wrote: [snip] These were the log messages, which show that stonith_admin did its job and CMAN was notified of the fencing: http://pastebin.com/jaH820Bv. Could you please look at the output of 'dlm_tool ls' and 'dlm_tool dump'? You probably have 'kern_stop'

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread emmanuel segura
Hello Willian The first thing i seen in your clvmd log it's this = WARNING: Locking disabled. Be careful! This could corrupt your metadata. = use this command lvmconf --enable-cluster and remember for

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread William Seligman
On 3/15/12 5:18 AM, emmanuel segura wrote: The first thing i seen in your clvmd log it's this = WARNING: Locking disabled. Be careful! This could corrupt your metadata. = I saw that too, and thought

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread William Seligman
On 3/15/12 3:43 AM, Vladislav Bogdanov wrote: 14.03.2012 00:42, William Seligman wrote: [snip] These were the log messages, which show that stonith_admin did its job and CMAN was notified of the fencing: http://pastebin.com/jaH820Bv. Could you please look at the output of 'dlm_tool ls'

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread emmanuel segura
yes william Now try clvmd -d and see what happen locking_type = 3 it's lvm cluster lock type Il giorno 15 marzo 2012 16:15, William Seligman selig...@nevis.columbia.edu ha scritto: On 3/15/12 5:18 AM, emmanuel segura wrote: The first thing i seen in your clvmd log it's this

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread William Seligman
On 3/15/12 11:50 AM, emmanuel segura wrote: yes william Now try clvmd -d and see what happen locking_type = 3 it's lvm cluster lock type Since you asked for confirmation, here it is: the output of 'clvmd -d' just now. http://pastebin.com/bne8piEw. I crashed the other node at Mar 15

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread emmanuel segura
Hello William Ho did you created your volume group give me the output of vgs command when the cluster it's up Il giorno 15 marzo 2012 17:06, William Seligman selig...@nevis.columbia.edu ha scritto: On 3/15/12 11:50 AM, emmanuel segura wrote: yes william Now try clvmd -d and see what

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread William Seligman
On 3/15/12 12:15 PM, emmanuel segura wrote: Ho did you created your volume group pvcreate /dev/drbd0 vgcreate -c y ADMIN /dev/drbd0 lvcreate -L 200G -n usr ADMIN # ... and so on # Nevis-HA is the cluster name I used in cluster.conf mkfs.gfs2 -p lock_dlm -j 2 -t Nevis_HA:usr /dev/ADMIN/usr #

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread emmanuel segura
Hello William I don't see any error and the answer for your question it's yes can you show me your /etc/cluster/cluster.conf and your crm configure show like that more later i can try to look if i found some fix Il giorno 15 marzo 2012 17:42, William Seligman selig...@nevis.columbia.edu ha

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread Vladislav Bogdanov
15.03.2012 18:43, William Seligman wrote: On 3/15/12 3:43 AM, Vladislav Bogdanov wrote: 14.03.2012 00:42, William Seligman wrote: [snip] These were the log messages, which show that stonith_admin did its job and CMAN was notified of the fencing: http://pastebin.com/jaH820Bv. Could you

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread William Seligman
On 3/15/12 12:55 PM, emmanuel segura wrote: I don't see any error and the answer for your question it's yes can you show me your /etc/cluster/cluster.conf and your crm configure show like that more later i can try to look if i found some fix Thanks for taking a look. My cluster.conf:

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread William Seligman
On 3/15/12 3:45 PM, Vladislav Bogdanov wrote: 15.03.2012 18:43, William Seligman wrote: On 3/15/12 3:43 AM, Vladislav Bogdanov wrote: 14.03.2012 00:42, William Seligman wrote: [snip] These were the log messages, which show that stonith_admin did its job and CMAN was notified of the

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread emmanuel segura
Ok William we can try to understand what happen when clvm hang edit the /etc/lvm/lvm.conf and change level = 7 in the log session and uncomment this line file = /var/log/lvm2.log Il giorno 15 marzo 2012 20:50, William Seligman selig...@nevis.columbia.edu ha scritto: On 3/15/12 12:55 PM,

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread Lars Marowsky-Bree
On 2012-03-15T15:59:21, William Seligman selig...@nevis.columbia.edu wrote: Could this be an issue? I've noticed that my fencing agent always seems to be called with action=reboot when a node is fenced. Why is it using 'reboot' and not 'off'? Is this the standard, or am I missing a

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread William Seligman
On 3/15/12 4:57 PM, emmanuel segura wrote: we can try to understand what happen when clvm hang edit the /etc/lvm/lvm.conf and change level = 7 in the log session and uncomment this line file = /var/log/lvm2.log Here's the tail end of the file (the original is 1.6M). Because there no

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread William Seligman
On 3/15/12 6:05 PM, William Seligman wrote: On 3/15/12 4:57 PM, emmanuel segura wrote: we can try to understand what happen when clvm hang edit the /etc/lvm/lvm.conf and change level = 7 in the log session and uncomment this line file = /var/log/lvm2.log Here's the tail end of the

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-15 Thread William Seligman
On 3/15/12 6:07 PM, William Seligman wrote: On 3/15/12 6:05 PM, William Seligman wrote: On 3/15/12 4:57 PM, emmanuel segura wrote: we can try to understand what happen when clvm hang edit the /etc/lvm/lvm.conf and change level = 7 in the log session and uncomment this line file =

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-14 Thread emmanuel segura
Hello William I think it's better you make clvmd start at boot chkconfig cman on ; chkconfig clvmd on Il giorno 13 marzo 2012 23:29, William Seligman selig...@nevis.columbia.edu ha scritto: On 3/13/12 5:50 PM, emmanuel segura wrote: So if you using cman why you use lsb::clvmd I

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-14 Thread William Seligman
On 3/14/12 6:02 AM, emmanuel segura wrote: I think it's better you make clvmd start at boot chkconfig cman on ; chkconfig clvmd on I've already tried it. It doesn't work. The problem is that my LVM information is on the drbd. If I start up clvmd before drbd, it won't find the logical

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-14 Thread emmanuel segura
Hello William i did new you are using drbd and i dont't know what type of configuration you using But it's better you try to start clvm with clvmd -d like thak we can see what it's the problem Il giorno 14 marzo 2012 14:02, William Seligman selig...@nevis.columbia.edu ha scritto: On 3/14/12

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-14 Thread William Seligman
On 3/14/12 9:26 AM, Lars Marowsky-Bree wrote: On 2012-03-14T09:02:59, William Seligmanselig...@nevis.columbia.edu wrote: To ask a slightly different question - why? Does your workload require / benefit from a dual-primary architecture? Most don't.

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-14 Thread William Seligman
On 3/14/12 9:20 AM, emmanuel segura wrote: Hello William i did new you are using drbd and i dont't know what type of configuration you using But it's better you try to start clvm with clvmd -d like thak we can see what it's the problem For what it's worth, here's the output of running

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-14 Thread William Seligman
On 3/14/12 12:43 PM, Dimitri Maziuk wrote: On 03/14/2012 11:08 AM, Lars Marowsky-Bree wrote: On 2012-03-14T11:41:53, William Seligman selig...@nevis.columbia.edu wrote: I'm mindful of the issues involved, such as those Lars Ellenberg brought up in his response. I need something that will

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-14 Thread Lars Marowsky-Bree
On 2012-03-14T18:22:42, William Seligman selig...@nevis.columbia.edu wrote: Now consider a primary-primary cluster. Both run the same resource. One fails. There's no failover here; the other box still runs the resource. In my case, the only thing that has to work is cloned cluster IP

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-14 Thread Dimitri Maziuk
On 03/14/2012 05:22 PM, William Seligman wrote: Now consider a primary-primary cluster. Both run the same resource. One fails. There's no failover here; the other box still runs the resource. In my case, the only thing that has to work is cloned cluster IP address, and that I've verified

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-13 Thread emmanuel segura
Sorry Willian But i think clvmd it must be used with ocf:lvm2:clvmd esample crm confgiure primitive clvmd ocf:lvm2:clvmd params daemon_timeout=30 clone cln_clvmd clvmd and rember clvmd depend on dlm, so for the dlm you sould same Il giorno 13 marzo 2012 17:29, William Seligman

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-13 Thread William Seligman
On 3/13/12 2:49 PM, emmanuel segura wrote: Sorry Willian But i think clvmd it must be used with ocf:lvm2:clvmd esample crm confgiure primitive clvmd ocf:lvm2:clvmd params daemon_timeout=30 clone cln_clvmd clvmd and rember clvmd depend on dlm, so for the dlm you sould same I

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-13 Thread William Seligman
On 3/13/12 12:29 PM, William Seligman wrote: I'm not sure if this is a Linux-HA question; please direct me to the appropriate list if it's not. I'm setting up a two-node cman+pacemaker+gfs2 cluster as described in Clusters From Scratch. Fencing is through forcibly rebooting a node by

Re: [Linux-HA] clvm/dlm/gfs2 hangs if a node crashes

2012-03-13 Thread emmanuel segura
Hello Willian So if you using cman why you use lsb::clvmd I think you are very confused Il giorno 13 marzo 2012 22:42, William Seligman selig...@nevis.columbia.edu ha scritto: On 3/13/12 12:29 PM, William Seligman wrote: I'm not sure if this is a Linux-HA question; please direct me to