"device busy" could be because you have a shell having that as the cwd.
Check:
ls -l /proc/[0-9]*/cwd
Georg Höllrigl wrote:
> Hello,
>
> I've several LUNs mounted in a 7 node cluster - one LUN which is only used on
> 4 of the nodes.
>
> It's impossible to umount this LUN - I'm always getting device is busy on my
> nodes.
>
> I'm using SuSE Linux Enterprise 10 SP 2 with Kernel 2.6.16.60-0.34-smp
>
> lsof doesn't show anything that could still be open on this volume.
>
>
> host1:~ # mount
> ....
> /dev/dm-10 on /srv/www/vhosts type ocfs2 (rw,_netdev,heartbeat=local)
>
> host1:~ # umount /dev/dm-10 -vvv -f
> Trying to umount /dev/dm-10
> umount2: Device or resource busy
> umount: /srv/www/vhosts: device is busy
> umount2: Device or resource busy
> umount: /srv/www/vhosts: device is busy
>
>
> Here are the Logfiles from the other node - which was blocking all requests
> until the node that
> umounted got evicted.
>
> Jul 30 09:31:12 host1-s-01 kernel: o2net: connection to node host1-s-03 (num
> 2) at 10.0.1.170:7777
> has been idle for 60.0 seconds, shutting it down.
> Jul 30 09:31:12 host1-s-01 kernel: (0,0):o2net_idle_timer:1476 here are some
> times that might help
> debug the situation: (tmr 1248939012.142271 now 1248939072.147815 dr
> 1248939012.142267 adv
> 1248939012.142271:1248939012.142272 func (1c9b2828
> :502) 1248939007.404799:1248939007.404802)
> Jul 30 09:31:12 host1-s-01 kernel: o2net: no longer connected to node
> host1-s-03 (num 2) at
> 10.0.1.170:7777
> Jul 30 09:31:12 host1-s-01 kernel: (16685,0):dlm_do_master_request:1360
> ERROR: link to 2 went down!
> Jul 30 09:31:12 host1-s-01 kernel: (16685,0):dlm_get_lock_resource:937 ERROR:
> status = -112
> Jul 30 09:31:12 host1-s-01 kernel: (16680,0):dlm_do_master_request:1360
> ERROR: link to 2 went down!
> Jul 30 09:31:12 host1-s-01 kernel: (16680,0):dlm_get_lock_resource:937 ERROR:
> status = -112
> Jul 30 09:31:12 host1-s-01 kernel: (16637,0):dlm_do_master_request:1360
> ERROR: link to 2 went down!
> Jul 30 09:31:12 host1-s-01 kernel: (16637,0):dlm_get_lock_resource:937 ERROR:
> status = -112
> Jul 30 09:31:13 host1-s-01 kernel: (16755,0):dlm_do_master_request:1360
> ERROR: link to 2 went down!
> Jul 30 09:31:13 host1-s-01 kernel: (16755,0):dlm_get_lock_resource:937 ERROR:
> status = -107
> Jul 30 09:32:12 host1-s-01 kernel: (5723,0):o2net_connect_expired:1637 ERROR:
> no connection
> established with node 2 after 60.0 seconds, giving up and returning errors.
> Jul 30 09:32:14 host1-s-01 kernel: (5764,0):ocfs2_dlm_eviction_cb:108 device
> (253,10): dlm has
> evicted node 2
> Jul 30 09:32:17 host1-s-01 kernel: (5723,0):ocfs2_dlm_eviction_cb:108 device
> (253,10): dlm has
> evicted node 2
> Jul 30 09:32:17 host1-s-01 kernel: (16685,0):dlm_restart_lock_mastery:1243
> ERROR: node down! 2
> Jul 30 09:32:17 host1-s-01 kernel: (16685,0):dlm_wait_for_lock_mastery:1060
> ERROR: status = -11
> Jul 30 09:32:17 host1-s-01 kernel: (16680,0):dlm_restart_lock_mastery:1243
> ERROR: node down! 2
> Jul 30 09:32:17 host1-s-01 kernel: (16680,0):dlm_wait_for_lock_mastery:1060
> ERROR: status = -11
> Jul 30 09:32:17 host1-s-01 kernel: (16637,0):dlm_restart_lock_mastery:1243
> ERROR: node down! 2
> Jul 30 09:32:17 host1-s-01 kernel: (16637,0):dlm_wait_for_lock_mastery:1060
> ERROR: status = -11
> Jul 30 09:32:18 host1-s-01 kernel: (16685,0):dlm_get_lock_resource:918
> D7581877783A4174A498C97DDC573E52:M0000000000000008bfed0600000000: at least
> one node (2) to recover
> before lock mastery can begin
> Jul 30 09:32:18 host1-s-01 kernel: (16680,0):dlm_get_lock_resource:918
> D7581877783A4174A498C97DDC573E52:N0000000003560f35: at least one node (2) to
> recover before lock
> mastery can begin
> Jul 30 09:32:18 host1-s-01 kernel: (16637,0):dlm_get_lock_resource:918
> D7581877783A4174A498C97DDC573E52:N0000000008172be2: at least one node (2) to
> recover before lock
> mastery can begin
> Jul 30 09:32:18 host1-s-01 kernel: (16755,0):dlm_restart_lock_mastery:1243
> ERROR: node down! 2
> Jul 30 09:32:18 host1-s-01 kernel: (16755,0):dlm_wait_for_lock_mastery:1060
> ERROR: status = -11
> Jul 30 09:32:19 host1-s-01 kernel: (16755,0):dlm_get_lock_resource:918
> D7581877783A4174A498C97DDC573E52:O000000000000000a40040b00000000: at least
> one node (2) to recover
> before lock mastery can begin
> Jul 30 09:33:01 host1-s-01 kernel: o2net: accepted connection from node
> host1-s-03 (num 2) at
> 10.0.1.170:7777
> Jul 30 09:33:05 host1-s-01 kernel: ocfs2_dlm: Node 2 joins domain
> D7581877783A4174A498C97DDC573E52
> Jul 30 09:33:05 host1-s-01 kernel: ocfs2_dlm: Nodes in domain
> ("D7581877783A4174A498C97DDC573E52"):
> 0 1 2 3
> Jul 30 09:33:09 host1-s-01 kernel: ocfs2_dlm: Node 2 joins domain
> F7929E2FBDB0487DA142467EB725FC22
> Jul 30 09:33:09 host1-s-01 kernel: ocfs2_dlm: Nodes in domain
> ("F7929E2FBDB0487DA142467EB725FC22"):
> 0 1 2 3
> Jul 30 09:33:13 host1-s-01 kernel: ocfs2_dlm: Node 2 joins domain
> 6831EB702AC04901A6D5BBE7EBE691AE
> Jul 30 09:33:13 host1-s-01 kernel: ocfs2_dlm: Nodes in domain
> ("6831EB702AC04901A6D5BBE7EBE691AE"):
> 0 1 2 3
> Jul 30 09:33:19 host1-s-01 kernel: ocfs2_dlm: Node 2 joins domain
> 0E7D6EE19D644648919028729AE662A1
> Jul 30 09:33:19 host1-s-01 kernel: ocfs2_dlm: Nodes in domain
> ("0E7D6EE19D644648919028729AE662A1"):
> 0 1 2 3 4 5 6
> Jul 30 09:33:23 host1-s-01 kernel: ocfs2_dlm: Node 2 joins domain
> 63865CE5EDE74713A6B3CECE2A3923C0
> Jul 30 09:33:23 host1-s-01 kernel: ocfs2_dlm: Nodes in domain
> ("63865CE5EDE74713A6B3CECE2A3923C0"):
> 0 1 2 3 4 5 6
> Jul 30 09:33:27 host1-s-01 kernel: ocfs2_dlm: Node 2 joins domain
> D5754A078396403FB841A798BE945A26
> Jul 30 09:33:27 host1-s-01 kernel: ocfs2_dlm: Nodes in domain
> ("D5754A078396403FB841A798BE945A26"):
> 0 1 2 3 4 5 6
> Jul 30 09:33:31 host1-s-01 kernel: ocfs2_dlm: Node 2 joins domain
> 976B1863D9114CFAA314354BB1235577
> Jul 30 09:33:31 host1-s-01 kernel: ocfs2_dlm: Nodes in domain
> ("976B1863D9114CFAA314354BB1235577"):
> 0 1 2 3 4 5 6
> Jul 30 09:33:35 host1-s-01 kernel: ocfs2_dlm: Node 2 joins domain
> 933698C044EF46D9A175057C523C2D1E
> Jul 30 09:33:35 host1-s-01 kernel: ocfs2_dlm: Nodes in domain
> ("933698C044EF46D9A175057C523C2D1E"):
> 0 1 2 3 4 5 6
>
>
> The strange thing is, that this only happens with one of the volumes -
> perhaps you could point me
> into the right direction how to fix this volume?
>
>
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users
