Hello Sunil, Yes, we monitored /proc/mounts and /etc/mtab and both look OK. I have attached the output of both of these at various stages in the test outlined below. Also, the -n option is not used on the mount.
Regards Mark Schloss Mark Schloss | Oracle DBA | Information Technology | x0013 -----Original Message----- From: Sunil Mushran [mailto:[EMAIL PROTECTED] Sent: Wednesday, 23 July 2008 12:13 PM To: Mark Schloss Cc: [email protected] Subject: Re: [Ocfs2-users] OCFS processes active after a umount [SEC=UNOFFICIAL] Did you monitor /proc/mounts as I had suggested. -----Original Message----- >From Mark Schloss <[EMAIL PROTECTED]> Sent Mon 7/21/2008 9:22 PM To Sunil Mushran <[EMAIL PROTECTED]> Cc [email protected] Subject Re: [Ocfs2-users] OCFS processes active after a umount [SEC=UNOFFICIAL] Hello Sunil, Thanks for the reply. The version is 1.2.9-1 and kernel is 2.6.18-92.1.1.el5. We have managed to reproduce the problem and it appears to be related to multipathing. We recently moved to multipathing the OCFS volumes. On the weekend we tested removing one path to the OCFS volumes and all was OK. When the path was restored the box still showed it as being unavailable. We found this to be caused by the multipath daemon not running as it had not been set to start automatically. After the daemon was started both paths to the OCFS volumes were shown as available, however, when trying to umount and offline OCFS we see the behaviour outlined originally. This is reproducible as follows - 1. Start with both paths to the OCFS volumes available 2. Shutdown the multipath daemon 3. Remove one path by disconnecting a cable to the switch 4. Restore the path by reconnecting cable 5. Start multipath daemon 6. Check both paths are available 7. Umount the OCFS file system (this returns immediately without the usual few second delay) 8. Offline OCFS - error is received Under normal circumstances, i.e.: when the multipath daemon is continuously available, losing a path, restoring a path, umount and offline all work as expected. Regards Mark Mark Schloss | Oracle DBA | Information Technology | x0013 -----Original Message----- From: Sunil Mushran [mailto:[EMAIL PROTECTED] Sent: Tuesday, 22 July 2008 3:43 AM To: Mark Schloss Cc: [email protected] Subject: Re: [Ocfs2-users] OCFS processes active after a umount [SEC=UNOFFICIAL] That is strange. Next time double check the mounts with: $ cat /proc/mounts The mount command prints the entries in /etc/mtab while the /proc/mounts dumps the information from the kernel. If those threads are there, it means the volume is still mounted. Two in this case. The entries in mtab are added by mount.ocfs2 and removed by umount. There is a chance that mount.ocfs2 was unable to add the entries in that file. Or, maybe one used the -n option to force that behavior. Which version/kernel is this? Sunil Mark Schloss wrote: > Hello, > > I have two OCFS file file systems mounted at /ocfs_1 and /ocfs_2. I > have unmounted both OCFS file systems and was trying to then offline > and unload OCFS. The offline command failed with - > > # ./o2cb offline > Stopping O2CB cluster ocfs2: Failed > Unable to stop cluster as heartbeat region still active Looking at the > processes on this box shows a number of OCFS processes are still > active - > > ps -ef |grep ocf > root 4704 23 0 Jul19 ? 00:00:00 [ocfs2_wq] > root 4705 23 0 Jul19 ? 00:00:00 [ocfs2vote-0] > root 4710 23 0 Jul19 ? 00:00:00 [ocfs2cmt-0] > root 4730 23 0 Jul19 ? 00:00:00 [ocfs2vote-1] > root 4735 23 0 Jul19 ? 00:00:00 [ocfs2cmt-1] > root 10214 3485 0 18:12 pts/2 00:00:00 grep ocf > > According to the FAQ, the ocfs2vote and ocfs2cmt processes should have > gone at the umount. > > Mount shows that there no OCFS file systems mounted - > > # mount |grep -i ocfs > ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw) > > What can I do to offline/unload OCFS? (offline force fails with the > same message as above) > > Regards > > Mark Schloss ********************************************************************** Please note that your email address is known to AUSTRAC for the purposes of communicating with you. The information transmitted in this e-mail is for the use of the intended recipient only and may contain confidential and/or legally privileged material. If you have received this information in error you must not disseminate, copy or take any action on it and we request that you delete all copies of this transmission together with attachments and notify the sender. This footnote also confirms that this email message has been swept for the presence of computer viruses. ********************************************************************** _______________________________________________ Ocfs2-users mailing list [email protected] http://oss.oracle.com/mailman/listinfo/ocfs2-users ********************************************************************** Please note that your email address is known to AUSTRAC for the purposes of communicating with you. The information transmitted in this e-mail is for the use of the intended recipient only and may contain confidential and/or legally privileged material. If you have received this information in error you must not disseminate, copy or take any action on it and we request that you delete all copies of this transmission together with attachments and notify the sender. This footnote also confirms that this email message has been swept for the presence of computer viruses. **********************************************************************
Before Test # cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw,data=ordered 0 0 /dev /dev tmpfs rw 0 0 /proc /proc proc rw 0 0 /sys /sys sysfs rw 0 0 /proc/bus/usb /proc/bus/usb usbfs rw 0 0 devpts /dev/pts devpts rw 0 0 /dev/md7 /app1 ext3 rw,nodev,data=ordered 0 0 /dev/md0 /boot ext3 rw,nosuid,nodev,data=ordered 0 0 tmpfs /dev/shm tmpfs rw 0 0 /dev/sda9 /diskdump ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/sdb9 /diskdump2 ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md4 /home ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md5 /opt ext3 rw,data=ordered 0 0 /dev/md9 /tmp ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md8 /u01 ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md3 /usr ext3 rw,data=ordered 0 0 /dev/md6 /var ext3 rw,nosuid,nodev,data=ordered 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0 oracleasmfs /dev/oracleasm oracleasmfs rw 0 0 configfs /sys/kernel/config configfs rw 0 0 ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0 /dev/mapper/ocfsrac1p1 /ocfs_1 ocfs2 rw 0 0 /dev/mapper/ocfsrac2p1 /ocfs_2 ocfs2 rw 0 0 # cat /etc/mtab /dev/md1 / ext3 rw 0 0 proc /proc proc rw 0 0 sysfs /sys sysfs rw 0 0 devpts /dev/pts devpts rw,gid=5,mode=620 0 0 /dev/md7 /app1 ext3 rw,nodev 0 0 /dev/md0 /boot ext3 rw,nosuid,nodev 0 0 tmpfs /dev/shm tmpfs rw 0 0 /dev/sda9 /diskdump ext3 rw,nosuid,nodev 0 0 /dev/sdb9 /diskdump2 ext3 rw,nosuid,nodev 0 0 /dev/md4 /home ext3 rw,nosuid,nodev 0 0 /dev/md5 /opt ext3 rw 0 0 /dev/md9 /tmp ext3 rw,nosuid,nodev 0 0 /dev/md8 /u01 ext3 rw,nosuid,nodev 0 0 /dev/md3 /usr ext3 rw 0 0 /dev/md6 /var ext3 rw,nosuid,nodev 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0 oracleasmfs /dev/oracleasm oracleasmfs rw 0 0 configfs /sys/kernel/config configfs rw 0 0 ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0 /dev/mapper/ocfsrac1p1 /ocfs_1 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0 /dev/mapper/ocfsrac2p1 /ocfs_2 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0 After cable removed # cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw,data=ordered 0 0 /dev /dev tmpfs rw 0 0 /proc /proc proc rw 0 0 /sys /sys sysfs rw 0 0 /proc/bus/usb /proc/bus/usb usbfs rw 0 0 devpts /dev/pts devpts rw 0 0 /dev/md7 /app1 ext3 rw,nodev,data=ordered 0 0 /dev/md0 /boot ext3 rw,nosuid,nodev,data=ordered 0 0 tmpfs /dev/shm tmpfs rw 0 0 /dev/sda9 /diskdump ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/sdb9 /diskdump2 ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md4 /home ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md5 /opt ext3 rw,data=ordered 0 0 /dev/md9 /tmp ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md8 /u01 ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md3 /usr ext3 rw,data=ordered 0 0 /dev/md6 /var ext3 rw,nosuid,nodev,data=ordered 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0 oracleasmfs /dev/oracleasm oracleasmfs rw 0 0 configfs /sys/kernel/config configfs rw 0 0 ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0 /dev/mapper/ocfsrac1p1 /ocfs_1 ocfs2 rw 0 0 /dev/mapper/ocfsrac2p1 /ocfs_2 ocfs2 rw 0 0 # cat /etc/mtab /dev/md1 / ext3 rw 0 0 proc /proc proc rw 0 0 sysfs /sys sysfs rw 0 0 devpts /dev/pts devpts rw,gid=5,mode=620 0 0 /dev/md7 /app1 ext3 rw,nodev 0 0 /dev/md0 /boot ext3 rw,nosuid,nodev 0 0 tmpfs /dev/shm tmpfs rw 0 0 /dev/sda9 /diskdump ext3 rw,nosuid,nodev 0 0 /dev/sdb9 /diskdump2 ext3 rw,nosuid,nodev 0 0 /dev/md4 /home ext3 rw,nosuid,nodev 0 0 /dev/md5 /opt ext3 rw 0 0 /dev/md9 /tmp ext3 rw,nosuid,nodev 0 0 /dev/md8 /u01 ext3 rw,nosuid,nodev 0 0 /dev/md3 /usr ext3 rw 0 0 /dev/md6 /var ext3 rw,nosuid,nodev 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0 oracleasmfs /dev/oracleasm oracleasmfs rw 0 0 configfs /sys/kernel/config configfs rw 0 0 ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0 /dev/mapper/ocfsrac1p1 /ocfs_1 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0 /dev/mapper/ocfsrac2p1 /ocfs_2 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0 After cable replaced and daemon start # cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw,data=ordered 0 0 /dev /dev tmpfs rw 0 0 /proc /proc proc rw 0 0 /sys /sys sysfs rw 0 0 /proc/bus/usb /proc/bus/usb usbfs rw 0 0 devpts /dev/pts devpts rw 0 0 /dev/md7 /app1 ext3 rw,nodev,data=ordered 0 0 /dev/md0 /boot ext3 rw,nosuid,nodev,data=ordered 0 0 tmpfs /dev/shm tmpfs rw 0 0 /dev/sda9 /diskdump ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/sdb9 /diskdump2 ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md4 /home ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md5 /opt ext3 rw,data=ordered 0 0 /dev/md9 /tmp ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md8 /u01 ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md3 /usr ext3 rw,data=ordered 0 0 /dev/md6 /var ext3 rw,nosuid,nodev,data=ordered 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0 oracleasmfs /dev/oracleasm oracleasmfs rw 0 0 configfs /sys/kernel/config configfs rw 0 0 ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0 /dev/mapper/ocfsrac1p1 /ocfs_1 ocfs2 rw 0 0 /dev/mapper/ocfsrac2p1 /ocfs_2 ocfs2 rw 0 0 # cat /etc/mtab /dev/md1 / ext3 rw 0 0 proc /proc proc rw 0 0 sysfs /sys sysfs rw 0 0 devpts /dev/pts devpts rw,gid=5,mode=620 0 0 /dev/md7 /app1 ext3 rw,nodev 0 0 /dev/md0 /boot ext3 rw,nosuid,nodev 0 0 tmpfs /dev/shm tmpfs rw 0 0 /dev/sda9 /diskdump ext3 rw,nosuid,nodev 0 0 /dev/sdb9 /diskdump2 ext3 rw,nosuid,nodev 0 0 /dev/md4 /home ext3 rw,nosuid,nodev 0 0 /dev/md5 /opt ext3 rw 0 0 /dev/md9 /tmp ext3 rw,nosuid,nodev 0 0 /dev/md8 /u01 ext3 rw,nosuid,nodev 0 0 /dev/md3 /usr ext3 rw 0 0 /dev/md6 /var ext3 rw,nosuid,nodev 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0 oracleasmfs /dev/oracleasm oracleasmfs rw 0 0 configfs /sys/kernel/config configfs rw 0 0 ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0 /dev/mapper/ocfsrac1p1 /ocfs_1 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0 /dev/mapper/ocfsrac2p1 /ocfs_2 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0 After umount #umount /ocfs_1 # cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw,data=ordered 0 0 /dev /dev tmpfs rw 0 0 /proc /proc proc rw 0 0 /sys /sys sysfs rw 0 0 /proc/bus/usb /proc/bus/usb usbfs rw 0 0 devpts /dev/pts devpts rw 0 0 /dev/md7 /app1 ext3 rw,nodev,data=ordered 0 0 /dev/md0 /boot ext3 rw,nosuid,nodev,data=ordered 0 0 tmpfs /dev/shm tmpfs rw 0 0 /dev/sda9 /diskdump ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/sdb9 /diskdump2 ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md4 /home ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md5 /opt ext3 rw,data=ordered 0 0 /dev/md9 /tmp ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md8 /u01 ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md3 /usr ext3 rw,data=ordered 0 0 /dev/md6 /var ext3 rw,nosuid,nodev,data=ordered 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0 oracleasmfs /dev/oracleasm oracleasmfs rw 0 0 configfs /sys/kernel/config configfs rw 0 0 ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0 /dev/mapper/ocfsrac2p1 /ocfs_2 ocfs2 rw 0 0 # cat /etc/mtab /dev/md1 / ext3 rw 0 0 proc /proc proc rw 0 0 sysfs /sys sysfs rw 0 0 devpts /dev/pts devpts rw,gid=5,mode=620 0 0 /dev/md7 /app1 ext3 rw,nodev 0 0 /dev/md0 /boot ext3 rw,nosuid,nodev 0 0 tmpfs /dev/shm tmpfs rw 0 0 /dev/sda9 /diskdump ext3 rw,nosuid,nodev 0 0 /dev/sdb9 /diskdump2 ext3 rw,nosuid,nodev 0 0 /dev/md4 /home ext3 rw,nosuid,nodev 0 0 /dev/md5 /opt ext3 rw 0 0 /dev/md9 /tmp ext3 rw,nosuid,nodev 0 0 /dev/md8 /u01 ext3 rw,nosuid,nodev 0 0 /dev/md3 /usr ext3 rw 0 0 /dev/md6 /var ext3 rw,nosuid,nodev 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0 oracleasmfs /dev/oracleasm oracleasmfs rw 0 0 configfs /sys/kernel/config configfs rw 0 0 ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0 /dev/mapper/ocfsrac2p1 /ocfs_2 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0 # umount /ocfs_2 # cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw,data=ordered 0 0 /dev /dev tmpfs rw 0 0 /proc /proc proc rw 0 0 /sys /sys sysfs rw 0 0 /proc/bus/usb /proc/bus/usb usbfs rw 0 0 devpts /dev/pts devpts rw 0 0 /dev/md7 /app1 ext3 rw,nodev,data=ordered 0 0 /dev/md0 /boot ext3 rw,nosuid,nodev,data=ordered 0 0 tmpfs /dev/shm tmpfs rw 0 0 /dev/sda9 /diskdump ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/sdb9 /diskdump2 ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md4 /home ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md5 /opt ext3 rw,data=ordered 0 0 /dev/md9 /tmp ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md8 /u01 ext3 rw,nosuid,nodev,data=ordered 0 0 /dev/md3 /usr ext3 rw,data=ordered 0 0 /dev/md6 /var ext3 rw,nosuid,nodev,data=ordered 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0 oracleasmfs /dev/oracleasm oracleasmfs rw 0 0 configfs /sys/kernel/config configfs rw 0 0 ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0 # cat /etc/mtab /dev/md1 / ext3 rw 0 0 proc /proc proc rw 0 0 sysfs /sys sysfs rw 0 0 devpts /dev/pts devpts rw,gid=5,mode=620 0 0 /dev/md7 /app1 ext3 rw,nodev 0 0 /dev/md0 /boot ext3 rw,nosuid,nodev 0 0 tmpfs /dev/shm tmpfs rw 0 0 /dev/sda9 /diskdump ext3 rw,nosuid,nodev 0 0 /dev/sdb9 /diskdump2 ext3 rw,nosuid,nodev 0 0 /dev/md4 /home ext3 rw,nosuid,nodev 0 0 /dev/md5 /opt ext3 rw 0 0 /dev/md9 /tmp ext3 rw,nosuid,nodev 0 0 /dev/md8 /u01 ext3 rw,nosuid,nodev 0 0 /dev/md3 /usr ext3 rw 0 0 /dev/md6 /var ext3 rw,nosuid,nodev 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0 oracleasmfs /dev/oracleasm oracleasmfs rw 0 0 configfs /sys/kernel/config configfs rw 0 0 ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0 Processes remaining after umount # ps -ef |grep ocfs root 15681 23 0 Jul22 ? 00:00:00 [ocfs2_wq] root 15683 23 0 Jul22 ? 00:00:00 [ocfs2vote-0] root 15688 23 0 Jul22 ? 00:00:00 [ocfs2cmt-0] root 16092 23 0 Jul22 ? 00:00:00 [ocfs2vote-1] root 16097 23 0 Jul22 ? 00:00:00 [ocfs2cmt-1] root 24479 16721 0 14:17 pts/0 00:00:00 grep ocfs FSTAB entries # cat /etc/fstab /dev/mapper/ocfsrac1p1 /ocfs_1 ocfs2 _netdev,datavolume,nointr 0 0 /dev/mapper/ocfsrac2p1 /ocfs_2 ocfs2 _netdev,datavolume,nointr 0 0
_______________________________________________ Ocfs2-users mailing list [email protected] http://oss.oracle.com/mailman/listinfo/ocfs2-users
