Re: [ceph-users] can not umount ceph osd partition
Hello, >>> I am using 0.94.5. When I try to umount partition and fsck it I have issue: >>> root@storage003:~# stop ceph-osd id=13 >>> ceph-osd stop/waiting >>> root@storage003:~# umount /var/lib/ceph/osd/ceph-13 >>> root@storage003:~# fsck -yf /dev/sdf >>> fsck from util-linux 2.20.1 >>> e2fsck 1.42.9 (4-Feb-2014) >>> /dev/sdf is in use. >>> e2fsck: Cannot continue, aborting. >>> >>> There is no /var/lib/ceph/osd/ceph-13 in /proc mounts. But no ability to >>> check >>> fs. >>> I can mount -o remount,rw, but I would like to umount device for maintenance >>> and, maybe, replace it. >>> >>> Why I can't umount? > >> is "lsof -n | grep /dev/sdf" give something ? > > Nothing. > >> and are you sure /dev/sdf is the disk for osd 13 ? > > Absolutelly. I have even tried fsck -yf /dev/disk/by-label/osd-13. No luck. > > Disk is mounted using LABEL in fstab, journal is symlink to > /dev/disk/by-partlabel/j-13. I think it's more linux related. could you try to look with lsof if something hold the device by the label or uuid instead of /dev/sdf ? you can try to delete the device from the scsi bus with something like : echo 1 > /sys/block//device/delete be careful, it is like removing the disk physically, if a process holds the device, you might expect that process gonna switch into kernel status "D+" . You won't be able to kill that process even by kill -9. To stop it, you will have to reboot the server. you can give a look here how to manipulate scsi bus: http://fibrevillage.com/storage/279-hot-add-remove-rescan-of-scsi-devices-on-linux you can install the package "scsitools" that provide rescan-scsi-bus.sh to rescan you scsi bus to get back your disk removed. http://manpages.ubuntu.com/manpages/precise/man8/rescan-scsi-bus.8.html hope that can help you -- Yoann Moulin EPFL IC-IT ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] can not umount ceph osd partition
Hello! On Thu, Feb 04, 2016 at 11:10:06AM +0100, yoann.moulin wrote: > Hello, I am using 0.94.5. When I try to umount partition and fsck it I have issue: root@storage003:~# stop ceph-osd id=13 ceph-osd stop/waiting root@storage003:~# umount /var/lib/ceph/osd/ceph-13 root@storage003:~# fsck -yf /dev/sdf fsck from util-linux 2.20.1 e2fsck 1.42.9 (4-Feb-2014) /dev/sdf is in use. e2fsck: Cannot continue, aborting. There is no /var/lib/ceph/osd/ceph-13 in /proc mounts. But no ability to check fs. I can mount -o remount,rw, but I would like to umount device for maintenance and, maybe, replace it. Why I can't umount? >> >>> is "lsof -n | grep /dev/sdf" give something ? >> >> Nothing. >> >>> and are you sure /dev/sdf is the disk for osd 13 ? >> >> Absolutelly. I have even tried fsck -yf /dev/disk/by-label/osd-13. No luck. >> >> Disk is mounted using LABEL in fstab, journal is symlink to >> /dev/disk/by-partlabel/j-13. > I think it's more linux related. Maybe. But I have it only on ceph boxes :( > could you try to look with lsof if something hold the device by the > label or uuid instead of /dev/sdf ? > you can try to delete the device from the scsi bus with something like : > echo 1 > /sys/block//device/delete > be careful, it is like removing the disk physically, if a process holds > the device, you might expect that process gonna switch into kernel > status "D+" . You won't be able to kill that process even by kill -9. To > stop it, you will have to reboot the server. > you can give a look here how to manipulate scsi bus: > http://fibrevillage.com/storage/279-hot-add-remove-rescan-of-scsi-devices-on-linux > you can install the package "scsitools" that provide rescan-scsi-bus.sh > to rescan you scsi bus to get back your disk removed. > http://manpages.ubuntu.com/manpages/precise/man8/rescan-scsi-bus.8.html > hope that can help you Thanx a lot! I will try to use partx -u (it sometimes helped me in past to re-read partitions from disk when gdisk was not able to update kernel's list of partitions) and software removing/inserting drive. If some processes fails into uninterruptible sleep, I will reboot node. It will be rebooted in any case if this will not help. If I investigate thomething it will be posted here. I think, it can affect other ceph users. -- WBR, Max A. Krasilnikov ColoCall Data Center ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] can not umount ceph osd partition
Hello! I am using 0.94.5. When I try to umount partition and fsck it I have issue: root@storage003:~# stop ceph-osd id=13 ceph-osd stop/waiting root@storage003:~# umount /var/lib/ceph/osd/ceph-13 root@storage003:~# fsck -yf /dev/sdf fsck from util-linux 2.20.1 e2fsck 1.42.9 (4-Feb-2014) /dev/sdf is in use. e2fsck: Cannot continue, aborting. There is no /var/lib/ceph/osd/ceph-13 in /proc mounts. But no ability to check fs. I can mount -o remount,rw, but I would like to umount device for maintenance and, maybe, replace it. Why I can't umount? -- WBR, Max A. Krasilnikov ColoCall Data Center ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] can not umount ceph osd partition
Hello, > I am using 0.94.5. When I try to umount partition and fsck it I have issue: > root@storage003:~# stop ceph-osd id=13 > ceph-osd stop/waiting > root@storage003:~# umount /var/lib/ceph/osd/ceph-13 > root@storage003:~# fsck -yf /dev/sdf > fsck from util-linux 2.20.1 > e2fsck 1.42.9 (4-Feb-2014) > /dev/sdf is in use. > e2fsck: Cannot continue, aborting. > > There is no /var/lib/ceph/osd/ceph-13 in /proc mounts. But no ability to check > fs. > I can mount -o remount,rw, but I would like to umount device for maintenance > and, maybe, replace it. > > Why I can't umount? is "lsof -n | grep /dev/sdf" give something ? and are you sure /dev/sdf is the disk for osd 13 ? -- Yoann Moulin EPFL IC-IT ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] can not umount ceph osd partition
Здравствуйте! On Wed, Feb 03, 2016 at 04:59:30PM +0100, yoann.moulin wrote: > Hello, >> I am using 0.94.5. When I try to umount partition and fsck it I have issue: >> root@storage003:~# stop ceph-osd id=13 >> ceph-osd stop/waiting >> root@storage003:~# umount /var/lib/ceph/osd/ceph-13 >> root@storage003:~# fsck -yf /dev/sdf >> fsck from util-linux 2.20.1 >> e2fsck 1.42.9 (4-Feb-2014) >> /dev/sdf is in use. >> e2fsck: Cannot continue, aborting. >> >> There is no /var/lib/ceph/osd/ceph-13 in /proc mounts. But no ability to >> check >> fs. >> I can mount -o remount,rw, but I would like to umount device for maintenance >> and, maybe, replace it. >> >> Why I can't umount? > is "lsof -n | grep /dev/sdf" give something ? Nothing. > and are you sure /dev/sdf is the disk for osd 13 ? Absolutelly. I have even tried fsck -yf /dev/disk/by-label/osd-13. No luck. Disk is mounted using LABEL in fstab, journal is symlink to /dev/disk/by-partlabel/j-13. -- WBR, Max A. Krasilnikov ColoCall Data Center ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
