More information: When the system is booted, for whatever reason udev doesn’t seem to find the devices used for OSD. However, once the system comes up, I can perform a “udevadm trigger –action=add” command and all the devices appear. Perhaps some sort of race condition? I am using a 95-ceph-osd.rules file for udev.
From: ceph-users [mailto:[email protected]] On Behalf Of O'Reilly, Dan Sent: Sunday, August 10, 2014 7:38 AM To: 'Andrew Woodward' Cc: ceph-users; [email protected] Subject: Re: [ceph-users] [Ceph-community] working ceph.conf file? Thanks for the information on the ceph.conf contents. I wasn’t clear if that we necessary or not. Here's the issue: I set up JBOD OSD disks with separate journals on a set of mirrored disks to provide some protection of the journals in case of a journal disk failure. That works fine - until you reboot the system. The journal entry in the OSD metadata points to /dev/disk/by-partuuid, which apparently isn’t preserved after a reboot. # ll /var/lib/ceph/osd/ceph-0 total 56 -rw-r--r-- 1 root root 502 Aug 7 09:05 activate.monmap -rw-r--r-- 1 root root 3 Aug 7 09:05 active -rw-r--r-- 1 root root 37 Aug 7 09:05 ceph_fsid drwxr-xr-x 81 root root 8192 Aug 8 10:27 current -rw-r--r-- 1 root root 37 Aug 7 09:05 fsid lrwxrwxrwx 1 root root 58 Aug 7 09:05 journal -> /dev/disk/by-partuuid/12f2c3db-524d-4cc0-a759-afea4a083237 -rw-r--r-- 1 root root 37 Aug 7 09:05 journal_uuid -rw------- 1 root root 56 Aug 7 09:05 keyring -rw-r--r-- 1 root root 21 Aug 7 09:05 magic -rw-r--r-- 1 root root 6 Aug 7 09:05 ready -rw-r--r-- 1 root root 4 Aug 7 09:05 store_version -rw-r--r-- 1 root root 42 Aug 7 09:05 superblock -rw-r--r-- 1 root root 0 Aug 7 09:06 sysvinit -rw-r--r-- 1 root root 2 Aug 7 09:05 whoami [NEW:note: root@tm1cldosdl01 on parent: /root] # ll /dev/disk/by-partuuid/12f2c3db-524d-4cc0-a759-afea4a083237 lrwxrwxrwx 1 root root 18 Aug 7 09:06 /dev/disk/by-partuuid/12f2c3db-524d-4cc0-a759-afea4a083237 -> ../../cciss/c0d1p1 Because the /dev/disk/by-partuuid directory no longer exists after a reboot, the mount of the OSD disk fails after a reboot because the journal partition can’t be determined. The fundamental problem is, then, how do I preserve the /dev/disk/by-partuuid directory between reboots? This is RHEL 6.5, by the way. Thanks! From: Andrew Woodward [mailto:[email protected]] Sent: Friday, August 08, 2014 3:57 PM To: O'Reilly, Dan Cc: [email protected]<mailto:[email protected]>; ceph-users Subject: Re: [Ceph-community] working ceph.conf file? Dan, It is not necessary to specify the OSD data in ceph.conf anymore. Ceph has two auto-start functions besides this method. udev rules: ceph uses a udev rule to scan and attempt to mount (and activate) partitions with specific GUID set for the partition typecode sgdisk --typecode=<<partation number>>:<<GUID>> /dev/<<disk>> the exact GUID's to use can be found https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules. These are set automaticly using ceph-disk (or ceph-deploy) if it creates the partition from an empty disk, in the case that it does not, you have to set them by hand, all be it should probably do this, or at least tell you should do this. ceph init script: the ceph init script will scan /var/lib/ceph/osd (or the otherwise configured location) for <cluster name>-<id> (default cluster name is ceph) folders and attempt to start the osd service for each of them if they look correct lastly, and possibly the most annoying way is that you can configure each OSD and path in ceph.conf, I don't have any good examples as the two prior are more flexible / require less config. On Fri, Aug 8, 2014 at 8:53 AM, O'Reilly, Dan <[email protected]<mailto:[email protected]>> wrote: Does anybody have a good sample ceph.conf file I can use for reference? I’m having a problem where OSD’s won’t come back up after a sysem reboot. Dan O'Reilly UNIX Systems Administration [cid:[email protected]] 9601 S. Meridian Blvd. Englewood, CO 80112 720-514-6293<tel:720-514-6293> _______________________________________________ Ceph-community mailing list [email protected]<mailto:[email protected]> http://lists.ceph.com/listinfo.cgi/ceph-community-ceph.com -- Andrew Mirantis Ceph community
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
