hrm.... with cciss try adding rootdelay=90 to your boot options. I'm not sure if it will delay udev for non-root, but I've heard of other people needing the delay for the / mount because the cciss devices might not be ready immediately
On Mon, Aug 11, 2014 at 7:54 AM, O'Reilly, Dan <[email protected]> wrote: > More information: > > > > When the system is booted, for whatever reason udev doesn’t seem to find > the devices used for OSD. However, once the system comes up, I can perform > a “udevadm trigger –action=add” command and all the devices appear. > Perhaps some sort of race condition? I am using a 95-ceph-osd.rules file > for udev. > > > > *From:* ceph-users [mailto:[email protected]] *On Behalf > Of *O'Reilly, Dan > *Sent:* Sunday, August 10, 2014 7:38 AM > *To:* 'Andrew Woodward' > *Cc:* ceph-users; [email protected] > *Subject:* Re: [ceph-users] [Ceph-community] working ceph.conf file? > > > > Thanks for the information on the ceph.conf contents. I wasn’t clear if > that we necessary or not. > > > > Here's the issue: > > > > I set up JBOD OSD disks with separate journals on a set of mirrored disks > to provide some protection of the journals in case of a journal disk > failure. That works fine - until you reboot the system. The journal entry > in the OSD metadata points to /dev/disk/by-partuuid, which apparently isn’t > preserved after a reboot. > > > > # ll /var/lib/ceph/osd/ceph-0 > > total 56 > > -rw-r--r-- 1 root root 502 Aug 7 09:05 activate.monmap > > -rw-r--r-- 1 root root 3 Aug 7 09:05 active > > -rw-r--r-- 1 root root 37 Aug 7 09:05 ceph_fsid > > drwxr-xr-x 81 root root 8192 Aug 8 10:27 current > > -rw-r--r-- 1 root root 37 Aug 7 09:05 fsid > > *lrwxrwxrwx 1 root root 58 Aug 7 09:05 journal -> > /dev/disk/by-partuuid/12f2c3db-524d-4cc0-a759-afea4a083237* > > -rw-r--r-- 1 root root 37 Aug 7 09:05 journal_uuid > > -rw------- 1 root root 56 Aug 7 09:05 keyring > > -rw-r--r-- 1 root root 21 Aug 7 09:05 magic > > -rw-r--r-- 1 root root 6 Aug 7 09:05 ready > > -rw-r--r-- 1 root root 4 Aug 7 09:05 store_version > > -rw-r--r-- 1 root root 42 Aug 7 09:05 superblock > > -rw-r--r-- 1 root root 0 Aug 7 09:06 sysvinit > > -rw-r--r-- 1 root root 2 Aug 7 09:05 whoami > > [NEW:note: root@tm1cldosdl01 on parent: /root] > > # ll /dev/disk/by-partuuid/12f2c3db-524d-4cc0-a759-afea4a083237 > > *lrwxrwxrwx 1 root root 18 Aug 7 09:06 > /dev/disk/by-partuuid/12f2c3db-524d-4cc0-a759-afea4a083237 -> > ../../cciss/c0d1p1* > > > > Because the /dev/disk/by-partuuid directory no longer exists after a > reboot, the mount of the OSD disk fails after a reboot because the journal > partition can’t be determined. The fundamental problem is, then, how do I > preserve the /dev/disk/by-partuuid directory between reboots? This is RHEL > 6.5, by the way. > > > > Thanks! > > > > *From:* Andrew Woodward [mailto:[email protected] <[email protected]>] > *Sent:* Friday, August 08, 2014 3:57 PM > *To:* O'Reilly, Dan > *Cc:* [email protected]; ceph-users > *Subject:* Re: [Ceph-community] working ceph.conf file? > > > > Dan, > > > > It is not necessary to specify the OSD data in ceph.conf anymore. Ceph has > two auto-start functions besides this method. > > > > udev rules: > > ceph uses a udev rule to scan and attempt to mount (and activate) > partitions with specific GUID set for the partition typecode > > sgdisk --typecode=<<partation number>>:<<GUID>> /dev/<<disk>> > > > > the exact GUID's to use can be found > https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules. These > are set automaticly using ceph-disk (or ceph-deploy) if it creates the > partition from an empty disk, in the case that it does not, you have to set > them by hand, all be it should probably do this, or at least tell you > should do this. > > > > ceph init script: > > the ceph init script will scan /var/lib/ceph/osd (or the otherwise > configured location) for <cluster name>-<id> (default cluster name is > ceph) folders and attempt to start the osd service for each of them if they > look correct > > > > lastly, and possibly the most annoying way is that you can configure each > OSD and path in ceph.conf, I don't have any good examples as the two prior > are more flexible / require less config. > > > > > > > > On Fri, Aug 8, 2014 at 8:53 AM, O'Reilly, Dan <[email protected]> > wrote: > > Does anybody have a good sample ceph.conf file I can use for reference? > I’m having a problem where OSD’s won’t come back up after a sysem reboot. > > > > Dan O'Reilly > > UNIX Systems Administration > > [image: cid:638154011@09122011-048B] > > 9601 S. Meridian Blvd. > > Englewood, CO 80112 > > 720-514-6293 > > > > > > > _______________________________________________ > Ceph-community mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-community-ceph.com > > > > > > -- > > Andrew > > Mirantis > > Ceph community > -- Andrew Mirantis Ceph community
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
