Re: [ceph-users] Journal symlink broken / Ceph 0.94.5 / CentOS 6.7
Hi Loic, Problem solved! So i just started to reinstall with the last known working conf - centos 6.6 (remember i updated to 6.7). During the install it complains about some "bios raid metadata" on one of the disks (/dev/sdc) and wants to hide it. Last time i just added some boot parameter to ignore this, so it would let me install the OS on this disk. So i thought...hmm, maybe this is the root cause to the problems? Abort re-install, back into the 6.7 install, and apply this fix to my /dev/sdc; https://kezhong.wordpress.com/2011/06/14/how-to-remove-bios-raid-metadata-from-disk-on-fedora/ Deleted both journals and osd's and re-created - and wupti - things are working :-) !!! So i guess having an old disk with "bios raid metadata" on it will disturb ceph. May ceph should include a check for this "bios raid metadata"? Someone besides me might decide to use old raid disks for their ceph setup ;-) Thank you very much for your kind help! Cheers, Jesper ***** On 18/12/2015 22:09, Jesper Thorhauge wrote: > Hi Loic, > > Getting closer! > > lrwxrwxrwx 1 root root 10 Dec 18 19:43 1e9d527f-0866-4284-b77c-c1cb04c5a168 > -> ../../sdc4 > lrwxrwxrwx 1 root root 10 Dec 18 19:43 c34d4694-b486-450d-b57f-da24255f0072 > -> ../../sdc3 > lrwxrwxrwx 1 root root 10 Dec 18 19:42 c83b5aa5-fe77-42f6-9415-25ca0266fb7f > -> ../../sdb1 > lrwxrwxrwx 1 root root 10 Dec 18 19:42 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 > -> ../../sda1 > > So symlinks are now working! Activating an OSD is a different story :-( > > "ceph-disk -vv activate /dev/sda1" gives me; > > INFO:ceph-disk:Running command: /sbin/blkid -p -s TYPE -ovalue -- /dev/sda1 > INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. > --lookup osd_mount_options_xfs > INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. > --lookup osd_fs_mount_options_xfs > DEBUG:ceph-disk:Mounting /dev/sda1 on /var/lib/ceph/tmp/mnt.A99cDp with > options noatime,inode64 > INFO:ceph-disk:Running command: /bin/mount -t xfs -o noatime,inode64 -- > /dev/sda1 /var/lib/ceph/tmp/mnt.A99cDp > DEBUG:ceph-disk:Cluster uuid is 07b5c90b-6cae-40c0-93b2-31e0ebad7315 > INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph > --show-config-value=fsid > DEBUG:ceph-disk:Cluster name is ceph > DEBUG:ceph-disk:OSD uuid is e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 > DEBUG:ceph-disk:OSD id is 6 > DEBUG:ceph-disk:Initializing OSD... > INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name > client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon > getmap -o /var/lib/ceph/tmp/mnt.A99cDp/activate.monmap > got monmap epoch 6 > INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster ceph --mkfs > --mkkey -i 6 --monmap /var/lib/ceph/tmp/mnt.A99cDp/activate.monmap --osd-data > /var/lib/ceph/tmp/mnt.A99cDp --osd-journal > /var/lib/ceph/tmp/mnt.A99cDp/journal --osd-uuid > e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 --keyring > /var/lib/ceph/tmp/mnt.A99cDp/keyring > HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device > 2015-12-18 21:58:12.489357 7f266d7b0800 -1 journal check: ondisk fsid > ---- doesn't match expected > e85f4d92-c8f1-4591-bd2a-aa43b80f58f6, invalid (someone else's?) journal > HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device > HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device > HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device > 2015-12-18 21:58:12.680566 7f266d7b0800 -1 > filestore(/var/lib/ceph/tmp/mnt.A99cDp) could not find > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory > 2015-12-18 21:58:12.865810 7f266d7b0800 -1 created object store > /var/lib/ceph/tmp/mnt.A99cDp journal /var/lib/ceph/tmp/mnt.A99cDp/journal for > osd.6 fsid 07b5c90b-6cae-40c0-93b2-31e0ebad7315 > 2015-12-18 21:58:12.865844 7f266d7b0800 -1 auth: error reading file: > /var/lib/ceph/tmp/mnt.A99cDp/keyring: can't open > /var/lib/ceph/tmp/mnt.A99cDp/keyring: (2) No such file or directory > 2015-12-18 21:58:12.865910 7f266d7b0800 -1 created new key in keyring > /var/lib/ceph/tmp/mnt.A99cDp/keyring > INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. > --lookup init > DEBUG:ceph-disk:Marking with init system sysvinit > DEBUG:ceph-disk:Authorizing OSD key... > INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name > client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring auth > add osd.6 -i /var/lib/ceph/tmp/mnt.A99cDp/keyring osd allow * mon allow > profile osd > Error EINVAL: entity osd.6 exists but key does not match > ERROR:ceph-disk:Fail
Re: [ceph-users] Journal symlink broken / Ceph 0.94.5 / CentOS 6.7
Hi Loic, searched around for possible udev bugs, and then tried to run "yum update". Udev did have a fresh update with the following version diffs; udev-147-2.63.el6_7.1.x86_64 --> udev-147-2.63.el6_7.1.x86_64 from what i can see this update fixes stuff related to symbolic links / external devices. /dev/sdc sits on external eSata. So... https://rhn.redhat.com/errata/RHBA-2015-1382.html will reboot tonight and get back :-) /jesper ***' I guess that's the problem you need to solve : why /dev/sdc does not generate udev events (different driver than /dev/sda maybe ?). Once it does, Ceph should work. A workaround could be to add somethink like: ceph-disk-udev 3 sdc3 sdc ceph-disk-udev 4 sdc4 sdc in /etc/rc.local. On 17/12/2015 12:01, Jesper Thorhauge wrote: > Nope, the previous post contained all that was in the boot.log :-( > > /Jesper > > ** > > - Den 17. dec 2015, kl. 11:53, Loic Dachary <l...@dachary.org> skrev: > > On 17/12/2015 11:33, Jesper Thorhauge wrote: >> Hi Loic, >> >> Sounds like something does go wrong when /dev/sdc3 shows up. Is there anyway >> i can debug this further? Log-files? Modify the .rules file...? > > Do you see traces of what happens when /dev/sdc3 shows up in boot.log ? > >> >> /Jesper >> >> >> >> The non-symlink files in /dev/disk/by-partuuid come to existence because of: >> >> * system boots >> * udev rule calls ceph-disk-udev via 95-ceph-osd.rules on /dev/sda1 >> * ceph-disk-udev creates the symlink >> /dev/disk/by-partuuid/c83b5aa5-fe77-42f6-9415-25ca0266fb7f -> ../../sdb1 >> * ceph-disk activate /dev/sda1 is mounted and finds a symlink to the journal >> journal -> /dev/disk/by-partuuid/1e9d527f-0866-4284-b77c-c1cb04c5a168 which >> does not yet exists because /dev/sdc udev rules have not been run yet >> * ceph-osd opens the journal in write mode and that creates the file >> /dev/disk/by-partuuid/1e9d527f-0866-4284-b77c-c1cb04c5a168 as a regular file >> * the file is empty and the osd fails to activate with the error you see >> (EINVAL because the file is empty) >> >> This is ok, supported and expected since there is no way to know which disk >> will show up first. >> >> When /dev/sdc shows up, the same logic will be triggered: >> >> * udev rule calls ceph-disk-udev via 95-ceph-osd.rules on /dev/sda1 >> * ceph-disk-udev creates the symlink >> /dev/disk/by-partuuid/1e9d527f-0866-4284-b77c-c1cb04c5a168 -> ../../sdc3 >> (overriding the file because ln -sf) >> * ceph-disk activate-journal /dev/sdc3 finds that >> c83b5aa5-fe77-42f6-9415-25ca0266fb7f is the data partition for that journal >> and mounts /dev/disk/by-partuuid/c83b5aa5-fe77-42f6-9415-25ca0266fb7f >> * ceph-osd opens the journal and all is well >> >> Except something goes wrong in your case, presumably because ceph-disk-udev >> is not called when /dev/sdc3 shows up ? >> >> On 17/12/2015 08:29, Jesper Thorhauge wrote: >>> Hi Loic, >>> >>> osd's are on /dev/sda and /dev/sdb, journal's is on /dev/sdc (sdc3 / sdc4). >>> >>> sgdisk for sda shows; >>> >>> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown) >>> Partition unique GUID: E85F4D92-C8F1-4591-BD2A-AA43B80F58F6 >>> First sector: 2048 (at 1024.0 KiB) >>> Last sector: 1953525134 (at 931.5 GiB) >>> Partition size: 1953523087 sectors (931.5 GiB) >>> Attribute flags: >>> Partition name: 'ceph data' >>> >>> for sdb >>> >>> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown) >>> Partition unique GUID: C83B5AA5-FE77-42F6-9415-25CA0266FB7F >>> First sector: 2048 (at 1024.0 KiB) >>> Last sector: 1953525134 (at 931.5 GiB) >>> Partition size: 1953523087 sectors (931.5 GiB) >>> Attribute flags: >>> Partition name: 'ceph data' >>> >>> for /dev/sdc3 >>> >>> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown) >>> Partition unique GUID: C34D4694-B486-450D-B57F-DA24255F0072 >>> First sector: 935813120 (at 446.2 GiB) >>> Last sector: 956293119 (at 456.0 GiB) >>> Partition size: 2048 sectors (9.8 GiB) >>> Attribute flags: >>> Partition name: 'ceph journal' >>> >>> for /dev/sdc4 >>> >>> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown) >>> Partition unique GUID: 1
Re: [ceph-users] Journal symlink broken / Ceph 0.94.5 / CentOS 6.7
Hi Loic, Damn, the updated udev didn't fix the problem :-( The rc.local workaround is also complaining; INFO:ceph-disk:Running command: /usr/bin/ceph-osd -i 0 --get-journal-uuid --osd-journal /dev/sdc3 libust[2648/2648]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305) HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device DEBUG:ceph-disk:Journal /dev/sdc3 has OSD UUID ---- INFO:ceph-disk:Running command: /sbin/blkid -p -s TYPE -ovalue -- /dev/disk/by-partuuid/---- error: /dev/disk/by-partuuid/----: No such file or directory ceph-disk: Cannot discover filesystem type: device /dev/disk/by-partuuid/----: Command '/sbin/blkid' returned non-zero exit status 2 INFO:ceph-disk:Running command: /usr/bin/ceph-osd -i 0 --get-journal-uuid --osd-journal /dev/sdc4 libust[2687/2687]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305) HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device DEBUG:ceph-disk:Journal /dev/sdc4 has OSD UUID ---- INFO:ceph-disk:Running command: /sbin/blkid -p -s TYPE -ovalue -- /dev/disk/by-partuuid/---- error: /dev/disk/by-partuuid/----: No such file or directory ceph-disk: Cannot discover filesystem type: device /dev/disk/by-partuuid/----: Command '/sbin/blkid' returned non-zero exit status 2 /dev/sdc1 and /dev/sdc2 contains the boot loader and OS, so driverwise i guess things are working :-) But "HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device" seems to be the underlying issue. Any thoughts? /Jesper * Hi Loic, searched around for possible udev bugs, and then tried to run "yum update". Udev did have a fresh update with the following version diffs; udev-147-2.63.el6_7.1.x86_64 --> udev-147-2.63.el6_7.1.x86_64 from what i can see this update fixes stuff related to symbolic links / external devices. /dev/sdc sits on external eSata. So... https://rhn.redhat.com/errata/RHBA-2015-1382.html will reboot tonight and get back :-) /jesper ***' I guess that's the problem you need to solve : why /dev/sdc does not generate udev events (different driver than /dev/sda maybe ?). Once it does, Ceph should work. A workaround could be to add somethink like: ceph-disk-udev 3 sdc3 sdc ceph-disk-udev 4 sdc4 sdc in /etc/rc.local. On 17/12/2015 12:01, Jesper Thorhauge wrote: > Nope, the previous post contained all that was in the boot.log :-( > > /Jesper > > ** > > - Den 17. dec 2015, kl. 11:53, Loic Dachary <l...@dachary.org> skrev: > > On 17/12/2015 11:33, Jesper Thorhauge wrote: >> Hi Loic, >> >> Sounds like something does go wrong when /dev/sdc3 shows up. Is there anyway >> i can debug this further? Log-files? Modify the .rules file...? > > Do you see traces of what happens when /dev/sdc3 shows up in boot.log ? > >> >> /Jesper >> >> >> >> The non-symlink files in /dev/disk/by-partuuid come to existence because of: >> >> * system boots >> * udev rule calls ceph-disk-udev via 95-ceph-osd.rules on /dev/sda1 >> * ceph-disk-udev creates the symlink >> /dev/disk/by-partuuid/c83b5aa5-fe77-42f6-9415-25ca0266fb7f -> ../../sdb1 >> * ceph-disk activate /dev/sda1 is mounted and finds a symlink to the journal >> journal -> /dev/disk/by-partuuid/1e9d527f-0866-4284-b77c-c1cb04c5a168 which >> does not yet exists because /dev/sdc udev rules have not been run yet >> * ceph-osd opens the journal in write mode and that creates the file >> /dev/disk/by-partuuid/1e9d527f-0866-4284-b77c-c1cb04c5a168 as a regular file >> * the file is empty and the osd fails to activate with the error you see >> (EINVAL because the file is empty) >> >> This is ok, supported and expected since there is no way to know which disk >> will show up first. >> >> When /dev/sdc shows up, the same logic will be triggered: >> >> * udev rule calls ceph-disk-udev via 95-ceph-osd.rules on /dev/sda1 >> * ceph-disk-udev creates the symlink >> /dev/disk/by-partuuid/1e9d527f-0866-4284-b77c-c1cb04c5a168 -> ../../sdc3 >> (overriding the file because ln -sf) >> * ceph-disk activate-journal /dev/sdc3 finds that >> c83b5aa5-fe77-42f6-9415-25ca0266fb7f is the data partition for that journal >> and mounts /dev/disk/by-partuuid/c83b5aa5-fe77-42f6-9415-25ca0266fb7f >
Re: [ceph-users] Journal symlink broken / Ceph 0.94.5 / CentOS 6.7
Hi Loic, Getting closer! lrwxrwxrwx 1 root root 10 Dec 18 19:43 1e9d527f-0866-4284-b77c-c1cb04c5a168 -> ../../sdc4 lrwxrwxrwx 1 root root 10 Dec 18 19:43 c34d4694-b486-450d-b57f-da24255f0072 -> ../../sdc3 lrwxrwxrwx 1 root root 10 Dec 18 19:42 c83b5aa5-fe77-42f6-9415-25ca0266fb7f -> ../../sdb1 lrwxrwxrwx 1 root root 10 Dec 18 19:42 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 -> ../../sda1 So symlinks are now working! Activating an OSD is a different story :-( "ceph-disk -vv activate /dev/sda1" gives me; INFO:ceph-disk:Running command: /sbin/blkid -p -s TYPE -ovalue -- /dev/sda1 INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs DEBUG:ceph-disk:Mounting /dev/sda1 on /var/lib/ceph/tmp/mnt.A99cDp with options noatime,inode64 INFO:ceph-disk:Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sda1 /var/lib/ceph/tmp/mnt.A99cDp DEBUG:ceph-disk:Cluster uuid is 07b5c90b-6cae-40c0-93b2-31e0ebad7315 INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid DEBUG:ceph-disk:Cluster name is ceph DEBUG:ceph-disk:OSD uuid is e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 DEBUG:ceph-disk:OSD id is 6 DEBUG:ceph-disk:Initializing OSD... INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/tmp/mnt.A99cDp/activate.monmap got monmap epoch 6 INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster ceph --mkfs --mkkey -i 6 --monmap /var/lib/ceph/tmp/mnt.A99cDp/activate.monmap --osd-data /var/lib/ceph/tmp/mnt.A99cDp --osd-journal /var/lib/ceph/tmp/mnt.A99cDp/journal --osd-uuid e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 --keyring /var/lib/ceph/tmp/mnt.A99cDp/keyring HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device 2015-12-18 21:58:12.489357 7f266d7b0800 -1 journal check: ondisk fsid ---- doesn't match expected e85f4d92-c8f1-4591-bd2a-aa43b80f58f6, invalid (someone else's?) journal HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device 2015-12-18 21:58:12.680566 7f266d7b0800 -1 filestore(/var/lib/ceph/tmp/mnt.A99cDp) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2015-12-18 21:58:12.865810 7f266d7b0800 -1 created object store /var/lib/ceph/tmp/mnt.A99cDp journal /var/lib/ceph/tmp/mnt.A99cDp/journal for osd.6 fsid 07b5c90b-6cae-40c0-93b2-31e0ebad7315 2015-12-18 21:58:12.865844 7f266d7b0800 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.A99cDp/keyring: can't open /var/lib/ceph/tmp/mnt.A99cDp/keyring: (2) No such file or directory 2015-12-18 21:58:12.865910 7f266d7b0800 -1 created new key in keyring /var/lib/ceph/tmp/mnt.A99cDp/keyring INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup init DEBUG:ceph-disk:Marking with init system sysvinit DEBUG:ceph-disk:Authorizing OSD key... INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring auth add osd.6 -i /var/lib/ceph/tmp/mnt.A99cDp/keyring osd allow * mon allow profile osd Error EINVAL: entity osd.6 exists but key does not match ERROR:ceph-disk:Failed to activate DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.A99cDp INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.A99cDp Traceback (most recent call last): File "/usr/sbin/ceph-disk", line 2994, in main() File "/usr/sbin/ceph-disk", line 2972, in main args.func(args) File "/usr/sbin/ceph-disk", line 2178, in main_activate init=args.mark_init, File "/usr/sbin/ceph-disk", line 1954, in mount_activate (osd_id, cluster) = activate(path, activate_key_template, init) File "/usr/sbin/ceph-disk", line 2153, in activate keyring=keyring, File "/usr/sbin/ceph-disk", line 1756, in auth_key 'mon', 'allow profile osd', File "/usr/sbin/ceph-disk", line 323, in command_check_call return subprocess.check_call(arguments) File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/usr/bin/ceph', '--cluster', 'ceph', '--name', 'client.bootstrap-osd', '--keyring', '/var/lib/ceph/bootstrap-osd/ceph.keyring', 'auth', 'add', 'osd.6', '-i', '/var/lib/ceph/tmp/mnt.A99cDp/keyring', 'osd', 'allow *', 'mon', 'allow profile osd']' returned non-zero exit status 22 Thanks! /Jesper *** Hi Jesper, The goal of the rc.local is twofold but mainly to ensure the /dev/disk/by-partuuid symlinks exists for the jou
Re: [ceph-users] Journal symlink broken / Ceph 0.94.5 / CentOS 6.7
Hi Loic,
Yep, 95-ceph-osd.rules contains exactly that...
***
And 95-ceph-osd.rules contains the following ?
# Check gpt partion for ceph tags and activate
ACTION=="add", SUBSYSTEM=="block", \
ENV{DEVTYPE}=="partition", \
ENV{ID_PART_TABLE_TYPE}=="gpt", \
RUN+="/usr/sbin/ceph-disk-udev $number $name $parent"
On 17/12/2015 08:29, Jesper Thorhauge wrote:
> Hi Loic,
>
> osd's are on /dev/sda and /dev/sdb, journal's is on /dev/sdc (sdc3 / sdc4).
>
> sgdisk for sda shows;
>
> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
> Partition unique GUID: E85F4D92-C8F1-4591-BD2A-AA43B80F58F6
> First sector: 2048 (at 1024.0 KiB)
> Last sector: 1953525134 (at 931.5 GiB)
> Partition size: 1953523087 sectors (931.5 GiB)
> Attribute flags:
> Partition name: 'ceph data'
>
> for sdb
>
> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
> Partition unique GUID: C83B5AA5-FE77-42F6-9415-25CA0266FB7F
> First sector: 2048 (at 1024.0 KiB)
> Last sector: 1953525134 (at 931.5 GiB)
> Partition size: 1953523087 sectors (931.5 GiB)
> Attribute flags:
> Partition name: 'ceph data'
>
> for /dev/sdc3
>
> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
> Partition unique GUID: C34D4694-B486-450D-B57F-DA24255F0072
> First sector: 935813120 (at 446.2 GiB)
> Last sector: 956293119 (at 456.0 GiB)
> Partition size: 2048 sectors (9.8 GiB)
> Attribute flags:
> Partition name: 'ceph journal'
>
> for /dev/sdc4
>
> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
> Partition unique GUID: 1E9D527F-0866-4284-B77C-C1CB04C5A168
> First sector: 956293120 (at 456.0 GiB)
> Last sector: 976773119 (at 465.8 GiB)
> Partition size: 2048 sectors (9.8 GiB)
> Attribute flags:
> Partition name: 'ceph journal'
>
> 60-ceph-partuuid-workaround.rules is located in /lib/udev/rules.d, so it
> seems correct to me.
>
> after a reboot, /dev/disk/by-partuuid is;
>
> -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168
> -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072
> lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f
> -> ../../sdb1
> lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6
> -> ../../sda1
>
> i dont know how to verify the symlink of the journal file - can you guide me
> on that one?
>
> Thank :-) !
>
> /Jesper
>
> **
>
> Hi,
>
> On 17/12/2015 07:53, Jesper Thorhauge wrote:
>> Hi,
>>
>> Some more information showing in the boot.log;
>>
>> 2015-12-16 07:35:33.289830 7f1b990ad800 -1
>> filestore(/var/lib/ceph/tmp/mnt.aWZTcE) mkjournal error creating journal on
>> /var/lib/ceph/tmp/mnt.aWZTcE/journal: (22) Invalid argument
>> 2015-12-16 07:35:33.289842 7f1b990ad800 -1 OSD::mkfs: ObjectStore::mkfs
>> failed with error -22
>> 2015-12-16 07:35:33.289883 7f1b990ad800 -1 ** ERROR: error creating empty
>> object store in /var/lib/ceph/tmp/mnt.aWZTcE: (22) Invalid argument
>> ERROR:ceph-disk:Failed to activate
>> ceph-disk: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs',
>> '--mkkey', '-i', '7', '--monmap',
>> '/var/lib/ceph/tmp/mnt.aWZTcE/activate.monmap', '--osd-data',
>> '/var/lib/ceph/tmp/mnt.aWZTcE', '--osd-journal',
>> '/var/lib/ceph/tmp/mnt.aWZTcE/journal', '--osd-uuid',
>> 'c83b5aa5-fe77-42f6-9415-25ca0266fb7f', '--keyring',
>> '/var/lib/ceph/tmp/mnt.aWZTcE/keyring']' returned non-zero exit status 1
>> ceph-disk: Error: One or more partitions failed to activate
>>
>> Maybe related to the "(22) Invalid argument" part..?
>
> After a reboot the symlinks are reconstructed and if they are still
> incorrect, it means there is an inconsistency somewhere else. To debug the
> problem, could you mount /dev/sda1 and verify the symlink of the journal file
> ? Then verify the content of /dev/disk/by-partuuid. And also display the
> partition information with sgdisk -i 1 /dev/sda and sgdisk -i 2 /dev/sda. Are
> you collocating your journal with the data, on the same disk ? Or are they on
> two different disks ?
>
> git log --no-merges --oneline tags/v0.94.3..tags/v0.94.5 udev
>
> shows nothing, meaning there has been no change to udev rules. There is one
> change related to the installation of the udev rules
> https://github.com/ceph/ceph/commit/4eb58ad2027148561d94bb43346b464b55d041a6.
> Could you double check 60-ceph-partuuid-workaround.rul
Re: [ceph-users] Journal symlink broken / Ceph 0.94.5 / CentOS 6.7
Hi Loic, Sounds like something does go wrong when /dev/sdc3 shows up. Is there anyway i can debug this further? Log-files? Modify the .rules file...? /Jesper The non-symlink files in /dev/disk/by-partuuid come to existence because of: * system boots * udev rule calls ceph-disk-udev via 95-ceph-osd.rules on /dev/sda1 * ceph-disk-udev creates the symlink /dev/disk/by-partuuid/c83b5aa5-fe77-42f6-9415-25ca0266fb7f -> ../../sdb1 * ceph-disk activate /dev/sda1 is mounted and finds a symlink to the journal journal -> /dev/disk/by-partuuid/1e9d527f-0866-4284-b77c-c1cb04c5a168 which does not yet exists because /dev/sdc udev rules have not been run yet * ceph-osd opens the journal in write mode and that creates the file /dev/disk/by-partuuid/1e9d527f-0866-4284-b77c-c1cb04c5a168 as a regular file * the file is empty and the osd fails to activate with the error you see (EINVAL because the file is empty) This is ok, supported and expected since there is no way to know which disk will show up first. When /dev/sdc shows up, the same logic will be triggered: * udev rule calls ceph-disk-udev via 95-ceph-osd.rules on /dev/sda1 * ceph-disk-udev creates the symlink /dev/disk/by-partuuid/1e9d527f-0866-4284-b77c-c1cb04c5a168 -> ../../sdc3 (overriding the file because ln -sf) * ceph-disk activate-journal /dev/sdc3 finds that c83b5aa5-fe77-42f6-9415-25ca0266fb7f is the data partition for that journal and mounts /dev/disk/by-partuuid/c83b5aa5-fe77-42f6-9415-25ca0266fb7f * ceph-osd opens the journal and all is well Except something goes wrong in your case, presumably because ceph-disk-udev is not called when /dev/sdc3 shows up ? On 17/12/2015 08:29, Jesper Thorhauge wrote: > Hi Loic, > > osd's are on /dev/sda and /dev/sdb, journal's is on /dev/sdc (sdc3 / sdc4). > > sgdisk for sda shows; > > Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown) > Partition unique GUID: E85F4D92-C8F1-4591-BD2A-AA43B80F58F6 > First sector: 2048 (at 1024.0 KiB) > Last sector: 1953525134 (at 931.5 GiB) > Partition size: 1953523087 sectors (931.5 GiB) > Attribute flags: > Partition name: 'ceph data' > > for sdb > > Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown) > Partition unique GUID: C83B5AA5-FE77-42F6-9415-25CA0266FB7F > First sector: 2048 (at 1024.0 KiB) > Last sector: 1953525134 (at 931.5 GiB) > Partition size: 1953523087 sectors (931.5 GiB) > Attribute flags: > Partition name: 'ceph data' > > for /dev/sdc3 > > Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown) > Partition unique GUID: C34D4694-B486-450D-B57F-DA24255F0072 > First sector: 935813120 (at 446.2 GiB) > Last sector: 956293119 (at 456.0 GiB) > Partition size: 2048 sectors (9.8 GiB) > Attribute flags: > Partition name: 'ceph journal' > > for /dev/sdc4 > > Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown) > Partition unique GUID: 1E9D527F-0866-4284-B77C-C1CB04C5A168 > First sector: 956293120 (at 456.0 GiB) > Last sector: 976773119 (at 465.8 GiB) > Partition size: 2048 sectors (9.8 GiB) > Attribute flags: > Partition name: 'ceph journal' > > 60-ceph-partuuid-workaround.rules is located in /lib/udev/rules.d, so it > seems correct to me. > > after a reboot, /dev/disk/by-partuuid is; > > -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168 > -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072 > lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f > -> ../../sdb1 > lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 > -> ../../sda1 > > i dont know how to verify the symlink of the journal file - can you guide me > on that one? > > Thank :-) ! > > /Jesper > > ** > > Hi, > > On 17/12/2015 07:53, Jesper Thorhauge wrote: >> Hi, >> >> Some more information showing in the boot.log; >> >> 2015-12-16 07:35:33.289830 7f1b990ad800 -1 >> filestore(/var/lib/ceph/tmp/mnt.aWZTcE) mkjournal error creating journal on >> /var/lib/ceph/tmp/mnt.aWZTcE/journal: (22) Invalid argument >> 2015-12-16 07:35:33.289842 7f1b990ad800 -1 OSD::mkfs: ObjectStore::mkfs >> failed with error -22 >> 2015-12-16 07:35:33.289883 7f1b990ad800 -1 ** ERROR: error creating empty >> object store in /var/lib/ceph/tmp/mnt.aWZTcE: (22) Invalid argument >> ERROR:ceph-disk:Failed to activate >> ceph-disk: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', >> '--mkkey', '-i', '7', '--monmap', >> '/var/lib/ceph/tmp/mnt.aWZTcE/activate.monmap', '
Re: [ceph-users] Journal symlink broken / Ceph 0.94.5 / CentOS 6.7
Nope, the previous post contained all that was in the boot.log :-( /Jesper ** - Den 17. dec 2015, kl. 11:53, Loic Dachary <l...@dachary.org> skrev: On 17/12/2015 11:33, Jesper Thorhauge wrote: > Hi Loic, > > Sounds like something does go wrong when /dev/sdc3 shows up. Is there anyway > i can debug this further? Log-files? Modify the .rules file...? Do you see traces of what happens when /dev/sdc3 shows up in boot.log ? > > /Jesper > > > > The non-symlink files in /dev/disk/by-partuuid come to existence because of: > > * system boots > * udev rule calls ceph-disk-udev via 95-ceph-osd.rules on /dev/sda1 > * ceph-disk-udev creates the symlink > /dev/disk/by-partuuid/c83b5aa5-fe77-42f6-9415-25ca0266fb7f -> ../../sdb1 > * ceph-disk activate /dev/sda1 is mounted and finds a symlink to the journal > journal -> /dev/disk/by-partuuid/1e9d527f-0866-4284-b77c-c1cb04c5a168 which > does not yet exists because /dev/sdc udev rules have not been run yet > * ceph-osd opens the journal in write mode and that creates the file > /dev/disk/by-partuuid/1e9d527f-0866-4284-b77c-c1cb04c5a168 as a regular file > * the file is empty and the osd fails to activate with the error you see > (EINVAL because the file is empty) > > This is ok, supported and expected since there is no way to know which disk > will show up first. > > When /dev/sdc shows up, the same logic will be triggered: > > * udev rule calls ceph-disk-udev via 95-ceph-osd.rules on /dev/sda1 > * ceph-disk-udev creates the symlink > /dev/disk/by-partuuid/1e9d527f-0866-4284-b77c-c1cb04c5a168 -> ../../sdc3 > (overriding the file because ln -sf) > * ceph-disk activate-journal /dev/sdc3 finds that > c83b5aa5-fe77-42f6-9415-25ca0266fb7f is the data partition for that journal > and mounts /dev/disk/by-partuuid/c83b5aa5-fe77-42f6-9415-25ca0266fb7f > * ceph-osd opens the journal and all is well > > Except something goes wrong in your case, presumably because ceph-disk-udev > is not called when /dev/sdc3 shows up ? > > On 17/12/2015 08:29, Jesper Thorhauge wrote: >> Hi Loic, >> >> osd's are on /dev/sda and /dev/sdb, journal's is on /dev/sdc (sdc3 / sdc4). >> >> sgdisk for sda shows; >> >> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown) >> Partition unique GUID: E85F4D92-C8F1-4591-BD2A-AA43B80F58F6 >> First sector: 2048 (at 1024.0 KiB) >> Last sector: 1953525134 (at 931.5 GiB) >> Partition size: 1953523087 sectors (931.5 GiB) >> Attribute flags: >> Partition name: 'ceph data' >> >> for sdb >> >> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown) >> Partition unique GUID: C83B5AA5-FE77-42F6-9415-25CA0266FB7F >> First sector: 2048 (at 1024.0 KiB) >> Last sector: 1953525134 (at 931.5 GiB) >> Partition size: 1953523087 sectors (931.5 GiB) >> Attribute flags: >> Partition name: 'ceph data' >> >> for /dev/sdc3 >> >> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown) >> Partition unique GUID: C34D4694-B486-450D-B57F-DA24255F0072 >> First sector: 935813120 (at 446.2 GiB) >> Last sector: 956293119 (at 456.0 GiB) >> Partition size: 2048 sectors (9.8 GiB) >> Attribute flags: >> Partition name: 'ceph journal' >> >> for /dev/sdc4 >> >> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown) >> Partition unique GUID: 1E9D527F-0866-4284-B77C-C1CB04C5A168 >> First sector: 956293120 (at 456.0 GiB) >> Last sector: 976773119 (at 465.8 GiB) >> Partition size: 2048 sectors (9.8 GiB) >> Attribute flags: >> Partition name: 'ceph journal' >> >> 60-ceph-partuuid-workaround.rules is located in /lib/udev/rules.d, so it >> seems correct to me. >> >> after a reboot, /dev/disk/by-partuuid is; >> >> -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168 >> -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072 >> lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f >> -> ../../sdb1 >> lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 >> -> ../../sda1 >> >> i dont know how to verify the symlink of the journal file - can you guide me >> on that one? >> >> Thank :-) ! >> >> /Jesper >> >> ** >> >> Hi, >> >> On 17/12/2015 07:53, Jesper Thorhauge wrote: >>> Hi, >>>
Re: [ceph-users] Journal symlink broken / Ceph 0.94.5 / CentOS 6.7
Hi, Some more information showing in the boot.log; 2015-12-16 07:35:33.289830 7f1b990ad800 -1 filestore(/var/lib/ceph/tmp/mnt.aWZTcE) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.aWZTcE/journal: (22) Invalid argument 2015-12-16 07:35:33.289842 7f1b990ad800 -1 OSD::mkfs: ObjectStore::mkfs failed with error -22 2015-12-16 07:35:33.289883 7f1b990ad800 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.aWZTcE: (22) Invalid argument ERROR:ceph-disk:Failed to activate ceph-disk: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '7', '--monmap', '/var/lib/ceph/tmp/mnt.aWZTcE/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.aWZTcE', '--osd-journal', '/var/lib/ceph/tmp/mnt.aWZTcE/journal', '--osd-uuid', 'c83b5aa5-fe77-42f6-9415-25ca0266fb7f', '--keyring', '/var/lib/ceph/tmp/mnt.aWZTcE/keyring']' returned non-zero exit status 1 ceph-disk: Error: One or more partitions failed to activate Maybe related to the "(22) Invalid argument" part..? /Jesper * Hi, I have done several reboots, and it did not lead to healthy symlinks :-( /Jesper Hi, On 16/12/2015 07:39, Jesper Thorhauge wrote: > Hi, > > A fresh server install on one of my nodes (and yum update) left me with > CentOS 6.7 / Ceph 0.94.5. All the other nodes are running Ceph 0.94.2. > > "ceph-disk prepare /dev/sda /dev/sdc" seems to work as expected, but > "ceph-disk activate / dev/sda1" fails. I have traced the problem to > "/dev/disk/by-partuuid", where the journal symlinks are broken; > > -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168 > -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072 > lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f > -> ../../sdb1 > lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 > -> ../../sda1 > > Re-creating them manually wont survive a reboot. Is this a problem with the > udev rules in Ceph 0.94.3+? This usually is a symptom of something else going wrong (i.e. it is possible to confuse the kernel into creating the wrong symbolic links). The correct symlinks should be set when you reboot. > Hope that somebody can help me :-) Please let us know if rebooting leads to healthy symlinks. Cheers > > Thanks! > > Best regards, > Jesper > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Loïc Dachary, Artisan Logiciel Libre ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Journal symlink broken / Ceph 0.94.5 / CentOS 6.7
Hi Loic, osd's are on /dev/sda and /dev/sdb, journal's is on /dev/sdc (sdc3 / sdc4). sgdisk for sda shows; Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown) Partition unique GUID: E85F4D92-C8F1-4591-BD2A-AA43B80F58F6 First sector: 2048 (at 1024.0 KiB) Last sector: 1953525134 (at 931.5 GiB) Partition size: 1953523087 sectors (931.5 GiB) Attribute flags: Partition name: 'ceph data' for sdb Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown) Partition unique GUID: C83B5AA5-FE77-42F6-9415-25CA0266FB7F First sector: 2048 (at 1024.0 KiB) Last sector: 1953525134 (at 931.5 GiB) Partition size: 1953523087 sectors (931.5 GiB) Attribute flags: Partition name: 'ceph data' for /dev/sdc3 Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown) Partition unique GUID: C34D4694-B486-450D-B57F-DA24255F0072 First sector: 935813120 (at 446.2 GiB) Last sector: 956293119 (at 456.0 GiB) Partition size: 2048 sectors (9.8 GiB) Attribute flags: Partition name: 'ceph journal' for /dev/sdc4 Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown) Partition unique GUID: 1E9D527F-0866-4284-B77C-C1CB04C5A168 First sector: 956293120 (at 456.0 GiB) Last sector: 976773119 (at 465.8 GiB) Partition size: 2048 sectors (9.8 GiB) Attribute flags: Partition name: 'ceph journal' 60-ceph-partuuid-workaround.rules is located in /lib/udev/rules.d, so it seems correct to me. after a reboot, /dev/disk/by-partuuid is; -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168 -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072 lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f -> ../../sdb1 lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 -> ../../sda1 i dont know how to verify the symlink of the journal file - can you guide me on that one? Thank :-) ! /Jesper ** Hi, On 17/12/2015 07:53, Jesper Thorhauge wrote: > Hi, > > Some more information showing in the boot.log; > > 2015-12-16 07:35:33.289830 7f1b990ad800 -1 > filestore(/var/lib/ceph/tmp/mnt.aWZTcE) mkjournal error creating journal on > /var/lib/ceph/tmp/mnt.aWZTcE/journal: (22) Invalid argument > 2015-12-16 07:35:33.289842 7f1b990ad800 -1 OSD::mkfs: ObjectStore::mkfs > failed with error -22 > 2015-12-16 07:35:33.289883 7f1b990ad800 -1 ** ERROR: error creating empty > object store in /var/lib/ceph/tmp/mnt.aWZTcE: (22) Invalid argument > ERROR:ceph-disk:Failed to activate > ceph-disk: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', > '--mkkey', '-i', '7', '--monmap', > '/var/lib/ceph/tmp/mnt.aWZTcE/activate.monmap', '--osd-data', > '/var/lib/ceph/tmp/mnt.aWZTcE', '--osd-journal', > '/var/lib/ceph/tmp/mnt.aWZTcE/journal', '--osd-uuid', > 'c83b5aa5-fe77-42f6-9415-25ca0266fb7f', '--keyring', > '/var/lib/ceph/tmp/mnt.aWZTcE/keyring']' returned non-zero exit status 1 > ceph-disk: Error: One or more partitions failed to activate > > Maybe related to the "(22) Invalid argument" part..? After a reboot the symlinks are reconstructed and if they are still incorrect, it means there is an inconsistency somewhere else. To debug the problem, could you mount /dev/sda1 and verify the symlink of the journal file ? Then verify the content of /dev/disk/by-partuuid. And also display the partition information with sgdisk -i 1 /dev/sda and sgdisk -i 2 /dev/sda. Are you collocating your journal with the data, on the same disk ? Or are they on two different disks ? git log --no-merges --oneline tags/v0.94.3..tags/v0.94.5 udev shows nothing, meaning there has been no change to udev rules. There is one change related to the installation of the udev rules https://github.com/ceph/ceph/commit/4eb58ad2027148561d94bb43346b464b55d041a6. Could you double check 60-ceph-partuuid-workaround.rules is installed where it should ? Cheers > > /Jesper > > * > > Hi, > > I have done several reboots, and it did not lead to healthy symlinks :-( > > /Jesper > > > > Hi, > > On 16/12/2015 07:39, Jesper Thorhauge wrote: >> Hi, >> >> A fresh server install on one of my nodes (and yum update) left me with >> CentOS 6.7 / Ceph 0.94.5. All the other nodes are running Ceph 0.94.2. >> >> "ceph-disk prepare /dev/sda /dev/sdc" seems to work as expected, but >> "ceph-disk activate / dev/sda1" fails. I have traced the problem to >> "/dev/disk/by-partuuid", where the journal symlinks are broken; >> >> -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168 >> -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d46
Re: [ceph-users] Journal symlink broken / Ceph 0.94.5 / CentOS 6.7
Hi, I have done several reboots, and it did not lead to healthy symlinks :-( /Jesper Hi, On 16/12/2015 07:39, Jesper Thorhauge wrote: > Hi, > > A fresh server install on one of my nodes (and yum update) left me with > CentOS 6.7 / Ceph 0.94.5. All the other nodes are running Ceph 0.94.2. > > "ceph-disk prepare /dev/sda /dev/sdc" seems to work as expected, but > "ceph-disk activate / dev/sda1" fails. I have traced the problem to > "/dev/disk/by-partuuid", where the journal symlinks are broken; > > -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168 > -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072 > lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f > -> ../../sdb1 > lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 > -> ../../sda1 > > Re-creating them manually wont survive a reboot. Is this a problem with the > udev rules in Ceph 0.94.3+? This usually is a symptom of something else going wrong (i.e. it is possible to confuse the kernel into creating the wrong symbolic links). The correct symlinks should be set when you reboot. > Hope that somebody can help me :-) Please let us know if rebooting leads to healthy symlinks. Cheers > > Thanks! > > Best regards, > Jesper > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Loïc Dachary, Artisan Logiciel Libre ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Journal symlink broken / Ceph 0.94.5 / CentOS 6.7
Hi, A fresh server install on one of my nodes (and yum update) left me with CentOS 6.7 / Ceph 0.94.5. All the other nodes are running Ceph 0.94.2. "ceph-disk prepare /dev/sda /dev/sdc" seems to work as expected, but "ceph-disk activate / dev/sda1" fails. I have traced the problem to "/dev/disk/by-partuuid", where the journal symlinks are broken; -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168 -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072 lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f -> ../../sdb1 lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 -> ../../sda1 Re-creating them manually wont survive a reboot. Is this a problem with the udev rules in Ceph 0.94.3+? Hope that somebody can help me :-) Thanks! Best regards, Jesper ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
