Re: [ceph-users] Missing udev rule for FC disks (Re: mkjournal error creating journal ... : (13) Permission denied)
Thanks a lot, Tom, glad this was already taken care of! Will keep the patch around until the official one somehow gets into my distribution. Ciao ciao Fulvio Original Message Subject: Re: [ceph-users] Missing udev rule for FC disks (Re: mkjournal error creating journal ... : (13) Permission denied) From: To: , Date: 1/22/2018 10:34 AM I believe I've recently spent some time with this issue, so I hope this is helpful. Apologies if it's an unrelated dm/udev/ceph-disk problem. https://lists.freedesktop.org/archives/systemd-devel/2017-July/039222.html The above email from last July explains the situation somewhat, with the outcome (as I understand it) being future versions of lvm/dm will have rules to create the necessary partuuid symlinks for dm devices. I'm unsure when that will make its way into various distribution lvm packages (I haven't checked up on this for a month or two actually). For now I've tested running with the new dm-disk.rules on the storage nodes that need it, which allowed ceph-disk to work as expected. Cheers Tom -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Fulvio Galeazzi Sent: 19 January 2018 15:46 To: ceph-users@lists.ceph.com Subject: [ceph-users] Missing udev rule for FC disks (Re: mkjournal error creating journal ... : (13) Permission denied) Hallo, apologies for reviving an old thread, but I just wasted again one full day as I had forgotten about this issue... To recap, udev rules nowadays do not (at least in my case, I am using disks served via FiberChannel) create the links /dev/disk/by-partuuid that ceph-disk expects. I see the "culprit" is this line in (am on CentOS, but Ubuntu has the same issue): /usr/lib/udev/rules.d/60-persistent-storage.rules . # skip rules for inappropriate block devices KERNEL=="fd*|mtd*|nbd*|gnbd*|btibm*|dm-*|md*|zram*|mmcblk[0-9]*rpmb", GOTO="persistent_storage_end" . stating that multipath'ed devices (called dm-*) should be skipped. I can happily live with the file mentioned below, but was wondering: - is there any hope that newer kernels may handle multipath devices properly? - as an alternative, could it be possible to update ceph-disk such that symlinks for journal use some other /dev/disk/by-? Thanks! Fulvio smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Missing udev rule for FC disks (Re: mkjournal error creating journal ... : (13) Permission denied)
I believe I've recently spent some time with this issue, so I hope this is
helpful. Apologies if it's an unrelated dm/udev/ceph-disk problem.
https://lists.freedesktop.org/archives/systemd-devel/2017-July/039222.html
The above email from last July explains the situation somewhat, with the
outcome (as I understand it) being future versions of lvm/dm will have rules to
create the necessary partuuid symlinks for dm devices.
I'm unsure when that will make its way into various distribution lvm packages
(I haven't checked up on this for a month or two actually). For now I've tested
running with the new dm-disk.rules on the storage nodes that need it, which
allowed ceph-disk to work as expected.
Cheers
Tom
-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Fulvio
Galeazzi
Sent: 19 January 2018 15:46
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Missing udev rule for FC disks (Re: mkjournal error
creating journal ... : (13) Permission denied)
Hallo,
apologies for reviving an old thread, but I just wasted again one full
day as I had forgotten about this issue...
To recap, udev rules nowadays do not (at least in my case, I am using disks
served via FiberChannel) create the links /dev/disk/by-partuuid that ceph-disk
expects.
I see the "culprit" is this line in (am on CentOS, but Ubuntu has the same
issue): /usr/lib/udev/rules.d/60-persistent-storage.rules
.
# skip rules for inappropriate block devices
KERNEL=="fd*|mtd*|nbd*|gnbd*|btibm*|dm-*|md*|zram*|mmcblk[0-9]*rpmb",
GOTO="persistent_storage_end"
.
stating that multipath'ed devices (called dm-*) should be skipped.
I can happily live with the file mentioned below, but was wondering:
- is there any hope that newer kernels may handle multipath devices
properly?
- as an alternative, could it be possible to update ceph-disk
such that symlinks for journal use some other
/dev/disk/by-?
Thanks!
Fulvio
On 3/16/2017 5:59 AM, Gunwoo Gim wrote:
> Thank you so much Peter. The 'udevadm trigger' after 'partprobe'
> triggered the udev rules and I've found out that even before the udev
> ruleset triggers the owner is already ceph:ceph.
>
> I've dug into ceph-disk a little more and found out that there is a
> symbolic link of
> /dev/disk/by-partuuid/120c536d-cb30-4cea-b607-dd347022a497 at
> [/dev/mapper/vg--hdd1-lv--hdd1p1(the_filestore_osd)]/journal and the
> source doesn't exist. though it exists in /dev/disk/by-parttypeuuid
> which has been populated by
> /lib/udev/rules.d/60-ceph-by-parttypeuuid.rules
>
> So I added this in /lib/udev/rules.d/60-ceph-by-parttypeuuid.rules:
> # when ceph-disk prepares a filestore osd it makes a symbolic link by
> disk/by-partuuid but LVM2 doesn't seem to populate /dev/disk/by-partuuid.
> ENV{ID_PART_ENTRY_SCHEME}=="gpt", ENV{ID_PART_ENTRY_TYPE}=="?*",
> ENV{ID_PART_ENTRY_UUID}=="?*",
> SYMLINK+="disk/by-partuuid/$env{ID_PART_ENTRY_UUID}"
> And finally got the osds all up and in. :D
>
> Yeah, It wasn't actually a permission problem, but the link just
> wasn't existing.
>
>
> ~ # ceph-disk -v activate /dev/mapper/vg--hdd1-lv--hdd1p1 ...
> mount: Mounting /dev/mapper/vg--hdd1-lv--hdd1p1 on
> /var/lib/ceph/tmp/mnt.ECAifr with options
> noatime,largeio,inode64,swalloc
> command_check_call: Running command: /bin/mount -t xfs -o
> noatime,largeio,inode64,swalloc -- /dev/mapper/vg--hdd1-lv--hdd1p1
> /var/lib/ceph/tmp/mnt.ECAifr
> mount: DIGGIN ls -al /var/lib/ceph/tmp/mnt.ECAifr
> mount: DIGGIN total 36
> drwxr-xr-x 3 ceph ceph 174 Mar 14 11:51 .
> drwxr-xr-x 6 ceph ceph 4096 Mar 16 11:30 ..
> -rw-r--r-- 1 root root 202 Mar 16 11:19 activate.monmap
> -rw-r--r-- 1 ceph ceph 37 Mar 14 11:45 ceph_fsid drwxr-xr-x 3 ceph
> ceph 39 Mar 14 11:51 current
> -rw-r--r-- 1 ceph ceph 37 Mar 14 11:45 fsid lrwxrwxrwx 1 ceph ceph
> 58 Mar 14 11:45 journal ->
> /dev/disk/by-partuuid/120c536d-cb30-4cea-b607-dd347022a497
> -rw-r--r-- 1 ceph ceph 37 Mar 14 11:45 journal_uuid
> -rw-r--r-- 1 ceph ceph 21 Mar 14 11:45 magic
> -rw-r--r-- 1 ceph ceph 4 Mar 14 11:51 store_version
> -rw-r--r-- 1 ceph ceph 53 Mar 14 11:51 superblock
> -rw-r--r-- 1 ceph ceph 2 Mar 14 11:51 whoami ...
> ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph',
> '--mkfs', '--mkkey', '-i', u'0', '--monmap',
> '/var/lib/ceph/tmp/mnt.ECAifr/activate.monmap', '- -osd-data',
> '/var/lib/ceph/tmp/mnt.ECAifr', '--osd-journal',
> '/var/lib/ceph/tmp/mnt.ECAifr/journal', '--o
[ceph-users] Missing udev rule for FC disks (Re: mkjournal error creating journal ... : (13) Permission denied)
Hallo,
apologies for reviving an old thread, but I just wasted again one
full day as I had forgotten about this issue...
To recap, udev rules nowadays do not (at least in my case, I am
using disks served via FiberChannel) create the links
/dev/disk/by-partuuid that ceph-disk expects.
I see the "culprit" is this line in (am on CentOS, but Ubuntu has the
same issue): /usr/lib/udev/rules.d/60-persistent-storage.rules
.
# skip rules for inappropriate block devices
KERNEL=="fd*|mtd*|nbd*|gnbd*|btibm*|dm-*|md*|zram*|mmcblk[0-9]*rpmb",
GOTO="persistent_storage_end"
.
stating that multipath'ed devices (called dm-*) should be skipped.
I can happily live with the file mentioned below, but was wondering:
- is there any hope that newer kernels may handle multipath devices
properly?
- as an alternative, could it be possible to update ceph-disk
such that symlinks for journal use some other
/dev/disk/by-?
Thanks!
Fulvio
On 3/16/2017 5:59 AM, Gunwoo Gim wrote:
Thank you so much Peter. The 'udevadm trigger' after 'partprobe'
triggered the udev rules and I've found out that even before the udev
ruleset triggers the owner is already ceph:ceph.
I've dug into ceph-disk a little more and found out that there is a
symbolic link of
/dev/disk/by-partuuid/120c536d-cb30-4cea-b607-dd347022a497 at
[/dev/mapper/vg--hdd1-lv--hdd1p1(the_filestore_osd)]/journal and the
source doesn't exist. though it exists in /dev/disk/by-parttypeuuid
which has been populated by /lib/udev/rules.d/60-ceph-by-parttypeuuid.rules
So I added this in /lib/udev/rules.d/60-ceph-by-parttypeuuid.rules:
# when ceph-disk prepares a filestore osd it makes a symbolic link by
disk/by-partuuid but LVM2 doesn't seem to populate /dev/disk/by-partuuid.
ENV{ID_PART_ENTRY_SCHEME}=="gpt", ENV{ID_PART_ENTRY_TYPE}=="?*",
ENV{ID_PART_ENTRY_UUID}=="?*",
SYMLINK+="disk/by-partuuid/$env{ID_PART_ENTRY_UUID}"
And finally got the osds all up and in. :D
Yeah, It wasn't actually a permission problem, but the link just
wasn't existing.
~ # ceph-disk -v activate /dev/mapper/vg--hdd1-lv--hdd1p1
...
mount: Mounting /dev/mapper/vg--hdd1-lv--hdd1p1 on
/var/lib/ceph/tmp/mnt.ECAifr with options noatime,largeio,inode64,swalloc
command_check_call: Running command: /bin/mount -t xfs -o
noatime,largeio,inode64,swalloc -- /dev/mapper/vg--hdd1-lv--hdd1p1
/var/lib/ceph/tmp/mnt.ECAifr
mount: DIGGIN ls -al /var/lib/ceph/tmp/mnt.ECAifr
mount: DIGGIN total 36
drwxr-xr-x 3 ceph ceph 174 Mar 14 11:51 .
drwxr-xr-x 6 ceph ceph 4096 Mar 16 11:30 ..
-rw-r--r-- 1 root root 202 Mar 16 11:19 activate.monmap
-rw-r--r-- 1 ceph ceph 37 Mar 14 11:45 ceph_fsid
drwxr-xr-x 3 ceph ceph 39 Mar 14 11:51 current
-rw-r--r-- 1 ceph ceph 37 Mar 14 11:45 fsid
lrwxrwxrwx 1 ceph ceph 58 Mar 14 11:45 journal ->
/dev/disk/by-partuuid/120c536d-cb30-4cea-b607-dd347022a497
-rw-r--r-- 1 ceph ceph 37 Mar 14 11:45 journal_uuid
-rw-r--r-- 1 ceph ceph 21 Mar 14 11:45 magic
-rw-r--r-- 1 ceph ceph 4 Mar 14 11:51 store_version
-rw-r--r-- 1 ceph ceph 53 Mar 14 11:51 superblock
-rw-r--r-- 1 ceph ceph 2 Mar 14 11:51 whoami
...
ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs',
'--mkkey', '-i', u'0', '--monmap',
'/var/lib/ceph/tmp/mnt.ECAifr/activate.monmap', '-
-osd-data', '/var/lib/ceph/tmp/mnt.ECAifr', '--osd-journal',
'/var/lib/ceph/tmp/mnt.ECAifr/journal', '--osd-uuid',
u'377c336b-278d-4caf-b2f5-592ac72cd9b6', '-
-keyring', '/var/lib/ceph/tmp/mnt.ECAifr/keyring', '--setuser', 'ceph',
'--setgroup', 'ceph'] failed : 2017-03-16 11:30:05.238725 7f918fbc0a40
-1 filestore(/v
ar/lib/ceph/tmp/mnt.ECAifr) mkjournal error creating journal on
/var/lib/ceph/tmp/mnt.ECAifr/journal: (13) Permission denied
2017-03-16 11:30:05.238756 7f918fbc0a40 -1 OSD::mkfs: ObjectStore::mkfs
failed with error -13
2017-03-16 11:30:05.238833 7f918fbc0a40 -1 ** ERROR: error creating
empty object store in /var/lib/ceph/tmp/mnt.ECAifr: (13) Permission denied
~ # blkid /dev/mapper/vg--*lv-*p* | grep
'120c536d-cb30-4cea-b607-dd347022a497'
/dev/mapper/vg--ssd1-lv--ssd1p1: PARTLABEL="ceph journal"
PARTUUID="120c536d-cb30-4cea-b607-dd347022a497"
~ # ls -al /dev/disk/by-id | grep dm-22
lrwxrwxrwx 1 root root 11 Mar 16 11:37 dm-name-vg--ssd1-lv--ssd1p1 ->
../../dm-22
lrwxrwxrwx 1 root root 11 Mar 16 11:37
dm-uuid-part1-LVM-n1SH1FvtfjgxJOMWN9aHurFvn2BpIsLZi89GWxA68hLmUQV6l5oyiEOPsFciRbKg
-> ../../dm-22
~ # ls -al /dev/disk/by-parttypeuuid | grep dm-22
lrwxrwxrwx 1 root root 11 Mar 16 11:37
45b0969e-9b03-4f30-b4c6-b4b80ceff106.120c536d-cb30-4cea-b607-dd347022a497 ->
../../dm-22
~ # ls -al /dev/disk/by-uuid | grep dm-22
~ # ls -al /dev/disk/by-partuuid/ | grep dm-22
~ # ls -al /dev/disk/by-path | grep dm-22
Best Regards,
Nicholas Gim.
On Wed, Mar 15, 2017 at 6:46 PM Peter Maloney
> wrote:
On 03/15/17 08:43, Gunwoo Gim wrote:
After a reboot, all the parti
