Re: [ceph-users] Fwd: Separate --block.wal --block.db bluestore not working as expected.
On 4/7/18 4:21 PM, Alfredo Deza wrote:
On Sat, Apr 7, 2018 at 11:59 AM, Gary Verhulp <ga...@gmoneylove.com> wrote:
I’m trying to create bluestore osds with separate --block.wal --block.db
devices on a write intensive SSD
I’ve split the SSD (/dev/sda) into two partditions sda1 and sda2 for db and
wal
I seems to me the osd uuid is getting changed and I’m only able to start the
last OSD
Do I need to create a new partition or logical volume on the SSD for each
OSD?
Correct! This is what is needed for each OSD. You are re-using the
same partitions for the other OSD which is why you are getting the
following message:
2018-04-06 19:45:43.730515 7fe91a9cfd00 -1 bluestore(/dev/sda1)
_check_or_set_bdev_label bdev /dev/sda1 fsid
eb6cbcb3-f644-4973-b745-0e4389ef719c does not match our fsid
9d7a103a-f590-4842-bd3d-e9da27c3fb09
I’m sure this is a simple fail in my understanding of how it is supposed to
be provisioned.
Any advice would be appreciated.
Thanks,
Gary
[root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdc
--block.wal /dev/sda2 --block.db /dev/sda1
Running command: sudo vgcreate --force --yes
ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034 /dev/sdc
stdout: Physical volume "/dev/sdc" successfully created.
stdout: Volume group "ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034"
successfully created
Running command: sudo lvcreate --yes -l 100%FREE -n
osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09
ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034
stdout: Logical volume "osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09"
created.
Running command: sudo mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1
Running command: chown -R ceph:ceph /dev/dm-2
Running command: sudo ln -s
/dev/ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034/osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09
/var/lib/ceph/osd/ceph-1/block
Running command: sudo ceph --cluster ceph --name client.bootstrap-osd
--keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/osd/ceph-1/activate.monmap
stderr: got monmap epoch 1
Running command: ceph-authtool /var/lib/ceph/osd/ceph-1/keyring
--create-keyring --name osd.1 --add-key
AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g==
stdout: creating /var/lib/ceph/osd/ceph-1/keyring
stdout: added entity osd.1 auth auth(auid = 18446744073709551615
key=AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g== with 0 caps)
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/
Running command: chown -R ceph:ceph /dev/sda2
Running command: chown -R ceph:ceph /dev/sda1
Running command: sudo ceph-osd --cluster ceph --osd-objectstore bluestore
--mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --key
--bluestore-block-wal-path
/dev/sda2 --bluestore-block-db-path /dev/sda1 --osd-data
/var/lib/ceph/osd/ceph-1/ --osd-uuid 9d7a103a-f590-4842-bd3d-e9da27c3fb09
--setuser ceph --setgroup ceph
stderr: 2018-04-06 19:41:44.519662 7f734f2e4d00 -1
bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode
label at offset 102: buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding
stderr: 2018-04-06 19:41:44.520939 7f734f2e4d00 -1
bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode
label at offset 102: buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding
stderr: 2018-04-06 19:41:44.521190 7f734f2e4d00 -1
bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode
label at offset 102: buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding
stderr: 2018-04-06 19:41:44.521454 7f734f2e4d00 -1
bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid
stderr: 2018-04-06 19:41:47.307648 7f734f2e4d00 -1 key
AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g==
stderr: 2018-04-06 19:41:48.068161 7f734f2e4d00 -1 created object store
/var/lib/ceph/osd/ceph-1/ for osd.1 fsid
1ff50434-64ad-42bd-9a70-1968e4a9a813
[root@osdhost osd]# ceph-bluestore-tool show-label --dev /dev/sda1
{
"/dev/sda1": {
"osd_uuid": "9d7a103a-f590-4842-bd3d-e9da27c3fb09",
"size": 200043171840,
"btime": "2018-04-06 19:41:44.523894",
"description": "bluefs db"
}
}
[root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdd
--block.wal /dev/sda2 --block.db /dev/sda1
Running command: sudo vgcreate --force --yes
ceph-cc91203d-de5c-4d27-8c48-a58663075e67 /dev/sdd
stdout: Physical volume "/dev/sdd" successfully created.
stdout: Volume group "ceph-cc91203d-de5c-4d27-8c48-a58663075e67"
successfully created
Running command: sudo lvcre
[ceph-users] Fwd: Separate --block.wal --block.db bluestore not working as expected.
I’m trying to create bluestore osds with separate --block.wal --block.db
devices on a write intensive SSD
I’ve split the SSD (/dev/sda) into two partditions sda1 and sda2 for db
and wal
I seems to me the osd uuid is getting changed and I’m only able to start
the last OSD
Do I need to create a new partition or logical volume on the SSD for
each OSD?
I’m sure this is a simple fail in my understanding of how it is supposed
to be provisioned.
Any advice would be appreciated.
Thanks,
Gary
[root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdc
--block.wal /dev/sda2 --block.db /dev/sda1
Running command: sudo vgcreate --force --yes
ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034 /dev/sdc
stdout: Physical volume "/dev/sdc" successfully created.
stdout: Volume group "ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034"
successfully created
Running command: sudo lvcreate --yes -l 100%FREE -n
osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09
ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034
stdout: Logical volume "osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09"
created.
Running command: sudo mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1
Running command: chown -R ceph:ceph /dev/dm-2
Running command: sudo ln -s
/dev/ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034/osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09
/var/lib/ceph/osd/ceph-1/block
Running command: sudo ceph --cluster ceph --name client.bootstrap-osd
--keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/osd/ceph-1/activate.monmap
stderr: got monmap epoch 1
Running command: ceph-authtool /var/lib/ceph/osd/ceph-1/keyring
--create-keyring --name osd.1 --add-key
AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g==
stdout: creating /var/lib/ceph/osd/ceph-1/keyring
stdout: added entity osd.1 auth auth(auid = 18446744073709551615
key=AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g== with 0 caps)
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/
Running command: chown -R ceph:ceph /dev/sda2
Running command: chown -R ceph:ceph /dev/sda1
Running command: sudo ceph-osd --cluster ceph --osd-objectstore
bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap
--key
--bluestore-block-wal-path /dev/sda2 --bluestore-block-db-path /dev/sda1
--osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid
9d7a103a-f590-4842-bd3d-e9da27c3fb09 --setuser ceph --setgroup ceph
stderr: 2018-04-06 19:41:44.519662 7f734f2e4d00 -1
bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to
decode label at offset 102: buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode
past end of struct encoding
stderr: 2018-04-06 19:41:44.520939 7f734f2e4d00 -1
bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to
decode label at offset 102: buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode
past end of struct encoding
stderr: 2018-04-06 19:41:44.521190 7f734f2e4d00 -1
bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to
decode label at offset 102: buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode
past end of struct encoding
stderr: 2018-04-06 19:41:44.521454 7f734f2e4d00 -1
bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid
stderr: 2018-04-06 19:41:47.307648 7f734f2e4d00 -1 key
AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g==
stderr: 2018-04-06 19:41:48.068161 7f734f2e4d00 -1 created object store
/var/lib/ceph/osd/ceph-1/ for osd.1 fsid
1ff50434-64ad-42bd-9a70-1968e4a9a813
[root@osdhost osd]# ceph-bluestore-tool show-label --dev /dev/sda1
{
"/dev/sda1": {
"osd_uuid": "9d7a103a-f590-4842-bd3d-e9da27c3fb09",
"size": 200043171840,
"btime": "2018-04-06 19:41:44.523894",
"description": "bluefs db"
}
}
[root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdd
--block.wal /dev/sda2 --block.db /dev/sda1
Running command: sudo vgcreate --force --yes
ceph-cc91203d-de5c-4d27-8c48-a58663075e67 /dev/sdd
stdout: Physical volume "/dev/sdd" successfully created.
stdout: Volume group "ceph-cc91203d-de5c-4d27-8c48-a58663075e67"
successfully created
Running command: sudo lvcreate --yes -l 100%FREE -n
osd-block-eb6cbcb3-f644-4973-b745-0e4389ef719c
ceph-cc91203d-de5c-4d27-8c48-a58663075e67
stdout: Logical volume "osd-block-eb6cbcb3-f644-4973-b745-0e4389ef719c"
created.
Running command: sudo mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-6
Running command: chown -R ceph:ceph /dev/dm-8
Running command: sudo ln -s
/dev/ceph-cc91203d-de5c-4d27-8c48-a58663075e67/osd-block-eb6cbcb3-f644-4973-b745-0e4389ef719c
/var/lib/ceph/osd/ceph-6/block
Running command: sudo ceph --cluster ceph --name client.bootstrap-osd
--keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
