Hi!
On luminous 12.2.2
I'm migrating some OSDs from filestore to bluestore using the "simple" method
as described in docs:
http://docs.ceph.com/docs/master/rados/operations/bluestore-migration/#convert-existing-osds
<http://docs.ceph.com/docs/master/rados/operations/bluestore-migration/#convert-existing-osds>
Mark out and Replace.
However, at 9.: ceph-volume create --bluestore --data $DEVICE --osd-id $ID
it seems to create the bluestore but it fails to authenticate with the old
osd-id auth.
(the command above is also missing lvm or simple)
I think it's related to this:
http://tracker.ceph.com/issues/22642 <http://tracker.ceph.com/issues/22642>
# ceph-volume lvm create --bluestore --data /dev/sdc --osd-id 0
Running command: sudo vgcreate --force --yes
ceph-efad7df8-721d-43d8-8d02-449406e70b90 /dev/sdc
stderr: WARNING: lvmetad is running but disabled. Restart lvmetad before
enabling it!
stdout: Physical volume "/dev/sdc" successfully created
stdout: Volume group "ceph-efad7df8-721d-43d8-8d02-449406e70b90" successfully
created
Running command: sudo lvcreate --yes -l 100%FREE -n
osd-block-138ce507-f28a-45bf-814c-7fa124a9d9b9
ceph-efad7df8-721d-43d8-8d02-449406e70b90
stderr: WARNING: lvmetad is running but disabled. Restart lvmetad before
enabling it!
stdout: Logical volume "osd-block-138ce507-f28a-45bf-814c-7fa124a9d9b9"
created.
Running command: sudo mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: chown -R ceph:ceph /dev/dm-4
Running command: sudo ln -s
/dev/ceph-efad7df8-721d-43d8-8d02-449406e70b90/osd-block-138ce507-f28a-45bf-814c-7fa124a9d9b9
/var/lib/ceph/osd/ceph-0/block
Running command: sudo ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/osd/ceph-0/activate.monmap
stderr: got monmap epoch 2
Running command: ceph-authtool /var/lib/ceph/osd/ceph-0/keyring
--create-keyring --name osd.0 --add-key XXXXXXXX
stdout: creating /var/lib/ceph/osd/ceph-0/keyring
stdout: added entity osd.0 auth auth(auid = 18446744073709551615 key= XXXXXXXX
with 0 caps)
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: sudo ceph-osd --cluster ceph --osd-objectstore bluestore
--mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --key
**************************************** --osd-data /var/lib/ceph/osd/ceph-0/
--osd-uuid 138ce507-f28a-45bf-814c-7fa124a9d9b9 --setuser ceph --setgroup ceph
stderr: 2018-01-26 14:59:10.039549 7fd7ef951cc0 -1
bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode
label at offset 102: buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end
of struct encoding
stderr: 2018-01-26 14:59:10.039744 7fd7ef951cc0 -1
bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode
label at offset 102: buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end
of struct encoding
stderr: 2018-01-26 14:59:10.039925 7fd7ef951cc0 -1
bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode
label at offset 102: buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end
of struct encoding
stderr: 2018-01-26 14:59:10.039984 7fd7ef951cc0 -1
bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
stderr: 2018-01-26 14:59:11.359951 7fd7ef951cc0 -1 key XXXXXXXX
stderr: 2018-01-26 14:59:11.888476 7fd7ef951cc0 -1 created object store
/var/lib/ceph/osd/ceph-0/ for osd.0 fsid efad7df8-721d-43d8-8d02-449406e70b90
Running command: sudo ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-efad7df8-721d-43d8-8d02-449406e70b90/osd-block-138ce507-f28a-45bf-814c-7fa124a9d9b9
--path /var/lib/ceph/osd/ceph-0
Running command: sudo ln -snf
/dev/ceph-efad7df8-721d-43d8-8d02-449406e70b90/osd-block-138ce507-f28a-45bf-814c-7fa124a9d9b9
/var/lib/ceph/osd/ceph-0/block
Running command: chown -R ceph:ceph /dev/dm-4
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: sudo systemctl enable
ceph-volume@lvm-0-138ce507-f28a-45bf-814c-7fa124a9d9b9
stderr: Created symlink from
/etc/systemd/system/multi-user.target.wants/[email protected]
to /lib/systemd/system/[email protected].
Running command: sudo systemctl start ceph-osd@0
ceph-osd.0.log shows:
2018-01-26 15:09:07.379039 7f545d3b9cc0 4 rocksdb:
[/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2859] Recovered from manifest
file:db/MANIFEST-000095 succeeded,manifest_file_number is 95, next_file_number
is 97, last_sequence is 21, log_number is 0,prev_log_number is
0,max_column_family is 0
2018-01-26 15:09:07.379046 7f545d3b9cc0 4 rocksdb:
[/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2867] Column family [default]
(ID 0), log number is 94
2018-01-26 15:09:07.379087 7f545d3b9cc0 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1516979347379083, "job": 1, "event": "recovery_started",
"log_files": [96]}
2018-01-26 15:09:07.379091 7f545d3b9cc0 4 rocksdb:
[/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #96 mode 0
2018-01-26 15:09:07.379102 7f545d3b9cc0 4 rocksdb:
[/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2395] Creating manifest 98
2018-01-26 15:09:07.380466 7f545d3b9cc0 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1516979347380463, "job": 1, "event": "recovery_finished"}
2018-01-26 15:09:07.381331 7f545d3b9cc0 4 rocksdb:
[/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:1063] DB pointer
0x556ecb8c3000
2018-01-26 15:09:07.381353 7f545d3b9cc0 1 bluestore(/var/lib/ceph/osd/ceph-0)
_open_db opened rocksdb path db options
compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
2018-01-26 15:09:07.381616 7f545d3b9cc0 1 freelist init
2018-01-26 15:09:07.381660 7f545d3b9cc0 1 bluestore(/var/lib/ceph/osd/ceph-0)
_open_alloc opening allocation metadata
2018-01-26 15:09:07.381679 7f545d3b9cc0 1 bluestore(/var/lib/ceph/osd/ceph-0)
_open_alloc loaded 447 G in 1 extents
2018-01-26 15:09:07.382077 7f545d3b9cc0 0 _get_class not permitted to load kvs
2018-01-26 15:09:07.382309 7f545d3b9cc0 0 <cls>
/build/ceph-12.2.2/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs
2018-01-26 15:09:07.382583 7f545d3b9cc0 0 _get_class not permitted to load sdk
2018-01-26 15:09:07.382827 7f545d3b9cc0 0 <cls>
/build/ceph-12.2.2/src/cls/hello/cls_hello.cc:296: loading cls_hello
2018-01-26 15:09:07.385755 7f545d3b9cc0 0 _get_class not permitted to load lua
2018-01-26 15:09:07.386073 7f545d3b9cc0 0 osd.0 0 crush map has features
288232575208783872, adjusting msgr requires for clients
2018-01-26 15:09:07.386078 7f545d3b9cc0 0 osd.0 0 crush map has features
288232575208783872 was 8705, adjusting msgr requires for mons
2018-01-26 15:09:07.386079 7f545d3b9cc0 0 osd.0 0 crush map has features
288232575208783872, adjusting msgr requires for osds
2018-01-26 15:09:07.386132 7f545d3b9cc0 0 osd.0 0 load_pgs
2018-01-26 15:09:07.386134 7f545d3b9cc0 0 osd.0 0 load_pgs opened 0 pgs
2018-01-26 15:09:07.386137 7f545d3b9cc0 0 osd.0 0 using weightedpriority op
queue with priority op cut off at 64.
2018-01-26 15:09:07.386580 7f545d3b9cc0 -1 osd.0 0 log_to_monitors
{default=true}
2018-01-26 15:09:07.388077 7f545d3b9cc0 -1 osd.0 0 init authentication failed:
(1) Operation not permitted
The old osd is still there.
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 2.60458 root default
-2 0.86819 host int1
0 ssd 0.43159 osd.0 destroyed 0 1.00000
3 ssd 0.43660 osd.3 up 1.00000 1.00000
-3 0.86819 host int2
1 ssd 0.43159 osd.1 up 1.00000 1.00000
4 ssd 0.43660 osd.4 up 1.00000 1.00000
-4 0.86819 host int3
2 ssd 0.43159 osd.2 up 1.00000 1.00000
5 ssd 0.43660 osd.5 up 1.00000 1.00000
What's the best course of action? Purging osd.0, zapping the device again and
creating without --osd-id set?
Kind Regards,
David Majchrzak_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
