Good evening,
since 2018 we have been using a custom script to create disks / partitions, because at the time both ceph-disk and ceph-volume exhibited bugs that made them unreliable for us. We recently re-tested ceph-volume and while it seems generally speaking [0] to work, using LVM seems to introduce an additional layer that is not needed from our side. The script we created is little less than 100 lines long and works by specifying the device and its device class: ./ceph-osd-create-start /dev/sdd ssd Everything else is determined by the script. As the script is very simple and work independently of any init system, we wanted to discuss whether it would make sense for anyone else to re-integrate something like it back into ceph upstream? We are aware that ceph-disk has been deprecated, however ceph-volume raw had some issues our setup [2]. The script itself can be found on [1]. At the end it has some ungleich specifics, but we'd be very open to remove that. Best regards, Nico [0] https://tracker.ceph.com/issues/47724 [1] https://code.ungleich.ch/ungleich-public/ungleich-tools/-/blob/master/ceph-osd-create-start [2] [00:19:45] server8.place6:/var/lib/ceph/osd# ceph-volume raw prepare --bluestore --data /dev/sdn Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d0fa7074-6cdf-4947-ac1e-e73dd0ec1fe8 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-3 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /bin/chown -R ceph:ceph /dev/sdn Running command: /bin/ln -s /dev/sdn /var/lib/ceph/osd/ceph-3/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-3/activate.monmap stderr: 2020-09-28 00:20:48.276 7f8c90f27700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2020-09-28 00:20:48.276 7f8c90f27700 -1 AuthRegistry(0x7f8c8c081d08) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx stderr: got monmap epoch 12 Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-3/keyring --create-keyring --name osd.3 --add-key AQA/EHFfINr/HhAAVFAP9NF2LLGFrJEGXbbMSw== stdout: creating /var/lib/ceph/osd/ceph-3/keyring added entity osd.3 auth(key=AQA/EHFfINr/HhAAVFAP9NF2LLGFrJEGXbbMSw==) Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/keyring Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid d0fa7074-6cdf-4947-ac1e-e73dd0ec1fe8 --setuser ceph --setgroup ceph stderr: 2020-09-28 00:20:48.760 7f6e34ebfd80 -1 bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid stderr: 2020-09-28 00:20:55.912 7f6e34ebfd80 -1 bdev(0x561a9a0f6700 /var/lib/ceph/osd/ceph-3//block) _lock flock failed on /var/lib/ceph/osd/ceph-3//block stderr: 2020-09-28 00:20:55.912 7f6e34ebfd80 -1 bdev(0x561a9a0f6700 /var/lib/ceph/osd/ceph-3//block) open failed to lock /var/lib/ceph/osd/ceph-3//block: (11) Resource temporarily unavailable stderr: 2020-09-28 00:20:55.912 7f6e34ebfd80 -1 OSD::mkfs: couldn't mount ObjectStore: error (11) Resource temporarily unavailable stderr: 2020-09-28 00:20:55.912 7f6e34ebfd80 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-3/: (11) Resource temporarily unavailable --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.3 --yes-i-really-mean-it stderr: 2020-09-28 00:20:56.072 7f3e446a9700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2020-09-28 00:20:56.072 7f3e446a9700 -1 AuthRegistry(0x7f3e3c081d08) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx stderr: purged osd.3 --> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid d0fa7074-6cdf-4947-ac1e-e73dd0ec1fe8 --setuser ceph --setgroup ceph [00:20:56] server8.place6:/var/lib/ceph/osd# -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
