Dear Paul and Alfredo, Downgrading to ceph-deploy 1.5.38 did not work either. I labeled the journal partition (i.e. /dev/nvme0n1p12) with parted, and it added a gpt partuuid to this specific partition:
/dev/nvme0n1p12: PTUUID="8a775205-1364-43d9-820e-c4d3a0d9f9e3" PTTYPE="gpt" PARTLABEL="ceph journal" PARTUUID="0fdff4d6-2833-4e6e-a832-9fb2452bc396” Then I retried with ceph-deploy 2.0 with the following command and it succeeded. Before labeling as gpt with parted, it failed. ceph-deploy osd create --filestore --journal /dev/nvme0n1p12 --data /dev/sdl ank-ceph10 If anyone encounters a problem like this, this solution worked for me. FYI. My best, Huseyin On 11 Jul 2018 20:31 +0300, Alfredo Deza <ad...@redhat.com>, wrote: > On Wed, Jul 11, 2018 at 12:57 PM, Huseyin Cotuk <hco...@gmail.com> wrote: > > Hi Paul, > > > > Thanks for your reply. I did not mention any special parameter while > > upgrading to luminous. So this ceph-deploy version is the one coming from > > the official debian luminous repo. > > That is because ceph-volume came out in Luminous and fully supports > most everything ceph-disk provided. > > Another alternative is to allow ceph-volume to take over the > management of your ceph-disk OSDs, although that is not provided via > ceph-deploy. > > In the meantime, you can still use ceph-disk in Luminous, but like > Paul mentioned it will require you to downgrade ceph-deploy to 1.5.39 > > > > I will try to downgrade ceph-deploy and try to add osd again. To prevent any > > inconsistency, maybe you can check the version of ceph-deploy package on the > > repository. > > > > My best, > > Huseyin > > On 11 Tem 2018 19:49 +0300, Paul Emmerich <paul.emmer...@croit.io>, wrote: > > > > BlueStore is really stable and mature nowadays. > > > > You seem to be using ceph-deploy 2.0.0 which I would not call mature and > > stable at the moment ;) > > Anyways, it uses ceph-volume instead of ceph-disk and I think you have to > > specify the actual partition here. > > But I'd just downgrade to ceph-deploy 1.5.39 when running Luminous (not a > > long-term solution as ceph-disk will > > be removed in Nautilus) > > > > Paul > > > > 2018-07-11 18:37 GMT+02:00 Huseyin Cotuk <hco...@gmail.com>: > > > > > > Hello everybody, > > > > > > I have just upgraded my ceph cluster from kraken to luminous. I just want > > > to go on with filestore based objectstore for OSDs until Redhat announces > > > bluestore as stable. It is still in technical preview. > > > > > > So my question is: “What is the right procedure of adding an filestore > > > based OSD into the existing cluster with an NVME journal?" > > > > > > My NVME journal contains ceph journal partitions for existing OSD created > > > with kraken. > > > > > > root@ank-ceph10:~# lsblk /dev/nvme0n1 > > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > > > nvme0n1 259:0 0 2.9T 0 disk > > > |-nvme0n1p1 259:1 0 40G 0 part > > > |-nvme0n1p2 259:2 0 40G 0 part > > > |-nvme0n1p3 259:3 0 40G 0 part > > > |-nvme0n1p4 259:4 0 40G 0 part > > > |-nvme0n1p5 259:5 0 40G 0 part > > > |-nvme0n1p6 259:6 0 40G 0 part > > > |-nvme0n1p7 259:7 0 40G 0 part > > > |-nvme0n1p8 259:8 0 40G 0 part > > > |-nvme0n1p9 259:9 0 40G 0 part > > > |-nvme0n1p10 259:10 0 40G 0 part > > > |-nvme0n1p11 259:11 0 40G 0 part > > > `-nvme0n1p12 259:12 0 40G 0 part > > > > > > When i try to add a new osd with the following command, > > > > > > ceph-deploy osd create --filestore --journal /dev/nvme0n1 --data /dev/sdl > > > ank-ceph10 > > > > > > I get the following error: > > > > > > [ank-ceph10][INFO ] Running command: sudo /usr/sbin/ceph-volume --cluster > > > ceph lvm create --filestore --data /dev/sdl --journal /dev/nvme0n1 > > > [ank-ceph10][WARNIN] --> RuntimeError: unable to use device > > > [ank-ceph10][DEBUG ] Running command: /usr/bin/ceph-authtool > > > --gen-print-key > > > [ank-ceph10][DEBUG ] Running command: /usr/bin/ceph --cluster ceph --name > > > client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring > > > -i - > > > osd new 2d203f03-e547-4a8a-9140-53f48ed52e06 > > > [ank-ceph10][DEBUG ] Running command: vgcreate --force --yes > > > ceph-35465726-457d-439d-9f59-a8a050f5a486 /dev/sdl > > > [ank-ceph10][DEBUG ] stderr: /run/lvm/lvmetad.socket: connect failed: No > > > such file or directory > > > [ank-ceph10][DEBUG ] WARNING: Failed to connect to lvmetad. Falling back > > > to internal scanning. > > > [ank-ceph10][DEBUG ] stdout: Physical volume "/dev/sdl" successfully > > > created > > > [ank-ceph10][DEBUG ] stdout: Volume group > > > "ceph-35465726-457d-439d-9f59-a8a050f5a486" successfully created > > > [ank-ceph10][DEBUG ] Running command: lvcreate --yes -l 100%FREE -n > > > osd-data-2d203f03-e547-4a8a-9140-53f48ed52e06 > > > ceph-35465726-457d-439d-9f59-a8a050f5a486 > > > [ank-ceph10][DEBUG ] stderr: /run/lvm/lvmetad.socket: connect failed: No > > > such file or directory > > > [ank-ceph10][DEBUG ] WARNING: Failed to connect to lvmetad. Falling back > > > to internal scanning. > > > [ank-ceph10][DEBUG ] stdout: Logical volume > > > "osd-data-2d203f03-e547-4a8a-9140-53f48ed52e06" created. > > > [ank-ceph10][DEBUG ] --> blkid could not detect a PARTUUID for device: > > > /dev/nvme0n1 > > > [ank-ceph10][DEBUG ] --> Was unable to complete a new OSD, will rollback > > > changes > > > [ank-ceph10][DEBUG ] --> OSD will be fully purged from the cluster, > > > because the ID was generated > > > [ank-ceph10][DEBUG ] Running command: ceph osd purge osd.119 > > > --yes-i-really-mean-it > > > [ank-ceph10][DEBUG ] stderr: purged osd.119 > > > [ank-ceph10][ERROR ] RuntimeError: command returned non-zero exit status: > > > 1 > > > [ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-volume > > > --cluster ceph lvm create --filestore --data /dev/sdl --journal > > > /dev/nvme0n1 > > > [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs > > > > > > AFAIU, blkid looks for PARTUUID for the journal device, but it does not > > > have one. Only partitions have. I do not want to format my journal device. > > > Any recommendation about this? > > > > > > root@ank-ceph10:~# blkid /dev/nvme0n1* > > > /dev/nvme0n1: PTUUID="a6431404-5693-4076-98c9-ffbe84224e1b" PTTYPE="gpt" > > > /dev/nvme0n1p1: PARTLABEL="ceph journal" > > > PARTUUID="43440fab-a30f-4e42-9c15-35f375dde033" > > > /dev/nvme0n1p10: PARTLABEL="ceph journal" > > > PARTUUID="c9c9f459-98a1-4a6a-9350-9942a6fc02f6" > > > /dev/nvme0n1p11: PARTLABEL="ceph journal" > > > PARTUUID="3f64ddc1-ac5d-4b7b-ace3-ad35d44e4fd3" > > > /dev/nvme0n1p12: PARTLABEL="ceph journal" > > > PARTUUID="0fdff4d6-2833-4e6e-a832-9fb2452bc396" > > > /dev/nvme0n1p2: PARTLABEL="ceph journal" > > > PARTUUID="5ce0b4e8-3571-4297-974a-9ef648fac1a8" > > > /dev/nvme0n1p3: PARTLABEL="ceph journal" > > > PARTUUID="228cee11-06e3-4691-963a-77e74e099716" > > > /dev/nvme0n1p4: PARTLABEL="ceph journal" > > > PARTUUID="b7c09c3e-e4ae-42be-8686-5daf9e40c407" > > > /dev/nvme0n1p5: PARTLABEL="ceph journal" > > > PARTUUID="60d9115c-ebb1-4eaf-85ae-31379a5e9450" > > > /dev/nvme0n1p6: PARTLABEL="ceph journal" > > > PARTUUID="5a057b30-b697-4598-84c0-1794c608d70c" > > > /dev/nvme0n1p7: PARTLABEL="ceph journal" > > > PARTUUID="c22c272d-5b75-40ca-970e-87b1b303944c" > > > /dev/nvme0n1p8: PARTLABEL="ceph journal" > > > PARTUUID="ed9fd194-1490-42b1-a2b4-ae36b2a4f8ce" > > > /dev/nvme0n1p9: PARTLABEL="ceph journal" > > > PARTUUID="d5589315-4e47-49c4-91f5-48e1a55011d2" > > > > > > While using kraken, I used to add OSDs with journals via the follwing > > > command: > > > > > > ceph-deploy osd prepare ank-ceph10:sdl:/dev/nvme0n1 > > > > > > > > > Thanks for any recommendation. > > > > > > Best regards, > > > > > > Dr. Huseyin COTUK > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > -- > > Paul Emmerich > > > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > > > croit GmbH > > Freseniusstr. 31h > > 81247 München > > www.croit.io > > Tel: +49 89 1896585 90 > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
