On Wed, Jul 11, 2018 at 3:33 PM, Huseyin Cotuk <hco...@gmail.com> wrote: > Dear Paul and Alfredo, > > Downgrading to ceph-deploy 1.5.38 did not work either. I labeled the journal > partition (i.e. /dev/nvme0n1p12) with parted, and it added a gpt partuuid to > this specific partition: > > /dev/nvme0n1p12: PTUUID="8a775205-1364-43d9-820e-c4d3a0d9f9e3" PTTYPE="gpt" > PARTLABEL="ceph journal" PARTUUID="0fdff4d6-2833-4e6e-a832-9fb2452bc396” > > Then I retried with ceph-deploy 2.0 with the following command and it > succeeded. Before labeling as gpt with parted, it failed. > > ceph-deploy osd create --filestore --journal /dev/nvme0n1p12 --data /dev/sdl > ank-ceph10 > > If anyone encounters a problem like this, this solution worked for me. FYI.
Right, with ceph-deploy version 2.0.X that will work perfectly, because it meets the requirements of ceph-volume. I was assuming you were wanting to keep deploying with ceph-disk. > > My best, > > Huseyin > On 11 Jul 2018 20:31 +0300, Alfredo Deza <ad...@redhat.com>, wrote: > > On Wed, Jul 11, 2018 at 12:57 PM, Huseyin Cotuk <hco...@gmail.com> wrote: > > Hi Paul, > > Thanks for your reply. I did not mention any special parameter while > upgrading to luminous. So this ceph-deploy version is the one coming from > the official debian luminous repo. > > > That is because ceph-volume came out in Luminous and fully supports > most everything ceph-disk provided. > > Another alternative is to allow ceph-volume to take over the > management of your ceph-disk OSDs, although that is not provided via > ceph-deploy. > > In the meantime, you can still use ceph-disk in Luminous, but like > Paul mentioned it will require you to downgrade ceph-deploy to 1.5.39 > > > I will try to downgrade ceph-deploy and try to add osd again. To prevent any > inconsistency, maybe you can check the version of ceph-deploy package on the > repository. > > My best, > Huseyin > On 11 Tem 2018 19:49 +0300, Paul Emmerich <paul.emmer...@croit.io>, wrote: > > BlueStore is really stable and mature nowadays. > > You seem to be using ceph-deploy 2.0.0 which I would not call mature and > stable at the moment ;) > Anyways, it uses ceph-volume instead of ceph-disk and I think you have to > specify the actual partition here. > But I'd just downgrade to ceph-deploy 1.5.39 when running Luminous (not a > long-term solution as ceph-disk will > be removed in Nautilus) > > Paul > > 2018-07-11 18:37 GMT+02:00 Huseyin Cotuk <hco...@gmail.com>: > > > Hello everybody, > > I have just upgraded my ceph cluster from kraken to luminous. I just want > to go on with filestore based objectstore for OSDs until Redhat announces > bluestore as stable. It is still in technical preview. > > So my question is: “What is the right procedure of adding an filestore > based OSD into the existing cluster with an NVME journal?" > > My NVME journal contains ceph journal partitions for existing OSD created > with kraken. > > root@ank-ceph10:~# lsblk /dev/nvme0n1 > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > nvme0n1 259:0 0 2.9T 0 disk > |-nvme0n1p1 259:1 0 40G 0 part > |-nvme0n1p2 259:2 0 40G 0 part > |-nvme0n1p3 259:3 0 40G 0 part > |-nvme0n1p4 259:4 0 40G 0 part > |-nvme0n1p5 259:5 0 40G 0 part > |-nvme0n1p6 259:6 0 40G 0 part > |-nvme0n1p7 259:7 0 40G 0 part > |-nvme0n1p8 259:8 0 40G 0 part > |-nvme0n1p9 259:9 0 40G 0 part > |-nvme0n1p10 259:10 0 40G 0 part > |-nvme0n1p11 259:11 0 40G 0 part > `-nvme0n1p12 259:12 0 40G 0 part > > When i try to add a new osd with the following command, > > ceph-deploy osd create --filestore --journal /dev/nvme0n1 --data /dev/sdl > ank-ceph10 > > I get the following error: > > [ank-ceph10][INFO ] Running command: sudo /usr/sbin/ceph-volume --cluster > ceph lvm create --filestore --data /dev/sdl --journal /dev/nvme0n1 > [ank-ceph10][WARNIN] --> RuntimeError: unable to use device > [ank-ceph10][DEBUG ] Running command: /usr/bin/ceph-authtool > --gen-print-key > [ank-ceph10][DEBUG ] Running command: /usr/bin/ceph --cluster ceph --name > client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - > osd new 2d203f03-e547-4a8a-9140-53f48ed52e06 > [ank-ceph10][DEBUG ] Running command: vgcreate --force --yes > ceph-35465726-457d-439d-9f59-a8a050f5a486 /dev/sdl > [ank-ceph10][DEBUG ] stderr: /run/lvm/lvmetad.socket: connect failed: No > such file or directory > [ank-ceph10][DEBUG ] WARNING: Failed to connect to lvmetad. Falling back > to internal scanning. > [ank-ceph10][DEBUG ] stdout: Physical volume "/dev/sdl" successfully > created > [ank-ceph10][DEBUG ] stdout: Volume group > "ceph-35465726-457d-439d-9f59-a8a050f5a486" successfully created > [ank-ceph10][DEBUG ] Running command: lvcreate --yes -l 100%FREE -n > osd-data-2d203f03-e547-4a8a-9140-53f48ed52e06 > ceph-35465726-457d-439d-9f59-a8a050f5a486 > [ank-ceph10][DEBUG ] stderr: /run/lvm/lvmetad.socket: connect failed: No > such file or directory > [ank-ceph10][DEBUG ] WARNING: Failed to connect to lvmetad. Falling back > to internal scanning. > [ank-ceph10][DEBUG ] stdout: Logical volume > "osd-data-2d203f03-e547-4a8a-9140-53f48ed52e06" created. > [ank-ceph10][DEBUG ] --> blkid could not detect a PARTUUID for device: > /dev/nvme0n1 > [ank-ceph10][DEBUG ] --> Was unable to complete a new OSD, will rollback > changes > [ank-ceph10][DEBUG ] --> OSD will be fully purged from the cluster, > because the ID was generated > [ank-ceph10][DEBUG ] Running command: ceph osd purge osd.119 > --yes-i-really-mean-it > [ank-ceph10][DEBUG ] stderr: purged osd.119 > [ank-ceph10][ERROR ] RuntimeError: command returned non-zero exit status: > 1 > [ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-volume > --cluster ceph lvm create --filestore --data /dev/sdl --journal /dev/nvme0n1 > [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs > > AFAIU, blkid looks for PARTUUID for the journal device, but it does not > have one. Only partitions have. I do not want to format my journal device. > Any recommendation about this? > > root@ank-ceph10:~# blkid /dev/nvme0n1* > /dev/nvme0n1: PTUUID="a6431404-5693-4076-98c9-ffbe84224e1b" PTTYPE="gpt" > /dev/nvme0n1p1: PARTLABEL="ceph journal" > PARTUUID="43440fab-a30f-4e42-9c15-35f375dde033" > /dev/nvme0n1p10: PARTLABEL="ceph journal" > PARTUUID="c9c9f459-98a1-4a6a-9350-9942a6fc02f6" > /dev/nvme0n1p11: PARTLABEL="ceph journal" > PARTUUID="3f64ddc1-ac5d-4b7b-ace3-ad35d44e4fd3" > /dev/nvme0n1p12: PARTLABEL="ceph journal" > PARTUUID="0fdff4d6-2833-4e6e-a832-9fb2452bc396" > /dev/nvme0n1p2: PARTLABEL="ceph journal" > PARTUUID="5ce0b4e8-3571-4297-974a-9ef648fac1a8" > /dev/nvme0n1p3: PARTLABEL="ceph journal" > PARTUUID="228cee11-06e3-4691-963a-77e74e099716" > /dev/nvme0n1p4: PARTLABEL="ceph journal" > PARTUUID="b7c09c3e-e4ae-42be-8686-5daf9e40c407" > /dev/nvme0n1p5: PARTLABEL="ceph journal" > PARTUUID="60d9115c-ebb1-4eaf-85ae-31379a5e9450" > /dev/nvme0n1p6: PARTLABEL="ceph journal" > PARTUUID="5a057b30-b697-4598-84c0-1794c608d70c" > /dev/nvme0n1p7: PARTLABEL="ceph journal" > PARTUUID="c22c272d-5b75-40ca-970e-87b1b303944c" > /dev/nvme0n1p8: PARTLABEL="ceph journal" > PARTUUID="ed9fd194-1490-42b1-a2b4-ae36b2a4f8ce" > /dev/nvme0n1p9: PARTLABEL="ceph journal" > PARTUUID="d5589315-4e47-49c4-91f5-48e1a55011d2" > > While using kraken, I used to add OSDs with journals via the follwing > command: > > ceph-deploy osd prepare ank-ceph10:sdl:/dev/nvme0n1 > > > Thanks for any recommendation. > > Best regards, > > Dr. Huseyin COTUK > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
