I have a new cluster and I'd like to put the DB on the NVMe device, but
only make it 30GB, then use 100GB of the rest of the NVMe as an OSD for the
RGW metadata pool.
I set up the disks like the conf below without the block_db_size and it
created all the LVs on the HDDs and one LV on the NVMe that took up all the
space.
I've tried using block_db_size in vars, and also as a property in the list
for each OSD disk but neither work.
With block_db_size in the vars I get:
failed: [sun-gcs02-osd01] (item={u'db': u'/dev/nvme0n1', u'data':
u'/dev/sda', u'crush_device_class': u'hdd'}) => changed=true
ansible_loop_var: item
cmd:
- docker
- run
- --rm
- --privileged
- --net=host
- --ipc=host
- --ulimit
- nofile=1024:1024
- -v
- /run/lock/lvm:/run/lock/lvm:z
- -v
- /var/run/udev/:/var/run/udev/:z
- -v
- /dev:/dev
- -v
- /etc/ceph:/etc/ceph:z
- -v
- /run/lvm/:/run/lvm/
- -v
- /var/lib/ceph/:/var/lib/ceph/:z
- -v
- /var/log/ceph/:/var/log/ceph/:z
- --entrypoint=ceph-volume
- docker.io/ceph/daemon:latest
- --cluster
- ceph
- lvm
- prepare
- --bluestore
- --data
- /dev/sda
- --block.db
- /dev/nvme0n1
- --crush-device-class
- hdd
delta: '0:00:05.004777'
end: '2019-08-28 23:26:39.074850'
item:
crush_device_class: hdd
data: /dev/sda
db: /dev/nvme0n1
msg: non-zero return code
rc: 1
start: '2019-08-28 23:26:34.070073'
stderr: '--> RuntimeError: unable to use device'
stderr_lines: <omitted>
stdout: |-
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd
--keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
bcc7b3c3-6203-47c7-9f34-7b2e2060bf59
Running command: /usr/sbin/vgcreate -s 1G --force --yes
ceph-76cd6a80-17dd-4a89-a35b-0844026bc9d4 /dev/sda
stdout: Physical volume "/dev/sda" successfully created.
stdout: Volume group "ceph-76cd6a80-17dd-4a89-a35b-0844026bc9d4"
successfully created
Running command: /usr/sbin/lvcreate --yes -l 100%FREE -n
osd-block-bcc7b3c3-6203-47c7-9f34-7b2e2060bf59
ceph-76cd6a80-17dd-4a89-a35b-0844026bc9d4
stdout: Logical volume "osd-block-bcc7b3c3-6203-47c7-9f34-7b2e2060bf59"
created.
--> blkid could not detect a PARTUUID for device: /dev/nvme0n1
--> Was unable to complete a new OSD, will rollback changes
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd
--keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.21
--yes-i-really-mean-it
stderr: purged osd.21
stdout_lines: <omitted>
... (One for each device) ...
And the LVs are created for all the HDD OSDs and none on the NVMe.
Looking through the code I don't see a way to set a size for the OSD, but
maybe I'm just missing it as I'm really new to Ansible.
osds:
hosts:
sun-gcs02-osd[01:43]:
sun-gcs02-osd[45:60]:
vars:
block_db_size: 32212254720
lvm_volumes:
- data: '/dev/sda'
db: '/dev/nvme0n1'
crush_device_class: 'hdd'
- data: '/dev/sdb'
db: '/dev/nvme0n1'
crush_device_class: 'hdd'
- data: '/dev/sdc'
db: '/dev/nvme0n1'
crush_device_class: 'hdd'
- data: '/dev/sdd'
db: '/dev/nvme0n1'
crush_device_class: 'hdd'
- data: '/dev/sde'
db: '/dev/nvme0n1'
crush_device_class: 'hdd'
- data: '/dev/sdf'
db: '/dev/nvme0n1'
crush_device_class: 'hdd'
- data: '/dev/sdg'
db: '/dev/nvme0n1'
crush_device_class: 'hdd'
- data: '/dev/sdh'
db: '/dev/nvme0n1'
crush_device_class: 'hdd'
- data: '/dev/sdi'
db: '/dev/nvme0n1'
crush_device_class: 'hdd'
- data: '/dev/sdj'
db: '/dev/nvme0n1'
crush_device_class: 'hdd'
- data: '/dev/sdk'
db: '/dev/nvme0n1'
crush_device_class: 'hdd'
- data: '/dev/sdl'
db: '/dev/nvme0n1'
crush_device_class: 'hdd'
- data: '/dev/nvme0n1' # Use the rest for metadata
crush_device_class: 'nvme'
With block_db_size set for each disk, I got an error during the parameter
checking phase in Ansible and no LVs were created.
Please help me understand how to configure what I would like to do.
Thank you,
Robert LeBlanc
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
