Ok, so this is how it works with LVM.
I was playing around a bit with the following config:
---
dummy:
osd_scenario: lvm
crush_device_class: "nvme"
osds_per_device: 4
devices:
- /dev/sde
lvm_volumes:
- data: /dev/sdc
db: db_osd1
db_vg: journal
crush_device_class: "hdd"
- data: /dev/sdd
db: db_osd2
db_vg: journal
crush_device_class: "ssd"
Another thing came up in my mind, so looks like the devices option if we want
to use separate faster drive for db+wal is only possible with nvme drive based
on this scenario from ceph-ansible doc:
“Other provisioning strategies are possible, by mixing spinning and solid state
devices, for example:
osd_scenario: lvm
devices:
- /dev/sda
- /dev/sdb
- /dev/nvme0n1
Similar to the initial example, this would end up producing 2 OSDs, but data
would be placed on the slower spinning drives (/dev/sda, and /dev/sdb) and
journals would be placed on the faster solid state device /dev/nvme0n1.”
I haven’t really found anything in ceph ansible, how it determines that which
drive is faster. So If I don’t have nvme, but I have ssd, it can automatically
recognize, that hey, sda,sdb,sdc are hdds, sde is ssd and it will put the db on
SSD?
What I’d like to achieve, I have let’s say this hardware:
OS: 480GB SSD SFF x 2 RAID 1
--------------------------------------------
Journal: 3,84TB LFF SSD
-------------------------------------------
SSD OSD: 3,84TB LFF SSD x 3
-------------------------------------------
HDD OSD: 6TB HDD LFF x 6 + 10TB HDD LFF x2
I’d like to deploy it with ceph-ansible with the devices option to put the db
for the HDDs on the 3.84TB journal drive.
---
dummy:
osd_scenario: lvm
crush_device_class: "hdd"
osds_per_device: 1
devices:
- /dev/sdc
- /dev/sdd
- /dev/sde
- /dev/sdf
- /dev/sdg
- /dev/sdh
- /dev/sdi
- /dev/sdj
- /dev/sdk // this is the 3.84TB ssd
lvm_volumes: // these would be the 3x3.84TB SSD OSDs
- data: /dev/sdl
- data: /dev/sdm
- data: /dev/sdn
I don’t have the HW yet to test, so that’s why I’m curious could this work
without NVME?
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: [email protected]<mailto:[email protected]>
---------------------------------------------------
From: Robert LeBlanc <[email protected]>
Sent: Thursday, April 30, 2020 7:03 AM
To: Szabo, Istvan (Agoda) <[email protected]>
Cc: [email protected]
Subject: Re: [ceph-users] ceph-ansible question
Email received from outside the company. If in doubt don't click links nor open
attachments!
________________________________
Yes, but they are just LVs, so you can not create them or delete them easily so
that it returns the space to the VG for something else.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Tue, Apr 28, 2020 at 6:55 PM Szabo, Istvan (Agoda)
<[email protected]<mailto:[email protected]>> wrote:
Hi,
So actually you’ve created in the DB VG many lv for the OSDs? This is that I
want to avoid actually, because if some of the osds are not is use it is still
holding the space, isn’t it?
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: [email protected]<mailto:[email protected]>
---------------------------------------------------
From: Robert LeBlanc <[email protected]<mailto:[email protected]>>
Sent: Wednesday, April 29, 2020 3:55 AM
To: Szabo, Istvan (Agoda)
<[email protected]<mailto:[email protected]>>
Cc: [email protected]<mailto:[email protected]>
Subject: Re: [ceph-users] ceph-ansible question
Email received from outside the company. If in doubt don't click links nor open
attachments!
________________________________
I'm sure there is a simpler way, but I wanted DBs of a certain size and a data
OSD on the NVMe as well. I wrote a script to create all the VGs and LVs of the
sizes that I wanted then added this to my Ansible inventory (I prefer to have
as much config in the inventory rather than scattered throughout a bunch of
files, this is the osd group.
osds:
hosts:
osd[01:43]:
osd[45:60]:
vars:
block_db_size: 32212254720
lvm_volumes:
- data: 'blk-00'
data_vg: 'ceph-blk-00'
db: 'db-00'
db_vg: 'ceph-db-00'
crush_device_class: 'hdd'
- data: 'blk-01'
data_vg: 'ceph-blk-01'
db: 'db-01'
db_vg: 'ceph-db-00'
crush_device_class: 'hdd'
- data: 'blk-02'
data_vg: 'ceph-blk-02'
db: 'db-02'
db_vg: 'ceph-db-00'
crush_device_class: 'hdd'
- data: 'blk-03'
data_vg: 'ceph-blk-03'
db: 'db-03'
db_vg: 'ceph-db-00'
crush_device_class: 'hdd'
- data: 'blk-04'
data_vg: 'ceph-blk-04'
db: 'db-04'
db_vg: 'ceph-db-00'
crush_device_class: 'hdd'
- data: 'blk-05'
data_vg: 'ceph-blk-05'
db: 'db-05'
db_vg: 'ceph-db-00'
crush_device_class: 'hdd'
- data: 'blk-06'
data_vg: 'ceph-blk-06'
db: 'db-06'
db_vg: 'ceph-db-00'
crush_device_class: 'hdd'
- data: 'blk-07'
data_vg: 'ceph-blk-07'
db: 'db-07'
db_vg: 'ceph-db-00'
crush_device_class: 'hdd'
- data: 'blk-08'
data_vg: 'ceph-blk-08'
db: 'db-08'
db_vg: 'ceph-db-00'
crush_device_class: 'hdd'
- data: 'blk-09'
data_vg: 'ceph-blk-09'
db: 'db-09'
db_vg: 'ceph-db-00'
crush_device_class: 'hdd'
- data: 'blk-10'
data_vg: 'ceph-blk-10'
db: 'db-10'
db_vg: 'ceph-db-00'
crush_device_class: 'hdd'
- data: 'blk-11'
data_vg: 'ceph-blk-11'
db: 'db-11'
db_vg: 'ceph-db-00'
crush_device_class: 'hdd'
- data: 'blk-12' # Use the rest for metadata
data_vg: 'ceph-db-00'
crush_device_class: 'nvme'
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Tue, Apr 28, 2020 at 3:14 AM Szabo, Istvan (Agoda)
<[email protected]<mailto:[email protected]>> wrote:
Hi,
I've tried to create ceph luminous cluster for testing porpuses with
ceph-ansible on my 3 machines hyperv vms, but I've got the below error with the
following with the following osd configuration:
---
dummy:
osd_scenario: lvm
lvm_volumes:
- data: osd1lv
data_vg: osd1
db: journal_lv
db_vg: journal
wal: journal_lv
wal_vg: journal
- data: osd2lv
data_vg: osd2
db: journal_lv
db_vg: journal
wal: journal_lv
wal_vg: journal
I have 3 disks, 2 for osd 1 for journal, I'd like to use the journal (sdb) for
both (sdc sdd) device.
I've also tried just simply use:
lvm_volumes:
- data: /dev/sdc
db: /dev/sdb
wal: /dev/sdb
- data: /dev/sdd
db: /dev/sdb
wal: /dev/sdb
But this one complains about partuuid not found for sdb.
I've also tried to create with parted a gpt partition to have the partuuid, but
that time it complains about gpt flag.
My setup is a centos 7, ansible 2.6, checked out the stable-3.2 ceph-ansible.
The all.yml is like this:
---
dummy:
ceph_origin: repository
ceph_repository: community
ceph_stable_release: luminous
monitor_interface: eth0
public_network: 172.19.15.0/24<http://172.19.15.0/24>
cluster_network: 192.168.1.0/24<http://192.168.1.0/24>
Is there anybody can show me a working configuration for ceph-volume with wal
and db configuration?
Thank you in advance.
The ansible dump is here:
failed: [ceph3] (item={'data': 'osd2lv', 'data_vg': 'osd2', 'db': 'journal_lv',
'db_vg': 'journal', 'wal': 'journal_lv', 'wal_vg': 'journal'}) => {"changed":
true, "cmd": ["ceph-volume", "--cluster", "ceph", "lvm", "create",
"--bluestore", "--data", "osd2/osd2lv", "--block.db", "journal/journal_lv",
"--block.wal", "journal/journal_lv"], "delta": "0:00:06.029411", "end":
"2020-04-28 16:48:13.252098", "item": {"data": "osd2lv", "data_vg": "osd2",
"db": "journal_lv", "db_vg": "journal", "wal": "journal_lv", "wal_vg":
"journal"}, "msg": "non-zero return code", "rc": 1, "start": "2020-04-28
16:48:07.222687", "stderr": "Running command: /bin/ceph-authtool
--gen-print-key\nRunning command: /bin/ceph --cluster ceph --name
client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i -
osd new 842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3\nRunning command:
/bin/ceph-authtool --gen-print-key\nRunning command: mount -t tmpfs tmpfs
/var/lib/ceph/osd/ceph-2\nRunning command: restorecon /var/l
ib/ceph/osd/ceph-2\nRunning command: chown -h ceph:ceph
/dev/osd2/osd2lv\nRunning command: chown -R ceph:ceph /dev/dm-4\nRunning
command: ln -s /dev/osd2/osd2lv /var/lib/ceph/osd/ceph-2/block\nRunning
command: ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/osd/ceph-2/activate.monmap\n stderr: got monmap epoch 1\nRunning
command: ceph-authtool /var/lib/ceph/osd/ceph-2/keyring --create-keyring --name
osd.2 --add-key AQDX+6de1vEIOhAA8Fy22wWp7Z0d/lnYWpgenA==\n stdout: creating
/var/lib/ceph/osd/ceph-2/keyring\nadded entity osd.2 auth auth(auid =
18446744073709551615 key=AQDX+6de1vEIOhAA8Fy22wWp7Z0d/lnYWpgenA== with 0
caps)\nRunning command: chown -R ceph:ceph
/var/lib/ceph/osd/ceph-2/keyring\nRunning command: chown -R ceph:ceph
/var/lib/ceph/osd/ceph-2/\nRunning command: chown -h ceph:ceph
/dev/journal/journal_lv\nRunning command: chown -R ceph:ceph /dev/dm-5\nRunning
command: chown -h ceph:ceph /dev/journal
/journal_lv\nRunning command: chown -R ceph:ceph /dev/dm-5\nRunning command:
/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap
/var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --bluestore-block-wal-path
/dev/journal/journal_lv --bluestore-block-db-path /dev/journal/journal_lv
--osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid
842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3 --setuser ceph --setgroup ceph\n stderr:
2020-04-28 16:48:11.895597 7fb89cb53ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/)
_read_fsid unparsable uuid\n stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t)
[with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread
7fb89cb53ec0 time 2020-04-28 16:48:12.908859\n stderr: /home/jenkins-build/b
uild/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
757: FAILED assert(available >= allocated)\n stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110)
[0x55f13e067ae0]\n stderr: 2: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]\n stderr: 3: (BlueFS::mount()+0x271)
[0x55f13dff3411]\n stderr: 4: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]\n stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]\n stderr: 6: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n
stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d,
int)+0x29b) [0x55f13dabc99b]\n stderr: 8: (main()+0x11a5) [0x55f13d9d4975]\n
stderr: 9: (__libc_start_main()+0xf5) [0x7fb899005505]\n
stderr: 10: (()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of the
executable, or `objdump -rdS <executable>` is needed to interpret this.\n
stderr: 2020-04-28 16:48:12.911181 7fb89cb53ec0 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t)
[with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread
7fb89cb53ec0 time 2020-04-28 16:48:12.908859\n stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
757: FAILED assert(available >= allocated)\n stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1: (ceph
::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110)
[0x55f13e067ae0]\n stderr: 2: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]\n stderr: 3: (BlueFS::mount()+0x271)
[0x55f13dff3411]\n stderr: 4: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]\n stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]\n stderr: 6: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n
stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d,
int)+0x29b) [0x55f13dabc99b]\n stderr: 8: (main()+0x11a5) [0x55f13d9d4975]\n
stderr: 9: (__libc_start_main()+0xf5) [0x7fb899005505]\n stderr: 10:
(()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.\n stderr: -253>
2020-04-28 16:48:11.895597 7fb89cb53ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/)
_read_fsid unparsable uuid\n stderr: 0> 2020-04-28 16:48:12.911181 7fb89cb53ec0
-1 /home/jenkins-build/build/work
space/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t)
[with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread
7fb89cb53ec0 time 2020-04-28 16:48:12.908859\n stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
757: FAILED assert(available >= allocated)\n stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110)
[0x55f13e067ae0]\n stderr: 2: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]\n stderr: 3: (BlueFS::mount()+0x271) [0x5
5f13dff3411]\n stderr: 4: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]\n stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]\n stderr: 6: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n
stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d,
int)+0x29b) [0x55f13dabc99b]\n stderr: 8: (main()+0x11a5) [0x55f13d9d4975]\n
stderr: 9: (__libc_start_main()+0xf5) [0x7fb899005505]\n stderr: 10:
(()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.\n stderr: *** Caught
signal (Aborted) **\n stderr: in thread 7fb89cb53ec0 thread_name:ceph-osd\n
stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e)
luminous (stable)\n stderr: 1: (()+0xa76e51) [0x55f13e028e51]\n stderr: 2:
(()+0xf5f0) [0x7fb899ff95f0]\n stderr: 3: (gsignal()+0x37) [0x7fb899019337]\n
stderr: 4: (abort()+0x148) [0x7fb89901aa28]\n stderr: 5:
(ceph::__ceph_assert_fail(char const*, char const*, int, c
har const*)+0x284) [0x55f13e067c54]\n stderr: 6:
(BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674)
[0x55f13e01ae54]\n stderr: 7: (BlueFS::mount()+0x271) [0x55f13dff3411]\n
stderr: 8: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 9:
(BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 10:
(BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 11:
(OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b)
[0x55f13dabc99b]\n stderr: 12: (main()+0x11a5) [0x55f13d9d4975]\n stderr: 13:
(__libc_start_main()+0xf5) [0x7fb899005505]\n stderr: 14: (()+0x4c4383)
[0x55f13da76383]\n stderr: 2020-04-28 16:48:12.914234 7fb89cb53ec0 -1 ***
Caught signal (Aborted) **\n stderr: in thread 7fb89cb53ec0
thread_name:ceph-osd\n stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1:
(()+0xa76e51) [0x55f13e028e51]\n stderr: 2: (()+0xf5f0) [0x7fb899ff95f0]\n
stderr: 3: (gsignal()+0x37) [0x7fb89901933
7]\n stderr: 4: (abort()+0x148) [0x7fb89901aa28]\n stderr: 5:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284)
[0x55f13e067c54]\n stderr: 6: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]\n stderr: 7: (BlueFS::mount()+0x271)
[0x55f13dff3411]\n stderr: 8: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]\n stderr: 9: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]\n stderr: 10: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n
stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d,
int)+0x29b) [0x55f13dabc99b]\n stderr: 12: (main()+0x11a5) [0x55f13d9d4975]\n
stderr: 13: (__libc_start_main()+0xf5) [0x7fb899005505]\n stderr: 14:
(()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.\n stderr: 0>
2020-04-28 16:48:12.914234 7fb89cb53ec0 -1 *** Caught signal (Aborted) **\n
stderr: in thread 7fb89cb53ec0 thread_name:ceph-o
sd\n stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e)
luminous (stable)\n stderr: 1: (()+0xa76e51) [0x55f13e028e51]\n stderr: 2:
(()+0xf5f0) [0x7fb899ff95f0]\n stderr: 3: (gsignal()+0x37) [0x7fb899019337]\n
stderr: 4: (abort()+0x148) [0x7fb89901aa28]\n stderr: 5:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284)
[0x55f13e067c54]\n stderr: 6: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]\n stderr: 7: (BlueFS::mount()+0x271)
[0x55f13dff3411]\n stderr: 8: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]\n stderr: 9: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]\n stderr: 10: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n
stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d,
int)+0x29b) [0x55f13dabc99b]\n stderr: 12: (main()+0x11a5) [0x55f13d9d4975]\n
stderr: 13: (__libc_start_main()+0xf5) [0x7fb899005505]\n stderr: 14:
(()+0x4c4383) [0x55f13da76383]\n stderr: NOTE:
a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.\n--> Was unable to complete a new OSD, will rollback
changes\n--> OSD will be fully purged from the cluster, because the ID was
generated\nRunning command: ceph osd purge osd.2 --yes-i-really-mean-it\n
stderr: purged osd.2\nTraceback (most recent call last):\n File
\"/sbin/ceph-volume\", line 9, in <module>\n
load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()\n
File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 38, in
__init__\n self.main(self.argv)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 59, in
newfunc\n return f(*a, **kw)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 149, in main\n
terminal.dispatch(self.mapper, subcommand_args)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 194, in
dispatch\n instance.main()\n File \"/usr/lib/python2.7/site-package
s/ceph_volume/devices/lvm/main.py\", line 40, in main\n
terminal.dispatch(self.mapper, self.argv)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 194, in
dispatch\n instance.main()\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py\", line
69, in main\n self.create(args)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in
is_root\n return func(*a, **kw)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py\", line
26, in create\n prepare_step.safe_prepare(args)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line
219, in safe_prepare\n self.prepare()\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in
is_root\n return func(*a, **kw)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line
320, in prepare\n osd_fsid,\n File \"/usr/lib/python2.7/site-packages/ce
ph_volume/devices/lvm/prepare.py\", line 119, in prepare_bluestore\n
db=db\n File \"/usr/lib/python2.7/site-packages/ceph_volume/util/prepare.py\",
line 397, in osd_mkfs_bluestore\n raise RuntimeError('Command failed with
exit code %s: %s' % (returncode, ' '.join(command)))\nRuntimeError: Command
failed with exit code -6: /bin/ceph-osd --cluster ceph --osd-objectstore
bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap
--keyfile - --bluestore-block-wal-path /dev/journal/journal_lv
--bluestore-block-db-path /dev/journal/journal_lv --osd-data
/var/lib/ceph/osd/ceph-2/ --osd-uuid 842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3
--setuser ceph --setgroup ceph", "stderr_lines": ["Running command:
/bin/ceph-authtool --gen-print-key", "Running command: /bin/ceph --cluster ceph
--name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
-i - osd new 842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3", "Running command:
/bin/ceph-authtool --gen-print-key", "Running comman
d: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2", "Running command:
restorecon /var/lib/ceph/osd/ceph-2", "Running command: chown -h ceph:ceph
/dev/osd2/osd2lv", "Running command: chown -R ceph:ceph /dev/dm-4", "Running
command: ln -s /dev/osd2/osd2lv /var/lib/ceph/osd/ceph-2/block", "Running
command: ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/osd/ceph-2/activate.monmap", " stderr: got monmap epoch 1",
"Running command: ceph-authtool /var/lib/ceph/osd/ceph-2/keyring
--create-keyring --name osd.2 --add-key
AQDX+6de1vEIOhAA8Fy22wWp7Z0d/lnYWpgenA==", " stdout: creating
/var/lib/ceph/osd/ceph-2/keyring", "added entity osd.2 auth auth(auid =
18446744073709551615 key=AQDX+6de1vEIOhAA8Fy22wWp7Z0d/lnYWpgenA== with 0
caps)", "Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring",
"Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/", "Running
command: chown -h ceph:ceph /dev/journal
/journal_lv", "Running command: chown -R ceph:ceph /dev/dm-5", "Running
command: chown -h ceph:ceph /dev/journal/journal_lv", "Running command: chown
-R ceph:ceph /dev/dm-5", "Running command: /bin/ceph-osd --cluster ceph
--osd-objectstore bluestore --mkfs -i 2 --monmap
/var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --bluestore-block-wal-path
/dev/journal/journal_lv --bluestore-block-db-path /dev/journal/journal_lv
--osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid
842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3 --setuser ceph --setgroup ceph", " stderr:
2020-04-28 16:48:11.895597 7fb89cb53ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/)
_read_fsid unparsable uuid", " stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t)
[with L1 = AllocatorLevel01L
oose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time 2020-04-28
16:48:12.908859", " stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
757: FAILED assert(available >= allocated)", " stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110)
[0x55f13e067ae0]", " stderr: 2: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 3: (BlueFS::mount()+0x271)
[0x55f13dff3411]", " stderr: 4: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]", " stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]", " stderr: 6: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]", "
stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid
_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 8: (main()+0x11a5)
[0x55f13d9d4975]", " stderr: 9: (__libc_start_main()+0xf5) [0x7fb899005505]", "
stderr: 10: (()+0x4c4383) [0x55f13da76383]", " stderr: NOTE: a copy of the
executable, or `objdump -rdS <executable>` is needed to interpret this.", "
stderr: 2020-04-28 16:48:12.911181 7fb89cb53ec0 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t)
[with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread
7fb89cb53ec0 time 2020-04-28 16:48:12.908859", " stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
757
: FAILED assert(available >= allocated)", " stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110)
[0x55f13e067ae0]", " stderr: 2: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 3: (BlueFS::mount()+0x271)
[0x55f13dff3411]", " stderr: 4: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]", " stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]", " stderr: 6: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]", "
stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d,
int)+0x29b) [0x55f13dabc99b]", " stderr: 8: (main()+0x11a5) [0x55f13d9d4975]",
" stderr: 9: (__libc_start_main()+0xf5) [0x7fb899005505]", " stderr: 10:
(()+0x4c4383) [0x55f13da76383]", " stderr: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.", " stderr: -253>
2020-04-28 16:48:
11.895597 7fb89cb53ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid
unparsable uuid", " stderr: 0> 2020-04-28 16:48:12.911181 7fb89cb53ec0 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t)
[with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread
7fb89cb53ec0 time 2020-04-28 16:48:12.908859", " stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
757: FAILED assert(available >= allocated)", " stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1:
(ceph::__ceph_assert_fail(char const*, char const
*, int, char const*)+0x110) [0x55f13e067ae0]", " stderr: 2:
(BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674)
[0x55f13e01ae54]", " stderr: 3: (BlueFS::mount()+0x271) [0x55f13dff3411]", "
stderr: 4: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]", " stderr: 5:
(BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]", " stderr: 6:
(BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]", " stderr: 7:
(OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b)
[0x55f13dabc99b]", " stderr: 8: (main()+0x11a5) [0x55f13d9d4975]", " stderr: 9:
(__libc_start_main()+0xf5) [0x7fb899005505]", " stderr: 10: (()+0x4c4383)
[0x55f13da76383]", " stderr: NOTE: a copy of the executable, or `objdump -rdS
<executable>` is needed to interpret this.", " stderr: *** Caught signal
(Aborted) **", " stderr: in thread 7fb89cb53ec0 thread_name:ceph-osd", "
stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e)
luminous (stable)", " stderr: 1: (()+0xa76e51) [0x55f13e02
8e51]", " stderr: 2: (()+0xf5f0) [0x7fb899ff95f0]", " stderr: 3:
(gsignal()+0x37) [0x7fb899019337]", " stderr: 4: (abort()+0x148)
[0x7fb89901aa28]", " stderr: 5: (ceph::__ceph_assert_fail(char const*, char
const*, int, char const*)+0x284) [0x55f13e067c54]", " stderr: 6:
(BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674)
[0x55f13e01ae54]", " stderr: 7: (BlueFS::mount()+0x271) [0x55f13dff3411]", "
stderr: 8: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]", " stderr: 9:
(BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]", " stderr: 10:
(BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]", " stderr: 11:
(OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b)
[0x55f13dabc99b]", " stderr: 12: (main()+0x11a5) [0x55f13d9d4975]", " stderr:
13: (__libc_start_main()+0xf5) [0x7fb899005505]", " stderr: 14: (()+0x4c4383)
[0x55f13da76383]", " stderr: 2020-04-28 16:48:12.914234 7fb89cb53ec0 -1 ***
Caught signal (Aborted) **", " stderr: in thread 7fb89cb53ec0 t
hread_name:ceph-osd", " stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1:
(()+0xa76e51) [0x55f13e028e51]", " stderr: 2: (()+0xf5f0) [0x7fb899ff95f0]", "
stderr: 3: (gsignal()+0x37) [0x7fb899019337]", " stderr: 4: (abort()+0x148)
[0x7fb89901aa28]", " stderr: 5: (ceph::__ceph_assert_fail(char const*, char
const*, int, char const*)+0x284) [0x55f13e067c54]", " stderr: 6:
(BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674)
[0x55f13e01ae54]", " stderr: 7: (BlueFS::mount()+0x271) [0x55f13dff3411]", "
stderr: 8: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]", " stderr: 9:
(BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]", " stderr: 10:
(BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]", " stderr: 11:
(OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b)
[0x55f13dabc99b]", " stderr: 12: (main()+0x11a5) [0x55f13d9d4975]", " stderr:
13: (__libc_start_main()+0xf5) [0x7fb899005505]", " stderr: 14:
(()+0x4c4383) [0x55f13da76383]", " stderr: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.", " stderr: 0>
2020-04-28 16:48:12.914234 7fb89cb53ec0 -1 *** Caught signal (Aborted) **", "
stderr: in thread 7fb89cb53ec0 thread_name:ceph-osd", " stderr: ceph version
12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", "
stderr: 1: (()+0xa76e51) [0x55f13e028e51]", " stderr: 2: (()+0xf5f0)
[0x7fb899ff95f0]", " stderr: 3: (gsignal()+0x37) [0x7fb899019337]", " stderr:
4: (abort()+0x148) [0x7fb89901aa28]", " stderr: 5:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284)
[0x55f13e067c54]", " stderr: 6: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 7: (BlueFS::mount()+0x271)
[0x55f13dff3411]", " stderr: 8: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]", " stderr: 9: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]", " stderr: 10: (BlueStore::mkfs()
+0xe5a) [0x55f13df0d3ba]", " stderr: 11: (OSD::mkfs(CephContext*,
ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", "
stderr: 12: (main()+0x11a5) [0x55f13d9d4975]", " stderr: 13:
(__libc_start_main()+0xf5) [0x7fb899005505]", " stderr: 14: (()+0x4c4383)
[0x55f13da76383]", " stderr: NOTE: a copy of the executable, or `objdump -rdS
<executable>` is needed to interpret this.", "--> Was unable to complete a new
OSD, will rollback changes", "--> OSD will be fully purged from the cluster,
because the ID was generated", "Running command: ceph osd purge osd.2
--yes-i-really-mean-it", " stderr: purged osd.2", "Traceback (most recent call
last):", " File \"/sbin/ceph-volume\", line 9, in <module>", "
load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()", "
File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 38, in
__init__", " self.main(self.argv)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line
59, in newfunc", " return f(*a, **kw)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 149, in main", "
terminal.dispatch(self.mapper, subcommand_args)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 194, in
dispatch", " instance.main()", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py\", line 40,
in main", " terminal.dispatch(self.mapper, self.argv)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 194, in
dispatch", " instance.main()", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py\", line
69, in main", " self.create(args)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in
is_root", " return func(*a, **kw)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py\", line
26, in create", " prepare_step.safe_prepare(args)", " File
\"/usr/lib/python2.7/site-packages/cep
h_volume/devices/lvm/prepare.py\", line 219, in safe_prepare", "
self.prepare()", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in
is_root", " return func(*a, **kw)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line
320, in prepare", " osd_fsid,", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line
119, in prepare_bluestore", " db=db", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/util/prepare.py\", line 397, in
osd_mkfs_bluestore", " raise RuntimeError('Command failed with exit code %s:
%s' % (returncode, ' '.join(command)))", "RuntimeError: Command failed with
exit code -6: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs
-i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile -
--bluestore-block-wal-path /dev/journal/journal_lv --bluestore-block-db-path
/dev/journal/journal_lv --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid 842c
bbe5-3b34-4ea4-95d4-b0c7cab5fce3 --setuser ceph --setgroup ceph"], "stdout":
"", "stdout_lines": []}
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e:
[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>
---------------------------------------------------
________________________________
This message is confidential and is for the sole use of the intended
recipient(s). It may also be privileged or otherwise protected by copyright or
other legal rules. If you have received it by mistake please let us know by
reply email and delete it from your system. It is prohibited to copy this
message or disclose its content to anyone. Any confidentiality or privilege is
not waived or lost by any mistaken delivery or unauthorized disclosure of the
message. All messages sent to and from Agoda may be monitored to ensure
compliance with company policies, to protect the company's interests and to
remove potential malware. Electronic messages may be intercepted, amended, lost
or deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- [email protected]<mailto:[email protected]>
To unsubscribe send an email to
[email protected]<mailto:[email protected]>
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
