Hi, we're trying to give ceph a try on our compute cluster Initial stress tests passed without problems, but over the weekend a couple of cosd processes died and now access to the ceph mount point blocks and mounting the ceph dir fails with
mount: 192.168.1.141:6789,192.168.1.145:6789,192.168.1.150:6789:/:
can't read superblock
Attempts to restart the cosd on the affected storage nodes fails with
# /usr/local/bin/cosd -f -i 6 -c /etc/ceph/ceph.conf
** WARNING: Ceph is still under heavy development, and is only
suitable for **
** testing and review. Do not trust it with important data.
**
starting osd6 at 0.0.0.0:6800/2685 osd_data /var/ceph/osd6
/var/ceph/osd6/journal
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted
Stracing the cosd process shows that it calls mmap() with silly values
for the "fd" and the "length" parameter:
mmap(NULL, 18446744073709436928, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
I briefly looked at the source code and noticed that raw_mmap_pages()
in include/buffer.h of seems to call mmap() with an unsigned int
rather than with a size_t as the second (length) parameter. Since
18446744073709436928 = 0xfffffffffffe4000
this looks like an integer overflow. But maybe it is just uninitialized
garbage.
I've tried the v0.20.2 and the testing branch of the ceph git
repo. Both versions of cosd show the same behaviour.
Our ceph file system 5.5T large, we have 7 cosds, 3 cmons and 3 cmds,
see the ceph.conf below for details.
Any idea how to get back the data? If you need further debugging info,
don't hesitate to ask.
Thanks
Andre
---
[global]
; enable secure authentication
auth supported = cephx
osd journal size = 100 ; measured in MB
; You need at least one monitor. You need at least three if you want to
; tolerate any node failures. Always create an odd number.
[mon]
mon data = /var/ceph/mon$id
; some minimal logging (just message traffic) to aid debugging
debug ms = 1
[mon0]
host = node141
mon addr = 192.168.1.141:6789
[mon1]
host = node145
mon addr = 192.168.1.145:6789
[mon2]
host = node150
mon addr = 192.168.1.150:6789
; You need at least one mds. Define two to get a standby.
[mds]
; where the mds keeps it's secret encryption keys
keyring = /var/ceph/keyring.$name
[mds0]
host = node141
[mds1]
host = node145
[mds2]
host = node150
; osd
; You need at least one. Two if you want data to be replicated.
; Define as many as you like.
[osd]
; This is where the btrfs volume will be mounted.
osd data = /var/ceph/osd$id
; Ideally, make this a separate disk or partition. A few GB
; is usually enough; more if you have fast disks. You can use
; a file under the osd data dir if need be
; (e.g. /data/osd$id/journal), but it will be slower than a
; separate disk or partition.
osd journal = /var/ceph/osd$id/journal
[osd0]
host = node141
[osd1]
host = node145
[osd2]
host = node150
[osd3]
host = node146
[osd4]
host = node147
[osd5]
host = node149
[osd6]
host = node142
--
The only person who always got his work done by Friday was Robinson Crusoe
signature.asc
Description: Digital signature
