I am also getting inconsistent object errors on a regular basis, about
1-2 every week or so for about 300GB of data. All OSDs are using XFS
filesystems. Some OSDs are individual 3TB internal hard drives and some
are external FC attached raid6 arrays. I am using this cluster to store
kvm images and I've noticed that the inconsistent objects always occur
on my two most recently created VM images, even though one of them is
hardly ever used (just a bare VM not put into production yet). This all
started about 4 months ago on 0.72 and now is continuing to occur on
version .80. I also changed the number of replicas from 2 to 3 for the
pool containing these images and that had no effect.
Here is an example log entry:
2014-06-24 18:11:51.683310 7faf44297700 0 log [ERR] : 4.b6 shard 0:
soid c539a8b6/rbd_data.9fdea2ae8944a.00000000000004e2/head//4 digest
2541762784 != known digest 3305022936
2014-06-24 18:11:52.107321 7faf50f60700 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) set_extsize: FSSETXATTR:
(22) Invalid argument
2014-06-24 18:11:52.215752 7faf5075f700 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) set_extsize: FSSETXATTR:
(22) Invalid argument
2014-06-24 18:11:52.365798 7faf50f60700 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) set_extsize: FSSETXATTR:
(22) Invalid argument
2014-06-24 18:11:52.674643 7faf5075f700 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) set_extsize: FSSETXATTR:
(22) Invalid argument
2014-06-24 18:11:52.749641 7faf50f60700 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) set_extsize: FSSETXATTR:
(22) Invalid argument
2014-06-24 18:11:55.194967 7faf5075f700 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) set_extsize: FSSETXATTR:
(22) Invalid argument
2014-06-24 18:11:55.259322 7faf50f60700 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) set_extsize: FSSETXATTR:
(22) Invalid argument
2014-06-24 18:11:55.526157 7faf5075f700 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) set_extsize: FSSETXATTR:
(22) Invalid argument
2014-06-24 18:11:55.547270 7faf44297700 0 log [ERR] : 4.b6 deep-scrub 0
missing, 1 inconsistent objects
2014-06-24 18:11:55.547282 7faf44297700 0 log [ERR] : 4.b6 deep-scrub 1
errors
Sometimes one of the objects has 0 size. I've also started getting the
FSSETXATTR errors recently, though I think that started after this
problem started. I've read elsewhere that these are harmless and will go
away in a future version. I also looked in the monitor logs but didn't
see any reference to inconsistent or scrubbed objects.
Kevin
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
