Hi, I am seeing an issue on one of our older ceph clusters (mimic 13.2.1) in an
erasure coded pool on bluestore OSDs in which we are seeing 1 inconsistent pg
and 1 scrub error. It should be noted that we have an ongoing rebalance of
misplaced data that predates this issue which came from flapping OSDs due to
OSD_NEARFULL OSD_TOOFULL warnings/errors we corrected by removing some user
data from ceph's rgw/s3 api interface (users "s3 objects" where deleted via the
s3 api).
If anyone has any suggestions or guidance for dealing with this it would be
very much appreciated. I've included all the relevant / helpful information I
can think of below, if there is any additional information that you think would
be helpful to me or you in providing suggestions please let me know.
$ sudo ceph -s
cluster:
id: 6fa7ec72-79fb-4f45-8b9f-ea5cdc7ab18d
health: HEALTH_ERR
248317/437145405 objects misplaced (0.057%)
1 scrub errors
Possible data damage: 1 pg inconsistent
services:
mon: 3 daemons, quorum HW-CEPHM-AT01,HW-CEPHM-AT02,HW-CEPHM-AT03
mgr: HW-CEPHM-AT02(active)
osd: 109 osds: 107 up, 106 in; 2 remapped pgs
rgw: 3 daemons active
data:
pools: 10 pools, 1380 pgs
objects: 54.70 M objects, 68 TiB
usage: 116 TiB used, 169 TiB / 285 TiB avail
pgs: 248317/437145405 objects misplaced (0.057%)
1374 active+clean
3 active+clean+scrubbing+deep
2 active+remapped+backfilling
1 active+clean+inconsistent
io:
client: 28 KiB/s rd, 306 KiB/s wr, 26 op/s rd, 30 op/s wr
recovery: 6.2 MiB/s, 4 objects/s
$ sudo ceph health detail
HEALTH_ERR 247241/437143405 objects misplaced (0.057%); 1 scrub errors;
Possible data damage: 1 pg inconsistent
OBJECT_MISPLACED 247241/437143405 objects misplaced (0.057%)
OSD_SCRUB_ERRORS 1 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
pg 7.1 is active+clean+inconsistent, acting [2,57,51,15,20,28,9,39]
Examination of osd logs shows the error is in osd.2
zgrep -Hn 'ERR' ceph-osd.2.log-20200614.gz
ceph-osd.2.log-20200614.gz:1292:2020-06-14 03:31:06.572 7f94591a9700 -1
log_channel(cluster) log [ERR] : 7.1s0 deep-scrub stat mismatch, got
213029/213030 objects, 0/0 clones, 213029/213030 dirty, 0/0 omap, 0/0 pinned,
0/0 hit_set_archive, 0/0 whiteouts, 292308615921/292308670959 bytes, 0/0
manifest objects, 0/0 hit_set_archive bytes.
ceph-osd.2.log-20200614.gz:1293:2020-06-14 03:31:06.572 7f94591a9700 -1
log_channel(cluster) log [ERR] : 7.1 deep-scrub 1 errors
All other OSDs appear to be clean of errors
The pg in question (7.1) has been instructed to repair/scrub/deep-scrub but I
do not see any indication in it's logs that it has done a scrub or repair (it
does log a deep-scrub which comes back OK) and listing inconsistent objects
seems to indicate no issues
$ sudo rados list-inconsistent-pg default.rgw.buckets.data
["7.1"]
$ sudo ceph pg repair 7.1
instructing pg 7.1s0 on osd.2 to repair
$ sudo ceph pg scrub 7.1
instructing pg 7.1s0 on osd.2 to scrub
$ sudo ceph pg deep-scrub 7.1
instructing pg 7.1s0 on osd.2 to deep-scrub
grep -HnEi 'scrub|repair|deep-scrub' ceph-osd.2.log
ceph-osd.2.log:118:2020-06-14 07:28:10.139 7f94599aa700 0
log_channel(cluster) log [DBG] : 7.91 deep-scrub starts
ceph-osd.2.log:177:2020-06-14 08:39:11.404 7f94599aa700 0
log_channel(cluster) log [DBG] : 7.91 deep-scrub ok
ceph-osd.2.log:322:2020-06-14 12:17:31.405 7f94579a6700 0
log_channel(cluster) log [DBG] : 13.135 deep-scrub starts
ceph-osd.2.log:323:2020-06-14 12:17:32.744 7f94579a6700 0
log_channel(cluster) log [DBG] : 13.135 deep-scrub ok
ceph-osd.2.log:387:2020-06-14 13:40:35.941 7f94591a9700 0
log_channel(cluster) log [DBG] : 7.d8 deep-scrub starts
ceph-osd.2.log:441:2020-06-14 14:49:06.111 7f94591a9700 0
log_channel(cluster) log [DBG] : 7.d8 deep-scrub ok
Only the last deep-scrub was manually triggered
$ sudo rados list-inconsistent-obj 7.1 --format=json-pretty
{
"epoch": 30869,
"inconsistents": []
}
$ sudo rados list-inconsistent-obj 7.1s0 --format=json-pretty
{
"epoch": 30869,
"inconsistents": []
}
I'm not sure why no inconsistents (empty set) are reported in the above
Chris Shultz
Global Systems Architect
1 Stiles Road
Suite 202
SalemNH03079
United States
[email protected]
(m) 774.270.2679
korewireless.com
Disclaimer
The information contained in this communication from the sender is
confidential. It is intended solely for use by the recipient and others
authorized to receive it. If you are not the recipient, you are hereby notified
that any disclosure, copying, distribution or taking action in relation of the
contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been
automatically archived by Mimecast Ltd, an innovator in Software as a Service
(SaaS) for business. Providing a safer and more useful place for your human
generated data. Specializing in; Security, archiving and compliance. To find
out more visit the Mimecast website.
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
