Incomplete means "Ceph detects that a placement group is missing a
necessary period of history from its log. If you see this state, report a
bug, and try to start any failed OSDs that may contain the needed
information".
In the PG query, it lists some OSDs that it's trying to probe:
"probing_osds": [
"10",
"13",
"15",
"25"],
"down_osds_we_would_probe": [],
Is one of those the OSD you replaced? If so, you might try ceph pg {pg-id}
mark_unfound_lost revert|delete. That command will lose data; it tells
Ceph to give up looking for data that it can't find, so you might want to
wait a bit.
There's also the possibility that your crushmap is still not correct. In
the history, I can see that you had bad CRUSH MAPs in the past. Stuff like
"recovery_state": [
{ "name": "Started\/Primary\/Peering",
"enter_time": "2014-10-21 12:18:48.482666",
"past_intervals": [
{ "first": 4663,
"last": 4685,
"maybe_went_rw": 1,
"up": [],
"acting": [
10,
25,
10,
-1]},
shows that CRUSH placed some data on osd.10 twice, which is a sign of a
bad crushmap. You might run through the crushtool testing at
http://ceph.com/docs/master/man/8/crushtool/, just to make sure everything
is kosher.
On Tue, Oct 21, 2014 at 7:04 PM, Chris Kitzmiller <[email protected]
> wrote:
> I've gotten myself into the position of having ~100 incomplete PGs. All of
> my OSDs are up+in (and I've restarted them all one by one).
>
> I was in the process of rebalancing after altering my CRUSH map when I
> lost an OSD backing disk. I replaced that OSD and it seemed to be
> backfilling well. During this time I noticed that I had 2 underperforming
> disks which were holding up the backfilling process. I set them out to try
> and get everything recovered but *I think* this is what caused some of my
> PGs to go incomplete. Since then I set those two underperformers back in
> and they're still backfilling now.
>
> Any help would be appreciated in troubleshooting these PGs. I'm not sure
> why they're incomplete or what to do about it. A query of one of my
> incomplete PGs can be found here: http://pastebin.com/raw.php?i=AJ3RMjz6
>
> _______________________________________________
> ceph-users mailing list
> [email protected]
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
