Re: [ceph-users] Stuck down+peering after host failure.
Hi,
I found another possible cause for your problem:
http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/#placement-group-down-peering-failure
I hope that I helped,
Denes.
On 12/11/2017 03:43 PM, Denes Dolhay wrote:
Hi Aaron!
There is an previous post about safely shutting down and restarting a
cluster:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-April/017378.html
To the problems at hand:
What size were you using?
Ceph can only obey the failure domain if it knows exactly which osd is
on which host. So are you sure that there were no errors in the "ceph
osd tree"? Maybe an osd physically in the offline node was logically
placed to another host in the tree?
I think you should:
-Query a downed pg, determine its acting group "ceph pg map {pg-num}"
and compare it to the tree
-Try to make the offline host boot up, or if it was damaged then add
it's osds to another host
If this did not help, then please include a ceph health, ceph osd map,
ceph pg map {faulty pg-num}, ceph pg {faulty pg-num} query
I hope that I helped,
Denes.
On 12/11/2017 03:02 PM, Aaron Bassett wrote:
Morning All,
I have a large-ish (16 node, 1100 osds) cluster I recent had to move from one DC to another.
Before shutting everything down, I set noout, norecover, and nobackfill, thinking this would
help everything stand back up again. Upon installation at the new DC, one of the nodes refused
to boot. With my crush rule having the failure domain as host, I did not think this would be a
problem. However, once I turned off noout, norecover, and nobackfille, everything else came up
and settled in, I still have 1545 pgs stuck down+peering. On other pgs, recovery and
backfilling are proceeding as expected, but these pgs appear to be permanently stuck. When
querying the down+peering pgs, they all mention pgs from the down node in
""down_osds_we_would_probe". I'm not sure why it *needs* to query these since
it should have two other copies on other nodes? I'm not sure if bringing everything up with
noout or norecover on confused things. Looking for advice...
Aaron
CONFIDENTIALITY NOTICE
This e-mail message and any attachments are only for the use of the intended
recipient and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended
recipient, any disclosure, distribution or other use of this e-mail message or
attachments is prohibited. If you have received this e-mail message in error,
please delete and notify the sender immediately. Thank you.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Stuck down+peering after host failure.
Hi Aaron!
There is an previous post about safely shutting down and restarting a
cluster:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-April/017378.html
To the problems at hand:
What size were you using?
Ceph can only obey the failure domain if it knows exactly which osd is
on which host. So are you sure that there were no errors in the "ceph
osd tree"? Maybe an osd physically in the offline node was logically
placed to another host in the tree?
I think you should:
-Query a downed pg, determine its acting group "ceph pg map {pg-num}"
and compare it to the tree
-Try to make the offline host boot up, or if it was damaged then add
it's osds to another host
If this did not help, then please include a ceph health, ceph osd map,
ceph pg map {faulty pg-num}, ceph pg {faulty pg-num} query
I hope that I helped,
Denes.
On 12/11/2017 03:02 PM, Aaron Bassett wrote:
Morning All,
I have a large-ish (16 node, 1100 osds) cluster I recent had to move from one DC to another.
Before shutting everything down, I set noout, norecover, and nobackfill, thinking this would
help everything stand back up again. Upon installation at the new DC, one of the nodes refused
to boot. With my crush rule having the failure domain as host, I did not think this would be a
problem. However, once I turned off noout, norecover, and nobackfille, everything else came up
and settled in, I still have 1545 pgs stuck down+peering. On other pgs, recovery and
backfilling are proceeding as expected, but these pgs appear to be permanently stuck. When
querying the down+peering pgs, they all mention pgs from the down node in
""down_osds_we_would_probe". I'm not sure why it *needs* to query these since
it should have two other copies on other nodes? I'm not sure if bringing everything up with
noout or norecover on confused things. Looking for advice...
Aaron
CONFIDENTIALITY NOTICE
This e-mail message and any attachments are only for the use of the intended
recipient and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended
recipient, any disclosure, distribution or other use of this e-mail message or
attachments is prohibited. If you have received this e-mail message in error,
please delete and notify the sender immediately. Thank you.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Stuck down+peering after host failure.
Morning All, I have a large-ish (16 node, 1100 osds) cluster I recent had to move from one DC to another. Before shutting everything down, I set noout, norecover, and nobackfill, thinking this would help everything stand back up again. Upon installation at the new DC, one of the nodes refused to boot. With my crush rule having the failure domain as host, I did not think this would be a problem. However, once I turned off noout, norecover, and nobackfille, everything else came up and settled in, I still have 1545 pgs stuck down+peering. On other pgs, recovery and backfilling are proceeding as expected, but these pgs appear to be permanently stuck. When querying the down+peering pgs, they all mention pgs from the down node in ""down_osds_we_would_probe". I'm not sure why it *needs* to query these since it should have two other copies on other nodes? I'm not sure if bringing everything up with noout or norecover on confused things. Looking for advice... Aaron CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
