Re: [ceph-users] why sudden (and brief) HEALTH_ERR
ok, thanks for the feedback Piotr and Dan! MJ On 4-10-2017 9:38, Dan van der Ster wrote: Since Jewel (AFAIR), when (re)starting OSDs, pg status is reset to "never contacted", resulting in "pgs are stuck inactive for more than 300 seconds" being reported until osds regain connections between themselves. Also, the last_active state isn't updated very regularly, as far as I can tell. On our cluster I have increased this timeout --mon_pg_stuck_threshold: 1800 (Which helps suppress these bogus HEALTH_ERR's) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] why sudden (and brief) HEALTH_ERR
On Wed, Oct 4, 2017 at 9:08 AM, Piotr Dałek wrote:
> On 17-10-04 08:51 AM, lists wrote:
>>
>> Hi,
>>
>> Yesterday I chowned our /var/lib/ceph ceph, to completely finalize our
>> jewel migration, and noticed something interesting.
>>
>> After I brought back up the OSDs I just chowned, the system had some
>> recovery to do. During that recovery, the system went to HEALTH_ERR for a
>> short moment:
>>
>> See below, for consecutive ceph -s outputs:
>>
>>> [..]
>>> root@pm2:~# ceph -s
>>> cluster 1397f1dc-7d94-43ea-ab12-8f8792eee9c1
>>> health HEALTH_ERR
>>> 2 pgs are stuck inactive for more than 300 seconds
>
>
> ^^ that.
>
>>> 761 pgs degraded
>>> 2 pgs recovering
>>> 181 pgs recovery_wait
>>> 2 pgs stuck inactive
>>> 273 pgs stuck unclean
>>> 543 pgs undersized
>>> recovery 1394085/8384166 objects degraded (16.628%)
>>> 4/24 in osds are down
>>> noout flag(s) set
>>> monmap e3: 3 mons at
>>> {0=10.10.89.1:6789/0,1=10.10.89.2:6789/0,2=10.10.89.3:6789/0}
>>> election epoch 256, quorum 0,1,2 0,1,2
>>> osdmap e10230: 24 osds: 20 up, 24 in; 543 remapped pgs
>>> flags noout,sortbitwise,require_jewel_osds
>>> pgmap v36531146: 1088 pgs, 2 pools, 10703 GB data, 2729 kobjects
>>> 32724 GB used, 56656 GB / 89380 GB avail
>>> 1394085/8384166 objects degraded (16.628%)
>>> 543 active+undersized+degraded
>>> 310 active+clean
>>> 181 active+recovery_wait+degraded
>>> 26 active+degraded
>>> 13 active
>>>9 activating+degraded
>>>4 activating
>>>2 active+recovering+degraded
>>> recovery io 133 MB/s, 37 objects/s
>>> client io 64936 B/s rd, 9935 kB/s wr, 0 op/s rd, 942 op/s wr
>>> [..]
>>
>> It was only very briefly, but it did worry me a bit, fortunately, we went
>> back to the expected HEALTH_WARN very quickly, and everything finished fine,
>> so I guess nothing to worry.
>>
>> But I'm curious: can anyone explain WHY we got a brief HEALTH_ERR?
>>
>> No smart errors, apply and commit latency are all within the expected
>> ranges, the systems basically is healthy.
>>
>> Curious :-)
>
>
> Since Jewel (AFAIR), when (re)starting OSDs, pg status is reset to "never
> contacted", resulting in "pgs are stuck inactive for more than 300 seconds"
> being reported until osds regain connections between themselves.
>
Also, the last_active state isn't updated very regularly, as far as I can tell.
On our cluster I have increased this timeout
--mon_pg_stuck_threshold: 1800
(Which helps suppress these bogus HEALTH_ERR's)
-- dan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] why sudden (and brief) HEALTH_ERR
On 17-10-04 08:51 AM, lists wrote:
Hi,
Yesterday I chowned our /var/lib/ceph ceph, to completely finalize our jewel
migration, and noticed something interesting.
After I brought back up the OSDs I just chowned, the system had some
recovery to do. During that recovery, the system went to HEALTH_ERR for a
short moment:
See below, for consecutive ceph -s outputs:
[..]
root@pm2:~# ceph -s
cluster 1397f1dc-7d94-43ea-ab12-8f8792eee9c1
health HEALTH_ERR
2 pgs are stuck inactive for more than 300 seconds
^^ that.
761 pgs degraded
2 pgs recovering
181 pgs recovery_wait
2 pgs stuck inactive
273 pgs stuck unclean
543 pgs undersized
recovery 1394085/8384166 objects degraded (16.628%)
4/24 in osds are down
noout flag(s) set
monmap e3: 3 mons at
{0=10.10.89.1:6789/0,1=10.10.89.2:6789/0,2=10.10.89.3:6789/0}
election epoch 256, quorum 0,1,2 0,1,2
osdmap e10230: 24 osds: 20 up, 24 in; 543 remapped pgs
flags noout,sortbitwise,require_jewel_osds
pgmap v36531146: 1088 pgs, 2 pools, 10703 GB data, 2729 kobjects
32724 GB used, 56656 GB / 89380 GB avail
1394085/8384166 objects degraded (16.628%)
543 active+undersized+degraded
310 active+clean
181 active+recovery_wait+degraded
26 active+degraded
13 active
9 activating+degraded
4 activating
2 active+recovering+degraded
recovery io 133 MB/s, 37 objects/s
client io 64936 B/s rd, 9935 kB/s wr, 0 op/s rd, 942 op/s wr
[..]
It was only very briefly, but it did worry me a bit, fortunately, we went
back to the expected HEALTH_WARN very quickly, and everything finished fine,
so I guess nothing to worry.
But I'm curious: can anyone explain WHY we got a brief HEALTH_ERR?
No smart errors, apply and commit latency are all within the expected
ranges, the systems basically is healthy.
Curious :-)
Since Jewel (AFAIR), when (re)starting OSDs, pg status is reset to "never
contacted", resulting in "pgs are stuck inactive for more than 300 seconds"
being reported until osds regain connections between themselves.
--
Piotr Dałek
piotr.da...@corp.ovh.com
https://www.ovh.com/us/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] why sudden (and brief) HEALTH_ERR
Hi,
Yesterday I chowned our /var/lib/ceph ceph, to completely finalize our
jewel migration, and noticed something interesting.
After I brought back up the OSDs I just chowned, the system had some
recovery to do. During that recovery, the system went to HEALTH_ERR for
a short moment:
See below, for consecutive ceph -s outputs:
root@pm2:~# ceph -s
cluster 1397f1dc-7d94-43ea-ab12-8f8792eee9c1
health HEALTH_WARN
1025 pgs degraded
1 pgs recovering
60 pgs recovery_wait
307 pgs stuck unclean
964 pgs undersized
recovery 2477548/8384034 objects degraded (29.551%)
6/24 in osds are down
noout flag(s) set
monmap e3: 3 mons at
{0=10.10.89.1:6789/0,1=10.10.89.2:6789/0,2=10.10.89.3:6789/0}
election epoch 256, quorum 0,1,2 0,1,2
osdmap e10222: 24 osds: 18 up, 24 in; 964 remapped pgs
flags noout,sortbitwise,require_jewel_osds
pgmap v36531103: 1088 pgs, 2 pools, 10703 GB data, 2729 kobjects
32723 GB used, 56657 GB / 89380 GB avail
2477548/8384034 objects degraded (29.551%)
964 active+undersized+degraded
63 active+clean
60 active+recovery_wait+degraded
1 active+recovering+degraded
recovery io 63410 kB/s, 15 objects/s
client io 4348 kB/s wr, 0 op/s rd, 630 op/s wr
root@pm2:~# ceph -s
cluster 1397f1dc-7d94-43ea-ab12-8f8792eee9c1
health HEALTH_WARN
942 pgs degraded
1 pgs recovering
118 pgs recovery_wait
297 pgs stuck unclean
823 pgs undersized
recovery 2104751/8384079 objects degraded (25.104%)
6/24 in osds are down
noout flag(s) set
monmap e3: 3 mons at
{0=10.10.89.1:6789/0,1=10.10.89.2:6789/0,2=10.10.89.3:6789/0}
election epoch 256, quorum 0,1,2 0,1,2
osdmap e10224: 24 osds: 18 up, 24 in; 823 remapped pgs
flags noout,sortbitwise,require_jewel_osds
pgmap v36531118: 1088 pgs, 2 pools, 10703 GB data, 2729 kobjects
32723 GB used, 56657 GB / 89380 GB avail
2104751/8384079 objects degraded (25.104%)
823 active+undersized+degraded
146 active+clean
118 active+recovery_wait+degraded
1 active+recovering+degraded
recovery io 61945 kB/s, 16 objects/s
client io 2718 B/s rd, 5997 kB/s wr, 0 op/s rd, 638 op/s wr
root@pm2:~# ceph -s
cluster 1397f1dc-7d94-43ea-ab12-8f8792eee9c1
health HEALTH_ERR
2 pgs are stuck inactive for more than 300 seconds
761 pgs degraded
2 pgs recovering
181 pgs recovery_wait
2 pgs stuck inactive
273 pgs stuck unclean
543 pgs undersized
recovery 1394085/8384166 objects degraded (16.628%)
4/24 in osds are down
noout flag(s) set
monmap e3: 3 mons at
{0=10.10.89.1:6789/0,1=10.10.89.2:6789/0,2=10.10.89.3:6789/0}
election epoch 256, quorum 0,1,2 0,1,2
osdmap e10230: 24 osds: 20 up, 24 in; 543 remapped pgs
flags noout,sortbitwise,require_jewel_osds
pgmap v36531146: 1088 pgs, 2 pools, 10703 GB data, 2729 kobjects
32724 GB used, 56656 GB / 89380 GB avail
1394085/8384166 objects degraded (16.628%)
543 active+undersized+degraded
310 active+clean
181 active+recovery_wait+degraded
26 active+degraded
13 active
9 activating+degraded
4 activating
2 active+recovering+degraded
recovery io 133 MB/s, 37 objects/s
client io 64936 B/s rd, 9935 kB/s wr, 0 op/s rd, 942 op/s wr
root@pm2:~# ceph -s
cluster 1397f1dc-7d94-43ea-ab12-8f8792eee9c1
health HEALTH_WARN
725 pgs degraded
27 pgs peering
2 pgs recovering
207 pgs recovery_wait
269 pgs stuck unclean
516 pgs undersized
recovery 1325870/8384202 objects degraded (15.814%)
3/24 in osds are down
noout flag(s) set
monmap e3: 3 mons at
{0=10.10.89.1:6789/0,1=10.10.89.2:6789/0,2=10.10.89.3:6789/0}
election epoch 256, quorum 0,1,2 0,1,2
osdmap e10233: 24 osds: 21 up, 24 in; 418 remapped pgs
flags noout,sortbitwise,require_jewel_osds
pgmap v36531161: 1088 pgs, 2 pools, 10703 GB data, 2729 kobjects
32724 GB used, 56656 GB / 89380 GB avail
1325870/8384202 objects degraded (15.814%)
516 active+undersized+degraded
336 active+clean
207 active+recovery_wait+degraded
27 peering
2 active+recovering+degraded
recovery io 62886 kB/s, 15 objects/s
client io 3586 kB/s wr, 0 op/s rd, 251 op/s wr
It was only very briefly, bu
