"340 osds: 101 up, 112 in" This is going to be your culprit. Your CRUSH
map is in a really weird state. How many OSDs do you have in this
cluster? When OSDs go down, secondary OSDs take over for it, but when OSDs
get marked out, the cluster re-balances to distribute the data according to
how many replicas your settings say it should have (remapped PGs). Your
cluster thinks it has 340 OSDs in total, it believes that 112 of them are
added to the cluster, but only 101 of them are currently up and running.
That means that it is trying to put all of your data onto those 101 OSDs.
Your settings to have 16k PGs is fine for 340 OSDs, but with 101 OSDs
you're getting the error of too many PGs per OSD.
So next steps:
1) How many OSDs do you expect to be in your Ceph cluster?
2) Did you bring your OSDs back up during your rolling restart testing
BEFORE
a) They were marked down in the cluster?
b) You moved onto the next node? Additionally, did you wait for all
backfilling to finish before you proceeded to the next node?
3) Do you have enough memory in your nodes or are your OSDs being killed by
OOM killer? I see that you have a lot of peering PGs in your status
output. That is indicative that the OSDs are continually restarting or
being marked down for not responding.
On Thu, May 18, 2017 at 2:41 PM nokia ceph <nokiacephus...@gmail.com> wrote:
> Hello,
>
>
> Env;- Bluestore EC 4+1 v11.2.0 RHEL7.3 16383 PG
>
>
> We did our resiliency testing and found OSD's keeps on flapping and
> cluster went to error state.
>
> What we did:-
>
> 1. we have 5 node cluster
> 2. poweroff/stop ceph.target on last node and waited everything seems to
> reach back to normal.
> 3. Then power up the last node and then we see this recovery stuck on
> remapped PG.
> ~~~
> osdmap e4829: 340 osds: 101 up, 112 in; 15011 *remapped pgs*
> *~~~*
> 4. Initially all osd's reach 340, at the same time this remapped value
> reached 16384 with OSD epoch value e818
> 5. Then after 1 or 2 hour we suspect that this remapped PG value keeps on
> incremnet/decrement results the osd's started failed one by one. While we
> tested with below patch also still no change.
> patch -
> https://github.com/ceph/ceph-ci/commit/wip-prune-past-intervals-kraken
>
>
> #ceph -s
> 2017-05-18 18:07:45.876586 7fd6bb87e700 -1 WARNING: the following
> dangerous and experimental features are enabled: bluestore,rocksdb
> 2017-05-18 18:07:45.900045 7fd6bb87e700 -1 WARNING: the following
> dangerous and experimental features are enabled: bluestore,rocksdb
> cluster cb55baa8-d5a5-442e-9aae-3fd83553824e
> health HEALTH_ERR
> 27056 pgs are stuck inactive for more than 300 seconds
> 744 pgs degraded
> 10944 pgs down
> 3919 pgs peering
> 11416 pgs stale
> 744 pgs stuck degraded
> 15640 pgs stuck inactive
> 11416 pgs stuck stale
> 16384 pgs stuck unclean
> 744 pgs stuck undersized
> 744 pgs undersized
> recovery 1279809/135206985 objects degraded (0.947%)
> too many PGs per OSD (731 > max 300)
> 11/112 in osds are down
> monmap e3: 5 mons at {PL6-CN1=
> 10.50.62.151:6789/0,PL6-CN2=10.50.62.152:6789/0,PL6-CN3=10.50.62.153:6789/0,PL6-CN4=10.50.62.154:6789/0,PL6-CN5=1
> 0.50.62.155:6789/0}
> election epoch 22, quorum 0,1,2,3,4
> PL6-CN1,PL6-CN2,PL6-CN3,PL6-CN4,PL6-CN5
> mgr no daemons active
> osdmap e4827: 340 osds: 101 up, 112 in; 15011 remapped pgs
> flags sortbitwise,require_jewel_osds,require_kraken_osds
> pgmap v83202: 16384 pgs, 1 pools, 52815 GB data, 26407 kobjects
> 12438 GB used, 331 TB / 343 TB avail
> 1279809/135206985 objects degraded (0.947%)
> 4512 stale+down+remapped
> 3060 down+remapped
> 2204 stale+down
> 2000 stale+remapped+peering
> 1259 stale+peering
> 1167 down
> 739 stale+active+undersized+degraded
> 702 stale+remapped
> 557 peering
> 102 remapped+peering
>
>
>
> # ceph pg stat
> 2017-05-18 18:09:18.345865 7fe2f72ec700 -1 WARNING: the following
> dangerous and experimental features are enabled: bluestore,rocksdb
> 2017-05-18 18:09:18.368566 7fe2f72ec700 -1 WARNING: the following
> dangerous and experimental features are enabled: bluestore,rocksdb
> v83204: 16384 pgs: 1 inactive, 1259 stale+peering, 75 remapped, 2000
> stale+remapped+peering, 102 remapped+peering, 2204 stale+down, 739
> stale+active+undersized+degraded, 1 down+remapped+peering, 702
> stale+remapped, 557 peering, 4512 stale+down+remapped, 3060 down+remapped,
> 5 active+undersized+degraded, 1167 down; 52815 GB data, 12438 GB used, 331
> TB / 343 TB avail; 1279809/135206985 objects degraded (0.947%)
>
>
>
> Randomly capture some pg value.
> ~~~
> 3.3ffc 1646 0 1715 0 0 3451912192
> 1646 1646 stale+active+undersized+degraded 2017-05-18 11:06:32.453158
> 846'1646 872:1634 [36,NONE,278,219,225] 36
> [36,NONE,278,219,225] 36 0'0 2017-05-18 07:14:30.303859
> 0'0 2017-05-18 07:14:30.303859
> 3.3ffb 1711 0 0 0 0 3588227072
> 1711 1711 down 2017-05-18 15:20:52.858840
> 846'1711 1602:1708 [150,161,NONE,NONE,83] 150
> [150,161,NONE,NONE,83] 150 0'0 2017-05-18
> 07:14:30.303838 0'0 2017-05-18 07:14:30.303838
> 3.3ffa 1617 0 0 0 0 3391094784
> 1617 1617 down+remapped 2017-05-18 17:12:54.943317
> 846'1617 2525:1637 [48,292,77,277,49] 48
> [48,NONE,NONE,277,49] 48 0'0 2017-05-18 07:14:30.303807
> 0'0 2017-05-18 07:14:30.303807
> 3.3ff9 1682 0 0 0 0 3527409664
> <(352)%20740-9664> 1682 1682 down+remapped
> 2017-05-18 16:16:42.223632 846'1682 2195:1678 [266,79,NONE,309,258]
> 266 [NONE,NONE,NONE,NONE,258] 258 0'0 2017-05-18
> 07:14:30.303793 0'0 2017-05-18 07:14:30.303793
> ~~~
>
> ceph.conf
>
> [mon]
> mon_osd_down_out_interval = 3600
> mon_osd_reporter_subtree_level=host
> mon_osd_down_out_subtree_limit=host
> mon_osd_min_down_reporters = 4
> mon_allow_pool_delete = true
> [osd]
> bluestore = true
> bluestore_cache_size = 107374182
> bluefs_buffered_io = true
> osd_op_threads = 24
> osd_op_num_shards = 5
> osd_op_num_threads_per_shard = 2
> osd_enable_op_tracker = false
> osd_scrub_begin_hour = 1
> osd_scrub_end_hour = 7
> osd_deep_scrub_interval = 3.154e+9
> osd_max_backfills = 3
> osd_recovery_max_active = 3
> osd_recovery_op_priority = 1
>
>
>
> # ceph osd stat
> 2017-05-18 18:10:11.864303 7fedc5a98700 -1 WARNING: the following
> dangerous and experimental features are enabled: bluestore,rocksdb
> 2017-05-18 18:10:11.887182 7fedc5a98700 -1 WARNING: the following
> dangerous and experimental features are enabled: bluestore,rocksdb
> osdmap e4829: 340 osds: 101 up, 112 in; 15011 remapped pgs ==<<<
> <<<<<<<<<<<<<< SEE this
> flags sortbitwise,require_jewel_osds,require_kraken_osds
>
> Is there any config directive which helps to skip the remapped PG count
> while recovery process. Does Luminous v12.0.3 fixed the OSD flap issue?
>
> Awaiting for your suggestions.
>
> Thanks
> Jayaram
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
