Hi, you can play with the following 2 parameters: osd_recovery_max_active osd_max_backfills
The higher the number the higher the number of PGs being processed at the same time. Regards Jean-Charles LOPEZ [email protected] JC Lopez Senior Technical Instructor, Global Storage Consulting Practice Red Hat, Inc. [email protected] <mailto:[email protected]> +1 408-680-6959 > On Sep 20, 2017, at 08:26, Jonas Jaszkowic <[email protected]> > wrote: > > Thank you, that is very helpful. I didn’t know about the osd_max_backfills > option. Recovery is now working faster. > > What is the best way to make recovery as fast as possible assuming that I do > not care about read/write speed? (Besides > setting osd_max_backfills as high as possible). Are there any important > options that I have to know? > > What is the best practice to deal with the issue recovery speed vs. > read/write speed during a recovery situation? Do you > have any suggestions/references/hints how to deal with such situations? > > >> Am 20.09.2017 um 16:45 schrieb David Turner <[email protected] >> <mailto:[email protected]>>: >> >> To help things look a little better, I would also stop the daemon for osd.6 >> and mark it down `ceph osd down 6`. Note that if the OSD is still running >> it will likely mark itself back up and in on its own. I don't think that >> the OSD still running and being up in the cluster is causing the issue, but >> it might. After that, I would increase how many PGs can recover at the same >> time by increasing osd_max_backfills `ceph tell osd.* injectargs >> '--osd_max_backfills=5'`. Note that for production you'll want to set this >> number to something that doesn't negatively impact your client IO, but high >> enough to help recover your cluster faster. You can figure out that number >> by increasing it 1 at a time and watching the OSD performance with `iostat >> -x 1` or something to see how heavily used the OSDs are during your normal >> usage and again during recover while testing the settings. For testing, you >> can set it as high as you'd like (probably no need to go above 20 as that >> will likely saturate your disks' performance) to get the PGs out of the wait >> status and into active recovery and backfilling. >> >> On Wed, Sep 20, 2017 at 10:03 AM Jonas Jaszkowic >> <[email protected] <mailto:[email protected]>> wrote: >> Output of ceph status: >> >> cluster 18e87fd8-17c1-4045-a1a2-07aac106f200 >> health HEALTH_WARN >> 1 pgs backfill_wait >> 56 pgs degraded >> 1 pgs recovering >> 55 pgs recovery_wait >> 56 pgs stuck degraded >> 57 pgs stuck unclean >> recovery 50570/1369003 objects degraded (3.694%) >> recovery 854/1369003 objects misplaced (0.062%) >> monmap e2: 1 mons at {ip-172-31-16-102=172.31.16.102:6789/0 >> <http://172.31.16.102:6789/0>} >> election epoch 4, quorum 0 ip-172-31-16-102 >> mgr active: ip-172-31-16-102 >> osdmap e247: 32 osds: 32 up, 31 in; 1 remapped pgs >> flags sortbitwise,require_jewel_osds,require_kraken_osds >> pgmap v10860: 256 pgs, 1 pools, 1975 GB data, 111 kobjects >> 2923 GB used, 6836 GB / 9760 GB avail >> 50570/1369003 objects degraded (3.694%) >> 854/1369003 objects misplaced (0.062%) >> 199 active+clean >> 55 active+recovery_wait+degraded >> 1 active+remapped+backfill_wait >> 1 active+recovering+degraded >> client io 513 MB/s rd, 131 op/s rd, 0 op/s wr >> >> Output of ceph osd tree: >> >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY >> -1 9.83984 root default >> -2 0.30750 host ip-172-31-24-96 >> 0 0.30750 osd.0 up 1.00000 1.00000 >> -3 0.30750 host ip-172-31-30-32 >> 1 0.30750 osd.1 up 1.00000 1.00000 >> -4 0.30750 host ip-172-31-28-36 >> 2 0.30750 osd.2 up 1.00000 1.00000 >> -5 0.30750 host ip-172-31-18-100 >> 3 0.30750 osd.3 up 1.00000 1.00000 >> -6 0.30750 host ip-172-31-25-240 >> 4 0.30750 osd.4 up 1.00000 1.00000 >> -7 0.30750 host ip-172-31-24-110 >> 5 0.30750 osd.5 up 1.00000 1.00000 >> -8 0.30750 host ip-172-31-20-245 >> 6 0.30750 osd.6 up 0 1.00000 >> -9 0.30750 host ip-172-31-17-241 >> 7 0.30750 osd.7 up 1.00000 1.00000 >> -10 0.30750 host ip-172-31-18-107 >> 8 0.30750 osd.8 up 1.00000 1.00000 >> -11 0.30750 host ip-172-31-21-170 >> 9 0.30750 osd.9 up 1.00000 1.00000 >> -12 0.30750 host ip-172-31-21-29 >> 10 0.30750 osd.10 up 1.00000 1.00000 >> -13 0.30750 host ip-172-31-23-220 >> 11 0.30750 osd.11 up 1.00000 1.00000 >> -14 0.30750 host ip-172-31-24-154 >> 12 0.30750 osd.12 up 1.00000 1.00000 >> -15 0.30750 host ip-172-31-26-25 >> 13 0.30750 osd.13 up 1.00000 1.00000 >> -16 0.30750 host ip-172-31-20-28 >> 14 0.30750 osd.14 up 1.00000 1.00000 >> -17 0.30750 host ip-172-31-23-90 >> 15 0.30750 osd.15 up 1.00000 1.00000 >> -18 0.30750 host ip-172-31-31-197 >> 16 0.30750 osd.16 up 1.00000 1.00000 >> -19 0.30750 host ip-172-31-29-195 >> 17 0.30750 osd.17 up 1.00000 1.00000 >> -20 0.30750 host ip-172-31-28-9 >> 18 0.30750 osd.18 up 1.00000 1.00000 >> -21 0.30750 host ip-172-31-25-199 >> 19 0.30750 osd.19 up 1.00000 1.00000 >> -22 0.30750 host ip-172-31-25-187 >> 20 0.30750 osd.20 up 1.00000 1.00000 >> -23 0.30750 host ip-172-31-31-57 >> 21 0.30750 osd.21 up 1.00000 1.00000 >> -24 0.30750 host ip-172-31-20-64 >> 22 0.30750 osd.22 up 1.00000 1.00000 >> -25 0.30750 host ip-172-31-26-255 >> 23 0.30750 osd.23 up 1.00000 1.00000 >> -26 0.30750 host ip-172-31-18-146 >> 24 0.30750 osd.24 up 1.00000 1.00000 >> -27 0.30750 host ip-172-31-22-16 >> 25 0.30750 osd.25 up 1.00000 1.00000 >> -28 0.30750 host ip-172-31-26-152 >> 26 0.30750 osd.26 up 1.00000 1.00000 >> -29 0.30750 host ip-172-31-24-215 >> 27 0.30750 osd.27 up 1.00000 1.00000 >> -30 0.30750 host ip-172-31-24-138 >> 28 0.30750 osd.28 up 1.00000 1.00000 >> -31 0.30750 host ip-172-31-24-10 >> 29 0.30750 osd.29 up 1.00000 1.00000 >> -32 0.30750 host ip-172-31-20-79 >> 30 0.30750 osd.30 up 1.00000 1.00000 >> -33 0.30750 host ip-172-31-23-140 >> 31 0.30750 osd.31 up 1.00000 1.00000 >> >> Output of ceph health detail: >> >> HEALTH_WARN 1 pgs backfill_wait; 55 pgs degraded; 1 pgs recovering; 54 pgs >> recovery_wait; 55 pgs stuck degraded; 56 pgs stuck unclean; recovery >> 49688/1369003 objects degraded (3.630%); recovery 854/1369003 objects >> misplaced (0.062%) >> pg 3.b is stuck unclean for 3620.478034, current state >> active+recovery_wait+degraded, last acting [30,11,31,20,17,3,1,25,28,29,7,24] >> pg 3.f is stuck unclean for 2574.807568, current state >> active+recovery_wait+degraded, last acting [27,26,3,0,18,19,11,10,9,17,8,21] >> pg 3.11 is stuck unclean for 5031.004347, current state >> active+recovery_wait+degraded, last acting [30,3,2,7,4,17,14,23,5,16,13,29] >> pg 3.24 is stuck unclean for 3611.733994, current state >> active+recovery_wait+degraded, last acting [12,14,17,30,1,16,24,3,27,22,0,18] >> pg 3.2f is stuck unclean for 5562.733823, current state >> active+recovery_wait+degraded, last acting [22,24,5,27,10,2,3,0,17,15,23,7] >> pg 3.3b is stuck unclean for 5000.158982, current state >> active+recovery_wait+degraded, last acting [19,11,16,4,28,30,21,8,31,5,13,27] >> pg 3.aa is stuck unclean for 4827.355024, current state >> active+recovery_wait+degraded, last acting [12,10,20,0,22,9,19,24,2,3,16,5] >> pg 3.79 is stuck unclean for 3652.909790, current state >> active+remapped+backfill_wait, last acting [25,1,4,19,23,6,5,2,27,12,16,8] >> pg 3.ab is stuck unclean for 5607.537767, current state >> active+recovery_wait+degraded, last acting [19,3,30,11,0,4,22,25,16,12,8,14] >> pg 3.1b is stuck unclean for 4704.402285, current state >> active+recovery_wait+degraded, last acting [17,10,7,27,16,26,23,1,11,9,0,14] >> pg 3.7a is stuck unclean for 4465.053715, current state >> active+recovery_wait+degraded, last acting [17,8,2,19,14,16,31,20,29,26,15,9] >> pg 3.49 is stuck unclean for 4052.718824, current state >> active+recovery_wait+degraded, last acting [9,3,1,16,8,7,11,14,19,13,12,18] >> pg 3.ac <http://3.ac/> is stuck unclean for 4940.338938, current state >> active+recovery_wait+degraded, last acting [8,2,3,0,5,10,18,12,16,7,17,1] >> pg 3.83 is stuck unclean for 4381.695898, current state >> active+recovery_wait+degraded, last acting >> [18,12,21,15,16,27,3,26,28,5,20,19] >> pg 3.52 is stuck unclean for 4337.289527, current state >> active+recovery_wait+degraded, last acting [17,24,20,23,4,14,18,27,8,22,9,31] >> pg 3.ae <http://3.ae/> is stuck unclean for 5107.221614, current state >> active+recovery_wait+degraded, last acting [27,3,25,7,11,9,8,30,13,23,0,2] >> pg 3.b4 is stuck unclean for 4687.534444, current state >> active+recovery_wait+degraded, last acting [7,30,10,19,31,23,13,1,18,25,28,0] >> pg 3.bd <http://3.bd/> is stuck unclean for 5099.627501, current state >> active+recovery_wait+degraded, last acting [4,3,15,7,23,17,9,31,20,12,21,24] >> pg 3.ad <http://3.ad/> is stuck unclean for 4907.243126, current state >> active+recovery_wait+degraded, last acting [24,8,1,21,30,27,25,13,7,0,11,19] >> pg 3.af <http://3.af/> is stuck unclean for 3950.747953, current state >> active+recovery_wait+degraded, last acting >> [30,23,25,8,11,15,13,14,18,24,0,21] >> pg 3.7e is stuck unclean for 3461.008617, current state >> active+recovery_wait+degraded, last acting >> [22,15,12,16,4,0,30,14,31,23,10,17] >> pg 3.97 is stuck unclean for 5330.236878, current state >> active+recovery_wait+degraded, last acting >> [19,10,24,14,13,21,2,8,31,29,30,20] >> pg 3.f2 is stuck unclean for 2731.659626, current state >> active+recovery_wait+degraded, last acting [31,13,2,16,0,14,3,29,1,26,7,10] >> pg 3.c6 is stuck unclean for 6306.423348, current state >> active+recovery_wait+degraded, last acting [0,10,28,31,12,4,5,25,24,13,2,18] >> pg 3.67 is stuck unclean for 5118.168893, current state >> active+recovery_wait+degraded, last acting [9,15,29,18,25,7,17,30,4,12,26,23] >> pg 3.b5 is stuck unclean for 4369.784919, current state >> active+recovery_wait+degraded, last acting [17,3,28,5,15,4,16,25,11,0,26,31] >> pg 3.5b is stuck unclean for 2621.626018, current state >> active+recovery_wait+degraded, last acting [4,15,14,30,28,1,12,10,2,29,24,18] >> pg 3.f8 is stuck unclean for 4522.911060, current state >> active+recovery_wait+degraded, last acting [18,3,2,29,26,9,17,5,22,13,31,21] >> pg 3.3c is stuck unclean for 3337.005364, current state >> active+recovery_wait+degraded, last acting >> [2,24,16,20,18,25,26,10,23,12,19,31] >> pg 3.ec <http://3.ec/> is stuck unclean for 5592.096810, current state >> active+recovery_wait+degraded, last acting [18,11,27,9,26,1,22,31,0,10,24,15] >> pg 3.92 is stuck unclean for 5533.331735, current state >> active+recovery_wait+degraded, last acting [13,11,31,0,22,12,9,10,28,3,21,2] >> pg 3.c0 is stuck unclean for 5214.160745, current state >> active+recovery_wait+degraded, last acting [12,28,31,30,2,24,15,22,9,18,3,25] >> pg 3.61 is stuck unclean for 3880.126824, current state >> active+recovery_wait+degraded, last acting [4,8,24,15,0,28,16,19,13,1,18,27] >> pg 3.eb is stuck unclean for 5268.977639, current state >> active+recovery_wait+degraded, last acting [8,25,1,16,30,14,18,9,21,24,4,7] >> pg 3.b8 is stuck unclean for 4399.307382, current state >> active+recovery_wait+degraded, last acting [16,21,28,30,7,17,1,2,14,8,0,13] >> pg 3.d1 is stuck unclean for 3577.663496, current state >> active+recovery_wait+degraded, last acting [0,28,22,31,20,4,11,10,2,1,25,24] >> pg 3.89 is stuck unclean for 5730.882619, current state >> active+recovery_wait+degraded, last acting [4,5,1,2,0,9,24,11,14,13,15,28] >> pg 3.e8 is stuck unclean for 6516.175205, current state >> active+recovery_wait+degraded, last acting [23,11,28,7,8,14,27,9,30,31,24,5] >> pg 3.34 is stuck unclean for 5472.972458, current state >> active+recovery_wait+degraded, last acting [19,8,16,31,27,22,18,0,30,4,1,11] >> pg 3.fa is stuck unclean for 3740.578030, current state >> active+recovering+degraded, last acting [2,1,18,17,25,19,23,24,3,8,12,30] >> pg 3.9b is stuck unclean for 4914.758904, current state >> active+recovery_wait+degraded, last acting [23,15,26,27,12,28,3,4,8,30,0,1] >> pg 3.91 is stuck unclean for 4486.518498, current state >> active+recovery_wait+degraded, last acting [9,14,27,11,1,23,7,17,5,16,18,0] >> pg 3.26 is stuck unclean for 5068.577531, current state >> active+recovery_wait+degraded, last acting [9,25,16,24,12,2,17,22,31,29,26,7] >> pg 3.85 is stuck unclean for 5229.745995, current state >> active+recovery_wait+degraded, last acting [12,3,22,30,16,2,20,28,8,0,25,4] >> pg 3.e0 is stuck unclean for 2662.946214, current state >> active+recovery_wait+degraded, last acting [17,28,20,19,4,9,15,25,16,0,23,22] >> pg 3.81 is stuck unclean for 4265.267581, current state >> active+recovery_wait+degraded, last acting [13,4,0,19,10,17,16,24,12,26,3,20] >> pg 3.14 is stuck unclean for 4366.392617, current state >> active+recovery_wait+degraded, last acting [20,0,18,16,30,25,12,31,4,3,5,29] >> pg 3.7b is stuck unclean for 5133.369388, current state >> active+recovery_wait+degraded, last acting [4,22,30,1,21,5,12,19,17,0,2,23] >> pg 3.78 is stuck unclean for 5286.596260, current state >> active+recovery_wait+degraded, last acting [23,26,18,3,13,14,19,7,0,12,25,17] >> pg 3.d0 is stuck unclean for 5293.763984, current state >> active+recovery_wait+degraded, last acting [24,10,4,12,2,25,9,23,8,15,29,7] >> pg 3.71 is stuck unclean for 4571.041709, current state >> active+recovery_wait+degraded, last acting [25,9,27,29,16,13,11,3,18,19,26,4] >> pg 3.ca <http://3.ca/> is stuck unclean for 5465.875924, current state >> active+recovery_wait+degraded, last acting [17,24,21,3,16,14,25,10,2,5,28,18] >> pg 3.6b is stuck unclean for 4627.831337, current state >> active+recovery_wait+degraded, last acting [21,1,4,20,27,7,17,24,3,0,29,25] >> pg 3.69 is stuck unclean for 4757.583113, current state >> active+recovery_wait+degraded, last acting [8,26,4,30,11,9,16,12,18,27,14,19] >> pg 3.5c is stuck unclean for 5362.827077, current state >> active+recovery_wait+degraded, last acting [14,29,4,1,19,17,9,0,3,16,24,2] >> pg 3.51 is stuck unclean for 2778.350320, current state >> active+recovery_wait+degraded, last acting >> [13,31,11,22,25,30,1,3,27,23,21,17] >> pg 3.b is stuck degraded for 2292.794292, current state >> active+recovery_wait+degraded, last acting [30,11,31,20,17,3,1,25,28,29,7,24] >> pg 3.f is stuck degraded for 2292.771080, current state >> active+recovery_wait+degraded, last acting [27,26,3,0,18,19,11,10,9,17,8,21] >> pg 3.11 is stuck degraded for 2292.797135, current state >> active+recovery_wait+degraded, last acting [30,3,2,7,4,17,14,23,5,16,13,29] >> pg 3.24 is stuck degraded for 2292.825615, current state >> active+recovery_wait+degraded, last acting [12,14,17,30,1,16,24,3,27,22,0,18] >> pg 3.2f is stuck degraded for 2292.787887, current state >> active+recovery_wait+degraded, last acting [22,24,5,27,10,2,3,0,17,15,23,7] >> pg 3.3b is stuck degraded for 2292.823674, current state >> active+recovery_wait+degraded, last acting [19,11,16,4,28,30,21,8,31,5,13,27] >> pg 3.3c is stuck degraded for 2292.813364, current state >> active+recovery_wait+degraded, last acting >> [2,24,16,20,18,25,26,10,23,12,19,31] >> pg 3.49 is stuck degraded for 2292.804643, current state >> active+recovery_wait+degraded, last acting [9,3,1,16,8,7,11,14,19,13,12,18] >> pg 3.51 is stuck degraded for 2292.798396, current state >> active+recovery_wait+degraded, last acting >> [13,31,11,22,25,30,1,3,27,23,21,17] >> pg 3.52 is stuck degraded for 2292.799715, current state >> active+recovery_wait+degraded, last acting [17,24,20,23,4,14,18,27,8,22,9,31] >> pg 3.5b is stuck degraded for 2292.776512, current state >> active+recovery_wait+degraded, last acting [4,15,14,30,28,1,12,10,2,29,24,18] >> pg 3.5c is stuck degraded for 2292.808334, current state >> active+recovery_wait+degraded, last acting [14,29,4,1,19,17,9,0,3,16,24,2] >> pg 3.69 is stuck degraded for 2292.809014, current state >> active+recovery_wait+degraded, last acting [8,26,4,30,11,9,16,12,18,27,14,19] >> pg 3.78 is stuck degraded for 2292.798826, current state >> active+recovery_wait+degraded, last acting [23,26,18,3,13,14,19,7,0,12,25,17] >> pg 3.1b is stuck degraded for 2292.798541, current state >> active+recovery_wait+degraded, last acting [17,10,7,27,16,26,23,1,11,9,0,14] >> pg 3.7a is stuck degraded for 2292.803093, current state >> active+recovery_wait+degraded, last acting [17,8,2,19,14,16,31,20,29,26,15,9] >> pg 3.14 is stuck degraded for 2292.793869, current state >> active+recovery_wait+degraded, last acting [20,0,18,16,30,25,12,31,4,3,5,29] >> pg 3.7b is stuck degraded for 2292.782484, current state >> active+recovery_wait+degraded, last acting [4,22,30,1,21,5,12,19,17,0,2,23] >> pg 3.7e is stuck degraded for 2292.774470, current state >> active+recovery_wait+degraded, last acting >> [22,15,12,16,4,0,30,14,31,23,10,17] >> pg 3.83 is stuck degraded for 2292.795022, current state >> active+recovery_wait+degraded, last acting >> [18,12,21,15,16,27,3,26,28,5,20,19] >> pg 3.26 is stuck degraded for 2292.807846, current state >> active+recovery_wait+degraded, last acting [9,25,16,24,12,2,17,22,31,29,26,7] >> pg 3.85 is stuck degraded for 2292.813155, current state >> active+recovery_wait+degraded, last acting [12,3,22,30,16,2,20,28,8,0,25,4] >> pg 3.91 is stuck degraded for 2292.810660, current state >> active+recovery_wait+degraded, last acting [9,14,27,11,1,23,7,17,5,16,18,0] >> pg 3.92 is stuck degraded for 2292.809843, current state >> active+recovery_wait+degraded, last acting [13,11,31,0,22,12,9,10,28,3,21,2] >> pg 3.97 is stuck degraded for 2292.782984, current state >> active+recovery_wait+degraded, last acting >> [19,10,24,14,13,21,2,8,31,29,30,20] >> pg 3.aa is stuck degraded for 2292.805640, current state >> active+recovery_wait+degraded, last acting [12,10,20,0,22,9,19,24,2,3,16,5] >> pg 3.ab is stuck degraded for 2292.766750, current state >> active+recovery_wait+degraded, last acting [19,3,30,11,0,4,22,25,16,12,8,14] >> pg 3.ac <http://3.ac/> is stuck degraded for 2292.817247, current state >> active+recovery_wait+degraded, last acting [8,2,3,0,5,10,18,12,16,7,17,1] >> pg 3.ad <http://3.ad/> is stuck degraded for 2292.811631, current state >> active+recovery_wait+degraded, last acting [24,8,1,21,30,27,25,13,7,0,11,19] >> pg 3.ae <http://3.ae/> is stuck degraded for 2292.765243, current state >> active+recovery_wait+degraded, last acting [27,3,25,7,11,9,8,30,13,23,0,2] >> pg 3.af <http://3.af/> is stuck degraded for 2292.785730, current state >> active+recovery_wait+degraded, last acting >> [30,23,25,8,11,15,13,14,18,24,0,21] >> pg 3.b4 is stuck degraded for 2292.807764, current state >> active+recovery_wait+degraded, last acting [7,30,10,19,31,23,13,1,18,25,28,0] >> pg 3.b5 is stuck degraded for 2292.802932, current state >> active+recovery_wait+degraded, last acting [17,3,28,5,15,4,16,25,11,0,26,31] >> pg 3.b8 is stuck degraded for 2292.789546, current state >> active+recovery_wait+degraded, last acting [16,21,28,30,7,17,1,2,14,8,0,13] >> pg 3.bd <http://3.bd/> is stuck degraded for 2292.777194, current state >> active+recovery_wait+degraded, last acting [4,3,15,7,23,17,9,31,20,12,21,24] >> pg 3.61 is stuck degraded for 2292.780051, current state >> active+recovery_wait+degraded, last acting [4,8,24,15,0,28,16,19,13,1,18,27] >> pg 3.c0 is stuck degraded for 2292.813792, current state >> active+recovery_wait+degraded, last acting [12,28,31,30,2,24,15,22,9,18,3,25] >> pg 3.67 is stuck degraded for 2292.810551, current state >> active+recovery_wait+degraded, last acting [9,15,29,18,25,7,17,30,4,12,26,23] >> pg 3.c6 is stuck degraded for 2292.813695, current state >> active+recovery_wait+degraded, last acting [0,10,28,31,12,4,5,25,24,13,2,18] >> pg 3.6b is stuck degraded for 2292.784572, current state >> active+recovery_wait+degraded, last acting [21,1,4,20,27,7,17,24,3,0,29,25] >> pg 3.ca <http://3.ca/> is stuck degraded for 2292.802657, current state >> active+recovery_wait+degraded, last acting [17,24,21,3,16,14,25,10,2,5,28,18] >> pg 3.71 is stuck degraded for 2292.745595, current state >> active+recovery_wait+degraded, last acting [25,9,27,29,16,13,11,3,18,19,26,4] >> pg 3.d0 is stuck degraded for 2292.810869, current state >> active+recovery_wait+degraded, last acting [24,10,4,12,2,25,9,23,8,15,29,7] >> pg 3.d1 is stuck degraded for 2292.797445, current state >> active+recovery_wait+degraded, last acting [0,28,22,31,20,4,11,10,2,1,25,24] >> pg 3.81 is stuck degraded for 2292.803404, current state >> active+recovery_wait+degraded, last acting [13,4,0,19,10,17,16,24,12,26,3,20] >> pg 3.e0 is stuck degraded for 2292.763504, current state >> active+recovery_wait+degraded, last acting [17,28,20,19,4,9,15,25,16,0,23,22] >> pg 3.89 is stuck degraded for 2292.779741, current state >> active+recovery_wait+degraded, last acting [4,5,1,2,0,9,24,11,14,13,15,28] >> pg 3.e8 is stuck degraded for 2292.762470, current state >> active+recovery_wait+degraded, last acting [23,11,28,7,8,14,27,9,30,31,24,5] >> pg 3.eb is stuck degraded for 2292.823880, current state >> active+recovery_wait+degraded, last acting [8,25,1,16,30,14,18,9,21,24,4,7] >> pg 3.ec <http://3.ec/> is stuck degraded for 2292.820337, current state >> active+recovery_wait+degraded, last acting [18,11,27,9,26,1,22,31,0,10,24,15] >> pg 3.f2 is stuck degraded for 2292.781004, current state >> active+recovery_wait+degraded, last acting [31,13,2,16,0,14,3,29,1,26,7,10] >> pg 3.f8 is stuck degraded for 2292.791270, current state >> active+recovery_wait+degraded, last acting [18,3,2,29,26,9,17,5,22,13,31,21] >> pg 3.34 is stuck degraded for 2292.824069, current state >> active+recovery_wait+degraded, last acting [19,8,16,31,27,22,18,0,30,4,1,11] >> pg 3.9b is stuck degraded for 2292.762747, current state >> active+recovery_wait+degraded, last acting [23,15,26,27,12,28,3,4,8,30,0,1] >> pg 3.fa is stuck degraded for 2292.808553, current state >> active+recovering+degraded, last acting [2,1,18,17,25,19,23,24,3,8,12,30] >> pg 3.fa is active+recovering+degraded, acting >> [2,1,18,17,25,19,23,24,3,8,12,30] >> pg 3.f8 is active+recovery_wait+degraded, acting >> [18,3,2,29,26,9,17,5,22,13,31,21] >> pg 3.f2 is active+recovery_wait+degraded, acting >> [31,13,2,16,0,14,3,29,1,26,7,10] >> pg 3.ec <http://3.ec/> is active+recovery_wait+degraded, acting >> [18,11,27,9,26,1,22,31,0,10,24,15] >> pg 3.eb is active+recovery_wait+degraded, acting >> [8,25,1,16,30,14,18,9,21,24,4,7] >> pg 3.e8 is active+recovery_wait+degraded, acting >> [23,11,28,7,8,14,27,9,30,31,24,5] >> pg 3.e0 is active+recovery_wait+degraded, acting >> [17,28,20,19,4,9,15,25,16,0,23,22] >> pg 3.d1 is active+recovery_wait+degraded, acting >> [0,28,22,31,20,4,11,10,2,1,25,24] >> pg 3.d0 is active+recovery_wait+degraded, acting >> [24,10,4,12,2,25,9,23,8,15,29,7] >> pg 3.ca <http://3.ca/> is active+recovery_wait+degraded, acting >> [17,24,21,3,16,14,25,10,2,5,28,18] >> pg 3.c6 is active+recovery_wait+degraded, acting >> [0,10,28,31,12,4,5,25,24,13,2,18] >> pg 3.c0 is active+recovery_wait+degraded, acting >> [12,28,31,30,2,24,15,22,9,18,3,25] >> pg 3.bd <http://3.bd/> is active+recovery_wait+degraded, acting >> [4,3,15,7,23,17,9,31,20,12,21,24] >> pg 3.b8 is active+recovery_wait+degraded, acting >> [16,21,28,30,7,17,1,2,14,8,0,13] >> pg 3.b5 is active+recovery_wait+degraded, acting >> [17,3,28,5,15,4,16,25,11,0,26,31] >> pg 3.b4 is active+recovery_wait+degraded, acting >> [7,30,10,19,31,23,13,1,18,25,28,0] >> pg 3.af <http://3.af/> is active+recovery_wait+degraded, acting >> [30,23,25,8,11,15,13,14,18,24,0,21] >> pg 3.ae <http://3.ae/> is active+recovery_wait+degraded, acting >> [27,3,25,7,11,9,8,30,13,23,0,2] >> pg 3.ad <http://3.ad/> is active+recovery_wait+degraded, acting >> [24,8,1,21,30,27,25,13,7,0,11,19] >> pg 3.ac <http://3.ac/> is active+recovery_wait+degraded, acting >> [8,2,3,0,5,10,18,12,16,7,17,1] >> pg 3.ab is active+recovery_wait+degraded, acting >> [19,3,30,11,0,4,22,25,16,12,8,14] >> pg 3.aa is active+recovery_wait+degraded, acting >> [12,10,20,0,22,9,19,24,2,3,16,5] >> pg 3.9b is active+recovery_wait+degraded, acting >> [23,15,26,27,12,28,3,4,8,30,0,1] >> pg 3.97 is active+recovery_wait+degraded, acting >> [19,10,24,14,13,21,2,8,31,29,30,20] >> pg 3.92 is active+recovery_wait+degraded, acting >> [13,11,31,0,22,12,9,10,28,3,21,2] >> pg 3.91 is active+recovery_wait+degraded, acting >> [9,14,27,11,1,23,7,17,5,16,18,0] >> pg 3.89 is active+recovery_wait+degraded, acting >> [4,5,1,2,0,9,24,11,14,13,15,28] >> pg 3.85 is active+recovery_wait+degraded, acting >> [12,3,22,30,16,2,20,28,8,0,25,4] >> pg 3.83 is active+recovery_wait+degraded, acting >> [18,12,21,15,16,27,3,26,28,5,20,19] >> pg 3.81 is active+recovery_wait+degraded, acting >> [13,4,0,19,10,17,16,24,12,26,3,20] >> pg 3.7e is active+recovery_wait+degraded, acting >> [22,15,12,16,4,0,30,14,31,23,10,17] >> pg 3.7b is active+recovery_wait+degraded, acting >> [4,22,30,1,21,5,12,19,17,0,2,23] >> pg 3.7a is active+recovery_wait+degraded, acting >> [17,8,2,19,14,16,31,20,29,26,15,9] >> pg 3.79 is active+remapped+backfill_wait, acting >> [25,1,4,19,23,6,5,2,27,12,16,8] >> pg 3.78 is active+recovery_wait+degraded, acting >> [23,26,18,3,13,14,19,7,0,12,25,17] >> pg 3.71 is active+recovery_wait+degraded, acting >> [25,9,27,29,16,13,11,3,18,19,26,4] >> pg 3.6b is active+recovery_wait+degraded, acting >> [21,1,4,20,27,7,17,24,3,0,29,25] >> pg 3.69 is active+recovery_wait+degraded, acting >> [8,26,4,30,11,9,16,12,18,27,14,19] >> pg 3.67 is active+recovery_wait+degraded, acting >> [9,15,29,18,25,7,17,30,4,12,26,23] >> pg 3.61 is active+recovery_wait+degraded, acting >> [4,8,24,15,0,28,16,19,13,1,18,27] >> pg 3.5c is active+recovery_wait+degraded, acting >> [14,29,4,1,19,17,9,0,3,16,24,2] >> pg 3.5b is active+recovery_wait+degraded, acting >> [4,15,14,30,28,1,12,10,2,29,24,18] >> pg 3.52 is active+recovery_wait+degraded, acting >> [17,24,20,23,4,14,18,27,8,22,9,31] >> pg 3.51 is active+recovery_wait+degraded, acting >> [13,31,11,22,25,30,1,3,27,23,21,17] >> pg 3.49 is active+recovery_wait+degraded, acting >> [9,3,1,16,8,7,11,14,19,13,12,18] >> pg 3.3c is active+recovery_wait+degraded, acting >> [2,24,16,20,18,25,26,10,23,12,19,31] >> pg 3.3b is active+recovery_wait+degraded, acting >> [19,11,16,4,28,30,21,8,31,5,13,27] >> pg 3.34 is active+recovery_wait+degraded, acting >> [19,8,16,31,27,22,18,0,30,4,1,11] >> pg 3.2f is active+recovery_wait+degraded, acting >> [22,24,5,27,10,2,3,0,17,15,23,7] >> pg 3.26 is active+recovery_wait+degraded, acting >> [9,25,16,24,12,2,17,22,31,29,26,7] >> pg 3.24 is active+recovery_wait+degraded, acting >> [12,14,17,30,1,16,24,3,27,22,0,18] >> pg 3.1b is active+recovery_wait+degraded, acting >> [17,10,7,27,16,26,23,1,11,9,0,14] >> pg 3.14 is active+recovery_wait+degraded, acting >> [20,0,18,16,30,25,12,31,4,3,5,29] >> pg 3.11 is active+recovery_wait+degraded, acting >> [30,3,2,7,4,17,14,23,5,16,13,29] >> pg 3.f is active+recovery_wait+degraded, acting >> [27,26,3,0,18,19,11,10,9,17,8,21] >> pg 3.b is active+recovery_wait+degraded, acting >> [30,11,31,20,17,3,1,25,28,29,7,24] >> recovery 49688/1369003 objects degraded (3.630%) >> recovery 854/1369003 objects misplaced (0.062%) >> >> >> >>> Am 19.09.2017 um 22:15 schrieb David Turner <[email protected] >>> <mailto:[email protected]>>: >>> >>> Can you please provide the output of `ceph status`, `ceph osd tree`, and >>> `ceph health detail`? Thank you. >>> >>> On Tue, Sep 19, 2017 at 2:59 PM Jonas Jaszkowic >>> <[email protected] <mailto:[email protected]>> >>> wrote: >>> Hi all, >>> >>> I have setup a Ceph cluster consisting of one monitor, 32 OSD hosts (1 OSD >>> of size 320GB per host) and 16 clients which are reading >>> and writing to the cluster. I have one erasure coded pool (shec plugin) >>> with k=8, m=4, c=3 and pg_num=256. Failure domain is host. >>> I am able to reach a HEALTH_OK state and everything is working as expected. >>> The pool was populated with >>> 114048 files of different sizes ranging from 1kB to 4GB. Total amount of >>> data in the pool was around 3TB. The capacity of the >>> pool was around 10TB. >>> >>> I want to evaluate how Ceph is rebalancing data in case of an OSD loss >>> while clients are still reading. To do so, I am killing one OSD on purpose >>> via ceph osd out <osd-id> without adding a new one, i.e. I have 31 OSDs >>> left. Ceph seems to notice this failure and starts to rebalance data >>> which I can observe with the ceph -w command. >>> >>> However, Ceph failed to rebalance the data. The recovering process seemed >>> to be stuck at a random point. I waited more than 12h but the >>> number of degraded objects did not reduce and some PGs were stuck. Why is >>> this happening? Based on the number of OSDs and the k,m,c values >>> there should be enough hosts and OSDs to be able to recover from a single >>> OSD failure? >>> >>> Thank you in advance! >>> _______________________________________________ >>> ceph-users mailing list >>> [email protected] <mailto:[email protected]> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> >> > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
