Quick turn around, Changing/injecting osd_recovery_sleep_hdd into the running SSD OSD’s on bluestore opened the floodgates.
> pool objects-ssd id 20 > recovery io 1512 MB/s, 21547 objects/s > > pool fs-metadata-ssd id 16 > recovery io 0 B/s, 6494 keys/s, 271 objects/s > client io 82325 B/s rd, 68146 B/s wr, 1 op/s rd, 0 op/s wr Graph of performance jump. Extremely marked. https://imgur.com/a/LZR9R <https://imgur.com/a/LZR9R> So at least we now have the gun to go with the smoke. Thanks for the help and appreciate you pointing me in some directions that I was able to use to figure out the issue. Adding to ceph.conf for future OSD conversions. Thanks, Reed > On Feb 26, 2018, at 4:12 PM, Reed Dier <[email protected]> wrote: > > For the record, I am not seeing a demonstrative fix by injecting the value of > 0 into the OSDs running. >> osd_recovery_sleep_hybrid = '0.000000' (not observed, change may require >> restart) > > If it does indeed need to be restarted, I will need to wait for the current > backfills to finish their process as restarting an OSD would bring me under > min_size. > > However, doing config show on the osd daemon appears to have taken the value > of 0. > >> ceph daemon osd.24 config show | grep recovery_sleep >> "osd_recovery_sleep": "0.000000", >> "osd_recovery_sleep_hdd": "0.100000", >> "osd_recovery_sleep_hybrid": "0.000000", >> "osd_recovery_sleep_ssd": "0.000000", > > > I may take the restart as an opportunity to also move to 12.2.3 at the same > time, since it is not expected that that should affect this issue. > > I could also attempt to change osd_recovery_sleep_hdd as well, since these > are ssd osd’s, it shouldn’t make a difference, but its a free move. > > Thanks, > > Reed > >> On Feb 26, 2018, at 3:42 PM, Gregory Farnum <[email protected] >> <mailto:[email protected]>> wrote: >> >> On Mon, Feb 26, 2018 at 12:26 PM Reed Dier <[email protected] >> <mailto:[email protected]>> wrote: >> I will try to set the hybrid sleeps to 0 on the affected OSDs as an interim >> solution to getting the metadata configured correctly. >> >> Yes, that's a good workaround as long as you don't have any actual hybrid >> OSDs (or aren't worried about them sleeping...I'm not sure if that setting >> came from experience or not). >> >> >> For reference, here is the complete metadata for osd.24, bluestore SATA SSD >> with NVMe block.db. >> >>> { >>> "id": 24, >>> "arch": "x86_64", >>> "back_addr": "", >>> "back_iface": "bond0", >>> "bluefs": "1", >>> "bluefs_db_access_mode": "blk", >>> "bluefs_db_block_size": "4096", >>> "bluefs_db_dev": "259:0", >>> "bluefs_db_dev_node": "nvme0n1", >>> "bluefs_db_driver": "KernelDevice", >>> "bluefs_db_model": "INTEL SSDPEDMD400G4 ", >>> "bluefs_db_partition_path": "/dev/nvme0n1p4", >>> "bluefs_db_rotational": "0", >>> "bluefs_db_serial": " ", >>> "bluefs_db_size": "16000221184", >>> "bluefs_db_type": "nvme", >>> "bluefs_single_shared_device": "0", >>> "bluefs_slow_access_mode": "blk", >>> "bluefs_slow_block_size": "4096", >>> "bluefs_slow_dev": "253:8", >>> "bluefs_slow_dev_node": "dm-8", >>> "bluefs_slow_driver": "KernelDevice", >>> "bluefs_slow_model": "", >>> "bluefs_slow_partition_path": "/dev/dm-8", >>> "bluefs_slow_rotational": "0", >>> "bluefs_slow_size": "1920378863616", >>> "bluefs_slow_type": "ssd", >>> "bluestore_bdev_access_mode": "blk", >>> "bluestore_bdev_block_size": "4096", >>> "bluestore_bdev_dev": "253:8", >>> "bluestore_bdev_dev_node": "dm-8", >>> "bluestore_bdev_driver": "KernelDevice", >>> "bluestore_bdev_model": "", >>> "bluestore_bdev_partition_path": "/dev/dm-8", >>> "bluestore_bdev_rotational": "0", >>> "bluestore_bdev_size": "1920378863616", >>> "bluestore_bdev_type": "ssd", >>> "ceph_version": "ceph version 12.2.2 >>> (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)", >>> "cpu": "Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz", >>> "default_device_class": "ssd", >>> "distro": "ubuntu", >>> "distro_description": "Ubuntu 16.04.3 LTS", >>> "distro_version": "16.04", >>> "front_addr": "", >>> "front_iface": "bond0", >>> "hb_back_addr": "", >>> "hb_front_addr": "", >>> "hostname": “host00", >>> "journal_rotational": "1", >>> "kernel_description": "#29~16.04.2-Ubuntu SMP Tue Jan 9 22:00:44 >>> UTC 2018", >>> "kernel_version": "4.13.0-26-generic", >>> "mem_swap_kb": "124999672", >>> "mem_total_kb": "131914008", >>> "os": "Linux", >>> "osd_data": "/var/lib/ceph/osd/ceph-24", >>> "osd_objectstore": "bluestore", >>> "rotational": "0" >>> } >> >> >> So it looks like it correctly guessed(?) the >> bluestore_bdev_type/default_device_class correctly (though it may have been >> an inherited value?), as did bluefs_db_type get set to nvme correctly. >> >> So I’m not sure why journal_rotational is still showing 1. >> Maybe something in the ceph-volume lvm piece that isn’t correctly setting >> that flag on OSD creation? >> Also seems like the journal_rotational field should have been deprecated in >> bluestore as bluefs_db_rotational should cover that, and if there were a WAL >> partition as well, I assume there would be something to the tune of >> bluefs_wal_rotational or something like that, and journal would never be >> used for bluestore? >> >> Thanks to both of you for helping diagnose this issue. I created a ticket >> and have a PR up to fix it: http://tracker.ceph.com/issues/23141 >> <http://tracker.ceph.com/issues/23141>, >> https://github.com/ceph/ceph/pull/20602 >> <https://github.com/ceph/ceph/pull/20602> >> >> Until that gets backported into another Luminous release you'll need to do >> some kind of workaround though. :/ >> -Greg >> >> >> Appreciate the help. >> >> Thanks, >> Reed >> >>> On Feb 26, 2018, at 1:28 PM, Gregory Farnum <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> On Mon, Feb 26, 2018 at 11:21 AM Reed Dier <[email protected] >>> <mailto:[email protected]>> wrote: >>> The ‘good perf’ that I reported below was the result of beginning 5 new >>> bluestore conversions which results in a leading edge of ‘good’ >>> performance, before trickling off. >>> >>> This performance lasted about 20 minutes, where it backfilled a small set >>> of PGs off of non-bluestore OSDs. >>> >>> Current performance is now hovering around: >>>> pool objects-ssd id 20 >>>> recovery io 14285 kB/s, 202 objects/s >>>> >>>> pool fs-metadata-ssd id 16 >>>> recovery io 0 B/s, 262 keys/s, 12 objects/s >>>> client io 412 kB/s rd, 67593 B/s wr, 5 op/s rd, 0 op/s wr >>> >>>> What are you referencing when you talk about recovery ops per second? >>> >>> These are recovery ops as reported by ceph -s or via stats exported via >>> influx plugin in mgr, and via local collectd collection. >>> >>>> Also, what are the values for osd_recovery_sleep_hdd and >>>> osd_recovery_sleep_hybrid, and can you validate via "ceph osd metadata" >>>> that your BlueStore SSD OSDs are correctly reporting both themselves and >>>> their journals as non-rotational? >>> >>> This yields more interesting results. >>> Pasting results for 3 sets of OSDs in this order >>> {0}hdd+nvme block.db >>> {24}ssd+nvme block.db >>> {59}ssd+nvme journal >>> >>>> ceph osd metadata | grep 'id\|rotational' >>>> "id": 0, >>>> "bluefs_db_rotational": "0", >>>> "bluefs_slow_rotational": "1", >>>> "bluestore_bdev_rotational": "1", >>>> "journal_rotational": "1", >>>> "rotational": “1" >>>> "id": 24, >>>> "bluefs_db_rotational": "0", >>>> "bluefs_slow_rotational": "0", >>>> "bluestore_bdev_rotational": "0", >>>> "journal_rotational": "1", >>>> "rotational": “0" >>>> "id": 59, >>>> "journal_rotational": "0", >>>> "rotational": “0" >>> >>> I wonder if it matters/is correct to see "journal_rotational": “1” for the >>> bluestore OSD’s {0,24} with nvme block.db. >>> >>> Hope this may be helpful in determining the root cause. >>> >>> If you have an SSD main store and a hard drive ("rotational") journal, the >>> OSD will insert recovery sleeps from the osd_recovery_sleep_hybrid config >>> option. By default that is .025 (seconds). >>> >>> I believe you can override the setting (I'm not sure how), but you really >>> want to correct that flag at the OS layer. Generally when we see this >>> there's a RAID card or something between the solid-state device and the >>> host which is lying about the state of the world. >>> -Greg >>> >>> >>> If it helps, all of the OSD’s were originally deployed with ceph-deploy, >>> but are now being redone with ceph-volume locally on each host. >>> >>> Thanks, >>> >>> Reed >>> >>>> On Feb 26, 2018, at 1:00 PM, Gregory Farnum <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> On Mon, Feb 26, 2018 at 9:12 AM Reed Dier <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> After my last round of backfills completed, I started 5 more bluestore >>>> conversions, which helped me recognize a very specific pattern of >>>> performance. >>>> >>>>> pool objects-ssd id 20 >>>>> recovery io 757 MB/s, 10845 objects/s >>>>> >>>>> pool fs-metadata-ssd id 16 >>>>> recovery io 0 B/s, 36265 keys/s, 1633 objects/s >>>>> client io 2544 kB/s rd, 36788 B/s wr, 1 op/s rd, 0 op/s wr >>>> >>>> The “non-throttled” backfills are only coming from filestore SSD OSD’s. >>>> When backfilling from bluestore SSD OSD’s, they appear to be throttled at >>>> the aforementioned <20 ops per OSD. >>>> >>>> Wait, is that the current state? What are you referencing when you talk >>>> about recovery ops per second? >>>> >>>> Also, what are the values for osd_recovery_sleep_hdd and >>>> osd_recovery_sleep_hybrid, and can you validate via "ceph osd metadata" >>>> that your BlueStore SSD OSDs are correctly reporting both themselves and >>>> their journals as non-rotational? >>>> -Greg >>>> >>>> >>>> This would corroborate why the first batch of SSD’s I migrated to >>>> bluestore were all at “full” speed, as all of the OSD’s they were >>>> backfilling from were filestore based, compared to increasingly bluestore >>>> backfill targets, leading to increasingly long backfill times as I move >>>> from one host to the next. >>>> >>>> Looking at the recovery settings, the recovery_sleep and >>>> recovery_sleep_ssd values across bluestore or filestore OSDs are showing >>>> as 0 values, which means no sleep/throttle if I am reading everything >>>> correctly. >>>> >>>>> sudo ceph daemon osd.73 config show | grep recovery >>>>> "osd_allow_recovery_below_min_size": "true", >>>>> "osd_debug_skip_full_check_in_recovery": "false", >>>>> "osd_force_recovery_pg_log_entries_factor": "1.300000", >>>>> "osd_min_recovery_priority": "0", >>>>> "osd_recovery_cost": "20971520", >>>>> "osd_recovery_delay_start": "0.000000", >>>>> "osd_recovery_forget_lost_objects": "false", >>>>> "osd_recovery_max_active": "35", >>>>> "osd_recovery_max_chunk": "8388608", >>>>> "osd_recovery_max_omap_entries_per_chunk": "64000", >>>>> "osd_recovery_max_single_start": "1", >>>>> "osd_recovery_op_priority": "3", >>>>> "osd_recovery_op_warn_multiple": "16", >>>>> "osd_recovery_priority": "5", >>>>> "osd_recovery_retry_interval": "30.000000", >>>>> "osd_recovery_sleep": "0.000000", >>>>> "osd_recovery_sleep_hdd": "0.100000", >>>>> "osd_recovery_sleep_hybrid": "0.025000", >>>>> "osd_recovery_sleep_ssd": "0.000000", >>>>> "osd_recovery_thread_suicide_timeout": "300", >>>>> "osd_recovery_thread_timeout": "30", >>>>> "osd_scrub_during_recovery": "false", >>>> >>>> >>>> As far as I know, the device class is configured correctly as far as I >>>> know, it all shows as ssd/hdd correctly in ceph osd tree. >>>> >>>> So hopefully this may be enough of a smoking gun to help narrow down where >>>> this may be stemming from. >>>> >>>> Thanks, >>>> >>>> Reed >>>> >>>>> On Feb 23, 2018, at 10:04 AM, David Turner <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Here is a [1] link to a ML thread tracking some slow backfilling on >>>>> bluestore. It came down to the backfill sleep setting for them. Maybe >>>>> it will help. >>>>> >>>>> [1] https://www.mail-archive.com/[email protected]/msg40256.html >>>>> <https://www.mail-archive.com/[email protected]/msg40256.html> >>>>> On Fri, Feb 23, 2018 at 10:46 AM Reed Dier <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> Probably unrelated, but I do keep seeing this odd negative objects >>>>> degraded message on the fs-metadata pool: >>>>> >>>>>> pool fs-metadata-ssd id 16 >>>>>> -34/3 objects degraded (-1133.333%) >>>>>> recovery io 0 B/s, 89 keys/s, 2 objects/s >>>>>> client io 51289 B/s rd, 101 kB/s wr, 0 op/s rd, 0 op/s wr >>>>> >>>>> Don’t mean to clutter the ML/thread, however it did seem odd, maybe its a >>>>> culprit? Maybe its some weird sampling interval issue thats been solved >>>>> in 12.2.3? >>>>> >>>>> Thanks, >>>>> >>>>> Reed >>>>> >>>>> >>>>>> On Feb 23, 2018, at 8:26 AM, Reed Dier <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> Below is ceph -s >>>>>> >>>>>>> cluster: >>>>>>> id: {id} >>>>>>> health: HEALTH_WARN >>>>>>> noout flag(s) set >>>>>>> 260610/1068004947 objects misplaced (0.024%) >>>>>>> Degraded data redundancy: 23157232/1068004947 objects >>>>>>> degraded (2.168%), 332 pgs unclean, 328 pgs degraded, 328 pgs undersized >>>>>>> >>>>>>> services: >>>>>>> mon: 3 daemons, quorum mon02,mon01,mon03 >>>>>>> mgr: mon03(active), standbys: mon02 >>>>>>> mds: cephfs-1/1/1 up {0=mon03=up:active}, 1 up:standby >>>>>>> osd: 74 osds: 74 up, 74 in; 332 remapped pgs >>>>>>> flags noout >>>>>>> >>>>>>> data: >>>>>>> pools: 5 pools, 5316 pgs >>>>>>> objects: 339M objects, 46627 GB >>>>>>> usage: 154 TB used, 108 TB / 262 TB avail >>>>>>> pgs: 23157232/1068004947 objects degraded (2.168%) >>>>>>> 260610/1068004947 objects misplaced (0.024%) >>>>>>> 4984 active+clean >>>>>>> 183 active+undersized+degraded+remapped+backfilling >>>>>>> 145 active+undersized+degraded+remapped+backfill_wait >>>>>>> 3 active+remapped+backfill_wait >>>>>>> 1 active+remapped+backfilling >>>>>>> >>>>>>> io: >>>>>>> client: 8428 kB/s rd, 47905 B/s wr, 130 op/s rd, 0 op/s wr >>>>>>> recovery: 37057 kB/s, 50 keys/s, 217 objects/s >>>>>> >>>>>> Also the two pools on the SSDs, are the objects pool at 4096 PG, and the >>>>>> fs-metadata pool at 32 PG. >>>>>> >>>>>>> Are you sure the recovery is actually going slower, or are the >>>>>>> individual ops larger or more expensive? >>>>>> >>>>>> The objects should not vary wildly in size. >>>>>> Even if they were differing in size, the SSDs are roughly idle in their >>>>>> current state of backfilling when examining wait in iotop, or atop, or >>>>>> sysstat/iostat. >>>>>> >>>>>> This compares to when I was fully saturating the SATA backplane with >>>>>> over 1000MB/s of writes to multiple disks when the backfills were going >>>>>> “full speed.” >>>>>> >>>>>> Here is a breakdown of recovery io by pool: >>>>>> >>>>>>> pool objects-ssd id 20 >>>>>>> recovery io 6779 kB/s, 92 objects/s >>>>>>> client io 3071 kB/s rd, 50 op/s rd, 0 op/s wr >>>>>>> >>>>>>> pool fs-metadata-ssd id 16 >>>>>>> recovery io 0 B/s, 28 keys/s, 2 objects/s >>>>>>> client io 109 kB/s rd, 67455 B/s wr, 1 op/s rd, 0 op/s wr >>>>>>> >>>>>>> pool cephfs-hdd id 17 >>>>>>> recovery io 40542 kB/s, 158 objects/s >>>>>>> client io 10056 kB/s rd, 142 op/s rd, 0 op/s wr >>>>>> >>>>>> So the 24 HDD’s are outperforming the 50 SSD’s for recovery and client >>>>>> traffic at the moment, which seems conspicuous to me. >>>>>> >>>>>> Most of the OSD’s with recovery ops to the SSDs are reporting 8-12 ops, >>>>>> with one OSD occasionally spiking up to 300-500 for a few minutes. Stats >>>>>> being pulled by both local CollectD instances on each node, as well as >>>>>> the Influx plugin in MGR as we evaluate that against collectd. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Reed >>>>>> >>>>>> >>>>>>> On Feb 22, 2018, at 6:21 PM, Gregory Farnum <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> What's the output of "ceph -s" while this is happening? >>>>>>> >>>>>>> Is there some identifiable difference between these two states, like >>>>>>> you get a lot of throughput on the data pools but then metadata >>>>>>> recovery is slower? >>>>>>> >>>>>>> Are you sure the recovery is actually going slower, or are the >>>>>>> individual ops larger or more expensive? >>>>>>> >>>>>>> My WAG is that recovering the metadata pool, composed mostly of >>>>>>> directories stored in omap objects, is going much slower for some >>>>>>> reason. You can adjust the cost of those individual ops some by >>>>>>> changing osd_recovery_max_omap_entries_per_chunk (default: 8096), but >>>>>>> I'm not sure which way you want to go or indeed if this has anything to >>>>>>> do with the problem you're seeing. (eg, it could be that reading out >>>>>>> the omaps is expensive, so you can get higher recovery op numbers by >>>>>>> turning down the number of entries per request, but not actually see >>>>>>> faster backfilling because you have to issue more requests.) >>>>>>> -Greg >>>>>>> >>>>>>> On Wed, Feb 21, 2018 at 2:57 PM Reed Dier <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> I am running into an odd situation that I cannot easily explain. >>>>>>> I am currently in the midst of destroy and rebuild of OSDs from >>>>>>> filestore to bluestore. >>>>>>> With my HDDs, I am seeing expected behavior, but with my SSDs I am >>>>>>> seeing unexpected behavior. The HDDs and SSDs are set in crush >>>>>>> accordingly. >>>>>>> >>>>>>> My path to replacing the OSDs is to set the noout, norecover, >>>>>>> norebalance flag, destroy the OSD, create the OSD back, (iterate n >>>>>>> times, all within a single failure domain), unset the flags, and let it >>>>>>> go. It finishes, rinse, repeat. >>>>>>> >>>>>>> For the SSD OSDs, they are SATA SSDs (Samsung SM863a) , 10 to a node, >>>>>>> with 2 NVMe drives (Intel P3700), 5 SATA SSDs to 1 NVMe drive, 16G >>>>>>> partitions for block.db (previously filestore journals). >>>>>>> 2x10GbE networking between the nodes. SATA backplane caps out at around >>>>>>> 10 Gb/s as its 2x 6 Gb/s controllers. Luminous 12.2.2. >>>>>>> >>>>>>> When the flags are unset, recovery starts and I see a very large rush >>>>>>> of traffic, however, after the first machine completed, the performance >>>>>>> tapered off at a rapid pace and trickles. Comparatively, I’m getting >>>>>>> 100-200 recovery ops on 3 HDDs, backfilling from 21 other HDDs, where >>>>>>> as I’m getting 150-250 recovery ops on 5 SSDs, backfilling from 40 >>>>>>> other SSDs. Every once in a while I will see a spike up to 500, 1000, >>>>>>> or even 2000 ops on the SSDs, often a few hundred recovery ops from one >>>>>>> OSD, and 8-15 ops from the others that are backfilling. >>>>>>> >>>>>>> This is a far cry from the more than 15-30k recovery ops that it >>>>>>> started off recovering with 1-3k recovery ops from a single OSD to the >>>>>>> backfilling OSD(s). And an even farther cry from the >15k recovery ops >>>>>>> I was sustaining for over an hour or more before. I was able to rebuild >>>>>>> a 1.9T SSD (1.1T used) in a little under an hour, and I could do about >>>>>>> 5 at a time and still keep it at roughly an hour to backfill all of >>>>>>> them, but then I hit a roadblock after the first machine, when I tried >>>>>>> to do 10 at a time (single machine). I am now still experiencing the >>>>>>> same thing on the third node, while doing 5 OSDs at a time. >>>>>>> >>>>>>> The pools associated with these SSDs are cephfs-metadata, as well as a >>>>>>> pure rados object pool we use for our own internal applications. Both >>>>>>> are size=3, min_size=2. >>>>>>> >>>>>>> It appears I am not the first to run into this, but it looks like there >>>>>>> was no resolution: >>>>>>> https://www.spinics.net/lists/ceph-users/msg41493.html >>>>>>> <https://www.spinics.net/lists/ceph-users/msg41493.html> >>>>>>> >>>>>>> Recovery parameters for the OSDs match what was in the previous thread, >>>>>>> sans the osd conf block listed. And current osd_max_backfills = 30 and >>>>>>> osd_recovery_max_active = 35. Very little activity on the OSDs during >>>>>>> this period, so should not be any contention for iops on the SSDs. >>>>>>> >>>>>>> The only oddity that I can attribute to things is that we had a few >>>>>>> periods of time where the disk load on one of the mons was high enough >>>>>>> to cause the mon to drop out of quorum for a brief amount of time, a >>>>>>> few times. But I wouldn’t think backfills would just get throttled due >>>>>>> to mons flapping. >>>>>>> >>>>>>> Hopefully someone has some experience or can steer me in a path to >>>>>>> improve the performance of the backfills so that I’m not stuck in >>>>>>> backfill purgatory longer than I need to be. >>>>>>> >>>>>>> Linking an imgur album with some screen grabs of the recovery ops over >>>>>>> time for the first machine, versus the second and third machines to >>>>>>> demonstrate the delta between them. >>>>>>> https://imgur.com/a/OJw4b <https://imgur.com/a/OJw4b> >>>>>>> >>>>>>> Also including a ceph osd df of the SSDs, highlighted in red are the >>>>>>> OSDs currently backfilling. Could this possibly be PG overdose? I don’t >>>>>>> ever run into ‘stuck activating’ PGs, its just painfully slow >>>>>>> backfills, like they are being throttled by ceph, that are causing me >>>>>>> to worry. Drives aren’t worn, <30 P/E cycles on the drives, so plenty >>>>>>> of life left in them. >>>>>>> >>>>>>> Thanks, >>>>>>> Reed >>>>>>> >>>>>>>> $ ceph osd df >>>>>>>> ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS >>>>>>>> 24 ssd 1.76109 1.00000 1803G 1094G 708G 60.69 1.08 260 >>>>>>>> 25 ssd 1.76109 1.00000 1803G 1136G 667G 63.01 1.12 271 >>>>>>>> 26 ssd 1.76109 1.00000 1803G 1018G 785G 56.46 1.01 243 >>>>>>>> 27 ssd 1.76109 1.00000 1803G 1065G 737G 59.10 1.05 253 >>>>>>>> 28 ssd 1.76109 1.00000 1803G 1026G 776G 56.94 1.02 245 >>>>>>>> 29 ssd 1.76109 1.00000 1803G 1132G 671G 62.79 1.12 270 >>>>>>>> 30 ssd 1.76109 1.00000 1803G 944G 859G 52.35 0.93 224 >>>>>>>> 31 ssd 1.76109 1.00000 1803G 1061G 742G 58.85 1.05 252 >>>>>>>> 32 ssd 1.76109 1.00000 1803G 1003G 799G 55.67 0.99 239 >>>>>>>> 33 ssd 1.76109 1.00000 1803G 1049G 753G 58.20 1.04 250 >>>>>>>> 34 ssd 1.76109 1.00000 1803G 1086G 717G 60.23 1.07 257 >>>>>>>> 35 ssd 1.76109 1.00000 1803G 978G 824G 54.26 0.97 232 >>>>>>>> 36 ssd 1.76109 1.00000 1803G 1057G 745G 58.64 1.05 252 >>>>>>>> 37 ssd 1.76109 1.00000 1803G 1025G 777G 56.88 1.01 244 >>>>>>>> 38 ssd 1.76109 1.00000 1803G 1047G 756G 58.06 1.04 250 >>>>>>>> 39 ssd 1.76109 1.00000 1803G 1031G 771G 57.20 1.02 246 >>>>>>>> 40 ssd 1.76109 1.00000 1803G 1029G 774G 57.07 1.02 245 >>>>>>>> 41 ssd 1.76109 1.00000 1803G 1033G 770G 57.28 1.02 245 >>>>>>>> 42 ssd 1.76109 1.00000 1803G 993G 809G 55.10 0.98 236 >>>>>>>> 43 ssd 1.76109 1.00000 1803G 1072G 731G 59.45 1.06 256 >>>>>>>> 44 ssd 1.76109 1.00000 1803G 1039G 763G 57.64 1.03 248 >>>>>>>> 45 ssd 1.76109 1.00000 1803G 992G 810G 55.06 0.98 236 >>>>>>>> 46 ssd 1.76109 1.00000 1803G 1068G 735G 59.23 1.06 254 >>>>>>>> 47 ssd 1.76109 1.00000 1803G 1020G 783G 56.57 1.01 242 >>>>>>>> 48 ssd 1.76109 1.00000 1803G 945G 857G 52.44 0.94 225 >>>>>>>> 49 ssd 1.76109 1.00000 1803G 649G 1154G 36.01 0.64 139 >>>>>>>> 50 ssd 1.76109 1.00000 1803G 426G 1377G 23.64 0.42 83 >>>>>>>> 51 ssd 1.76109 1.00000 1803G 610G 1193G 33.84 0.60 131 >>>>>>>> 52 ssd 1.76109 1.00000 1803G 558G 1244G 30.98 0.55 118 >>>>>>>> 53 ssd 1.76109 1.00000 1803G 731G 1072G 40.54 0.72 161 >>>>>>>> 54 ssd 1.74599 1.00000 1787G 859G 928G 48.06 0.86 229 >>>>>>>> 55 ssd 1.74599 1.00000 1787G 942G 844G 52.74 0.94 252 >>>>>>>> 56 ssd 1.74599 1.00000 1787G 928G 859G 51.94 0.93 246 >>>>>>>> 57 ssd 1.74599 1.00000 1787G 1039G 748G 58.15 1.04 277 >>>>>>>> 58 ssd 1.74599 1.00000 1787G 963G 824G 53.87 0.96 255 >>>>>>>> 59 ssd 1.74599 1.00000 1787G 909G 877G 50.89 0.91 241 >>>>>>>> 60 ssd 1.74599 1.00000 1787G 1039G 748G 58.15 1.04 277 >>>>>>>> 61 ssd 1.74599 1.00000 1787G 892G 895G 49.91 0.89 238 >>>>>>>> 62 ssd 1.74599 1.00000 1787G 927G 859G 51.90 0.93 245 >>>>>>>> 63 ssd 1.74599 1.00000 1787G 864G 922G 48.39 0.86 229 >>>>>>>> 64 ssd 1.74599 1.00000 1787G 968G 819G 54.16 0.97 257 >>>>>>>> 65 ssd 1.74599 1.00000 1787G 892G 894G 49.93 0.89 237 >>>>>>>> 66 ssd 1.74599 1.00000 1787G 951G 836G 53.23 0.95 252 >>>>>>>> 67 ssd 1.74599 1.00000 1787G 878G 908G 49.16 0.88 232 >>>>>>>> 68 ssd 1.74599 1.00000 1787G 899G 888G 50.29 0.90 238 >>>>>>>> 69 ssd 1.74599 1.00000 1787G 948G 839G 53.04 0.95 252 >>>>>>>> 70 ssd 1.74599 1.00000 1787G 914G 873G 51.15 0.91 246 >>>>>>>> 71 ssd 1.74599 1.00000 1787G 1004G 782G 56.21 1.00 266 >>>>>>>> 72 ssd 1.74599 1.00000 1787G 812G 974G 45.47 0.81 216 >>>>>>>> 73 ssd 1.74599 1.00000 1787G 932G 855G 52.15 0.93 247 >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list >>>>>>> [email protected] <mailto:[email protected]> >>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>>> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> [email protected] <mailto:[email protected]> >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> [email protected] <mailto:[email protected]> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> >> >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
