-----Original message-----
> From:Alwin Antreich <[email protected]>
> Sent: Thursday 6th September 2018 18:36
> To: ceph-users <[email protected]>
> Cc: Menno Zonneveld <[email protected]>; Marc Roos <[email protected]>
> Subject: Re: [ceph-users] Rados performance inconsistencies, lower than
> expected performance
>
> On Thu, Sep 06, 2018 at 05:15:26PM +0200, Marc Roos wrote:
> >
> > It is idle, testing still, running a backup's at night on it.
> > How do you fill up the cluster so you can test between empty and full?
> > Do you have a "ceph df" from empty and full?
> >
> > I have done another test disabling new scrubs on the rbd.ssd pool (but
> > still 3 on hdd) with:
> > ceph tell osd.* injectargs --osd_max_backfills=0
> > Again getting slower towards the end.
> > Bandwidth (MB/sec): 395.749
> > Average Latency(s): 0.161713
> In the results you both had, the latency is twice as high as in our
> tests [1]. That can already make quiet some difference. Depending on the
> actual hardware used, there may or may not be the possibility for good
> optimisation.
>
> As a start, you could test the disks with fio, as shown in our benchmark
> paper, to get some results for comparison. The forum thread [1] has
> some benchmarks from other users for comparison.
>
> [1] https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/
Thanks for the suggestion, I redid the fio test and one server seem to be
causing trouble.
When I initially tested our SSD's according to the benchmark paper our Intel
SSD's performed more or less equal to the Samsung SSD's used.
from fio.log
fio: (groupid=0, jobs=1): err= 0: pid=3606315: Mon Sep 10 11:12:36 2018
write: io=4005.9MB, bw=68366KB/s, iops=17091, runt= 60001msec
slat (usec): min=5, max=252, avg= 5.76, stdev= 0.66
clat (usec): min=6, max=949, avg=51.72, stdev= 9.54
lat (usec): min=54, max=955, avg=57.48, stdev= 9.56
However one of the other machines (with identical SSD's) now performs poorly
compared to the others with these results
fio: (groupid=0, jobs=1): err= 0: pid=3893600: Mon Sep 10 11:15:17 2018
write: io=1258.8MB, bw=51801KB/s, iops=12950, runt= 24883msec
slat (usec): min=5, max=259, avg= 6.17, stdev= 0.78
clat (usec): min=53, max=857, avg=69.77, stdev=13.11
lat (usec): min=70, max=863, avg=75.93, stdev=13.17
I'll first resolve the slower machine before doing more testing as this surely
won't help overall performance.
> --
> Cheers,
> Alwin
Thanks!,
Menno
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
