Re: [ceph-users] kRBD write performance for high IO use cases

[Tyler Bishop]

Running 3.10 but I don't think i can change the depth on this older kernel.

I see that config option on my 4.9 test machine.  I wonder if that will
help a lot!  My cluster has wait but it seems entirely limited by the RBD
client.. the OSD are not busy and i don't really have any iowait at all.
_

*Tyler Bishop*
EST 2007


O: 513-299-7108 x1000
M: 513-646-5809
http://BeyondHosting.net 


This email is intended only for the recipient(s) above and/or
otherwise authorized personnel. The information contained herein and
attached is confidential and the property of Beyond Hosting. Any
unauthorized copying, forwarding, printing, and/or disclosing
any information related to this email is prohibited. If you received this
message in error, please contact the sender and destroy all copies of this
email and any attachment(s).


On Sat, Sep 8, 2018 at 4:56 AM Ilya Dryomov  wrote:

> On Sat, Sep 8, 2018 at 1:52 AM Tyler Bishop
>  wrote:
> >
> > I have a fairly large cluster running ceph bluestore with extremely fast
> SAS ssd for the metadata.  Doing FIO benchmarks I am getting 200k-300k
> random write iops but during sustained workloads of ElasticSearch my
> clients seem to hit a wall of around 1100 IO/s per RBD device.  I've tried
> 1 RBD and 4 RBD devices and I still only get 1100 IO per device, so 4
> devices gets me around 4k.
> >
> > Is there some sort of setting that limits each RBD devices performance?
> I've tried playing with nr_requests but that don't seem to change it at
> all... I'm just looking for another 20-30% performance on random write
> io... I even thought about doing raid 0 across 4-8 rbd devices just to get
> the io performance.
>
> What is the I/O profile of that workload?  How did you arrive at the
> 20-30% number?
>
> Which kernel are you running?  Increasing nr_requests doesn't actually
> increase the queue depth, at least on anything moderately recent.  You
> need to map with queue_depth=X for that, see [1] for details.
>
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b55841807fb864eccca0167650a65722fd7cd553
>
> Thanks,
>
> Ilya
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] kRBD write performance for high IO use cases

[Ilya Dryomov]

On Sat, Sep 8, 2018 at 1:52 AM Tyler Bishop
 wrote:
>
> I have a fairly large cluster running ceph bluestore with extremely fast SAS 
> ssd for the metadata.  Doing FIO benchmarks I am getting 200k-300k random 
> write iops but during sustained workloads of ElasticSearch my clients seem to 
> hit a wall of around 1100 IO/s per RBD device.  I've tried 1 RBD and 4 RBD 
> devices and I still only get 1100 IO per device, so 4 devices gets me around 
> 4k.
>
> Is there some sort of setting that limits each RBD devices performance?  I've 
> tried playing with nr_requests but that don't seem to change it at all... I'm 
> just looking for another 20-30% performance on random write io... I even 
> thought about doing raid 0 across 4-8 rbd devices just to get the io 
> performance.

What is the I/O profile of that workload?  How did you arrive at the
20-30% number?

Which kernel are you running?  Increasing nr_requests doesn't actually
increase the queue depth, at least on anything moderately recent.  You
need to map with queue_depth=X for that, see [1] for details.

[1] 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b55841807fb864eccca0167650a65722fd7cd553

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] kRBD write performance for high IO use cases

[Tyler Bishop]

I have a fairly large cluster running ceph bluestore with extremely fast
SAS ssd for the metadata.  Doing FIO benchmarks I am getting 200k-300k
random write iops but during sustained workloads of ElasticSearch my
clients seem to hit a wall of around 1100 IO/s per RBD device.  I've tried
1 RBD and 4 RBD devices and I still only get 1100 IO per device, so 4
devices gets me around 4k.

Is there some sort of setting that limits each RBD devices performance?
I've tried playing with nr_requests but that don't seem to change it at
all... I'm just looking for another 20-30% performance on random write
io... I even thought about doing raid 0 across 4-8 rbd devices just to get
the io performance.

Thoughts?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com