Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-20 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > William Josefsson > Sent: 20 October 2016 10:25 > To: Nick Fisk <n...@fisk.me.uk> > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] RBD with SSD journa

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-20 Thread William Josefsson
On Mon, Oct 17, 2016 at 6:16 PM, Nick Fisk wrote: > Did you also set /check the c-states, this can have a large impact as well? Hi Nick. I did try intel_idle.max_cstate=0, and I've got quite a significant improvement as attached below. Thanks for this advice! This is still with

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-18 Thread Christian Balzer
Hello, Note that the tests below were done on a VM with RBD cache disabled, so the "direct=1" flag in FIO had a similar impact to "sync=1". If your databases are MySQL, Oracle or something else that can use O_DIRECT, RBD caching can improve things dramatically for you (with the same risks that

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-18 Thread William Josefsson
Thx Christian for elaborating on this appreciate it, I will rerun some of my benchmarks and take your advice into consideration. I have also found maximum performance recommendations for the dell 730xd bios settings, hope these make sense: http://pasteboard.co/guHVMQVly.jpg I will set all these

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-18 Thread William Josefsson
ceph-users-boun...@lists.ceph.com] On Behalf Of >>> William Josefsson >>> Sent: 17 October 2016 10:39 >>> To: n...@fisk.me.uk >>> Cc: ceph-users@lists.ceph.com >>> Subject: Re: [ceph-users] RBD with SSD journals and SAS OSDs >>> >>> hi n

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-18 Thread William Josefsson
: ceph-users@lists.ceph.com >> Subject: Re: [ceph-users] RBD with SSD journals and SAS OSDs >> >> hi nick, I earlier did cpupower frequency-set --cpu-governor performance on >> all my hosts, which bumped all CPUs up to almost max >> speed or more. > > Did you als

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-17 Thread Christian Balzer
Hello, As I had this written mostly already and since it covers some points Nick raised in more detail, here we go. On Mon, 17 Oct 2016 16:30:48 +0800 William Josefsson wrote: > Thx Christian for helping troubleshooting the latency issues. I have > attached my fio job template below. >

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-17 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > William Josefsson > Sent: 17 October 2016 10:39 > To: n...@fisk.me.uk > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] RBD with SSD journals and SAS OSDs &g

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-17 Thread William Josefsson
z : 2614.125 On Mon, Oct 17, 2016 at 5:17 PM, Nick Fisk <n...@fisk.me.uk> wrote: >> -Original Message- >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >> William Josefsson >> Sent: 17 October 2016 09:31 >> To: Christian

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-17 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > William Josefsson > Sent: 17 October 2016 09:31 > To: Christian Balzer <ch...@gol.com> > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] RBD with SSD jou

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-17 Thread William Josefsson
Thx Christian for helping troubleshooting the latency issues. I have attached my fio job template below. I thought to eliminate the factor that the VM is the bottleneck, I've created a 128GB 32 cCPU flavor. Here's the latest fio benchmark. http://pastebin.ca/raw/3729693 I'm trying to benchmark

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-16 Thread Christian Balzer
Hello, On Sun, 16 Oct 2016 19:07:17 +0800 William Josefsson wrote: > Ok thanks for sharing. yes my journals are Intel S3610 200GB, which I > partition in 4 partitions each ~45GB. When I ceph-deploy I declare > these as the journals of the OSDs. > The size (45GB) of these journals is only going

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-16 Thread William Josefsson
Ok thanks for sharing. yes my journals are Intel S3610 200GB, which I partition in 4 partitions each ~45GB. When I ceph-deploy I declare these as the journals of the OSDs. I was trying to understand the blocking, and how much my SAS OSDs affected my performance. I have a total of 9 hosts, 158

Re: [ceph-users] RBD with SSD journals and SAS OSDs

2016-10-16 Thread Christian Balzer
Hello, On Sun, 16 Oct 2016 15:03:24 +0800 William Josefsson wrote: > Hi list, while I know that writes in the RADOS backend are sync() can > anyone please explain when the cluster will return on a write call for > RBD from VMs? Will data be considered synced one written to the > journal or all