Or separate the journals as this will bring the workload down on the spinners to 3Xrather than 6X
From: Marek Dohojda [mailto:mdoho...@altitudedigital.com] Sent: Tuesday, November 24, 2015 1:24 PM To: Nick Fisk Cc: Alan Johnson; ceph-users@lists.ceph.com Subject: Re: [ceph-users] Performance question Crad I think you are 100% correct: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 0.00 369.00 33.00 1405.00 132.00 135656.00 188.86 5.61 4.02 21.94 3.60 0.70 100.00 I was kinda wondering that this maybe the case, which is why I was wondering if I should be doing too much in terms of troubleshooting. So basically what you are saying I need to wait for new version? Thank you very much everybody! On Tue, Nov 24, 2015 at 9:35 AM, Nick Fisk <n...@fisk.me.uk<mailto:n...@fisk.me.uk>> wrote: You haven’t stated what size replication you are running. Keep in mind that with a replication factor of 3, you will be writing 6x the amount of data down to disks than what the benchmark says (3x replication x2 for data+journal write). You might actually be near the hardware maximums. What does iostat looks like whilst you are running rados bench, are the disks getting maxed out? From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com<mailto:ceph-users-boun...@lists.ceph.com>] On Behalf Of Marek Dohojda Sent: 24 November 2015 16:27 To: Alan Johnson <al...@supermicro.com<mailto:al...@supermicro.com>> Cc: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Performance question 7 total servers, 20 GIG pipe between servers, both reads and writes. The network itself has plenty of pipe left, it is averaging 40Mbits/s Rados Bench SAS 30 writes Total time run: 30.591927 Total writes made: 386 Write size: 4194304 Bandwidth (MB/sec): 50.471 Stddev Bandwidth: 48.1052 Max bandwidth (MB/sec): 160 Min bandwidth (MB/sec): 0 Average Latency: 1.25908 Stddev Latency: 2.62018 Max latency: 21.2809 Min latency: 0.029227 Rados Bench SSD writes Total time run: 20.425192 Total writes made: 1405 Write size: 4194304 Bandwidth (MB/sec): 275.150 Stddev Bandwidth: 122.565 Max bandwidth (MB/sec): 576 Min bandwidth (MB/sec): 0 Average Latency: 0.231803 Stddev Latency: 0.190978 Max latency: 0.981022 Min latency: 0.0265421 As you can see SSD is better but not as much as I would expect SSD to be. On Tue, Nov 24, 2015 at 9:10 AM, Alan Johnson <al...@supermicro.com<mailto:al...@supermicro.com>> wrote: Hard to know without more config details such as no of servers, network – GigE or !0 GigE, also not sure how you are measuring, (reads or writes) you could try RADOS bench as a baseline, I would expect more performance with 7 X 10K spinners journaled to SSDs. The fact that SSDs did not perform much better may mean to a bottleneck elsewhere – network perhaps? From: Marek Dohojda [mailto:mdoho...@altitudedigital.com<mailto:mdoho...@altitudedigital.com>] Sent: Tuesday, November 24, 2015 10:37 AM To: Alan Johnson Cc: Haomai Wang; ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Performance question Yeah they are, that is one thing I was planning on changing, What I am really interested at the moment, is vague expected performance. I mean is 100MB around normal, very low, or "could be better"? On Tue, Nov 24, 2015 at 8:02 AM, Alan Johnson <al...@supermicro.com<mailto:al...@supermicro.com>> wrote: Are the journals on the same device – it might be better to use the SSDs for journaling since you are not getting better performance with SSDs? From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com<mailto:ceph-users-boun...@lists.ceph.com>] On Behalf Of Marek Dohojda Sent: Monday, November 23, 2015 10:24 PM To: Haomai Wang Cc: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Performance question Sorry I should have specified SAS is the 100 MB :) , but to be honest SSD isn't much faster. On Mon, Nov 23, 2015 at 7:38 PM, Haomai Wang <haomaiw...@gmail.com<mailto:haomaiw...@gmail.com>> wrote: On Tue, Nov 24, 2015 at 10:35 AM, Marek Dohojda <mdoho...@altitudedigital.com<mailto:mdoho...@altitudedigital.com>> wrote: > No SSD and SAS are in two separate pools. > > On Mon, Nov 23, 2015 at 7:30 PM, Haomai Wang > <haomaiw...@gmail.com<mailto:haomaiw...@gmail.com>> wrote: >> >> On Tue, Nov 24, 2015 at 10:23 AM, Marek Dohojda >> <mdoho...@altitudedigital.com<mailto:mdoho...@altitudedigital.com>> wrote: >> > I have a Hammer Ceph cluster on 7 nodes with total 14 OSDs. 7 of which >> > are >> > SSD and 7 of which are SAS 10K drives. I get typically about 100MB IO >> > rates >> > on this cluster. So which pool you get with 100 MB? >> >> You mixed up sas and ssd in one pool? >> >> > >> > I have a simple question. Is 100MB within my configuration what I >> > should >> > expect, or should it be higher? I am not sure if I should be looking for >> > issues, or just accept what I have. >> > >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> >> > http://xo4t.mj.am/link/xo4t/rsxjit1/1/NlEqhua2rOHxmXdiOCL_wA/aHR0cDovL2xpc3RzLmNlcGguY29tL2xpc3RpbmZvLmNnaS9jZXBoLXVzZXJzLWNlcGguY29t >> > >> >> >> >> -- >> Best Regards, >> >> Wheat > > -- Best Regards, Wheat
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
