On Thu, Sep 13, 2012 at 12:25:36AM +0200, Mark Nelson wrote:
> On 09/12/2012 03:08 PM, Dieter Kasper wrote:
> > On Mon, Sep 10, 2012 at 10:39:58PM +0200, Mark Nelson wrote:
> >> On 09/10/2012 03:15 PM, Mike Ryan wrote:
> >>> *Disclaimer*: these results are an investigation into potential
> >>> bottlenecks in RADOS.
> > I appreciate this investigation very much !
> >
> >>> The test setup is wholly unrealistic, and these
> >>> numbers SHOULD NOT be used as an indication of the performance of OSDs,
> >>> messaging, RADOS, or ceph in general.
> >>>
> >>>
> >>> Executive summary: rados bench has some internal bottleneck. Once that's
> >>> cleared up, we're still having some issues saturating a single
> >>> connection to an OSD. Having 2-3 connection in parallel alleviates that
> >>> (either by having> 1 OSD or having multiple bencher clients).
> >>>
> >>>
> >>> I've run three separate tests: msbench, smalliobench, and rados bench.
> >>> In all cases I was trying to determine where bottleneck(s) exist. All
> >>> the tests were run on a machine with 192 GB of RAM. The backing stores
> >>> for all OSDs and journals are RAMdisks. The stores are running XFS.
> >>>
> >>> smalliobench: I ran tests varying the number of OSDs and bencher
> >>> clients. In all cases, the number of PG's per OSD is 100.
> >>>
> >>> OSD Bencher Throughput (mbyte/sec)
> >>> 1 1 510
> >>> 1 2 800
> >>> 1 3 850
> >>> 2 1 640
> >>> 2 2 660
> >>> 2 3 670
> >>> 3 1 780
> >>> 3 2 820
> >>> 3 3 870
> >>> 4 1 850
> >>> 4 2 970
> >>> 4 3 990
> >>>
> >>> Note: these numbers are fairly fuzzy. I eyeballed them and they're only
> >>> really accurate to about 10 mbyte/sec. The small IO bencher was run with
> >>> 100 ops in flight, 4 mbyte io's, 4 mbyte files.
> >>>
> >>> msbench: ran tests trying to determine max throughput of raw messaging
> >>> layer. Varied the number of concurrently connected msbench clients and
> >>> measured aggregate throughput. Take-away: a messaging client can very
> >>> consistently push 400-500 mbytes/sec through a single socket.
> >>>
> >>> Clients Throughput (mbyte/sec)
> >>> 1 520
> >>> 2 880
> >>> 3 1300
> >>> 4 1900
> >>>
> >>> Finally, rados bench, which seems to have its own bottleneck. Running
> >>> varying numbers of these, each client seems to get 250 mbyte/sec up till
> >>> the aggregate rate is around 1000 mbyte/sec (appx line speed as measured
> >>> by iperf). These were run on a pool with 100 PGs/OSD.
> >>>
> >>> Clients Throughput (mbyte/sec)
> >>> 1 250
> >>> 2 500
> >>> 3 750
> >>> 4 1000 (very fuzzy, probably 1000 +/- 75)
> >>> 5 1000, seems to level out here
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >>> the body of a message to [email protected]
> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>
> >> Hi guys,
> >>
> >> Some background on all of this:
> >>
> >> We've been doing some performance testing at Inktank and noticed that
> >> performance with a single rados bench instance was plateauing at between
> >> 600-700MB/s.
> >
> > 4-nodes with 10GbE interconnect; journals in RAM-Disk; replica=2
> >
> > # rados bench -p pbench 20 write
> > Maintaining 16 concurrent writes of 4194304 bytes for at least 20 seconds.
> > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > 0 0 0 0 0 0 - 0
> > 1 16 288 272 1087.81 1088 0.051123 0.0571643
> > 2 16 579 563 1125.85 1164 0.045729 0.0561784
> > 3 16 863 847 1129.19 1136 0.042012 0.0560869
> > 4 16 1150 1134 1133.87 1148 0.05466 0.0559281
> > 5 16 1441 1425 1139.87 1164 0.036852 0.0556809
> > 6 16 1733 1717 1144.54 1168 0.054594 0.0556124
> > 7 16 2007 1991 1137.59 1096 0.04454 0.0556698
> > 8 16 2290 2274 1136.88 1132 0.046777 0.0560103
> > 9 16 2580 2564 1139.44 1160 0.073328 0.0559353
> > 10 16 2871 2855 1141.88 1164 0.034091 0.0558576
> > 11 16 3158 3142 1142.43 1148 0.250688 0.0558404
> > 12 16 3445 3429 1142.88 1148 0.046941 0.0558071
> > 13 16 3726 3710 1141.42 1124 0.054092 0.0559
> > 14 16 4014 3998 1142.17 1152 0.03531 0.0558533
> > 15 16 4298 4282 1141.75 1136 0.040005 0.0559383
> > 16 16 4582 4566 1141.39 1136 0.048431 0.0559162
> > 17 16 4859 4843 1139.42 1108 0.045805 0.0559891
> > 18 16 5145 5129 1139.66 1144 0.046805 0.0560177
> > 19 16 5422 5406 1137.99 1108 0.037295 0.0561341
> > 2012-09-08 14:36:32.460311min lat: 0.029503 max lat: 0.47757 avg lat:
> > 0.0561424
> > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > 20 16 5701 5685 1136.89 1116 0.041493 0.0561424
> > Total time run: 20.197129
> > Total writes made: 5702
> > Write size: 4194304
> > Bandwidth (MB/sec): 1129.269
> >
> > Stddev Bandwidth: 23.7487
> > Max bandwidth (MB/sec): 1168
> > Min bandwidth (MB/sec): 1088
> > Average Latency: 0.0564675
> > Stddev Latency: 0.0327582
> > Max latency: 0.47757
> > Min latency: 0.029503
> >
> >
> > Best Regards,
> > -Dieter
> >
>
> Well look at that! :) Now I've gotta figure out what the difference is.
> How fast are the CPUs in your rados bench machine there?
One CPU socket in each node:
model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
MemTotal: 32856332 kB
>
> Also, I should mention that at these speeds, we noticed that crc32c
> calculations were actually having a pretty big effect.
perf report
Events: 39K cycles
+ 26.29% ceph-osd ceph-osd [.] 0x45e60b
+ 4.74% ceph-osd [kernel.kallsyms] [k]
copy_user_generic_string
+ 3.37% ceph-mon ceph-mon [.]
MHeartbeat::decode_payload()
+ 2.88% ceph-osd [kernel.kallsyms] [k] futex_wake
+ 2.61% swapper [kernel.kallsyms] [k] intel_idle
+ 2.34% ceph-osd [kernel.kallsyms] [k] __memcpy
+ 1.71% ceph-osd libc-2.11.3.so [.] memcpy
+ 1.70% ceph-osd [kernel.kallsyms] [k]
__copy_user_nocache
+ 1.66% ceph-osd [kernel.kallsyms] [k] futex_requeue
+ 1.33% ceph-mon ceph-mon [.]
MOSDOpReply::~MOSDOpReply()
+ 1.18% ceph-mon libc-2.11.3.so [.] memcpy
+ 1.16% ceph-mon ceph-mon [.]
MOSDPGInfo::decode_payload()
+ 0.97% ceph-osd [kernel.kallsyms] [k] futex_wake_op
+ 0.86% ceph-mon ceph-mon [.]
MExportDirDiscoverAck::print(std::ostream&) const
+ 0.79% ceph-osd [kernel.kallsyms] [k] _raw_spin_lock
+ 0.74% ceph-mon ceph-mon [.]
MOSDPing::decode_payload()
+ 0.52% ceph-osd libtcmalloc.so.0.3.0 [.] operator
new(unsigned long)
+ 0.51% ceph-mon ceph-mon [.]
MDiscover::print(std::ostream&) const
+ 0.48% ceph-osd [xfs] [k]
xfs_bmap_add_extent
+ 0.43% ceph-mon [kernel.kallsyms] [k]
copy_user_generic_string
+ 0.39% ceph-osd [kernel.kallsyms] [k]
iov_iter_fault_in_readable
Regards,
-Dieter
> Turning them off
> gave us a 10% performance boost. We're looking at faster
> implementations now.
>
> Mark
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
