Has anyone else a 10GE<->IB Lustre router? What are the typical performance numbers? How close do you get to 1GB/s?
Michael Am 10.09.2010 17:55, schrieb Michael Kluge: > And here are my params: > > r...@doss05:/home/tests/lnet# for F in /sys/module/lnet/parameters/* ; > do echo -n "$F: "; cat $F ; done > /sys/module/lnet/parameters/accept: secure > /sys/module/lnet/parameters/accept_backlog: 127 > /sys/module/lnet/parameters/accept_port: 988 > /sys/module/lnet/parameters/accept_timeout: 5 > /sys/module/lnet/parameters/auto_down: 1 > /sys/module/lnet/parameters/avoid_asym_router_failure: 0 > /sys/module/lnet/parameters/check_routers_before_use: 0 > /sys/module/lnet/parameters/config_on_load: 0 > /sys/module/lnet/parameters/dead_router_check_interval: 0 > /sys/module/lnet/parameters/forwarding: enabled > /sys/module/lnet/parameters/ip2nets: > /sys/module/lnet/parameters/large_router_buffers: 512 > /sys/module/lnet/parameters/live_router_check_interval: 0 > /sys/module/lnet/parameters/local_nid_dist_zero: 1 > /sys/module/lnet/parameters/networks: tcp0(eth2),o2ib(ib1) > /sys/module/lnet/parameters/peer_buffer_credits: 0 > /sys/module/lnet/parameters/portals_compatibility: none > /sys/module/lnet/parameters/router_ping_timeout: 50 > /sys/module/lnet/parameters/routes: > /sys/module/lnet/parameters/small_router_buffers: 8192 > /sys/module/lnet/parameters/tiny_router_buffers: 1024 > > I have not used ip2nets but configure routing but put explict routing > statements into the modprobe.d/ files. Is that OK? > > > Michael > > > Am 10.09.2010 um 17:48 schrieb Michael Kluge: > >> OK, IB back to back is at 1,2 GB/s, 10GE back to back at 950 MB/s, >> with additional lnet router I see 550 MB/s. Time for lnet tuning? >> >> Michael >> >>> Hi Andreas, >>> >>> Am 10.09.2010 um 16:35 schrieb Andreas Dilger: >>> >>>> On 2010-09-10, at 08:23, Michael Kluge wrote: >>>>> I have a Lustre 1.8.3 setup where I'd like to some lnet router >>>>> performance tests with routing between DDR IB<->10GE networks. >>>>> Currently I have three nodes, one with DDR IB, one with 10GE and >>>>> one with both that does the routing. A first short lnet test shows >>>>> 520-550 MB/s performance. >>>>> >>>>> Has anyone an idea which of the variables of the lnet module are >>>>> worth playing with to get this number a bit closer to 1GB/s? >>>> >>>> I would start by testing the performance on just the 10GigE side, >>>> and then separately on the IB side, to verify you are getting the >>>> expected performance from the components before trying them both >>>> together. Often it is necessary to tune the ethernet send/receive >>>> buffers. >>> >>> Ethernet back to back is at 950 MB/s. I have not looked at IB back to >>> back yet. >>> >>> >>> Michael >>> >>> -- >>> >>> Michael Kluge, M.Sc. >>> >>> Technische Universität Dresden >>> Center for Information Services and >>> High Performance Computing (ZIH) >>> D-01062 Dresden >>> Germany >>> >>> Contact: >>> Willersbau, Room WIL A 208 >>> Phone: (+49) 351 463-34217 >>> Fax: (+49) 351 463-37773 >>> e-mail: [email protected] <mailto:[email protected]> >>> WWW: http://www.tu-dresden.de/zih >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> [email protected] <mailto:[email protected]> >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> >> -- >> >> Michael Kluge, M.Sc. >> >> Technische Universität Dresden >> Center for Information Services and >> High Performance Computing (ZIH) >> D-01062 Dresden >> Germany >> >> Contact: >> Willersbau, Room WIL A 208 >> Phone: (+49) 351 463-34217 >> Fax: (+49) 351 463-37773 >> e-mail: [email protected] <mailto:[email protected]> >> WWW: http://www.tu-dresden.de/zih >> >> _______________________________________________ >> Lustre-discuss mailing list >> [email protected] <mailto:[email protected]> >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > -- > > Michael Kluge, M.Sc. > > Technische Universität Dresden > Center for Information Services and > High Performance Computing (ZIH) > D-01062 Dresden > Germany > > Contact: > Willersbau, Room WIL A 208 > Phone: (+49) 351 463-34217 > Fax: (+49) 351 463-37773 > e-mail: [email protected] <mailto:[email protected]> > WWW: http://www.tu-dresden.de/zih > > > > _______________________________________________ > Lustre-discuss mailing list > [email protected] > http://lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
