And here are my params: r...@doss05:/home/tests/lnet# for F in /sys/module/lnet/parameters/* ; do echo -n "$F: "; cat $F ; done /sys/module/lnet/parameters/accept: secure /sys/module/lnet/parameters/accept_backlog: 127 /sys/module/lnet/parameters/accept_port: 988 /sys/module/lnet/parameters/accept_timeout: 5 /sys/module/lnet/parameters/auto_down: 1 /sys/module/lnet/parameters/avoid_asym_router_failure: 0 /sys/module/lnet/parameters/check_routers_before_use: 0 /sys/module/lnet/parameters/config_on_load: 0 /sys/module/lnet/parameters/dead_router_check_interval: 0 /sys/module/lnet/parameters/forwarding: enabled /sys/module/lnet/parameters/ip2nets: /sys/module/lnet/parameters/large_router_buffers: 512 /sys/module/lnet/parameters/live_router_check_interval: 0 /sys/module/lnet/parameters/local_nid_dist_zero: 1 /sys/module/lnet/parameters/networks: tcp0(eth2),o2ib(ib1) /sys/module/lnet/parameters/peer_buffer_credits: 0 /sys/module/lnet/parameters/portals_compatibility: none /sys/module/lnet/parameters/router_ping_timeout: 50 /sys/module/lnet/parameters/routes: /sys/module/lnet/parameters/small_router_buffers: 8192 /sys/module/lnet/parameters/tiny_router_buffers: 1024
I have not used ip2nets but configure routing but put explict routing statements into the modprobe.d/ files. Is that OK? Michael Am 10.09.2010 um 17:48 schrieb Michael Kluge: > OK, IB back to back is at 1,2 GB/s, 10GE back to back at 950 MB/s, with > additional lnet router I see 550 MB/s. Time for lnet tuning? > > Michael > >> Hi Andreas, >> >> Am 10.09.2010 um 16:35 schrieb Andreas Dilger: >> >>> On 2010-09-10, at 08:23, Michael Kluge wrote: >>>> I have a Lustre 1.8.3 setup where I'd like to some lnet router performance >>>> tests with routing between DDR IB<->10GE networks. Currently I have three >>>> nodes, one with DDR IB, one with 10GE and one with both that does the >>>> routing. A first short lnet test shows 520-550 MB/s performance. >>>> >>>> Has anyone an idea which of the variables of the lnet module are worth >>>> playing with to get this number a bit closer to 1GB/s? >>> >>> I would start by testing the performance on just the 10GigE side, and then >>> separately on the IB side, to verify you are getting the expected >>> performance from the components before trying them both together. Often it >>> is necessary to tune the ethernet send/receive buffers. >> >> Ethernet back to back is at 950 MB/s. I have not looked at IB back to back >> yet. >> >> >> Michael >> >> -- >> >> Michael Kluge, M.Sc. >> >> Technische Universität Dresden >> Center for Information Services and >> High Performance Computing (ZIH) >> D-01062 Dresden >> Germany >> >> Contact: >> Willersbau, Room WIL A 208 >> Phone: (+49) 351 463-34217 >> Fax: (+49) 351 463-37773 >> e-mail: [email protected] >> WWW: http://www.tu-dresden.de/zih >> >> _______________________________________________ >> Lustre-discuss mailing list >> [email protected] >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > -- > > Michael Kluge, M.Sc. > > Technische Universität Dresden > Center for Information Services and > High Performance Computing (ZIH) > D-01062 Dresden > Germany > > Contact: > Willersbau, Room WIL A 208 > Phone: (+49) 351 463-34217 > Fax: (+49) 351 463-37773 > e-mail: [email protected] > WWW: http://www.tu-dresden.de/zih > > _______________________________________________ > Lustre-discuss mailing list > [email protected] > http://lists.lustre.org/mailman/listinfo/lustre-discuss -- Michael Kluge, M.Sc. Technische Universität Dresden Center for Information Services and High Performance Computing (ZIH) D-01062 Dresden Germany Contact: Willersbau, Room WIL A 208 Phone: (+49) 351 463-34217 Fax: (+49) 351 463-37773 e-mail: [email protected] WWW: http://www.tu-dresden.de/zih
_______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
