Hi Markus, I have a working test setup of this, seems it just depends on having the correct modprobe-confs for the involved machines.
On the MDS/OSS we have /etc/modprobe.d/lustre.conf with > options lnet networks=o2ib0(ib0) routes="tcp1 192.168.0.[3-4]@o2ib0" dead_router_check_interval=300 where 192.168.0.[3-4] are the (IBo)IPs of the two LNET-routers we installed. On the LNET-router box, we have > options lnet forwarding="enabled" > options lnet accept=all > options lnet networks="o2ib0(ib0),tcp1(eth0)" accept_timeout=15 And on the clients in the ethernet part, we have > options lnet networks=tcp1(eth0) routes="o2ib 10.1.1.[15,18]@tcp1" dead_router_check_interval=300 where 10.1.1.[15,18] are the IPs of the LNET routers on the ethernet side. Cheers, Thomas On 12/19/2011 02:46 PM, Markus Kirsch wrote: > Hello, > > > > I have a big problem finding an example how to setup a lustre-router. > > Maybe someone can direct me to the right direction. > > > > Inside the server room we have a lustre storage system with 1 combined > MDS/OSS node and 3 OSS nodes. > > Around 100 compute nodes connect to this lustre storage over IB. So > right now all lnet communication is done > > over IB. We have to connect now a small number of workstations outside > the server room to the lustre storage system. > > I would prefer to do this with a lustre router, instead of enabling > communication for the whole system over IB and TCP. > > > > > ################################################## > # server room # > # +---------+ # > # | MDS/OSS | # > # +---------+ # > # | OSS |\ # > +-------------+ > # +---------+ \ # | > workstation | > # | OSS | \ # > /+-------------+ > # +---------+ \+-----------+ +--------+ # +------------+/ > # | OSS | | IB-switch |--IB--| router |--TCP--| TCP-switch | > # +---------+ /+-----------+ +--------+ # +------------+\ > # / # > \+-------------+ > # +---------+ / # | > workstation | > # | clients |/ # > +-------------+ > # +---------+ # > # # > ################################################## > > Markus Kirsch > Consulting > > ict ag - Aachen > information · communication · technology > a Fujitsu Company > Kackertstraße 20 > 52072 Aachen > www.ictag.net > www.hpcline.de > www.ictag-shop.de > > Tel: +49 241 88949 156 > Fax: +49 241 88949 110 > Mobil: +49 177 88949 22 > Mail: [email protected] > > > > _______________________________________________ > Lustre-discuss mailing list > [email protected] > http://lists.lustre.org/mailman/listinfo/lustre-discuss -- -------------------------------------------------------------------- Thomas Roth Department: Informationstechnologie Location: SB3 1.262 Phone: +49-6159-71 1453 Fax: +49-6159-71 2986 GSI Helmholtzzentrum für Schwerionenforschung GmbH Planckstraße 1 64291 Darmstadt www.gsi.de Gesellschaft mit beschränkter Haftung Sitz der Gesellschaft: Darmstadt Handelsregister: Amtsgericht Darmstadt, HRB 1528 Geschäftsführung: Professor Dr. Dr. h.c. Horst Stöcker, Dr. Hartmut Eickhoff Vorsitzende des Aufsichtsrates: Dr. Beatrix Vierkorn-Rudolph Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
