On Wed, Nov 11, 2009 at 9:20 PM, Isaac Huang <[email protected]> wrote:
> On Wed, Nov 11, 2009 at 04:07:39PM -0600, Daneil Goodman wrote: > > Hello list, > > By searching the archive, I found a similar message dated back in > > January 2008 -- How do you make an MGS/OSS listen on 2 NICs? Looks > like > > there is no final solution and I am facing the similar situation and > > need your help. > > I am running centos 5 on both server (MGS, MDS and OSS are on same > > node) and clients: 2.6.18-128.1.6.el5_lustre.1.8.0.1smp. To simplify > > the issue, suppose the network is consist of one lustre server node > and > > two lustre client nodes. The server node has two NICs: eth0(100Mb) and > > eth1(1Gb), each client node only has one NIC:eth0. The network layout > > is as below. > > Server node eth0: 72.203.10.1 (Public network) <==> Switch1 <==> > > Public node eth0: 72.203.10.2 (Public network) > > Server node eth1: 192.168.10.1 (Internal network) <==> Switch2 <==> > > Private node eth0: 192.168.10.2 (Internal network) > > Both SELinux and Fireware are turned off. Public node does not know > > Private node, but Private node do knows Public node. > > The modprobe.conf likes the following: > > On server: options lnet networks="tcp0(eth0),tcp1(eth1)" > > On clients: options lnet networks=tcp <--- since there is only one > > I think you'd need to make clients in the 72.203.10.* network use tcp0 > and clients in the 192.168.10.* tcp1. To create a uniform module > It did the trick! Thanks! option that works across the whole cluster, 'ip2nets' is your friend: > > options lnet 'ip2nets="tcp0(eth0) 72.203.10.*; tcp1(eth1) > 192.168.10.[1-10]; tcp1(eth0) 192.168.10.[100-200]" > > (assuming that servers are 192.168.10.[1-10] and clients are > 192.168.10.[100-200].) > > There are three small issues about ip2nets: 1. Looks like LNET does not like the single quotes 'ip2nets="tcp0(eth0) 72.203.10.*; tcp1(eth1) 192.168.10.*"'. It says lnet: Unknown parameter `'ip2nets' After removed single quotes, I can load lnet module. 2. According to my observation, on public network, to mount /data using above ip2nets option is slower than networks option. 3. On private network node, I cannot start LNET using ip2nets option [r...@private ~]# lsmod |grep lnet lnet 273084 1 ksocklnd libcfs 136180 2 ksocklnd,lnet [r...@private ~]# lctl network configure LNET configure error 100: Network is down /var/log/messages shows: LustreError: 31927:0:(socklnd.c:2545:ksocknal_startup()) Interface eth1 is down LustreError: 105-4: Error -100 starting up LNI tcp But if you use networks option (options lnet networks=tcp1), it works well. Do you think what is the problem? Thanks, Goodman > Isaacc >
_______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
