Hello, I have problems with my lnet configuration on lustre 2.11.0 everything starts just fine but after a while lnet auto discovers peers and it adds the tcp network interface of my OSSes and clients so that clients start to write on lustre partition using tcp and no more o2ib. I use and need tcp just to contact the MDS, and o2ib for contacting OSSes. this configuration has always been working with Lustre 2.10.*
I tried to switch off auto peer dicovery but it did not work. I Als otried not tto use lnet.conf at all and just to use /etc/modprobe/lustre.conf with opions lnet networks=o2ib(ib0),tcp(eth0) but seems like lustre 2.11.0 does not like it anymore. so I went back to lnet.conf but I can't make it stop to auto discover tcp interfaces. after a while the tcp interfaces starts to appear while I did not configure it to do so. And they overcome the usage of o2ib. how I can prevent the usage of tcp interfaces on my OSS and clients side giving priority to the o2ib interface ? lnetctl export | grep tcp tcp bonding: 0 - net type: tcp - nid: 172.21.42.211@tcp tcp bonding: 0 tcp bonding: 0 * - primary nid: 172.21.42.202@tcp** ** - nid: 172.21.42.202@tcp* - primary nid: 172.21.42.213@tcp - nid: 172.21.42.213@tcp so 172.21.42.202@tcp is used instead of the infiniband interface, and this is discovered automatically. This is the configuration on my OSS where 172.21.42.213 is the MDS. net: - net type: tcp local NI(s): - nid: 172.21.42.211@tcp status: up interfaces: 0: enp1s0f0 - net type: o2ib local NI(s): - nid: 172.21.52.86@o2ib status: up interfaces: 0: ib0 peer: - primary nid: 172.21.42.213@tcp Multi-Rail: False peer ni: - nid: 172.21.42.213@tcp state: NA - primary nid: 172.21.52.126@o2ib Multi-Rail: False peer ni: - nid: 172.21.52.126@o2ib state: NA - primary nid: 172.21.52.127@o2ib Multi-Rail: False peer ni: - nid: 172.21.52.127@o2ib state: NA - primary nid: 172.21.52.128@o2ib Multi-Rail: False peer ni: - nid: 172.21.52.128@o2ib state: NA - primary nid: 172.21.52.129@o2ib Multi-Rail: False peer ni: - nid: 172.21.52.129@o2ib state: NA - primary nid: 172.21.52.130@o2ib Multi-Rail: False peer ni: - nid: 172.21.52.130@o2ib state: NA - primary nid: 172.21.52.131@o2ib Multi-Rail: False peer ni: - nid: 172.21.52.131@o2ib state: NA global: numa_range: 0 discovery: 0 thanks a lot Rick
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
