On Oct 21, 2010, at 9:48 AM, Joe Landman wrote: > On 10/21/2010 09:37 AM, Brock Palen wrote: >> We recently added a new oss, it has 1 1Gb interface and 1 10Gb >> interface, >> >> The 10Gb interface is eth4 10.164.0.166 The 1Gb interface is eth0 >> 10.164.0.10 > > They look like they are on the same subnet if you are using /24 ...
You are correct Both interfaces are on the same subnet: [r...@oss4-gb ~]# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.164.0.0 * 255.255.248.0 U 0 0 0 eth0 10.164.0.0 * 255.255.248.0 U 0 0 0 eth4 169.254.0.0 * 255.255.0.0 U 0 0 0 eth4 default 10.164.0.1 0.0.0.0 UG 0 0 0 eth0 There is no way to mask the lustre service away from the 1Gb interface? > >> >> In modprobe.conf I have: >> >> options lnet networks=tcp0(eth4) >> >> lctl list_nids 10.164.0....@tcp >> >>> From a host I run: >> >> lctl which_nid oss4 10.164.0....@tcp >> >> But yet I still see traffic over eth0 the 1Gb management network, >> might higher than I would expect (upto 100MB/s) The management >> interface is oss4-gb So If I do from a client: >> >> lctl which_nid oss4-gb 10.164.0...@tcp >> >> Why If I have netwroks=tcp0(eth4) and that list_nids showa only the >> 10Gb interface, do I have so much traffic over the 1Gb interface? >> There is some traffic on the 10Gb interface, but I would like to tell >> lustre 'don't use the 1Gb interface'. > > If they are on the same subnet, its possible that the 1GbE sees the arp > response first. And then its pretty much guaranteed to have the traffic > go out that port. > > If your subnets are different, this shouldn't be the issue. > >> >> Thanks! >> >> Brock Palen www.umich.edu/~brockp Center for Advanced Computing >> [email protected] (734)936-1985 >> >> >> >> _______________________________________________ Lustre-discuss >> mailing list [email protected] >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics Inc. > email: [email protected] > web : http://scalableinformatics.com > http://scalableinformatics.com/jackrabbit > phone: +1 734 786 8423 x121 > fax : +1 866 888 3112 > cell : +1 734 612 4615 > _______________________________________________ > Lustre-discuss mailing list > [email protected] > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
