Re: [Lustre-discuss] Seeking LNET router recomendations
I don't think so, because any time two NIDs map to the same node, LNET will pick a 'best' interface (based on hop count and other things) and always use that interface. In the normal case, you would subnet to separate the interfaces and use both, but you can't in this case as ib0 is the same IP for boh. Really would be best to have a second IP for TCP. cliffw On Wed, Mar 28, 2012 at 9:01 AM, Hayes, Bob bob.ha...@intel.com wrote: When we added ‘options lnet networks=o2ib(ib0),tcp1(ib0)’ to the MDS and the OSS’s, communications from the nodes using tcp1 would not be returned by the OSS’s. We need to use two protocols over the same interface. Is this possible? ** ** *Bob Hayes* HPC Sys. Admin. Intel Corp Software Services Group/DRD/CRT-DC* *** DP3-307-H7Tel: (253)371-3040 2800 N Center DrFax: (253)371-4647 DuPont WA 98327 *bob.ha...@intel.com mallick.arigap...@intel.com* ** ** *From:* Cliff White [mailto:cli...@whamcloud.com] *Sent:* Wednesday, March 21, 2012 9:28 AM *To:* Hayes, Bob *Cc:* lustre-discuss@lists.lustre.org *Subject:* Re: [Lustre-discuss] Seeking LNET router recomendations ** ** Or to put it another way, if your OSS systems can already 'see' both IB and IPoIB networks the most cost effective, high performance solution would be to add the necessary interface and put your MDS/MGS on both networks also. No need for routers, no performance impact. cliffw ** ** On Tue, Mar 13, 2012 at 11:25 AM, Hayes, Bob bob.ha...@intel.com wrote:* *** Are there any recommendations or guidelines for sizing a LNET routing facility. ~400 nodes, 8 OSS (dual socket E5 w/48GB RAM), 24 OST (10spindle RAID6 over SRP), 1 MGS/MDT How much load does LNET routing put on a system? If I make the 8 OSS systems do double duty as IB to IPoIB routers, will it have much impact on performance? *Bob Hayes* HPC Sys. Admin. Intel Corp Software Services Group/DRD/CRT-DC* *** DP3-307-H7Tel: (253)371-3040 2800 N Center DrFax: (253)371-4647 DuPont WA 98327 *bob.ha...@intel.com mallick.arigap...@intel.com* ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ** ** -- cliffw Support Guy WhamCloud, Inc. www.whamcloud.com ** ** ** ** -- cliffw Support Guy WhamCloud, Inc. www.whamcloud.com ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Seeking LNET router recomendations
When we added 'options lnet networks=o2ib(ib0),tcp1(ib0)' to the MDS and the OSS's, communications from the nodes using tcp1 would not be returned by the OSS's. We need to use two protocols over the same interface. Is this possible? Bob Hayes HPC Sys. Admin. Intel Corp Software Services Group/DRD/CRT-DC DP3-307-H7Tel: (253)371-3040 2800 N Center DrFax: (253)371-4647 DuPont WA 98327 bob.ha...@intel.commailto:mallick.arigap...@intel.com From: Cliff White [mailto:cli...@whamcloud.com] Sent: Wednesday, March 21, 2012 9:28 AM To: Hayes, Bob Cc: lustre-discuss@lists.lustre.org Subject: Re: [Lustre-discuss] Seeking LNET router recomendations Or to put it another way, if your OSS systems can already 'see' both IB and IPoIB networks the most cost effective, high performance solution would be to add the necessary interface and put your MDS/MGS on both networks also. No need for routers, no performance impact. cliffw On Tue, Mar 13, 2012 at 11:25 AM, Hayes, Bob bob.ha...@intel.commailto:bob.ha...@intel.com wrote: Are there any recommendations or guidelines for sizing a LNET routing facility. ~400 nodes, 8 OSS (dual socket E5 w/48GB RAM), 24 OST (10spindle RAID6 over SRP), 1 MGS/MDT How much load does LNET routing put on a system? If I make the 8 OSS systems do double duty as IB to IPoIB routers, will it have much impact on performance? Bob Hayes HPC Sys. Admin. Intel Corp Software Services Group/DRD/CRT-DC DP3-307-H7Tel: (253)371-3040tel:%28253%29371-3040 2800 N Center DrFax: (253)371-4647tel:%28253%29371-4647 DuPont WA 98327 bob.ha...@intel.commailto:mallick.arigap...@intel.com ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.orgmailto:Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss -- cliffw Support Guy WhamCloud, Inc. www.whamcloud.comhttp://www.whamcloud.com ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Seeking LNET router recomendations
Are there any recommendations or guidelines for sizing a LNET routing facility. ~400 nodes, 8 OSS (dual socket E5 w/48GB RAM), 24 OST (10spindle RAID6 over SRP), 1 MGS/MDT How much load does LNET routing put on a system? If I make the 8 OSS systems do double duty as IB to IPoIB routers, will it have much impact on performance? Bob Hayes HPC Sys. Admin. Intel Corp Software Services Group/DRD/CRT-DC DP3-307-H7Tel: (253)371-3040 2800 N Center DrFax: (253)371-4647 DuPont WA 98327 bob.ha...@intel.commailto:mallick.arigap...@intel.com ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Seeking LNET router recomendations
- An OSS really can't be a router, an OSS is an endpoint. Topologically, it shouldn't work, you should re-think network layout. - routing does place a load on the system, nodes doing routing should be dedicated to routing. - Load depends on traffic, basically you would have two hardware network interfaces, and ideally would be sending max traffic through both. Impact would depend on hardware types, etc. WIth ~400 nodes, you would want a 'pool' of routers, size of pool would depend on your usage. cliffw On Tue, Mar 13, 2012 at 11:25 AM, Hayes, Bob bob.ha...@intel.com wrote: Are there any recommendations or guidelines for sizing a LNET routing facility. ~400 nodes, 8 OSS (dual socket E5 w/48GB RAM), 24 OST (10spindle RAID6 over SRP), 1 MGS/MDT How much load does LNET routing put on a system? If I make the 8 OSS systems do double duty as IB to IPoIB routers, will it have much impact on performance? ** ** *Bob Hayes* HPC Sys. Admin. Intel Corp Software Services Group/DRD/CRT-DC* *** DP3-307-H7Tel: (253)371-3040 2800 N Center DrFax: (253)371-4647 DuPont WA 98327 *bob.ha...@intel.com mallick.arigap...@intel.com* ** ** ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss -- cliffw Support Guy WhamCloud, Inc. www.whamcloud.com ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Seeking LNET router recomendations
Or to put it another way, if your OSS systems can already 'see' both IB and IPoIB networks the most cost effective, high performance solution would be to add the necessary interface and put your MDS/MGS on both networks also. No need for routers, no performance impact. cliffw On Tue, Mar 13, 2012 at 11:25 AM, Hayes, Bob bob.ha...@intel.com wrote: Are there any recommendations or guidelines for sizing a LNET routing facility. ~400 nodes, 8 OSS (dual socket E5 w/48GB RAM), 24 OST (10spindle RAID6 over SRP), 1 MGS/MDT How much load does LNET routing put on a system? If I make the 8 OSS systems do double duty as IB to IPoIB routers, will it have much impact on performance? ** ** *Bob Hayes* HPC Sys. Admin. Intel Corp Software Services Group/DRD/CRT-DC* *** DP3-307-H7Tel: (253)371-3040 2800 N Center DrFax: (253)371-4647 DuPont WA 98327 *bob.ha...@intel.com mallick.arigap...@intel.com* ** ** ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss -- cliffw Support Guy WhamCloud, Inc. www.whamcloud.com ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
