Seems like you are on the correct track. This is similar to my setup. subnett'ed daemon 10GbE, 1GbE with main being QDR RDMA, admin 1GbE. To my mind the most important part is Setting "privateSubnetOverride" to 1. This allows both your 1GbE and your 40GbE to be on a private subnet. Serving block over public IPs just seems wrong on SO many levels. Whether truly private/internal or not. And how many people use public IPs internally? Wait, maybe I don't want to know...
Using 'verbsRdma enable' for your FDR seems to override Daemon node name for block, at least in my experience. I love the fallback to 10GbE and then 1GbE in case of disaster when using IB. Lately we seem to be generating bugs in OpenSM at a frightening rate so that has been _extremely_ helpful. Now if we could just monitor when it happens more easily than running mmfsadm test verbs conn, say by logging a failure of RDMA? Ed OSC ________________________________________ From: [email protected] [[email protected]] on behalf of Simon Thompson (Research Computing - IT Services) [[email protected]] Sent: Monday, September 01, 2014 3:44 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] GPFS admin host name vs subnets I was just reading through the docs at: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General+Parallel+File+System+(GPFS)/page/GPFS+Network+Communication+Overview And was wondering about using admin host name bs using subnets. My reading of the page is that if say I have a 1GbE network and a 40GbE network, I could have an admin host name on the 1GbE network. But equally from the docs, it looks like I could also use subnets to achieve the same whilst allowing the admin network to be a fall back for data if necessary. For example, create the cluster using the primary name on the 1GbE network, then use the subnets property to use set the network on the 40GbE network as the first and the network on the 1GbE network as the second in the list, thus GPFS data will pass over the 40GbE network in preference and the 1GbE network will, by default only be used for admin traffic as the admin host name will just be the name of the host on the 1GbE network. Is my reading of the docs correct? Or do I really want to be creating the cluster using the 40GbE network hostnames and set the admin node name to the name of the 1GbE network interface? (there's actually also an FDR switch in there somewhere for verbs as well) Thanks Simon _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
