Hi everybody, We have a 128 nodes (8 cores/node) 4x DDR IB cluster with 2:1 oversubscription and I use the IB net for:
- OpenMPI - Lustre - Admin (may change in future) I'am very interested in using IB QoS, as in the near future I'm deploying ADM processors having then 24 cores /node so I want to put a barrier to trafic so as no trafic (specially OpenMPI) is starved by others (specially Lustre I/O). So I read all the documentation I could get (http://www.mail-archive.com/[email protected]/msg04092.html was really very helpful) and made the configuration showed bellow. I'll really be very grateful if someone in the the list could tell me his/her opinion on the proposed configuration bellow. Any comment will be welcomed, even if the whole think is a complete nonsense, as no one in my zone (as far as I know) is using IB and QoS and is really painful. Personal doubts: - Am I taking properly into account 'latency' considerations for ? - Any need to define 'QoS Switch Port 0 options'?. - Is it interesting to make a difference for CAs and switches external ports configuration? - Not really very important to follow strictly the rule 'the weighting values for each VL should be multiples of 64', at least in vlarb_high? - Other 'weights suggested? Thanks in Advance ----- /etc/opensm/qos-policy.conf -------------------- # SL asignation to Flows. GUIDs are Port GUIDs qos-ulps default :0 # default SL (OPENMPI) any, target-port-guid 0x0002c90200279295 :1 # SL for Lustre MDT any, target-port-guid 0x0002c9020029fda9,0x0002c90200285ed5 :2 # SL for Lustre OSTs ipoib :3 # SL for Administration end-qos-ulps ----- /etc/opensm/opensm.conf ----------------------- # # QoS OPTIONS # # Enable QoS setup qos FALSE # QoS policy file to be used qos_policy_file /etc/opensm/qos-policy.conf # QoS default options qos_max_vls 4 qos_high_limit 4 qos_vlarb_high 0:128,1:64,2:0,3:0 qos_vlarb_low 0:192,1:16,2:64,3:8 qos_sl2vl 0,1,2,3,15,15,15,15,15,15,15,15,15,15,15,15 # QoS CA options qos_max_vls 4 qos_high_limit 4 qos_vlarb_high 0:128,1:64,2:0,3:0 qos_vlarb_low 0:192,1:16,2:64,3:8 qos_sl2vl 0,1,2,3,15,15,15,15,15,15,15,15,15,15,15,15 # QoS Switch Port 0 options #qos_sw0_max_vls 0 #qos_sw0_high_limit -1 #qos_sw0_vlarb_high (null) #qos_sw0_vlarb_low (null) #qos_sw0_sl2vl (null) # QoS Switch external ports options qos_swe_max_vls 4 qos_swe_high_limit 255 qos_swe_vlarb_high 0:192,1:16,2:64,3:8 qos_swe_vlarb_low 0:0,1:0,2:0,3:0 qos_swe_sl2vl 0,1,2,3,15,15,15,15,15,15,15,15,15,15,15,15 -- Ramiro Alba Centre Tecnològic de Tranferència de Calor http://www.cttc.upc.edu Escola Tècnica Superior d'Enginyeries Industrial i Aeronàutica de Terrassa Colom 11, E-08222, Terrassa, Barcelona, Spain Tel: (+34) 93 739 86 46 -- Aquest missatge ha estat analitzat per MailScanner a la cerca de virus i d'altres continguts perillosos, i es considera que est� net.
_______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
