Hi, On Tue, Sep 9, 2008 at 9:01 AM, Matthew Trzyna <[EMAIL PROTECTED]> wrote: > Hello > > > A "Basic Fabric Diagram" at the end. > > > I am working with a customer implementing a large IB fabric and is > encountering problems with OpenSM (OFED 1.3) when they added a new 264 node > cluster (with its own 288 port IB switch) to their existing cluster. Two > more 264 clusters are planned to be added in the near future. They recently > moved to SLES 10 SP1 and OFED 1.3 (before adding the new cluster) and had > not been experiencing these problems before. > > Could you help provide answers to the questions listed below? Additional > information about the configuration including a basic fabric diagram are > provided after the questions. > > What parameters should be set on the non-SM nodes that affect how the Subnet > Administrator functions? > What parameters should be set on the SM node(s) that affect how the Subnet > Administrator functions? And, what parameters should be removed from the SM > node(s)? (ie. ib_sa paths_per_dest=0x7f) > How should SM failover be setup? How many failover SM's should be > configured? This must happen quickly and transparently or GPFS will die > everywhere due to timeouts if this takes too long).
What is quickly enough ? > Are there SA (Subnet Administrator) commands that should not be executed on > a large "live" fabric? (ie. "saquery -p") > Should GPFS be configured "off" on the SM node(s)? > Do you know of any other OpenSM implementations that have 5 (or more) 288 > port IB switches that might have already encountered/resolved some of these > issues? There are some deployments with multiple large switches deployed. Not sure what you mean by issues; I see questions above. > The following problem that is being encountered may also be SA/SM related. A > node (NodeX) may be seen (through IPoIB) by all but a few nodes (NodesA-G). > A ping from those node (NodesA-G) to NodeX returns "Destination Host > Unreachable". A ping from NodeX to NodesA-G works. Sounds like perhaps those nodes were unable to join the broadcast group perhaps due to a rate issue. -- Hal > -------------------------------------------------------------------------------------------------- > > System Information > > Here is the current opensm.conf file: (See attached file: opensm.conf) > > It is the default configuration from the OFED 1.3 build with "priority" > added at the bottom. Note that the /etc/init.d/opensmd sources > /etc/sysconfig/opensm not etc/sysconfig/opensm.conf (opensm.conf was just > copied to opensm). There are a couple of "proposed" settings that are > commented out, that were found them on the web. > > Following are the present settings that may affect the Fabric: > > /etc/infiniband/openib.conf > SET_IPOIB_CM=no > > /etc/modprobe.conf.local > options ib_ipoib send_queue_size=512 recv_queue_size=512 > options ib_sa paths_per_dest=0x7f > > /etc/sysctl.conf > net.ipv4.neigh.ib0.base_reachable_time = 1200 > net.ipv4.neigh.default.gc_thresh3 = 3072 > net.ipv4.neigh.default.gc_thresh2 = 2500 > net.ipv4.neigh.default.gc_thresh1 = 2048 > > /etc/sysconfig/opensm > All defaults as supplied with OFED 1.3 OpenSM > > > ------------------------------------------------------- > > > Basic Fabric Diagram > > +----------+ > |Top Level |-------------------+ 20 IO nodes > +-----------------| 288 port |----------------+ 16 Viual nodes > | | IB Sw |------------+ | 2 Admin nodes > | +------| |---+ | | (SM nodes) > | | +----------+ | | | 4 Support nodes > | | | | | | > | | | | | | > 24 24 24 24 24 24 <--uplinks > | | | | | | > | | | | | +------+ > | | | | | | > |(BASE) |(SCU1) |(SCU2) |(SCU3) |(SCU4) |(SCU5) > +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ > |288-port| |288-port| |288-port| |288-port| |288-port| |288-port| > | IB Sw | | IB Sw | | IB Sw | | IB Sw | | IB Sw | | IB Sw | > +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ > 140-nodes 264-nodes 264-nodes 264-nodes 264-nodes 264-nodes > WhiteBox Dell Dell IBM IBM IBM (future) > > NOTE: SCU4 is not currently connected to the Top Level Switch. > We'd like to address these issues before making that connection. > > Subnet Managers are configured on nodes connected to the > Top Leval Switch. > > Let me know if you need any more information. > > Any help you could provide would be most appreciated. > > Thanks. > > Matt Trzyna > IBM Linux Cluster Enablement > 3039 Cornwallis Rd. > RTP, NC 27709 > e-mail: [EMAIL PROTECTED] > Office: (919) 254-9917 Tie Line: 444 > > _______________________________________________ > general mailing list > [email protected] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
