On Wed, Jun 13, 2007 at 11:03:09AM -0400, Jeff Squyres wrote: > Hey Gleb -- > > Can you explain the rationale for this change? Is there a reason why > the bandwidths reported by the IBV API are not sufficient? Are you > trying to do creative things with multi-LID scenarios (perhaps QOS- > like things)? If so, this looks like a good idea, but I'm not sure > that MCA params are the right vehicle for it. Sometimes HCA bandwidth depend on PCI-E slot it resides in. I have a setup here with 4 HCA 2 of them have bandwidth of 1500 and other 2 1150. In multi-lid setup I want to do more fancy things. Like, for instance, create fast path only for base lid and have its latency value to be lower to send all eager traffic to it.
> > I'm not particularly fond of creating variable MCA parameters after > the btl open call because they won't show up in ompi_info. Should we > do something else if you want to override bandwidths, perhaps > something similar to the HCA params file? If you recall, we made the > HCA params stuff a file because of exactly this issue: we didn't want > variable MCA parameters and we didn't want to have to make the user > type out a potentially long set of complex MCA parameters that were > dependent upon the exact environment that the user was running in. I modeled this after TCP BTL so blame George :) I actually like this notation, but obviously it has its shortcomings. I am open to other ideas if they can be implemented in a short time frame. > > > > On Jun 13, 2007, at 8:47 AM, g...@osl.iu.edu wrote: > > > Author: gleb > > Date: 2007-06-13 08:47:38 EDT (Wed, 13 Jun 2007) > > New Revision: 15041 > > URL: https://svn.open-mpi.org/trac/ompi/changeset/15041 > > > > Log: > > Allow to configure bandwidth and latency with finer granularity. > > > > Set bandwidth for all ports of mthca0: > > --mca btl_openib_bandwidth_mthca0 1000 > > > > Set bandwidth for port 1 of mthca1: > > --mca btl_openib_bandwidth_mthca1:1 1000 > > > > Set latency for port 2 lid 123 on mthca0: > > --mca btl_openib_latency_mthca0:2:123 20 > > > > Text files modified: > > trunk/ompi/mca/btl/openib/btl_openib_component.c | 43 ++++++++ > > ++++++++++++++++++++++++++++++++ > > 1 files changed, 43 insertions(+), 0 deletions(-) > > > > Modified: trunk/ompi/mca/btl/openib/btl_openib_component.c > > ====================================================================== > > ======== > > --- trunk/ompi/mca/btl/openib/btl_openib_component.c (original) > > +++ trunk/ompi/mca/btl/openib/btl_openib_component.c 2007-06-13 > > 08:47:38 EDT (Wed, 13 Jun 2007) > > @@ -299,6 +299,14 @@ > > openib_reg->mr = NULL; > > return OMPI_SUCCESS; > > } > > +static inline int param_register_int(const char* param_name, int > > default_value) > > +{ > > + int param_value = default_value; > > + int id = mca_base_param_register_int("btl", "openib", > > param_name, NULL, > > + default_value); > > + mca_base_param_lookup_int(id, ¶m_value); > > + return param_value; > > +} > > > > static int init_one_port(opal_list_t *btl_list, > > mca_btl_openib_hca_t *hca, > > uint8_t port_num, uint16_t pkey_index, > > @@ -331,6 +339,7 @@ > > for(lid = ib_port_attr->lid; > > lid < ib_port_attr->lid + lmc; lid++){ > > for(i = 0; i < mca_btl_openib_component.btls_per_lid; i++){ > > + char param[40]; > > openib_btl = malloc(sizeof(mca_btl_openib_module_t)); > > if(NULL == openib_btl) { > > BTL_ERROR(("Failed malloc: %s:%d\n", __FILE__, > > __LINE__)); > > @@ -353,6 +362,40 @@ > > openib_btl->ib_reg[MCA_BTL_TAG_BTL].cbfunc = > > btl_openib_control; > > openib_btl->ib_reg[MCA_BTL_TAG_BTL].cbdata = NULL; > > > > + /* Check bandwidth configured for this HCA */ > > + sprintf(param, "bandwidth_%s", ibv_get_device_name(hca- > > >ib_dev)); > > + openib_btl->super.btl_bandwidth = > > + param_register_int(param, openib_btl- > > >super.btl_bandwidth); > > + > > + /* Check bandwidth configured for this HCA/port */ > > + sprintf(param, "bandwidth_%s:%d", ibv_get_device_name > > (hca->ib_dev), > > + port_num); > > + openib_btl->super.btl_bandwidth = > > + param_register_int(param, openib_btl- > > >super.btl_bandwidth); > > + > > + /* Check bandwidth configured for this HCA/port/LID */ > > + sprintf(param, "bandwidth_%s:%d:%d", > > + ibv_get_device_name(hca->ib_dev), port_num, lid); > > + openib_btl->super.btl_bandwidth = > > + param_register_int(param, openib_btl- > > >super.btl_bandwidth); > > + > > + /* Check latency configured for this HCA */ > > + sprintf(param, "latency_%s", ibv_get_device_name(hca- > > >ib_dev)); > > + openib_btl->super.btl_latency = > > + param_register_int(param, openib_btl- > > >super.btl_latency); > > + > > + /* Check latency configured for this HCA/port */ > > + sprintf(param, "latency_%s:%d", ibv_get_device_name > > (hca->ib_dev), > > + port_num); > > + openib_btl->super.btl_latency = > > + param_register_int(param, openib_btl- > > >super.btl_latency); > > + > > + /* Check latency configured for this HCA/port/LID */ > > + sprintf(param, "latency_%s:%d:%d", ibv_get_device_name > > (hca->ib_dev), > > + port_num, lid); > > + openib_btl->super.btl_latency = > > + param_register_int(param, openib_btl- > > >super.btl_latency); > > + > > /* Auto-detect the port bandwidth */ > > if (0 == openib_btl->super.btl_bandwidth) { > > /* To calculate the bandwidth available on this port, > > _______________________________________________ > > svn-full mailing list > > svn-f...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/svn-full > > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Gleb.
