On Oct 14, 2015, at 12:48 PM, Nathan Hjelm <hje...@lanl.gov> wrote: > > I think this is from a known issue. Try applying this and run again: > > https://github.com/open-mpi/ompi/commit/952d01db70eab4cbe11ff4557434acaa928685a4.patch
The good news is that if this fixes your problem, the fix is already included in the upcoming v1.10.1 release. > -Nathan > > On Wed, Oct 14, 2015 at 06:33:07PM +0200, Paul Kapinos wrote: >> Dear Open MPI developer, >> >> We're puzzled by reproducible performance (bandwidth) penalty observed when >> comparing measurements via InfibiBand between two nodes, OpenMPI/1.10.0 >> compiled with *GCC/5.2* instead of GCC 4.8 and Intel compiler. >> >> Take a look at the attached picture of two measurements of NetPIPE >> http://bitspjoule.org/netpipe/ benchmark done with one MPI rank per node, >> communicating via QDR InfiniBand (y axis: Mbps, y axis: sample number) >> >> Up to sample 64 (8195 bytes message size) the achieved performance is >> virtually the same; from sample 65 (12285 bytes, *less* than 12k) the >> version of GCC compiled using GCC 5.2 suffer form 20%+ penalty in bandwidth. >> >> The result is reproducible and independent from nodes and ever linux >> distribution (both Scientific Linux 6 and CentOS 7 have the same results). >> Both C and Fortran benchmarks offer the very same behaviour so it is *not* >> an f08 issue. >> >> The acchieved bandwidth is definitely IB-range (gigabytes per second), the >> communication is running via InfinfiBand in all cases (no failback to IP, >> huh). >> >> The compile line is the same; the output of ompi_info --all and --params is >> the very same (cf. attachments) up to added support for fortran-08 in /5.2 >> version. >> >> We know about existence of 'eager_limit' parameter, which is *not* changed >> and is 12288 in both versions (this is *less* that the first distinguishing >> sample). >> >> Again, for us the *only* difference is usage of other (new) GCC release. >> >> Any idea about this 20%+ bandwidth loss? >> >> Best >> >> Paul Kapinos >> -- >> Dipl.-Inform. Paul Kapinos - High Performance Computing, >> RWTH Aachen University, IT Center >> Seffenter Weg 23, D 52074 Aachen (Germany) >> Tel: +49 241/80-24915 > > >> MCA btl: parameter "btl_openib_verbose" (current value: >> "false", data source: default, level: 9 dev/all, type: bool) >> Output some verbose OpenIB BTL information (0 = no >> output, nonzero = output) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_warn_no_device_params_found" >> (current value: "true", data source: default, level: 9 dev/all, type: bool, >> synonyms: btl_openib_warn_no_hca_params_found) >> Warn when no device-specific parameters are found >> in the INI file specified by the btl_openib_device_param_files MCA parameter >> (0 = do not warn; any other value = warn) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_warn_no_hca_params_found" >> (current value: "true", data source: default, level: 9 dev/all, type: bool, >> deprecated, synonym of: btl_openib_warn_no_device_params_found) >> Warn when no device-specific parameters are found >> in the INI file specified by the btl_openib_device_param_files MCA parameter >> (0 = do not warn; any other value = warn) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_warn_default_gid_prefix" >> (current value: "true", data source: default, level: 9 dev/all, type: bool) >> Warn when there is more than one active ports and >> at least one of them connected to the network with only default GID prefix >> configured (0 = do not warn; any other value = warn) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_warn_nonexistent_if" (current >> value: "true", data source: default, level: 9 dev/all, type: bool) >> Warn if non-existent devices and/or ports are >> specified in the btl_openib_if_[in|ex]clude MCA parameters (0 = do not warn; >> any other value = warn) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_abort_not_enough_reg_mem" >> (current value: "false", data source: default, level: 9 dev/all, type: bool) >> If there is not enough registered memory available >> on the system for Open MPI to function properly, Open MPI will issue a >> warning. If this MCA parameter is set to true, then Open MPI will also >> abort all MPI jobs (0 = warn, but do not abort; any other value = warn and >> abort) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_poll_cq_batch" (current >> value: "256", data source: default, level: 9 dev/all, type: unsigned) >> Retrieve up to poll_cq_batch completions from CQ >> MCA btl: parameter "btl_openib_device_param_files" (current >> value: >> "/opt/MPI/openmpi-1.10.0/linux/gcc/share/openmpi/mca-btl-openib-device-params.ini", >> data source: default, level: 9 dev/all, type: string, synonyms: >> btl_openib_hca_param_files) >> Colon-delimited list of INI-style files that >> contain device vendor/part-specific parameters (use semicolon for Windows) >> MCA btl: parameter "btl_openib_hca_param_files" (current >> value: >> "/opt/MPI/openmpi-1.10.0/linux/gcc/share/openmpi/mca-btl-openib-device-params.ini", >> data source: default, level: 9 dev/all, type: string, deprecated, synonym >> of: btl_openib_device_param_files) >> Colon-delimited list of INI-style files that >> contain device vendor/part-specific parameters (use semicolon for Windows) >> MCA btl: parameter "btl_openib_device_type" (current value: >> "all", data source: default, level: 9 dev/all, type: int) >> Specify to only use IB or iWARP network adapters >> (infiniband = only use InfiniBand HCAs; iwarp = only use iWARP NICs; all = >> use any available adapters) >> Valid values: 0:"infiniband", 0:"ib", 1:"iwarp", >> 1:"iw", 2:"all" >> MCA btl: parameter "btl_openib_max_btls" (current value: >> "-1", data source: default, level: 9 dev/all, type: int) >> Maximum number of device ports to use (-1 = use all >> available, otherwise must be >= 1) >> MCA btl: parameter "btl_openib_free_list_num" (current >> value: "8", data source: default, level: 9 dev/all, type: int) >> Initial size of free lists (must be >= 1) >> MCA btl: parameter "btl_openib_free_list_max" (current >> value: "-1", data source: default, level: 9 dev/all, type: int) >> Maximum size of free lists (-1 = infinite, >> otherwise must be >= 0) >> MCA btl: parameter "btl_openib_free_list_inc" (current >> value: "32", data source: default, level: 9 dev/all, type: int) >> Increment size of free lists (must be >= 1) >> MCA btl: parameter "btl_openib_mpool" (current value: >> "grdma", data source: default, level: 9 dev/all, type: string) >> Name of the memory pool to be used (it is unlikely >> that you will ever want to change this) >> MCA btl: parameter "btl_openib_reg_mru_len" (current value: >> "16", data source: default, level: 9 dev/all, type: int) >> Length of the registration cache most recently used >> list (must be >= 1) >> MCA btl: parameter "btl_openib_cq_size" (current value: >> "8192", data source: default, level: 9 dev/all, type: int, synonyms: >> btl_openib_ib_cq_size) >> Minimum size of the OpenFabrics completion queue >> (CQs are automatically sized based on the number of peer MPI processes; this >> value determines the *minimum* size of all CQs) >> MCA btl: parameter "btl_openib_ib_cq_size" (current value: >> "8192", data source: default, level: 9 dev/all, type: int, deprecated, >> synonym of: btl_openib_cq_size) >> Minimum size of the OpenFabrics completion queue >> (CQs are automatically sized based on the number of peer MPI processes; this >> value determines the *minimum* size of all CQs) >> MCA btl: parameter "btl_openib_max_inline_data" (current >> value: "-1", data source: default, level: 9 dev/all, type: int, synonyms: >> btl_openib_ib_max_inline_data) >> Maximum size of inline data segment (-1 = run-time >> probe to discover max value, otherwise must be >= 0). If not explicitly set, >> use max_inline_data from the INI file containing device-specific parameters >> MCA btl: parameter "btl_openib_ib_max_inline_data" (current >> value: "-1", data source: default, level: 9 dev/all, type: int, deprecated, >> synonym of: btl_openib_max_inline_data) >> Maximum size of inline data segment (-1 = run-time >> probe to discover max value, otherwise must be >= 0). If not explicitly set, >> use max_inline_data from the INI file containing device-specific parameters >> MCA btl: parameter "btl_openib_pkey" (current value: "0", >> data source: default, level: 9 dev/all, type: unsigned, synonyms: >> btl_openib_ib_pkey_val) >> OpenFabrics partition key (pkey) value. Unsigned >> integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be >> masked against the maximum allowable IB partition key value (0x7fff) >> MCA btl: parameter "btl_openib_ib_pkey_val" (current value: >> "0", data source: default, level: 9 dev/all, type: unsigned, deprecated, >> synonym of: btl_openib_pkey) >> OpenFabrics partition key (pkey) value. Unsigned >> integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be >> masked against the maximum allowable IB partition key value (0x7fff) >> MCA btl: parameter "btl_openib_psn" (current value: "0", >> data source: default, level: 9 dev/all, type: unsigned, synonyms: >> btl_openib_ib_psn) >> OpenFabrics packet sequence starting number (must >> be >= 0) >> MCA btl: parameter "btl_openib_ib_psn" (current value: "0", >> data source: default, level: 9 dev/all, type: unsigned, deprecated, synonym >> of: btl_openib_psn) >> OpenFabrics packet sequence starting number (must >> be >= 0) >> MCA btl: parameter "btl_openib_ib_qp_ous_rd_atom" (current >> value: "4", data source: default, level: 9 dev/all, type: unsigned) >> InfiniBand outstanding atomic reads (must be >= 0) >> MCA btl: parameter "btl_openib_mtu" (current value: "1k", >> data source: default, level: 9 dev/all, type: int, synonyms: >> btl_openib_ib_mtu) >> OpenFabrics MTU, in bytes (if not specified in INI >> files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048 >> bytes, 5=4096 bytes >> Valid values: 1:"256B", 2:"512B", 3:"1k", 4:"2k", >> 5:"4k" >> MCA btl: parameter "btl_openib_ib_mtu" (current value: "1k", >> data source: default, level: 9 dev/all, type: int, deprecated, synonym of: >> btl_openib_mtu) >> OpenFabrics MTU, in bytes (if not specified in INI >> files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048 >> bytes, 5=4096 bytes >> Valid values: 1:"256B", 2:"512B", 3:"1k", 4:"2k", >> 5:"4k" >> MCA btl: parameter "btl_openib_ib_min_rnr_timer" (current >> value: "25", data source: default, level: 9 dev/all, type: unsigned) >> InfiniBand minimum "receiver not ready" timer, in >> seconds (must be >= 0 and <= 31) >> MCA btl: parameter "btl_openib_ib_timeout" (current value: >> "24", data source: environment, level: 9 dev/all, type: unsigned) >> InfiniBand transmit timeout, plugged into formula: >> 4.096 microseconds * (2^btl_openib_ib_timeout) (must be >= 0 and <= 31) >> MCA btl: parameter "btl_openib_ib_retry_count" (current >> value: "7", data source: default, level: 9 dev/all, type: unsigned) >> InfiniBand transmit retry count (must be >= 0 and >> <= 7) >> MCA btl: parameter "btl_openib_ib_rnr_retry" (current value: >> "7", data source: default, level: 9 dev/all, type: unsigned) >> InfiniBand "receiver not ready" retry count; >> applies *only* to SRQ/XRC queues. PP queues use RNR retry values of 0 >> because Open MPI performs software flow control to guarantee that RNRs never >> occur (must be >= 0 and <= 7; 7 = "infinite") >> MCA btl: parameter "btl_openib_ib_max_rdma_dst_ops" (current >> value: "4", data source: default, level: 9 dev/all, type: unsigned) >> InfiniBand maximum pending RDMA destination >> operations (must be >= 0) >> MCA btl: parameter "btl_openib_ib_service_level" (current >> value: "0", data source: default, level: 9 dev/all, type: unsigned) >> InfiniBand service level (must be >= 0 and <= 15) >> MCA btl: parameter "btl_openib_ib_path_record_service_level" >> (current value: "0", data source: default, level: 9 dev/all, type: unsigned) >> Enable getting InfiniBand service level from >> PathRecord (must be >= 0, 0 = disabled, positive = try to get the service >> level from PathRecord) >> MCA btl: parameter "btl_openib_use_eager_rdma" (current >> value: "-1", data source: default, level: 9 dev/all, type: int) >> Use RDMA for eager messages (-1 = use device >> default, 0 = do not use eager RDMA, 1 = use eager RDMA) >> MCA btl: parameter "btl_openib_eager_rdma_threshold" >> (current value: "16", data source: default, level: 9 dev/all, type: int) >> Use RDMA for short messages after this number of >> messages are received from a given peer (must be >= 1) >> MCA btl: parameter "btl_openib_max_eager_rdma" (current >> value: "16", data source: default, level: 9 dev/all, type: int) >> Maximum number of peers allowed to use RDMA for >> short messages (RDMA is used for all long messages, except if explicitly >> disabled, such as with the "dr" pml) (must be >= 0) >> MCA btl: parameter "btl_openib_eager_rdma_num" (current >> value: "17", data source: default, level: 9 dev/all, type: int) >> Number of RDMA buffers to allocate for small >> messages (must be >= 1) >> MCA btl: parameter "btl_openib_btls_per_lid" (current value: >> "1", data source: default, level: 9 dev/all, type: unsigned) >> Number of BTLs to create for each InfiniBand LID >> (must be >= 1) >> MCA btl: parameter "btl_openib_max_lmc" (current value: "1", >> data source: default, level: 9 dev/all, type: unsigned) >> Maximum number of LIDs to use for each device port >> (must be >= 0, where 0 = use all available) >> MCA btl: parameter "btl_openib_enable_apm_over_lmc" (current >> value: "0", data source: default, level: 9 dev/all, type: int) >> Maximum number of alternative paths for each device >> port (must be >= -1, where 0 = disable apm, -1 = all available alternative >> paths ) >> MCA btl: parameter "btl_openib_enable_apm_over_ports" >> (current value: "0", data source: default, level: 9 dev/all, type: int) >> Enable alternative path migration (APM) over >> different ports of the same device (must be >= 0, where 0 = disable APM over >> ports, 1 = enable APM over ports of the same device) >> MCA btl: parameter "btl_openib_use_async_event_thread" >> (current value: "true", data source: default, level: 9 dev/all, type: bool) >> If nonzero, use the thread that will handle >> InfiniBand asynchronous events >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_enable_srq_resize" (current >> value: "true", data source: default, level: 9 dev/all, type: bool) >> Enable/Disable on demand SRQ resize. (0 = without >> resizing, nonzero = with resizing) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_rroce_enable" (current value: >> "false", data source: default, level: 9 dev/all, type: bool) >> Enable/Disable routing between different subnets(0 >> = disable, nonzero = enable) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_buffer_alignment" (current >> value: "64", data source: default, level: 9 dev/all, type: unsigned) >> Preferred communication buffer alignment, in bytes >> (must be > 0 and power of two) >> MCA btl: parameter "btl_openib_cq_poll_ratio" (current >> value: "100", data source: default, level: 9 dev/all, type: unsigned) >> How often to poll high priority CQ versus low >> priority CQ >> MCA btl: parameter "btl_openib_eager_rdma_poll_ratio" >> (current value: "100", data source: default, level: 9 dev/all, type: >> unsigned) >> How often to poll eager RDMA channel versus CQ >> MCA btl: parameter "btl_openib_hp_cq_poll_per_progress" >> (current value: "10", data source: default, level: 9 dev/all, type: unsigned) >> Max number of completion events to process for each >> call of BTL progress engine >> MCA btl: parameter "btl_openib_max_hw_msg_size" (current >> value: "0", data source: default, level: 9 dev/all, type: unsigned) >> Maximum size (in bytes) of a single fragment of a >> long message when using the RDMA protocols (must be > 0 and <= hw >> capabilities). >> MCA btl: parameter >> "btl_openib_allow_max_memory_registration" (current value: "true", data >> source: default, level: 9 dev/all, type: bool) >> Allow maximum possible memory to register with HCA >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_memory_registration_verbose" >> (current value: "0", data source: default, level: 9 dev/all, type: int) >> Output some verbose memory registration information >> (0 = no output, nonzero = output) >> MCA btl: parameter "btl_openib_ignore_locality" (current >> value: "0", data source: default, level: 9 dev/all, type: int) >> Ignore any locality information and use all devices >> (0 = use locality informaiton and use only close devices, nonzero = ignore >> locality information) >> MCA btl: informational "btl_openib_have_fork_support" >> (current value: "true", data source: default, level: 9 dev/all, type: bool) >> Whether the OpenFabrics stack supports applications >> that invoke the "fork()" system call or not (0 = no, 1 = yes). Note that >> this value does NOT indicate whether the system being run on supports >> "fork()" with OpenFabrics applications or not. >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_exclusivity" (current value: >> "1024", data source: default, level: 7 dev/basic, type: unsigned) >> BTL exclusivity (must be >= 0) >> MCA btl: parameter "btl_openib_flags" (current value: "310", >> data source: default, level: 5 tuner/detail, type: unsigned) >> BTL bit flags (general flags: SEND=1, PUT=2, GET=4, >> SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by >> the "dr" PML (ignored by others): ACK=16, CHECKSUM=32, RDMA_COMPLETION=128; >> flags only used by the "bfo" PML (ignored by others): FAILOVER_SUPPORT=512) >> MCA btl: parameter "btl_openib_rndv_eager_limit" (current >> value: "12288", data source: default, level: 4 tuner/basic, type: size_t) >> Size (in bytes, including header) of "phase 1" >> fragment sent for all large messages (must be >= 0 and <= eager_limit) >> MCA btl: parameter "btl_openib_eager_limit" (current value: >> "12288", data source: default, level: 4 tuner/basic, type: size_t) >> Maximum size (in bytes, including header) of >> "short" messages (must be >= 1). >> MCA btl: parameter "btl_openib_max_send_size" (current >> value: "65536", data source: default, level: 4 tuner/basic, type: size_t) >> Maximum size (in bytes) of a single "phase 2" >> fragment of a long message when using the pipeline protocol (must be >= 1) >> MCA btl: parameter "btl_openib_rdma_pipeline_send_length" >> (current value: "1048576", data source: default, level: 4 tuner/basic, type: >> size_t) >> Length of the "phase 2" portion of a large message >> (in bytes) when using the pipeline protocol. This part of the message will >> be split into fragments of size max_send_size and sent using send/receive >> semantics (must be >= 0; only relevant when the PUT flag is set) >> MCA btl: parameter "btl_openib_rdma_pipeline_frag_size" >> (current value: "1048576", data source: default, level: 4 tuner/basic, type: >> size_t) >> Maximum size (in bytes) of a single "phase 3" >> fragment from a long message when using the pipeline protocol. These >> fragments will be sent using RDMA semantics (must be >= 1; only relevant >> when the PUT flag is set) >> MCA btl: parameter "btl_openib_min_rdma_pipeline_size" >> (current value: "1060864", data source: default, level: 4 tuner/basic, type: >> size_t) >> Messages smaller than this size (in bytes) will not >> use the RDMA pipeline protocol. Instead, they will be split into fragments >> of max_send_size and sent using send/receive semantics (must be >=0, and is >> automatically adjusted up to at least >> (eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT flag >> is set) >> MCA btl: parameter "btl_openib_latency" (current value: "4", >> data source: default, level: 5 tuner/detail, type: unsigned) >> Approximate latency of interconnect (0 = >> auto-detect value at run-time [not supported in all BTL modules], >= 1 = >> latency in microseconds) >> MCA btl: parameter "btl_openib_bandwidth" (current value: >> "0", data source: default, level: 5 tuner/detail, type: unsigned) >> Approximate maximum bandwidth of interconnect (0 = >> auto-detect value at run-time [not supported in all BTL modules], >= 1 = >> bandwidth in Mbps) >> MCA btl: parameter "btl_openib_receive_queues" (current >> value: >> "P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64", >> data source: default, level: 9 dev/all, type: string) >> Colon-delimited, comma-delimited list of receive >> queues: P,4096,8,6,4:P,32768,8,6,4 >> MCA btl: parameter "btl_openib_if_include" (current value: >> "", data source: default, level: 9 dev/all, type: string) >> Comma-delimited list of devices/ports to be used >> (e.g. "mthca0,mthca1:2"; empty value means to use all ports found). >> Mutually exclusive with btl_openib_if_exclude. >> MCA btl: parameter "btl_openib_if_exclude" (current value: >> "", data source: default, level: 9 dev/all, type: string) >> Comma-delimited list of device/ports to be excluded >> (empty value means to not exclude any ports). Mutually exclusive with >> btl_openib_if_include. >> MCA btl: parameter "btl_openib_ipaddr_include" (current >> value: "", data source: default, level: 9 dev/all, type: string) >> Comma-delimited list of IP Addresses to be used >> (e.g. "192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_exclude. >> MCA btl: parameter "btl_openib_ipaddr_exclude" (current >> value: "", data source: default, level: 9 dev/all, type: string) >> Comma-delimited list of IP Addresses to be excluded >> (e.g. "192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_include. >> MCA btl: parameter "btl_openib_gid_index" (current value: >> "0", data source: default, level: 9 dev/all, type: int) >> GID index to use on verbs device ports >> MCA btl: parameter "btl_openib_memalign" (current value: >> "32", data source: default, level: 9 dev/all, type: int) >> [64 | 32 | 0] - Enable (64bit or 32bit)/Disable(0) >> memoryalignment for all malloc calls if btl openib is used. >> MCA btl: parameter "btl_openib_memalign_threshold" (current >> value: "12288", data source: default, level: 9 dev/all, type: size_t) >> Allocating memory more than >> btl_openib_memalign_threshholdbytes will automatically be aligned to the >> value of btl_openib_memalign bytes.memalign_threshhold defaults to the same >> value as mca_btl_openib_eager_limit. >> MCA btl: parameter "btl_openib_cpc_include" (current value: >> "", data source: default, level: 9 dev/all, type: string) >> Method used to select OpenFabrics connections >> (valid values: rdmacm,udcm) >> MCA btl: parameter "btl_openib_cpc_exclude" (current value: >> "", data source: default, level: 9 dev/all, type: string) >> Method used to exclude OpenFabrics connections >> (valid values: rdmacm,udcm) >> MCA btl: parameter "btl_openib_connect_rdmacm_priority" >> (current value: "30", data source: default, level: 9 dev/all, type: int) >> The selection method priority for rdma_cm >> MCA btl: parameter "btl_openib_connect_rdmacm_port" (current >> value: "0", data source: default, level: 9 dev/all, type: unsigned) >> The selection method port for rdma_cm >> MCA btl: parameter >> "btl_openib_connect_rdmacm_resolve_timeout" (current value: "30000", data >> source: default, level: 9 dev/all, type: int) >> The timeout (in miliseconds) for address and route >> resolution >> MCA btl: parameter "btl_openib_connect_rdmacm_retry_count" >> (current value: "20", data source: default, level: 9 dev/all, type: int) >> Maximum number of times rdmacm will retry route >> resolution >> MCA btl: parameter >> "btl_openib_connect_rdmacm_reject_causes_connect_error" (current value: >> "false", data source: default, level: 9 dev/all, type: bool) >> The drivers for some devices are buggy such that an >> RDMA REJECT action may result in a CONNECT_ERROR event instead of a REJECTED >> event. Setting this MCA parameter to true tells Open MPI to treat >> CONNECT_ERROR events on connections where a REJECT is expected as a REJECT >> (default: false) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_connect_udcm_priority" >> (current value: "63", data source: default, level: 9 dev/all, type: int) >> Priority of the udcm connection method >> MCA btl: parameter "btl_openib_connect_udcm_recv_count" >> (current value: "512", data source: default, level: 9 dev/all, type: int) >> Number of registered buffers to post >> MCA btl: parameter "btl_openib_connect_udcm_timeout" >> (current value: "500000", data source: default, level: 9 dev/all, type: int) >> Ack timeout for udcm connection messages >> MCA btl: parameter "btl_openib_connect_udcm_max_retry" >> (current value: "25", data source: default, level: 9 dev/all, type: int) >> Maximum number of times to retry sending a udcm >> connection message > >> MCA btl: parameter "btl_openib_verbose" (current value: >> "false", data source: default, level: 9 dev/all, type: bool) >> Output some verbose OpenIB BTL information (0 = no >> output, nonzero = output) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_warn_no_device_params_found" >> (current value: "true", data source: default, level: 9 dev/all, type: bool, >> synonyms: btl_openib_warn_no_hca_params_found) >> Warn when no device-specific parameters are found >> in the INI file specified by the btl_openib_device_param_files MCA parameter >> (0 = do not warn; any other value = warn) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_warn_no_hca_params_found" >> (current value: "true", data source: default, level: 9 dev/all, type: bool, >> deprecated, synonym of: btl_openib_warn_no_device_params_found) >> Warn when no device-specific parameters are found >> in the INI file specified by the btl_openib_device_param_files MCA parameter >> (0 = do not warn; any other value = warn) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_warn_default_gid_prefix" >> (current value: "true", data source: default, level: 9 dev/all, type: bool) >> Warn when there is more than one active ports and >> at least one of them connected to the network with only default GID prefix >> configured (0 = do not warn; any other value = warn) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_warn_nonexistent_if" (current >> value: "true", data source: default, level: 9 dev/all, type: bool) >> Warn if non-existent devices and/or ports are >> specified in the btl_openib_if_[in|ex]clude MCA parameters (0 = do not warn; >> any other value = warn) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_abort_not_enough_reg_mem" >> (current value: "false", data source: default, level: 9 dev/all, type: bool) >> If there is not enough registered memory available >> on the system for Open MPI to function properly, Open MPI will issue a >> warning. If this MCA parameter is set to true, then Open MPI will also >> abort all MPI jobs (0 = warn, but do not abort; any other value = warn and >> abort) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_poll_cq_batch" (current >> value: "256", data source: default, level: 9 dev/all, type: unsigned) >> Retrieve up to poll_cq_batch completions from CQ >> MCA btl: parameter "btl_openib_device_param_files" (current >> value: >> "/opt/MPI/openmpi-1.10.0/linux/gcc_5.2.0/share/openmpi/mca-btl-openib-device-params.ini", >> data source: default, level: 9 dev/all, type: string, synonyms: >> btl_openib_hca_param_files) >> Colon-delimited list of INI-style files that >> contain device vendor/part-specific parameters (use semicolon for Windows) >> MCA btl: parameter "btl_openib_hca_param_files" (current >> value: >> "/opt/MPI/openmpi-1.10.0/linux/gcc_5.2.0/share/openmpi/mca-btl-openib-device-params.ini", >> data source: default, level: 9 dev/all, type: string, deprecated, synonym >> of: btl_openib_device_param_files) >> Colon-delimited list of INI-style files that >> contain device vendor/part-specific parameters (use semicolon for Windows) >> MCA btl: parameter "btl_openib_device_type" (current value: >> "all", data source: default, level: 9 dev/all, type: int) >> Specify to only use IB or iWARP network adapters >> (infiniband = only use InfiniBand HCAs; iwarp = only use iWARP NICs; all = >> use any available adapters) >> Valid values: 0:"infiniband", 0:"ib", 1:"iwarp", >> 1:"iw", 2:"all" >> MCA btl: parameter "btl_openib_max_btls" (current value: >> "-1", data source: default, level: 9 dev/all, type: int) >> Maximum number of device ports to use (-1 = use all >> available, otherwise must be >= 1) >> MCA btl: parameter "btl_openib_free_list_num" (current >> value: "8", data source: default, level: 9 dev/all, type: int) >> Initial size of free lists (must be >= 1) >> MCA btl: parameter "btl_openib_free_list_max" (current >> value: "-1", data source: default, level: 9 dev/all, type: int) >> Maximum size of free lists (-1 = infinite, >> otherwise must be >= 0) >> MCA btl: parameter "btl_openib_free_list_inc" (current >> value: "32", data source: default, level: 9 dev/all, type: int) >> Increment size of free lists (must be >= 1) >> MCA btl: parameter "btl_openib_mpool" (current value: >> "grdma", data source: default, level: 9 dev/all, type: string) >> Name of the memory pool to be used (it is unlikely >> that you will ever want to change this) >> MCA btl: parameter "btl_openib_reg_mru_len" (current value: >> "16", data source: default, level: 9 dev/all, type: int) >> Length of the registration cache most recently used >> list (must be >= 1) >> MCA btl: parameter "btl_openib_cq_size" (current value: >> "8192", data source: default, level: 9 dev/all, type: int, synonyms: >> btl_openib_ib_cq_size) >> Minimum size of the OpenFabrics completion queue >> (CQs are automatically sized based on the number of peer MPI processes; this >> value determines the *minimum* size of all CQs) >> MCA btl: parameter "btl_openib_ib_cq_size" (current value: >> "8192", data source: default, level: 9 dev/all, type: int, deprecated, >> synonym of: btl_openib_cq_size) >> Minimum size of the OpenFabrics completion queue >> (CQs are automatically sized based on the number of peer MPI processes; this >> value determines the *minimum* size of all CQs) >> MCA btl: parameter "btl_openib_max_inline_data" (current >> value: "-1", data source: default, level: 9 dev/all, type: int, synonyms: >> btl_openib_ib_max_inline_data) >> Maximum size of inline data segment (-1 = run-time >> probe to discover max value, otherwise must be >= 0). If not explicitly set, >> use max_inline_data from the INI file containing device-specific parameters >> MCA btl: parameter "btl_openib_ib_max_inline_data" (current >> value: "-1", data source: default, level: 9 dev/all, type: int, deprecated, >> synonym of: btl_openib_max_inline_data) >> Maximum size of inline data segment (-1 = run-time >> probe to discover max value, otherwise must be >= 0). If not explicitly set, >> use max_inline_data from the INI file containing device-specific parameters >> MCA btl: parameter "btl_openib_pkey" (current value: "0", >> data source: default, level: 9 dev/all, type: unsigned, synonyms: >> btl_openib_ib_pkey_val) >> OpenFabrics partition key (pkey) value. Unsigned >> integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be >> masked against the maximum allowable IB partition key value (0x7fff) >> MCA btl: parameter "btl_openib_ib_pkey_val" (current value: >> "0", data source: default, level: 9 dev/all, type: unsigned, deprecated, >> synonym of: btl_openib_pkey) >> OpenFabrics partition key (pkey) value. Unsigned >> integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be >> masked against the maximum allowable IB partition key value (0x7fff) >> MCA btl: parameter "btl_openib_psn" (current value: "0", >> data source: default, level: 9 dev/all, type: unsigned, synonyms: >> btl_openib_ib_psn) >> OpenFabrics packet sequence starting number (must >> be >= 0) >> MCA btl: parameter "btl_openib_ib_psn" (current value: "0", >> data source: default, level: 9 dev/all, type: unsigned, deprecated, synonym >> of: btl_openib_psn) >> OpenFabrics packet sequence starting number (must >> be >= 0) >> MCA btl: parameter "btl_openib_ib_qp_ous_rd_atom" (current >> value: "4", data source: default, level: 9 dev/all, type: unsigned) >> InfiniBand outstanding atomic reads (must be >= 0) >> MCA btl: parameter "btl_openib_mtu" (current value: "1k", >> data source: default, level: 9 dev/all, type: int, synonyms: >> btl_openib_ib_mtu) >> OpenFabrics MTU, in bytes (if not specified in INI >> files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048 >> bytes, 5=4096 bytes >> Valid values: 1:"256B", 2:"512B", 3:"1k", 4:"2k", >> 5:"4k" >> MCA btl: parameter "btl_openib_ib_mtu" (current value: "1k", >> data source: default, level: 9 dev/all, type: int, deprecated, synonym of: >> btl_openib_mtu) >> OpenFabrics MTU, in bytes (if not specified in INI >> files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048 >> bytes, 5=4096 bytes >> Valid values: 1:"256B", 2:"512B", 3:"1k", 4:"2k", >> 5:"4k" >> MCA btl: parameter "btl_openib_ib_min_rnr_timer" (current >> value: "25", data source: default, level: 9 dev/all, type: unsigned) >> InfiniBand minimum "receiver not ready" timer, in >> seconds (must be >= 0 and <= 31) >> MCA btl: parameter "btl_openib_ib_timeout" (current value: >> "24", data source: environment, level: 9 dev/all, type: unsigned) >> InfiniBand transmit timeout, plugged into formula: >> 4.096 microseconds * (2^btl_openib_ib_timeout) (must be >= 0 and <= 31) >> MCA btl: parameter "btl_openib_ib_retry_count" (current >> value: "7", data source: default, level: 9 dev/all, type: unsigned) >> InfiniBand transmit retry count (must be >= 0 and >> <= 7) >> MCA btl: parameter "btl_openib_ib_rnr_retry" (current value: >> "7", data source: default, level: 9 dev/all, type: unsigned) >> InfiniBand "receiver not ready" retry count; >> applies *only* to SRQ/XRC queues. PP queues use RNR retry values of 0 >> because Open MPI performs software flow control to guarantee that RNRs never >> occur (must be >= 0 and <= 7; 7 = "infinite") >> MCA btl: parameter "btl_openib_ib_max_rdma_dst_ops" (current >> value: "4", data source: default, level: 9 dev/all, type: unsigned) >> InfiniBand maximum pending RDMA destination >> operations (must be >= 0) >> MCA btl: parameter "btl_openib_ib_service_level" (current >> value: "0", data source: default, level: 9 dev/all, type: unsigned) >> InfiniBand service level (must be >= 0 and <= 15) >> MCA btl: parameter "btl_openib_ib_path_record_service_level" >> (current value: "0", data source: default, level: 9 dev/all, type: unsigned) >> Enable getting InfiniBand service level from >> PathRecord (must be >= 0, 0 = disabled, positive = try to get the service >> level from PathRecord) >> MCA btl: parameter "btl_openib_use_eager_rdma" (current >> value: "-1", data source: default, level: 9 dev/all, type: int) >> Use RDMA for eager messages (-1 = use device >> default, 0 = do not use eager RDMA, 1 = use eager RDMA) >> MCA btl: parameter "btl_openib_eager_rdma_threshold" >> (current value: "16", data source: default, level: 9 dev/all, type: int) >> Use RDMA for short messages after this number of >> messages are received from a given peer (must be >= 1) >> MCA btl: parameter "btl_openib_max_eager_rdma" (current >> value: "16", data source: default, level: 9 dev/all, type: int) >> Maximum number of peers allowed to use RDMA for >> short messages (RDMA is used for all long messages, except if explicitly >> disabled, such as with the "dr" pml) (must be >= 0) >> MCA btl: parameter "btl_openib_eager_rdma_num" (current >> value: "17", data source: default, level: 9 dev/all, type: int) >> Number of RDMA buffers to allocate for small >> messages (must be >= 1) >> MCA btl: parameter "btl_openib_btls_per_lid" (current value: >> "1", data source: default, level: 9 dev/all, type: unsigned) >> Number of BTLs to create for each InfiniBand LID >> (must be >= 1) >> MCA btl: parameter "btl_openib_max_lmc" (current value: "1", >> data source: default, level: 9 dev/all, type: unsigned) >> Maximum number of LIDs to use for each device port >> (must be >= 0, where 0 = use all available) >> MCA btl: parameter "btl_openib_enable_apm_over_lmc" (current >> value: "0", data source: default, level: 9 dev/all, type: int) >> Maximum number of alternative paths for each device >> port (must be >= -1, where 0 = disable apm, -1 = all available alternative >> paths ) >> MCA btl: parameter "btl_openib_enable_apm_over_ports" >> (current value: "0", data source: default, level: 9 dev/all, type: int) >> Enable alternative path migration (APM) over >> different ports of the same device (must be >= 0, where 0 = disable APM over >> ports, 1 = enable APM over ports of the same device) >> MCA btl: parameter "btl_openib_use_async_event_thread" >> (current value: "true", data source: default, level: 9 dev/all, type: bool) >> If nonzero, use the thread that will handle >> InfiniBand asynchronous events >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_enable_srq_resize" (current >> value: "true", data source: default, level: 9 dev/all, type: bool) >> Enable/Disable on demand SRQ resize. (0 = without >> resizing, nonzero = with resizing) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_rroce_enable" (current value: >> "false", data source: default, level: 9 dev/all, type: bool) >> Enable/Disable routing between different subnets(0 >> = disable, nonzero = enable) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_buffer_alignment" (current >> value: "64", data source: default, level: 9 dev/all, type: unsigned) >> Preferred communication buffer alignment, in bytes >> (must be > 0 and power of two) >> MCA btl: parameter "btl_openib_cq_poll_ratio" (current >> value: "100", data source: default, level: 9 dev/all, type: unsigned) >> How often to poll high priority CQ versus low >> priority CQ >> MCA btl: parameter "btl_openib_eager_rdma_poll_ratio" >> (current value: "100", data source: default, level: 9 dev/all, type: >> unsigned) >> How often to poll eager RDMA channel versus CQ >> MCA btl: parameter "btl_openib_hp_cq_poll_per_progress" >> (current value: "10", data source: default, level: 9 dev/all, type: unsigned) >> Max number of completion events to process for each >> call of BTL progress engine >> MCA btl: parameter "btl_openib_max_hw_msg_size" (current >> value: "0", data source: default, level: 9 dev/all, type: unsigned) >> Maximum size (in bytes) of a single fragment of a >> long message when using the RDMA protocols (must be > 0 and <= hw >> capabilities). >> MCA btl: parameter >> "btl_openib_allow_max_memory_registration" (current value: "true", data >> source: default, level: 9 dev/all, type: bool) >> Allow maximum possible memory to register with HCA >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_memory_registration_verbose" >> (current value: "0", data source: default, level: 9 dev/all, type: int) >> Output some verbose memory registration information >> (0 = no output, nonzero = output) >> MCA btl: parameter "btl_openib_ignore_locality" (current >> value: "0", data source: default, level: 9 dev/all, type: int) >> Ignore any locality information and use all devices >> (0 = use locality informaiton and use only close devices, nonzero = ignore >> locality information) >> MCA btl: informational "btl_openib_have_fork_support" >> (current value: "true", data source: default, level: 9 dev/all, type: bool) >> Whether the OpenFabrics stack supports applications >> that invoke the "fork()" system call or not (0 = no, 1 = yes). Note that >> this value does NOT indicate whether the system being run on supports >> "fork()" with OpenFabrics applications or not. >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_exclusivity" (current value: >> "1024", data source: default, level: 7 dev/basic, type: unsigned) >> BTL exclusivity (must be >= 0) >> MCA btl: parameter "btl_openib_flags" (current value: "310", >> data source: default, level: 5 tuner/detail, type: unsigned) >> BTL bit flags (general flags: SEND=1, PUT=2, GET=4, >> SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by >> the "dr" PML (ignored by others): ACK=16, CHECKSUM=32, RDMA_COMPLETION=128; >> flags only used by the "bfo" PML (ignored by others): FAILOVER_SUPPORT=512) >> MCA btl: parameter "btl_openib_rndv_eager_limit" (current >> value: "12288", data source: default, level: 4 tuner/basic, type: size_t) >> Size (in bytes, including header) of "phase 1" >> fragment sent for all large messages (must be >= 0 and <= eager_limit) >> MCA btl: parameter "btl_openib_eager_limit" (current value: >> "12288", data source: default, level: 4 tuner/basic, type: size_t) >> Maximum size (in bytes, including header) of >> "short" messages (must be >= 1). >> MCA btl: parameter "btl_openib_max_send_size" (current >> value: "65536", data source: default, level: 4 tuner/basic, type: size_t) >> Maximum size (in bytes) of a single "phase 2" >> fragment of a long message when using the pipeline protocol (must be >= 1) >> MCA btl: parameter "btl_openib_rdma_pipeline_send_length" >> (current value: "1048576", data source: default, level: 4 tuner/basic, type: >> size_t) >> Length of the "phase 2" portion of a large message >> (in bytes) when using the pipeline protocol. This part of the message will >> be split into fragments of size max_send_size and sent using send/receive >> semantics (must be >= 0; only relevant when the PUT flag is set) >> MCA btl: parameter "btl_openib_rdma_pipeline_frag_size" >> (current value: "1048576", data source: default, level: 4 tuner/basic, type: >> size_t) >> Maximum size (in bytes) of a single "phase 3" >> fragment from a long message when using the pipeline protocol. These >> fragments will be sent using RDMA semantics (must be >= 1; only relevant >> when the PUT flag is set) >> MCA btl: parameter "btl_openib_min_rdma_pipeline_size" >> (current value: "1060864", data source: default, level: 4 tuner/basic, type: >> size_t) >> Messages smaller than this size (in bytes) will not >> use the RDMA pipeline protocol. Instead, they will be split into fragments >> of max_send_size and sent using send/receive semantics (must be >=0, and is >> automatically adjusted up to at least >> (eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT flag >> is set) >> MCA btl: parameter "btl_openib_latency" (current value: "4", >> data source: default, level: 5 tuner/detail, type: unsigned) >> Approximate latency of interconnect (0 = >> auto-detect value at run-time [not supported in all BTL modules], >= 1 = >> latency in microseconds) >> MCA btl: parameter "btl_openib_bandwidth" (current value: >> "0", data source: default, level: 5 tuner/detail, type: unsigned) >> Approximate maximum bandwidth of interconnect (0 = >> auto-detect value at run-time [not supported in all BTL modules], >= 1 = >> bandwidth in Mbps) >> MCA btl: parameter "btl_openib_receive_queues" (current >> value: >> "P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64", >> data source: default, level: 9 dev/all, type: string) >> Colon-delimited, comma-delimited list of receive >> queues: P,4096,8,6,4:P,32768,8,6,4 >> MCA btl: parameter "btl_openib_if_include" (current value: >> "", data source: default, level: 9 dev/all, type: string) >> Comma-delimited list of devices/ports to be used >> (e.g. "mthca0,mthca1:2"; empty value means to use all ports found). >> Mutually exclusive with btl_openib_if_exclude. >> MCA btl: parameter "btl_openib_if_exclude" (current value: >> "", data source: default, level: 9 dev/all, type: string) >> Comma-delimited list of device/ports to be excluded >> (empty value means to not exclude any ports). Mutually exclusive with >> btl_openib_if_include. >> MCA btl: parameter "btl_openib_ipaddr_include" (current >> value: "", data source: default, level: 9 dev/all, type: string) >> Comma-delimited list of IP Addresses to be used >> (e.g. "192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_exclude. >> MCA btl: parameter "btl_openib_ipaddr_exclude" (current >> value: "", data source: default, level: 9 dev/all, type: string) >> Comma-delimited list of IP Addresses to be excluded >> (e.g. "192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_include. >> MCA btl: parameter "btl_openib_gid_index" (current value: >> "0", data source: default, level: 9 dev/all, type: int) >> GID index to use on verbs device ports >> MCA btl: parameter "btl_openib_memalign" (current value: >> "32", data source: default, level: 9 dev/all, type: int) >> [64 | 32 | 0] - Enable (64bit or 32bit)/Disable(0) >> memoryalignment for all malloc calls if btl openib is used. >> MCA btl: parameter "btl_openib_memalign_threshold" (current >> value: "12288", data source: default, level: 9 dev/all, type: size_t) >> Allocating memory more than >> btl_openib_memalign_threshholdbytes will automatically be aligned to the >> value of btl_openib_memalign bytes.memalign_threshhold defaults to the same >> value as mca_btl_openib_eager_limit. >> MCA btl: parameter "btl_openib_cpc_include" (current value: >> "", data source: default, level: 9 dev/all, type: string) >> Method used to select OpenFabrics connections >> (valid values: rdmacm,udcm) >> MCA btl: parameter "btl_openib_cpc_exclude" (current value: >> "", data source: default, level: 9 dev/all, type: string) >> Method used to exclude OpenFabrics connections >> (valid values: rdmacm,udcm) >> MCA btl: parameter "btl_openib_connect_rdmacm_priority" >> (current value: "30", data source: default, level: 9 dev/all, type: int) >> The selection method priority for rdma_cm >> MCA btl: parameter "btl_openib_connect_rdmacm_port" (current >> value: "0", data source: default, level: 9 dev/all, type: unsigned) >> The selection method port for rdma_cm >> MCA btl: parameter >> "btl_openib_connect_rdmacm_resolve_timeout" (current value: "30000", data >> source: default, level: 9 dev/all, type: int) >> The timeout (in miliseconds) for address and route >> resolution >> MCA btl: parameter "btl_openib_connect_rdmacm_retry_count" >> (current value: "20", data source: default, level: 9 dev/all, type: int) >> Maximum number of times rdmacm will retry route >> resolution >> MCA btl: parameter >> "btl_openib_connect_rdmacm_reject_causes_connect_error" (current value: >> "false", data source: default, level: 9 dev/all, type: bool) >> The drivers for some devices are buggy such that an >> RDMA REJECT action may result in a CONNECT_ERROR event instead of a REJECTED >> event. Setting this MCA parameter to true tells Open MPI to treat >> CONNECT_ERROR events on connections where a REJECT is expected as a REJECT >> (default: false) >> Valid values: 0: f|false|disabled, 1: t|true|enabled >> MCA btl: parameter "btl_openib_connect_udcm_priority" >> (current value: "63", data source: default, level: 9 dev/all, type: int) >> Priority of the udcm connection method >> MCA btl: parameter "btl_openib_connect_udcm_recv_count" >> (current value: "512", data source: default, level: 9 dev/all, type: int) >> Number of registered buffers to post >> MCA btl: parameter "btl_openib_connect_udcm_timeout" >> (current value: "500000", data source: default, level: 9 dev/all, type: int) >> Ack timeout for udcm connection messages >> MCA btl: parameter "btl_openib_connect_udcm_max_retry" >> (current value: "25", data source: default, level: 9 dev/all, type: int) >> Maximum number of times to retry sending a udcm >> connection message > > > > >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2015/10/18162.php > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/10/18163.php -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
