I think this is from a known issue. Try applying this and run again: https://github.com/open-mpi/ompi/commit/952d01db70eab4cbe11ff4557434acaa928685a4.patch
-Nathan On Wed, Oct 14, 2015 at 06:33:07PM +0200, Paul Kapinos wrote: > Dear Open MPI developer, > > We're puzzled by reproducible performance (bandwidth) penalty observed when > comparing measurements via InfibiBand between two nodes, OpenMPI/1.10.0 > compiled with *GCC/5.2* instead of GCC 4.8 and Intel compiler. > > Take a look at the attached picture of two measurements of NetPIPE > http://bitspjoule.org/netpipe/ benchmark done with one MPI rank per node, > communicating via QDR InfiniBand (y axis: Mbps, y axis: sample number) > > Up to sample 64 (8195 bytes message size) the achieved performance is > virtually the same; from sample 65 (12285 bytes, *less* than 12k) the > version of GCC compiled using GCC 5.2 suffer form 20%+ penalty in bandwidth. > > The result is reproducible and independent from nodes and ever linux > distribution (both Scientific Linux 6 and CentOS 7 have the same results). > Both C and Fortran benchmarks offer the very same behaviour so it is *not* > an f08 issue. > > The acchieved bandwidth is definitely IB-range (gigabytes per second), the > communication is running via InfinfiBand in all cases (no failback to IP, > huh). > > The compile line is the same; the output of ompi_info --all and --params is > the very same (cf. attachments) up to added support for fortran-08 in /5.2 > version. > > We know about existence of 'eager_limit' parameter, which is *not* changed > and is 12288 in both versions (this is *less* that the first distinguishing > sample). > > Again, for us the *only* difference is usage of other (new) GCC release. > > Any idea about this 20%+ bandwidth loss? > > Best > > Paul Kapinos > -- > Dipl.-Inform. Paul Kapinos - High Performance Computing, > RWTH Aachen University, IT Center > Seffenter Weg 23, D 52074 Aachen (Germany) > Tel: +49 241/80-24915 > MCA btl: parameter "btl_openib_verbose" (current value: > "false", data source: default, level: 9 dev/all, type: bool) > Output some verbose OpenIB BTL information (0 = no > output, nonzero = output) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_warn_no_device_params_found" > (current value: "true", data source: default, level: 9 dev/all, type: bool, > synonyms: btl_openib_warn_no_hca_params_found) > Warn when no device-specific parameters are found > in the INI file specified by the btl_openib_device_param_files MCA parameter > (0 = do not warn; any other value = warn) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_warn_no_hca_params_found" > (current value: "true", data source: default, level: 9 dev/all, type: bool, > deprecated, synonym of: btl_openib_warn_no_device_params_found) > Warn when no device-specific parameters are found > in the INI file specified by the btl_openib_device_param_files MCA parameter > (0 = do not warn; any other value = warn) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_warn_default_gid_prefix" > (current value: "true", data source: default, level: 9 dev/all, type: bool) > Warn when there is more than one active ports and > at least one of them connected to the network with only default GID prefix > configured (0 = do not warn; any other value = warn) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_warn_nonexistent_if" (current > value: "true", data source: default, level: 9 dev/all, type: bool) > Warn if non-existent devices and/or ports are > specified in the btl_openib_if_[in|ex]clude MCA parameters (0 = do not warn; > any other value = warn) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_abort_not_enough_reg_mem" > (current value: "false", data source: default, level: 9 dev/all, type: bool) > If there is not enough registered memory available > on the system for Open MPI to function properly, Open MPI will issue a > warning. If this MCA parameter is set to true, then Open MPI will also abort > all MPI jobs (0 = warn, but do not abort; any other value = warn and abort) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_poll_cq_batch" (current > value: "256", data source: default, level: 9 dev/all, type: unsigned) > Retrieve up to poll_cq_batch completions from CQ > MCA btl: parameter "btl_openib_device_param_files" (current > value: > "/opt/MPI/openmpi-1.10.0/linux/gcc/share/openmpi/mca-btl-openib-device-params.ini", > data source: default, level: 9 dev/all, type: string, synonyms: > btl_openib_hca_param_files) > Colon-delimited list of INI-style files that > contain device vendor/part-specific parameters (use semicolon for Windows) > MCA btl: parameter "btl_openib_hca_param_files" (current > value: > "/opt/MPI/openmpi-1.10.0/linux/gcc/share/openmpi/mca-btl-openib-device-params.ini", > data source: default, level: 9 dev/all, type: string, deprecated, synonym > of: btl_openib_device_param_files) > Colon-delimited list of INI-style files that > contain device vendor/part-specific parameters (use semicolon for Windows) > MCA btl: parameter "btl_openib_device_type" (current value: > "all", data source: default, level: 9 dev/all, type: int) > Specify to only use IB or iWARP network adapters > (infiniband = only use InfiniBand HCAs; iwarp = only use iWARP NICs; all = > use any available adapters) > Valid values: 0:"infiniband", 0:"ib", 1:"iwarp", > 1:"iw", 2:"all" > MCA btl: parameter "btl_openib_max_btls" (current value: > "-1", data source: default, level: 9 dev/all, type: int) > Maximum number of device ports to use (-1 = use all > available, otherwise must be >= 1) > MCA btl: parameter "btl_openib_free_list_num" (current > value: "8", data source: default, level: 9 dev/all, type: int) > Initial size of free lists (must be >= 1) > MCA btl: parameter "btl_openib_free_list_max" (current > value: "-1", data source: default, level: 9 dev/all, type: int) > Maximum size of free lists (-1 = infinite, > otherwise must be >= 0) > MCA btl: parameter "btl_openib_free_list_inc" (current > value: "32", data source: default, level: 9 dev/all, type: int) > Increment size of free lists (must be >= 1) > MCA btl: parameter "btl_openib_mpool" (current value: > "grdma", data source: default, level: 9 dev/all, type: string) > Name of the memory pool to be used (it is unlikely > that you will ever want to change this) > MCA btl: parameter "btl_openib_reg_mru_len" (current value: > "16", data source: default, level: 9 dev/all, type: int) > Length of the registration cache most recently used > list (must be >= 1) > MCA btl: parameter "btl_openib_cq_size" (current value: > "8192", data source: default, level: 9 dev/all, type: int, synonyms: > btl_openib_ib_cq_size) > Minimum size of the OpenFabrics completion queue > (CQs are automatically sized based on the number of peer MPI processes; this > value determines the *minimum* size of all CQs) > MCA btl: parameter "btl_openib_ib_cq_size" (current value: > "8192", data source: default, level: 9 dev/all, type: int, deprecated, > synonym of: btl_openib_cq_size) > Minimum size of the OpenFabrics completion queue > (CQs are automatically sized based on the number of peer MPI processes; this > value determines the *minimum* size of all CQs) > MCA btl: parameter "btl_openib_max_inline_data" (current > value: "-1", data source: default, level: 9 dev/all, type: int, synonyms: > btl_openib_ib_max_inline_data) > Maximum size of inline data segment (-1 = run-time > probe to discover max value, otherwise must be >= 0). If not explicitly set, > use max_inline_data from the INI file containing device-specific parameters > MCA btl: parameter "btl_openib_ib_max_inline_data" (current > value: "-1", data source: default, level: 9 dev/all, type: int, deprecated, > synonym of: btl_openib_max_inline_data) > Maximum size of inline data segment (-1 = run-time > probe to discover max value, otherwise must be >= 0). If not explicitly set, > use max_inline_data from the INI file containing device-specific parameters > MCA btl: parameter "btl_openib_pkey" (current value: "0", > data source: default, level: 9 dev/all, type: unsigned, synonyms: > btl_openib_ib_pkey_val) > OpenFabrics partition key (pkey) value. Unsigned > integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be > masked against the maximum allowable IB partition key value (0x7fff) > MCA btl: parameter "btl_openib_ib_pkey_val" (current value: > "0", data source: default, level: 9 dev/all, type: unsigned, deprecated, > synonym of: btl_openib_pkey) > OpenFabrics partition key (pkey) value. Unsigned > integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be > masked against the maximum allowable IB partition key value (0x7fff) > MCA btl: parameter "btl_openib_psn" (current value: "0", > data source: default, level: 9 dev/all, type: unsigned, synonyms: > btl_openib_ib_psn) > OpenFabrics packet sequence starting number (must > be >= 0) > MCA btl: parameter "btl_openib_ib_psn" (current value: "0", > data source: default, level: 9 dev/all, type: unsigned, deprecated, synonym > of: btl_openib_psn) > OpenFabrics packet sequence starting number (must > be >= 0) > MCA btl: parameter "btl_openib_ib_qp_ous_rd_atom" (current > value: "4", data source: default, level: 9 dev/all, type: unsigned) > InfiniBand outstanding atomic reads (must be >= 0) > MCA btl: parameter "btl_openib_mtu" (current value: "1k", > data source: default, level: 9 dev/all, type: int, synonyms: > btl_openib_ib_mtu) > OpenFabrics MTU, in bytes (if not specified in INI > files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048 > bytes, 5=4096 bytes > Valid values: 1:"256B", 2:"512B", 3:"1k", 4:"2k", > 5:"4k" > MCA btl: parameter "btl_openib_ib_mtu" (current value: "1k", > data source: default, level: 9 dev/all, type: int, deprecated, synonym of: > btl_openib_mtu) > OpenFabrics MTU, in bytes (if not specified in INI > files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048 > bytes, 5=4096 bytes > Valid values: 1:"256B", 2:"512B", 3:"1k", 4:"2k", > 5:"4k" > MCA btl: parameter "btl_openib_ib_min_rnr_timer" (current > value: "25", data source: default, level: 9 dev/all, type: unsigned) > InfiniBand minimum "receiver not ready" timer, in > seconds (must be >= 0 and <= 31) > MCA btl: parameter "btl_openib_ib_timeout" (current value: > "24", data source: environment, level: 9 dev/all, type: unsigned) > InfiniBand transmit timeout, plugged into formula: > 4.096 microseconds * (2^btl_openib_ib_timeout) (must be >= 0 and <= 31) > MCA btl: parameter "btl_openib_ib_retry_count" (current > value: "7", data source: default, level: 9 dev/all, type: unsigned) > InfiniBand transmit retry count (must be >= 0 and > <= 7) > MCA btl: parameter "btl_openib_ib_rnr_retry" (current value: > "7", data source: default, level: 9 dev/all, type: unsigned) > InfiniBand "receiver not ready" retry count; > applies *only* to SRQ/XRC queues. PP queues use RNR retry values of 0 > because Open MPI performs software flow control to guarantee that RNRs never > occur (must be >= 0 and <= 7; 7 = "infinite") > MCA btl: parameter "btl_openib_ib_max_rdma_dst_ops" (current > value: "4", data source: default, level: 9 dev/all, type: unsigned) > InfiniBand maximum pending RDMA destination > operations (must be >= 0) > MCA btl: parameter "btl_openib_ib_service_level" (current > value: "0", data source: default, level: 9 dev/all, type: unsigned) > InfiniBand service level (must be >= 0 and <= 15) > MCA btl: parameter "btl_openib_ib_path_record_service_level" > (current value: "0", data source: default, level: 9 dev/all, type: unsigned) > Enable getting InfiniBand service level from > PathRecord (must be >= 0, 0 = disabled, positive = try to get the service > level from PathRecord) > MCA btl: parameter "btl_openib_use_eager_rdma" (current > value: "-1", data source: default, level: 9 dev/all, type: int) > Use RDMA for eager messages (-1 = use device > default, 0 = do not use eager RDMA, 1 = use eager RDMA) > MCA btl: parameter "btl_openib_eager_rdma_threshold" > (current value: "16", data source: default, level: 9 dev/all, type: int) > Use RDMA for short messages after this number of > messages are received from a given peer (must be >= 1) > MCA btl: parameter "btl_openib_max_eager_rdma" (current > value: "16", data source: default, level: 9 dev/all, type: int) > Maximum number of peers allowed to use RDMA for > short messages (RDMA is used for all long messages, except if explicitly > disabled, such as with the "dr" pml) (must be >= 0) > MCA btl: parameter "btl_openib_eager_rdma_num" (current > value: "17", data source: default, level: 9 dev/all, type: int) > Number of RDMA buffers to allocate for small > messages (must be >= 1) > MCA btl: parameter "btl_openib_btls_per_lid" (current value: > "1", data source: default, level: 9 dev/all, type: unsigned) > Number of BTLs to create for each InfiniBand LID > (must be >= 1) > MCA btl: parameter "btl_openib_max_lmc" (current value: "1", > data source: default, level: 9 dev/all, type: unsigned) > Maximum number of LIDs to use for each device port > (must be >= 0, where 0 = use all available) > MCA btl: parameter "btl_openib_enable_apm_over_lmc" (current > value: "0", data source: default, level: 9 dev/all, type: int) > Maximum number of alternative paths for each device > port (must be >= -1, where 0 = disable apm, -1 = all available alternative > paths ) > MCA btl: parameter "btl_openib_enable_apm_over_ports" > (current value: "0", data source: default, level: 9 dev/all, type: int) > Enable alternative path migration (APM) over > different ports of the same device (must be >= 0, where 0 = disable APM over > ports, 1 = enable APM over ports of the same device) > MCA btl: parameter "btl_openib_use_async_event_thread" > (current value: "true", data source: default, level: 9 dev/all, type: bool) > If nonzero, use the thread that will handle > InfiniBand asynchronous events > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_enable_srq_resize" (current > value: "true", data source: default, level: 9 dev/all, type: bool) > Enable/Disable on demand SRQ resize. (0 = without > resizing, nonzero = with resizing) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_rroce_enable" (current value: > "false", data source: default, level: 9 dev/all, type: bool) > Enable/Disable routing between different subnets(0 > = disable, nonzero = enable) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_buffer_alignment" (current > value: "64", data source: default, level: 9 dev/all, type: unsigned) > Preferred communication buffer alignment, in bytes > (must be > 0 and power of two) > MCA btl: parameter "btl_openib_cq_poll_ratio" (current > value: "100", data source: default, level: 9 dev/all, type: unsigned) > How often to poll high priority CQ versus low > priority CQ > MCA btl: parameter "btl_openib_eager_rdma_poll_ratio" > (current value: "100", data source: default, level: 9 dev/all, type: unsigned) > How often to poll eager RDMA channel versus CQ > MCA btl: parameter "btl_openib_hp_cq_poll_per_progress" > (current value: "10", data source: default, level: 9 dev/all, type: unsigned) > Max number of completion events to process for each > call of BTL progress engine > MCA btl: parameter "btl_openib_max_hw_msg_size" (current > value: "0", data source: default, level: 9 dev/all, type: unsigned) > Maximum size (in bytes) of a single fragment of a > long message when using the RDMA protocols (must be > 0 and <= hw > capabilities). > MCA btl: parameter > "btl_openib_allow_max_memory_registration" (current value: "true", data > source: default, level: 9 dev/all, type: bool) > Allow maximum possible memory to register with HCA > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_memory_registration_verbose" > (current value: "0", data source: default, level: 9 dev/all, type: int) > Output some verbose memory registration information > (0 = no output, nonzero = output) > MCA btl: parameter "btl_openib_ignore_locality" (current > value: "0", data source: default, level: 9 dev/all, type: int) > Ignore any locality information and use all devices > (0 = use locality informaiton and use only close devices, nonzero = ignore > locality information) > MCA btl: informational "btl_openib_have_fork_support" > (current value: "true", data source: default, level: 9 dev/all, type: bool) > Whether the OpenFabrics stack supports applications > that invoke the "fork()" system call or not (0 = no, 1 = yes). Note that this > value does NOT indicate whether the system being run on supports "fork()" > with OpenFabrics applications or not. > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_exclusivity" (current value: > "1024", data source: default, level: 7 dev/basic, type: unsigned) > BTL exclusivity (must be >= 0) > MCA btl: parameter "btl_openib_flags" (current value: "310", > data source: default, level: 5 tuner/detail, type: unsigned) > BTL bit flags (general flags: SEND=1, PUT=2, GET=4, > SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by > the "dr" PML (ignored by others): ACK=16, CHECKSUM=32, RDMA_COMPLETION=128; > flags only used by the "bfo" PML (ignored by others): FAILOVER_SUPPORT=512) > MCA btl: parameter "btl_openib_rndv_eager_limit" (current > value: "12288", data source: default, level: 4 tuner/basic, type: size_t) > Size (in bytes, including header) of "phase 1" > fragment sent for all large messages (must be >= 0 and <= eager_limit) > MCA btl: parameter "btl_openib_eager_limit" (current value: > "12288", data source: default, level: 4 tuner/basic, type: size_t) > Maximum size (in bytes, including header) of > "short" messages (must be >= 1). > MCA btl: parameter "btl_openib_max_send_size" (current > value: "65536", data source: default, level: 4 tuner/basic, type: size_t) > Maximum size (in bytes) of a single "phase 2" > fragment of a long message when using the pipeline protocol (must be >= 1) > MCA btl: parameter "btl_openib_rdma_pipeline_send_length" > (current value: "1048576", data source: default, level: 4 tuner/basic, type: > size_t) > Length of the "phase 2" portion of a large message > (in bytes) when using the pipeline protocol. This part of the message will > be split into fragments of size max_send_size and sent using send/receive > semantics (must be >= 0; only relevant when the PUT flag is set) > MCA btl: parameter "btl_openib_rdma_pipeline_frag_size" > (current value: "1048576", data source: default, level: 4 tuner/basic, type: > size_t) > Maximum size (in bytes) of a single "phase 3" > fragment from a long message when using the pipeline protocol. These > fragments will be sent using RDMA semantics (must be >= 1; only relevant when > the PUT flag is set) > MCA btl: parameter "btl_openib_min_rdma_pipeline_size" > (current value: "1060864", data source: default, level: 4 tuner/basic, type: > size_t) > Messages smaller than this size (in bytes) will not > use the RDMA pipeline protocol. Instead, they will be split into fragments > of max_send_size and sent using send/receive semantics (must be >=0, and is > automatically adjusted up to at least > (eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT flag > is set) > MCA btl: parameter "btl_openib_latency" (current value: "4", > data source: default, level: 5 tuner/detail, type: unsigned) > Approximate latency of interconnect (0 = > auto-detect value at run-time [not supported in all BTL modules], >= 1 = > latency in microseconds) > MCA btl: parameter "btl_openib_bandwidth" (current value: > "0", data source: default, level: 5 tuner/detail, type: unsigned) > Approximate maximum bandwidth of interconnect (0 = > auto-detect value at run-time [not supported in all BTL modules], >= 1 = > bandwidth in Mbps) > MCA btl: parameter "btl_openib_receive_queues" (current > value: > "P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64", > data source: default, level: 9 dev/all, type: string) > Colon-delimited, comma-delimited list of receive > queues: P,4096,8,6,4:P,32768,8,6,4 > MCA btl: parameter "btl_openib_if_include" (current value: > "", data source: default, level: 9 dev/all, type: string) > Comma-delimited list of devices/ports to be used > (e.g. "mthca0,mthca1:2"; empty value means to use all ports found). Mutually > exclusive with btl_openib_if_exclude. > MCA btl: parameter "btl_openib_if_exclude" (current value: > "", data source: default, level: 9 dev/all, type: string) > Comma-delimited list of device/ports to be excluded > (empty value means to not exclude any ports). Mutually exclusive with > btl_openib_if_include. > MCA btl: parameter "btl_openib_ipaddr_include" (current > value: "", data source: default, level: 9 dev/all, type: string) > Comma-delimited list of IP Addresses to be used > (e.g. "192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_exclude. > MCA btl: parameter "btl_openib_ipaddr_exclude" (current > value: "", data source: default, level: 9 dev/all, type: string) > Comma-delimited list of IP Addresses to be excluded > (e.g. "192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_include. > MCA btl: parameter "btl_openib_gid_index" (current value: > "0", data source: default, level: 9 dev/all, type: int) > GID index to use on verbs device ports > MCA btl: parameter "btl_openib_memalign" (current value: > "32", data source: default, level: 9 dev/all, type: int) > [64 | 32 | 0] - Enable (64bit or 32bit)/Disable(0) > memoryalignment for all malloc calls if btl openib is used. > MCA btl: parameter "btl_openib_memalign_threshold" (current > value: "12288", data source: default, level: 9 dev/all, type: size_t) > Allocating memory more than > btl_openib_memalign_threshholdbytes will automatically be aligned to the > value of btl_openib_memalign bytes.memalign_threshhold defaults to the same > value as mca_btl_openib_eager_limit. > MCA btl: parameter "btl_openib_cpc_include" (current value: > "", data source: default, level: 9 dev/all, type: string) > Method used to select OpenFabrics connections > (valid values: rdmacm,udcm) > MCA btl: parameter "btl_openib_cpc_exclude" (current value: > "", data source: default, level: 9 dev/all, type: string) > Method used to exclude OpenFabrics connections > (valid values: rdmacm,udcm) > MCA btl: parameter "btl_openib_connect_rdmacm_priority" > (current value: "30", data source: default, level: 9 dev/all, type: int) > The selection method priority for rdma_cm > MCA btl: parameter "btl_openib_connect_rdmacm_port" (current > value: "0", data source: default, level: 9 dev/all, type: unsigned) > The selection method port for rdma_cm > MCA btl: parameter > "btl_openib_connect_rdmacm_resolve_timeout" (current value: "30000", data > source: default, level: 9 dev/all, type: int) > The timeout (in miliseconds) for address and route > resolution > MCA btl: parameter "btl_openib_connect_rdmacm_retry_count" > (current value: "20", data source: default, level: 9 dev/all, type: int) > Maximum number of times rdmacm will retry route > resolution > MCA btl: parameter > "btl_openib_connect_rdmacm_reject_causes_connect_error" (current value: > "false", data source: default, level: 9 dev/all, type: bool) > The drivers for some devices are buggy such that an > RDMA REJECT action may result in a CONNECT_ERROR event instead of a REJECTED > event. Setting this MCA parameter to true tells Open MPI to treat > CONNECT_ERROR events on connections where a REJECT is expected as a REJECT > (default: false) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_connect_udcm_priority" > (current value: "63", data source: default, level: 9 dev/all, type: int) > Priority of the udcm connection method > MCA btl: parameter "btl_openib_connect_udcm_recv_count" > (current value: "512", data source: default, level: 9 dev/all, type: int) > Number of registered buffers to post > MCA btl: parameter "btl_openib_connect_udcm_timeout" > (current value: "500000", data source: default, level: 9 dev/all, type: int) > Ack timeout for udcm connection messages > MCA btl: parameter "btl_openib_connect_udcm_max_retry" > (current value: "25", data source: default, level: 9 dev/all, type: int) > Maximum number of times to retry sending a udcm > connection message > MCA btl: parameter "btl_openib_verbose" (current value: > "false", data source: default, level: 9 dev/all, type: bool) > Output some verbose OpenIB BTL information (0 = no > output, nonzero = output) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_warn_no_device_params_found" > (current value: "true", data source: default, level: 9 dev/all, type: bool, > synonyms: btl_openib_warn_no_hca_params_found) > Warn when no device-specific parameters are found > in the INI file specified by the btl_openib_device_param_files MCA parameter > (0 = do not warn; any other value = warn) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_warn_no_hca_params_found" > (current value: "true", data source: default, level: 9 dev/all, type: bool, > deprecated, synonym of: btl_openib_warn_no_device_params_found) > Warn when no device-specific parameters are found > in the INI file specified by the btl_openib_device_param_files MCA parameter > (0 = do not warn; any other value = warn) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_warn_default_gid_prefix" > (current value: "true", data source: default, level: 9 dev/all, type: bool) > Warn when there is more than one active ports and > at least one of them connected to the network with only default GID prefix > configured (0 = do not warn; any other value = warn) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_warn_nonexistent_if" (current > value: "true", data source: default, level: 9 dev/all, type: bool) > Warn if non-existent devices and/or ports are > specified in the btl_openib_if_[in|ex]clude MCA parameters (0 = do not warn; > any other value = warn) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_abort_not_enough_reg_mem" > (current value: "false", data source: default, level: 9 dev/all, type: bool) > If there is not enough registered memory available > on the system for Open MPI to function properly, Open MPI will issue a > warning. If this MCA parameter is set to true, then Open MPI will also abort > all MPI jobs (0 = warn, but do not abort; any other value = warn and abort) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_poll_cq_batch" (current > value: "256", data source: default, level: 9 dev/all, type: unsigned) > Retrieve up to poll_cq_batch completions from CQ > MCA btl: parameter "btl_openib_device_param_files" (current > value: > "/opt/MPI/openmpi-1.10.0/linux/gcc_5.2.0/share/openmpi/mca-btl-openib-device-params.ini", > data source: default, level: 9 dev/all, type: string, synonyms: > btl_openib_hca_param_files) > Colon-delimited list of INI-style files that > contain device vendor/part-specific parameters (use semicolon for Windows) > MCA btl: parameter "btl_openib_hca_param_files" (current > value: > "/opt/MPI/openmpi-1.10.0/linux/gcc_5.2.0/share/openmpi/mca-btl-openib-device-params.ini", > data source: default, level: 9 dev/all, type: string, deprecated, synonym > of: btl_openib_device_param_files) > Colon-delimited list of INI-style files that > contain device vendor/part-specific parameters (use semicolon for Windows) > MCA btl: parameter "btl_openib_device_type" (current value: > "all", data source: default, level: 9 dev/all, type: int) > Specify to only use IB or iWARP network adapters > (infiniband = only use InfiniBand HCAs; iwarp = only use iWARP NICs; all = > use any available adapters) > Valid values: 0:"infiniband", 0:"ib", 1:"iwarp", > 1:"iw", 2:"all" > MCA btl: parameter "btl_openib_max_btls" (current value: > "-1", data source: default, level: 9 dev/all, type: int) > Maximum number of device ports to use (-1 = use all > available, otherwise must be >= 1) > MCA btl: parameter "btl_openib_free_list_num" (current > value: "8", data source: default, level: 9 dev/all, type: int) > Initial size of free lists (must be >= 1) > MCA btl: parameter "btl_openib_free_list_max" (current > value: "-1", data source: default, level: 9 dev/all, type: int) > Maximum size of free lists (-1 = infinite, > otherwise must be >= 0) > MCA btl: parameter "btl_openib_free_list_inc" (current > value: "32", data source: default, level: 9 dev/all, type: int) > Increment size of free lists (must be >= 1) > MCA btl: parameter "btl_openib_mpool" (current value: > "grdma", data source: default, level: 9 dev/all, type: string) > Name of the memory pool to be used (it is unlikely > that you will ever want to change this) > MCA btl: parameter "btl_openib_reg_mru_len" (current value: > "16", data source: default, level: 9 dev/all, type: int) > Length of the registration cache most recently used > list (must be >= 1) > MCA btl: parameter "btl_openib_cq_size" (current value: > "8192", data source: default, level: 9 dev/all, type: int, synonyms: > btl_openib_ib_cq_size) > Minimum size of the OpenFabrics completion queue > (CQs are automatically sized based on the number of peer MPI processes; this > value determines the *minimum* size of all CQs) > MCA btl: parameter "btl_openib_ib_cq_size" (current value: > "8192", data source: default, level: 9 dev/all, type: int, deprecated, > synonym of: btl_openib_cq_size) > Minimum size of the OpenFabrics completion queue > (CQs are automatically sized based on the number of peer MPI processes; this > value determines the *minimum* size of all CQs) > MCA btl: parameter "btl_openib_max_inline_data" (current > value: "-1", data source: default, level: 9 dev/all, type: int, synonyms: > btl_openib_ib_max_inline_data) > Maximum size of inline data segment (-1 = run-time > probe to discover max value, otherwise must be >= 0). If not explicitly set, > use max_inline_data from the INI file containing device-specific parameters > MCA btl: parameter "btl_openib_ib_max_inline_data" (current > value: "-1", data source: default, level: 9 dev/all, type: int, deprecated, > synonym of: btl_openib_max_inline_data) > Maximum size of inline data segment (-1 = run-time > probe to discover max value, otherwise must be >= 0). If not explicitly set, > use max_inline_data from the INI file containing device-specific parameters > MCA btl: parameter "btl_openib_pkey" (current value: "0", > data source: default, level: 9 dev/all, type: unsigned, synonyms: > btl_openib_ib_pkey_val) > OpenFabrics partition key (pkey) value. Unsigned > integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be > masked against the maximum allowable IB partition key value (0x7fff) > MCA btl: parameter "btl_openib_ib_pkey_val" (current value: > "0", data source: default, level: 9 dev/all, type: unsigned, deprecated, > synonym of: btl_openib_pkey) > OpenFabrics partition key (pkey) value. Unsigned > integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be > masked against the maximum allowable IB partition key value (0x7fff) > MCA btl: parameter "btl_openib_psn" (current value: "0", > data source: default, level: 9 dev/all, type: unsigned, synonyms: > btl_openib_ib_psn) > OpenFabrics packet sequence starting number (must > be >= 0) > MCA btl: parameter "btl_openib_ib_psn" (current value: "0", > data source: default, level: 9 dev/all, type: unsigned, deprecated, synonym > of: btl_openib_psn) > OpenFabrics packet sequence starting number (must > be >= 0) > MCA btl: parameter "btl_openib_ib_qp_ous_rd_atom" (current > value: "4", data source: default, level: 9 dev/all, type: unsigned) > InfiniBand outstanding atomic reads (must be >= 0) > MCA btl: parameter "btl_openib_mtu" (current value: "1k", > data source: default, level: 9 dev/all, type: int, synonyms: > btl_openib_ib_mtu) > OpenFabrics MTU, in bytes (if not specified in INI > files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048 > bytes, 5=4096 bytes > Valid values: 1:"256B", 2:"512B", 3:"1k", 4:"2k", > 5:"4k" > MCA btl: parameter "btl_openib_ib_mtu" (current value: "1k", > data source: default, level: 9 dev/all, type: int, deprecated, synonym of: > btl_openib_mtu) > OpenFabrics MTU, in bytes (if not specified in INI > files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048 > bytes, 5=4096 bytes > Valid values: 1:"256B", 2:"512B", 3:"1k", 4:"2k", > 5:"4k" > MCA btl: parameter "btl_openib_ib_min_rnr_timer" (current > value: "25", data source: default, level: 9 dev/all, type: unsigned) > InfiniBand minimum "receiver not ready" timer, in > seconds (must be >= 0 and <= 31) > MCA btl: parameter "btl_openib_ib_timeout" (current value: > "24", data source: environment, level: 9 dev/all, type: unsigned) > InfiniBand transmit timeout, plugged into formula: > 4.096 microseconds * (2^btl_openib_ib_timeout) (must be >= 0 and <= 31) > MCA btl: parameter "btl_openib_ib_retry_count" (current > value: "7", data source: default, level: 9 dev/all, type: unsigned) > InfiniBand transmit retry count (must be >= 0 and > <= 7) > MCA btl: parameter "btl_openib_ib_rnr_retry" (current value: > "7", data source: default, level: 9 dev/all, type: unsigned) > InfiniBand "receiver not ready" retry count; > applies *only* to SRQ/XRC queues. PP queues use RNR retry values of 0 > because Open MPI performs software flow control to guarantee that RNRs never > occur (must be >= 0 and <= 7; 7 = "infinite") > MCA btl: parameter "btl_openib_ib_max_rdma_dst_ops" (current > value: "4", data source: default, level: 9 dev/all, type: unsigned) > InfiniBand maximum pending RDMA destination > operations (must be >= 0) > MCA btl: parameter "btl_openib_ib_service_level" (current > value: "0", data source: default, level: 9 dev/all, type: unsigned) > InfiniBand service level (must be >= 0 and <= 15) > MCA btl: parameter "btl_openib_ib_path_record_service_level" > (current value: "0", data source: default, level: 9 dev/all, type: unsigned) > Enable getting InfiniBand service level from > PathRecord (must be >= 0, 0 = disabled, positive = try to get the service > level from PathRecord) > MCA btl: parameter "btl_openib_use_eager_rdma" (current > value: "-1", data source: default, level: 9 dev/all, type: int) > Use RDMA for eager messages (-1 = use device > default, 0 = do not use eager RDMA, 1 = use eager RDMA) > MCA btl: parameter "btl_openib_eager_rdma_threshold" > (current value: "16", data source: default, level: 9 dev/all, type: int) > Use RDMA for short messages after this number of > messages are received from a given peer (must be >= 1) > MCA btl: parameter "btl_openib_max_eager_rdma" (current > value: "16", data source: default, level: 9 dev/all, type: int) > Maximum number of peers allowed to use RDMA for > short messages (RDMA is used for all long messages, except if explicitly > disabled, such as with the "dr" pml) (must be >= 0) > MCA btl: parameter "btl_openib_eager_rdma_num" (current > value: "17", data source: default, level: 9 dev/all, type: int) > Number of RDMA buffers to allocate for small > messages (must be >= 1) > MCA btl: parameter "btl_openib_btls_per_lid" (current value: > "1", data source: default, level: 9 dev/all, type: unsigned) > Number of BTLs to create for each InfiniBand LID > (must be >= 1) > MCA btl: parameter "btl_openib_max_lmc" (current value: "1", > data source: default, level: 9 dev/all, type: unsigned) > Maximum number of LIDs to use for each device port > (must be >= 0, where 0 = use all available) > MCA btl: parameter "btl_openib_enable_apm_over_lmc" (current > value: "0", data source: default, level: 9 dev/all, type: int) > Maximum number of alternative paths for each device > port (must be >= -1, where 0 = disable apm, -1 = all available alternative > paths ) > MCA btl: parameter "btl_openib_enable_apm_over_ports" > (current value: "0", data source: default, level: 9 dev/all, type: int) > Enable alternative path migration (APM) over > different ports of the same device (must be >= 0, where 0 = disable APM over > ports, 1 = enable APM over ports of the same device) > MCA btl: parameter "btl_openib_use_async_event_thread" > (current value: "true", data source: default, level: 9 dev/all, type: bool) > If nonzero, use the thread that will handle > InfiniBand asynchronous events > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_enable_srq_resize" (current > value: "true", data source: default, level: 9 dev/all, type: bool) > Enable/Disable on demand SRQ resize. (0 = without > resizing, nonzero = with resizing) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_rroce_enable" (current value: > "false", data source: default, level: 9 dev/all, type: bool) > Enable/Disable routing between different subnets(0 > = disable, nonzero = enable) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_buffer_alignment" (current > value: "64", data source: default, level: 9 dev/all, type: unsigned) > Preferred communication buffer alignment, in bytes > (must be > 0 and power of two) > MCA btl: parameter "btl_openib_cq_poll_ratio" (current > value: "100", data source: default, level: 9 dev/all, type: unsigned) > How often to poll high priority CQ versus low > priority CQ > MCA btl: parameter "btl_openib_eager_rdma_poll_ratio" > (current value: "100", data source: default, level: 9 dev/all, type: unsigned) > How often to poll eager RDMA channel versus CQ > MCA btl: parameter "btl_openib_hp_cq_poll_per_progress" > (current value: "10", data source: default, level: 9 dev/all, type: unsigned) > Max number of completion events to process for each > call of BTL progress engine > MCA btl: parameter "btl_openib_max_hw_msg_size" (current > value: "0", data source: default, level: 9 dev/all, type: unsigned) > Maximum size (in bytes) of a single fragment of a > long message when using the RDMA protocols (must be > 0 and <= hw > capabilities). > MCA btl: parameter > "btl_openib_allow_max_memory_registration" (current value: "true", data > source: default, level: 9 dev/all, type: bool) > Allow maximum possible memory to register with HCA > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_memory_registration_verbose" > (current value: "0", data source: default, level: 9 dev/all, type: int) > Output some verbose memory registration information > (0 = no output, nonzero = output) > MCA btl: parameter "btl_openib_ignore_locality" (current > value: "0", data source: default, level: 9 dev/all, type: int) > Ignore any locality information and use all devices > (0 = use locality informaiton and use only close devices, nonzero = ignore > locality information) > MCA btl: informational "btl_openib_have_fork_support" > (current value: "true", data source: default, level: 9 dev/all, type: bool) > Whether the OpenFabrics stack supports applications > that invoke the "fork()" system call or not (0 = no, 1 = yes). Note that this > value does NOT indicate whether the system being run on supports "fork()" > with OpenFabrics applications or not. > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_exclusivity" (current value: > "1024", data source: default, level: 7 dev/basic, type: unsigned) > BTL exclusivity (must be >= 0) > MCA btl: parameter "btl_openib_flags" (current value: "310", > data source: default, level: 5 tuner/detail, type: unsigned) > BTL bit flags (general flags: SEND=1, PUT=2, GET=4, > SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by > the "dr" PML (ignored by others): ACK=16, CHECKSUM=32, RDMA_COMPLETION=128; > flags only used by the "bfo" PML (ignored by others): FAILOVER_SUPPORT=512) > MCA btl: parameter "btl_openib_rndv_eager_limit" (current > value: "12288", data source: default, level: 4 tuner/basic, type: size_t) > Size (in bytes, including header) of "phase 1" > fragment sent for all large messages (must be >= 0 and <= eager_limit) > MCA btl: parameter "btl_openib_eager_limit" (current value: > "12288", data source: default, level: 4 tuner/basic, type: size_t) > Maximum size (in bytes, including header) of > "short" messages (must be >= 1). > MCA btl: parameter "btl_openib_max_send_size" (current > value: "65536", data source: default, level: 4 tuner/basic, type: size_t) > Maximum size (in bytes) of a single "phase 2" > fragment of a long message when using the pipeline protocol (must be >= 1) > MCA btl: parameter "btl_openib_rdma_pipeline_send_length" > (current value: "1048576", data source: default, level: 4 tuner/basic, type: > size_t) > Length of the "phase 2" portion of a large message > (in bytes) when using the pipeline protocol. This part of the message will > be split into fragments of size max_send_size and sent using send/receive > semantics (must be >= 0; only relevant when the PUT flag is set) > MCA btl: parameter "btl_openib_rdma_pipeline_frag_size" > (current value: "1048576", data source: default, level: 4 tuner/basic, type: > size_t) > Maximum size (in bytes) of a single "phase 3" > fragment from a long message when using the pipeline protocol. These > fragments will be sent using RDMA semantics (must be >= 1; only relevant when > the PUT flag is set) > MCA btl: parameter "btl_openib_min_rdma_pipeline_size" > (current value: "1060864", data source: default, level: 4 tuner/basic, type: > size_t) > Messages smaller than this size (in bytes) will not > use the RDMA pipeline protocol. Instead, they will be split into fragments > of max_send_size and sent using send/receive semantics (must be >=0, and is > automatically adjusted up to at least > (eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT flag > is set) > MCA btl: parameter "btl_openib_latency" (current value: "4", > data source: default, level: 5 tuner/detail, type: unsigned) > Approximate latency of interconnect (0 = > auto-detect value at run-time [not supported in all BTL modules], >= 1 = > latency in microseconds) > MCA btl: parameter "btl_openib_bandwidth" (current value: > "0", data source: default, level: 5 tuner/detail, type: unsigned) > Approximate maximum bandwidth of interconnect (0 = > auto-detect value at run-time [not supported in all BTL modules], >= 1 = > bandwidth in Mbps) > MCA btl: parameter "btl_openib_receive_queues" (current > value: > "P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64", > data source: default, level: 9 dev/all, type: string) > Colon-delimited, comma-delimited list of receive > queues: P,4096,8,6,4:P,32768,8,6,4 > MCA btl: parameter "btl_openib_if_include" (current value: > "", data source: default, level: 9 dev/all, type: string) > Comma-delimited list of devices/ports to be used > (e.g. "mthca0,mthca1:2"; empty value means to use all ports found). Mutually > exclusive with btl_openib_if_exclude. > MCA btl: parameter "btl_openib_if_exclude" (current value: > "", data source: default, level: 9 dev/all, type: string) > Comma-delimited list of device/ports to be excluded > (empty value means to not exclude any ports). Mutually exclusive with > btl_openib_if_include. > MCA btl: parameter "btl_openib_ipaddr_include" (current > value: "", data source: default, level: 9 dev/all, type: string) > Comma-delimited list of IP Addresses to be used > (e.g. "192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_exclude. > MCA btl: parameter "btl_openib_ipaddr_exclude" (current > value: "", data source: default, level: 9 dev/all, type: string) > Comma-delimited list of IP Addresses to be excluded > (e.g. "192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_include. > MCA btl: parameter "btl_openib_gid_index" (current value: > "0", data source: default, level: 9 dev/all, type: int) > GID index to use on verbs device ports > MCA btl: parameter "btl_openib_memalign" (current value: > "32", data source: default, level: 9 dev/all, type: int) > [64 | 32 | 0] - Enable (64bit or 32bit)/Disable(0) > memoryalignment for all malloc calls if btl openib is used. > MCA btl: parameter "btl_openib_memalign_threshold" (current > value: "12288", data source: default, level: 9 dev/all, type: size_t) > Allocating memory more than > btl_openib_memalign_threshholdbytes will automatically be aligned to the > value of btl_openib_memalign bytes.memalign_threshhold defaults to the same > value as mca_btl_openib_eager_limit. > MCA btl: parameter "btl_openib_cpc_include" (current value: > "", data source: default, level: 9 dev/all, type: string) > Method used to select OpenFabrics connections > (valid values: rdmacm,udcm) > MCA btl: parameter "btl_openib_cpc_exclude" (current value: > "", data source: default, level: 9 dev/all, type: string) > Method used to exclude OpenFabrics connections > (valid values: rdmacm,udcm) > MCA btl: parameter "btl_openib_connect_rdmacm_priority" > (current value: "30", data source: default, level: 9 dev/all, type: int) > The selection method priority for rdma_cm > MCA btl: parameter "btl_openib_connect_rdmacm_port" (current > value: "0", data source: default, level: 9 dev/all, type: unsigned) > The selection method port for rdma_cm > MCA btl: parameter > "btl_openib_connect_rdmacm_resolve_timeout" (current value: "30000", data > source: default, level: 9 dev/all, type: int) > The timeout (in miliseconds) for address and route > resolution > MCA btl: parameter "btl_openib_connect_rdmacm_retry_count" > (current value: "20", data source: default, level: 9 dev/all, type: int) > Maximum number of times rdmacm will retry route > resolution > MCA btl: parameter > "btl_openib_connect_rdmacm_reject_causes_connect_error" (current value: > "false", data source: default, level: 9 dev/all, type: bool) > The drivers for some devices are buggy such that an > RDMA REJECT action may result in a CONNECT_ERROR event instead of a REJECTED > event. Setting this MCA parameter to true tells Open MPI to treat > CONNECT_ERROR events on connections where a REJECT is expected as a REJECT > (default: false) > Valid values: 0: f|false|disabled, 1: t|true|enabled > MCA btl: parameter "btl_openib_connect_udcm_priority" > (current value: "63", data source: default, level: 9 dev/all, type: int) > Priority of the udcm connection method > MCA btl: parameter "btl_openib_connect_udcm_recv_count" > (current value: "512", data source: default, level: 9 dev/all, type: int) > Number of registered buffers to post > MCA btl: parameter "btl_openib_connect_udcm_timeout" > (current value: "500000", data source: default, level: 9 dev/all, type: int) > Ack timeout for udcm connection messages > MCA btl: parameter "btl_openib_connect_udcm_max_retry" > (current value: "25", data source: default, level: 9 dev/all, type: int) > Maximum number of times to retry sending a udcm > connection message > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/10/18162.php
