It's Intel Fabric Suite, a software distribution for True Scale that contains drivers, infiniband stack, PSM, and more:
https://downloadcenter.intel.com/Detail_Desc.aspx?DwnldID=24129&lang=eng Andrew > -----Original Message----- > From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Adrian > Reber > Sent: Monday, November 10, 2014 1:19 PM > To: Open MPI Developers > Subject: Re: [OMPI devel] 1.8.3 and PSM errors > > What is IFS? > > On Mon, Nov 10, 2014 at 09:12:41PM +0000, Friedley, Andrew wrote: > > Hi Adrian, > > > > Yes, I suggest trying either RH support or Intel's support at > ibsupp...@intel.com. They might have seen this problem before. Since > you're running the RHEL versions of PSM and related software, one thing you > could try is IFS. I think I was running IFS 7.3.0, so that's a difference > between > your setup and mine. At the least, it may help support nail down the issue. > > > > Andrew > > > > > -----Original Message----- > > > From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Adrian > > > Reber > > > Sent: Monday, November 10, 2014 12:39 PM > > > To: Open MPI Developers > > > Subject: Re: [OMPI devel] 1.8.3 and PSM errors > > > > > > Andrew, > > > > > > thanks for looking into this. I was able to reproduce this error on > > > RHEL 7 with PSM provided by RHEL: > > > > > > infinipath-psm-3.2-2_ga8c3e3e_open.2.el7.x86_64 > > > infinipath-psm-devel-3.2-2_ga8c3e3e_open.2.el7.x86_64 > > > > > > $ mpirun -np 32 mpi_test_suite -t "environment" > > > > > > mpi_test_suite:4877 terminated with signal 11 at PC=7f5a2f4a2150 > > > SP=7fff9e0ce770. Backtrace: > > > /lib64/libpsm_infinipath.so.1(ips_proto_connect+0x630)[0x7f5a2f4a215 > > > 0] > > > /lib64/libpsm_infinipath.so.1(ips_ptl_connect+0x3a)[0x7f5a2f49919a] > > > > /lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x3e7)[0x7f5a2f491727 > > > ] > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > 4.8/lib/libmpi.so.1(ompi_mtl_psm_add_procs+0x1f3)[0x7f5a30054cf3] > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > 4.8/lib/libmpi.so.1(ompi_comm_get_rprocs+0x49a)[0x7f5a2ff221da] > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > 4.8/lib/libmpi.so.1(PMPI_Intercomm_create+0x2f2)[0x7f5a2ff51832] > > > mpi_test_suite[0x469420] > > > mpi_test_suite[0x441d8e] > > > /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f5a2f8ffaf5] > > > mpi_test_suite[0x405349] > > > > > > Source RPM : infinipath-psm-3.2-2_ga8c3e3e_open.2.el7.src.rpm > > > Build Date : Tue 04 Mar 2014 02:45:41 AM CET Build Host : x86- > > > 025.build.eng.bos.redhat.com Relocations : /usr > > > Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> > > > Vendor : Red Hat, Inc. > > > URL : http://www.openfabrics.org/downloads/infinipath- > psm/infinipath- > > > psm-3.2-2_ga8c3e3e_open.tar.gz > > > Summary : QLogic PSM Libraries > > > > > > Is this supposed to work? Or is this something Red Hat has to fix? > > > > > > Adrian > > > > > > On Mon, Oct 27, 2014 at 10:22:08PM +0000, Friedley, Andrew wrote: > > > > Hi Adrian, > > > > > > > > I'm unable to reproduce here with OMPI v1.8.3 (I assume you're > > > > doing this > > > with one 8-core node): > > > > > > > > $ mpirun -np 32 -mca pml cm -mca mtl psm ./mpi_test_suite -t > > > "environment" > > > > (Rank:0) tst_test_array[0]:Status > > > > (Rank:0) tst_test_array[1]:Request_Null > > > > (Rank:0) tst_test_array[2]:Type_dup > > > > (Rank:0) tst_test_array[3]:Get_version Number of failed tests:0 > > > > > > > > Works with various np from 8 to 32. Your original case: > > > > > > > > $ mpirun -np 32 ./mpi_test_suite -t "All,^io,^one-sided" > > > > > > > > Runs for a while and eventually hits send cancellation errors. > > > > > > > > Any chance you could try updating your infinipath libraries? > > > > > > > > Andrew > > > > > > > > > -----Original Message----- > > > > > From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of > > > > > Adrian Reber > > > > > Sent: Monday, October 27, 2014 9:11 AM > > > > > To: Open MPI Developers > > > > > Subject: Re: [OMPI devel] 1.8.3 and PSM errors > > > > > > > > > > This is a simpler test setup: > > > > > > > > > > On 8 core machines this works: > > > > > > > > > > $ mpirun -np 8 mpi_test_suite -t "environment" > > > > > [...] > > > > > Number of failed tests:0 > > > > > > > > > > Using 9 or more cores it fails: > > > > > > > > > > $ mpirun -np 9 mpi_test_suite -t "environment" > > > > > > > > > > mpi_test_suite:20293 terminated with signal 11 at > > > > > PC=2b6d107fa9a4 SP=7fff06431a70. Backtrace: > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b6 > > > > > d107 > > > > > fa9a > > > > > 4] > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b6d107e > > > > > b1 > > > > > 72] > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > > 4.9/lib/libmpi.so.1(ompi_mtl_psm_add_procs+0x1a4)[0x2b6d0fa6e384 > > > > > ] > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_comm_get_rprocs+0x2fa)[0x2b6d0f93376a] > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Intercomm_create+0x332)[0x2b6d0f963d42] > > > > > mpi_test_suite[0x46cd00] > > > > > mpi_test_suite[0x44434c] > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b6d10047d5d] > > > > > mpi_test_suite[0x4058e9] > > > > > ------------------------------------------------------- > > > > > Primary job terminated normally, but 1 process returned a > > > > > non-zero exit code.. Per user-direction, the job has been aborted. > > > > > ------------------------------------------------------- > > > > > > > > > > mpi_test_suite:11212 terminated with signal 11 at > > > > > PC=2b2c27d0d9a4 SP=7ffff5020430. Backtrace: > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b2 > > > > > c27d > > > > > 0d9a > > > > > 4] > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b2c > > > > > 27cf > > > > > e17 > > > > > 2] > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > > 4.9/lib/libmpi.so.1(ompi_mtl_psm_add_procs+0x1a4)[0x2b2c26f81384 > > > > > ] > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_comm_get_rprocs+0x2fa)[0x2b2c26e4676a] > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Intercomm_create+0x332)[0x2b2c26e76d42] > > > > > mpi_test_suite[0x46cd00] > > > > > mpi_test_suite[0x44434c] > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b2c2755ad5d] > > > > > mpi_test_suite[0x4058e9] > > > > > ---------------------------------------------------------------- > > > > > ---- > > > > > ------ mpirun detected that one or more processes exited with > > > > > non-zero status, thus causing the job to be terminated. The > > > > > first process to do so was: > > > > > > > > > > Process name: [[47415,1],0] > > > > > Exit code: 1 > > > > > ---------------------------------------------------------------- > > > > > ---- > > > > > ------ > > > > > > > > > > > > > > > > > > > > On Mon, Oct 27, 2014 at 08:27:17AM -0700, Ralph Castain wrote: > > > > > > I’m afraid I can’t quite decipher from all this what actually > > > > > > fails. Of course, > > > > > PSM doesn’t support dynamic operations like comm_spawn or > > > > > connect_accept, so if you are running those tests that just > > > > > won’t work. Is that the heart of the problem here? > > > > > > > > > > > > > > > > > > > On Oct 27, 2014, at 1:40 AM, Adrian Reber <adr...@lisas.de> > wrote: > > > > > > > > > > > > > > Running Open MPI 1.8.3 with PSM does not seem to work right > > > > > > > now > > > at all. > > > > > > > I am getting the same errors also on trunk from my newly set up > MTT. > > > > > > > Before trying to debug this I just wanted to make sure this > > > > > > > is not a configuration error. I have following PSM packages > installed: > > > > > > > > > > > > > > infinipath-devel-3.1.1-363.1140_rhel6_qlc.noarch > > > > > > > infinipath-libs-3.1.1-363.1140_rhel6_qlc.x86_64 > > > > > > > infinipath-3.1.1-363.1140_rhel6_qlc.x86_64 > > > > > > > > > > > > > > with 1.6.5 I do not see PSM errors and the test suite fails much > later: > > > > > > > > > > > > > > P2P tests Many-to-one with MPI_Iprobe (MPI_ANY_SOURCE) > > > (21/48), > > > > > comm > > > > > > > Intracomm merged of the Halved Intercomm (13/13), type > > > > > > > MPI_TYPE_MIX_ARRAY (28/29) P2P tests Many-to-one with > > > MPI_Iprobe > > > > > > > (MPI_ANY_SOURCE) (21/48), comm Intracomm merged of the > > > > > > > Halved Intercomm (13/13), type MPI_TYPE_MIX_LB_UB (29/29) > > > > > > > n050304:5.0.Cannot cancel send requests (req=0x2ad8ba881f80) > > > > > > > P2P tests Many-to-one with Isend and Cancellation (22/48), > > > > > > > comm MPI_COMM_WORLD (1/13), type MPI_CHAR (1/29) > > > n050304:2.0.Cannot > > > > > > > cancel send requests (req=0x2b25143fbd88) n050302:7.0.Cannot > > > > > > > cancel send requests (req=0x2b4d95eb0f80) n050301:4.0.Cannot > > > > > > > cancel send requests (req=0x2adf03e14f80) n050304:4.0.Cannot > > > > > > > cancel send requests (req=0x2ad877257ed8) n050301:6.0.Cannot > > > > > > > cancel send requests (req=0x2ba47634af80) n050304:8.0.Cannot > > > > > > > cancel send requests (req=0x2ae8ac16cf80) n050302:3.0.Cannot > > > > > > > cancel send requests (req=0x2ab81dcb4d88) n050303:4.0.Cannot > > > > > > > cancel send requests (req=0x2b9ef4ef8f80) n050303:2.0.Cannot > > > > > > > cancel send requests (req=0x2ab0f03f9f80) n050302:9.0.Cannot > > > > > > > cancel send requests (req=0x2b214f9ebed8) n050301:2.0.Cannot > > > > > > > cancel send requests (req=0x2b31302d4f80) n050302:4.0.Cannot > > > > > > > cancel send requests (req=0x2b0581bd3f80) n050301:8.0.Cannot > > > > > > > cancel send requests (req=0x2ae53776bf80) n050303:6.0.Cannot > > > > > > > cancel send requests (req=0x2b13eeb78f80) n050304:7.0.Cannot > > > > > > > cancel send requests (req=0x2b4e99715f80) n050304:9.0.Cannot > > > > > > > cancel send requests (req=0x2b10429c2f80) n050304:3.0.Cannot > > > > > > > cancel send requests (req=0x2b9196f5fe30) n050304:6.0.Cannot > > > > > > > cancel send requests (req=0x2b30d6c69ed8) n050301:9.0.Cannot > > > > > > > cancel send requests (req=0x2b93c9e04f80) n050303:9.0.Cannot > > > > > > > cancel send requests (req=0x2ab4d6ce0f80) n050301:5.0.Cannot > > > > > > > cancel send requests (req=0x2b6ad851ef80) n050303:3.0.Cannot > > > > > > > cancel send requests (req=0x2b8ef52a0f80) n050301:3.0.Cannot > > > > > > > cancel send requests (req=0x2b277a4aff80) n050303:7.0.Cannot > > > > > > > cancel send requests (req=0x2ba570fa9f80) n050301:7.0.Cannot > > > > > > > cancel send requests (req=0x2ba707dfbf80) n050302:2.0.Cannot > > > > > > > cancel send requests (req=0x2b90f2e51e30) n050303:5.0.Cannot > > > > > > > cancel send requests (req=0x2b1250ba8f80) n050302:8.0.Cannot > > > > > > > cancel send requests (req=0x2b22e0129ed8) n050303:8.0.Cannot > > > > > > > cancel send requests (req=0x2b6609792f80) n050302:6.0.Cannot > > > > > > > cancel send requests (req=0x2b2b6081af80) n050302:5.0.Cannot > > > > > > > cancel send requests (req=0x2ab24f6f1f80) > > > > > > > ------------------------------------------------------------ > > > > > > > ---- > > > > > > > ---- > > > > > > > ------ mpirun has exited due to process rank 14 with PID > > > > > > > 4496 on node n050303 exiting improperly. There are two > > > > > > > reasons this could > > > occur: > > > > > > > > > > > > > > 1. this process did not call "init" before exiting, but > > > > > > > others in the job did. This can cause a job to hang > > > > > > > indefinitely while it waits for all processes to call > > > > > > > "init". By rule, if one process calls "init", then ALL > > > > > > > processes must call "init" prior to > > > termination. > > > > > > > > > > > > > > 2. this process called "init", but exited without calling > > > > > > > "finalize". > > > > > > > By rule, all processes that call "init" MUST call "finalize" > > > > > > > prior to exiting or it will be considered an "abnormal > > > > > > > termination" > > > > > > > > > > > > > > This may have caused other processes in the application to > > > > > > > be terminated by signals sent by mpirun (as reported here). > > > > > > > ------------------------------------------------------------ > > > > > > > ---- > > > > > > > ---- > > > > > > > ------ > > > > > > > [adrian@n050304 mpi_test_suite]$ > > > > > > > > > > > > > > and this are my PSM errors with 1.8.3: > > > > > > > > > > > > > > [adrian@n050304 mpi_test_suite]$ mpirun -np 32 > > > > > > > mpi_test_suite -t > > > > > "All,^io,^one-sided" > > > > > > > > > > > > > > mpi_test_suite:8904 terminated with signal 11 at > > > > > > > PC=2b08466239a4 > > > > > SP=7ffff03c6e30. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:16905 terminated with signal 11 at > > > > > > > PC=2ae4cad209a4 > > > > > SP=7fffceefa730. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:3171 terminated with signal 11 at > > > > > > > PC=2b57daafe9a4 > > > > > SP=7fff5c4b3af0. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:16906 terminated with signal 11 at > > > > > > > PC=2b4c9fa019a4 > > > > > SP=7fffe916c330. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:3172 terminated with signal 11 at > > > > > > > PC=2b6dde92e9a4 > > > > > SP=7fff04cf1730. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:16907 terminated with signal 11 at > > > > > > > PC=2ad6eb8589a4 > > > > > SP=7fffc30d02f0. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:3173 terminated with signal 11 at > > > > > > > PC=2b2e4aec89a4 > > > > > SP=7fffa054e230. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:16908 terminated with signal 11 at > > > > > > > PC=2b4e6e5589a4 > > > > > SP=7fff68c7a1f0. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:3174 terminated with signal 11 at > > > > > > > PC=2b7049b279a4 > > > > > SP=7fff99a49f70. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:16909 terminated with signal 11 at > > > > > > > PC=2b252219d9a4 > > > > > SP=7fff72a0c6b0. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:3175 terminated with signal 11 at > > > > > > > PC=2ac8d5caf9a4 > > > > > SP=7fff6d7a63f0. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:16910 terminated with signal 11 at > > > > > > > PC=2b7f83fc49a4 > > > > > SP=7fffb95b89b0. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:3176 terminated with signal 11 at > > > > > > > PC=2b11438da9a4 > > > > > SP=7fffe626f270. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:16903 terminated with signal 11 at > > > > > > > PC=2ac5249249a4 > > > > > SP=7fff8874af30. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:3177 terminated with signal 11 at > > > > > > > PC=2ab6154549a4 > > > > > SP=7fffbf6ff430. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:16904 terminated with signal 11 at > > > > > > > PC=2ad0265099a4 > > > > > SP=7fff89fea470. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:3178 terminated with signal 11 at > > > > > > > PC=2b606b1a79a4 > > > > > SP=7fff20240db0. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:4458 terminated with signal 11 at > > > > > > > PC=2b593ef029a4 > > > > > SP=7fff4f48b470. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:4459 terminated with signal 11 at > > > > > > > PC=2b06dde559a4 > > > > > SP=7fffd771a4f0. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:4460 terminated with signal 11 at > > > > > > > PC=2ba7904cb9a4 > > > > > SP=7fff9694c8b0. Backtrace: > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2ab > > > > > > > 6154 > > > > > > > 549a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ab61544 > > > > > > > 5172] > > > > > > > > > > > > > > mpi_test_suite:4461 terminated with signal 11 at > > > > > > > PC=2b26799fd9a4 > > > > > SP=7fff70f69eb0. Backtrace: > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b1 > > > > > > > 1438 > > > > > > > da9a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b11438c > > > > > > > b172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b1142b4e384] > > > > > > > > > > > > > > mpi_test_suite:4462 terminated with signal 11 at > > > > > > > PC=2b15418e19a4 > > > > > SP=7fff858425b0. Backtrace: > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2ab6146c8384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2ab61458d76a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2ab6145bdd42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ab614ca1d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > > > > > > > > mpi_test_suite:4463 terminated with signal 11 at > > > > > > > PC=2b43082919a4 > > > > > SP=7fff2ea8a530. Backtrace: > > > > > > > > > > > > > > mpi_test_suite:4464 terminated with signal 11 at > > > > > > > PC=2adc01fe89a4 > > > > > SP=7fff0de9d4b0. Backtrace: > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b1142a1376a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b1142a43d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > > > > > > > > mpi_test_suite:4465 terminated with signal 11 at > > > > > > > PC=2b477a1819a4 > > > > > SP=7fffd33831b0. Backtrace: > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b1143127d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b4 > > > > > > > 3082 > > > > > > > 919a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b430828 > > > > > > > 2172] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b4 > > > > > > > 77a1 > > > > > > > 819a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b477a17 > > > > > > > 2172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b47793f5384] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b2 > > > > > > > 6799 > > > > > > > fd9a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b26799e > > > > > > > e172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b2678c71384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b47792ba76a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b47792ead42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b4307505384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b43073ca76a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b43073fad42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b2678b3676a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b2678b66d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b4307aded5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b47799ced5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b267924ad5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b2 > > > > > > > e4ae > > > > > > > c89a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b2e4aeb > > > > > > > 9172] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2ac > > > > > > > 8d5c > > > > > > > af9a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ac8d5ca > > > > > > > 0172] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b5 > > > > > > > 7daa > > > > > > > fe9a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b57daae > > > > > > > f172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b57d9d72384] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b0 > > > > > > > 8466 > > > > > > > 239a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b084661 > > > > > > > 4172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b0845897384] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b5 > > > > > > > 93ef > > > > > > > 029a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b593eef > > > > > > > 3172] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b7 > > > > > > > 049b > > > > > > > 279a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b7049b1 > > > > > > > 8172] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b4 > > > > > > > e6e5 > > > > > > > 589a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b4e6e54 > > > > > > > 9172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b084575c76a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b084578cd42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b0845e70d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b1 > > > > > > > 5418 > > > > > > > e19a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b15418d > > > > > > > 2172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b2e4a13c384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b2e4a00176a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b2e4a031d42] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b4e6d7cc384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b593e176384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b593e03b76a] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b6 > > > > > > > 06b1 > > > > > > > a79a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b606b19 > > > > > > > 8172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b606a41b384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b4e6d69176a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b1540b55384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b1540a1a76a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b57d9c3776a] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b0 > > > > > > > 6dde > > > > > > > 559a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b06dde4 > > > > > > > 6172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b06dd0c9384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b06dcf8e76a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b06dcfbed42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2ac8d4f23384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2ac8d4de876a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b4e6d6c1d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b4e6dda5d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b1540a4ad42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b154112ed5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b6 > > > > > > > dde9 > > > > > > > 2e9a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b6dde91 > > > > > > > f172] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2ad > > > > > > > 6eb8 > > > > > > > 589a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ad6eb84 > > > > > > > 9172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b593e06bd42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b593e74fd5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b606a2e076a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b606a310d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b606a9f4d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b06dd6a2d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b7048d9b384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b7048c6076a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b7048c90d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2ac > > > > > > > 5249 > > > > > > > 249a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ac52491 > > > > > > > 5172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2ac523b98384] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2ad > > > > > > > c01f > > > > > > > e89a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2adc01fd > > > > > > > 9172] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b2e4a715d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2ad6eaacc384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2ad6ea99176a] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2ba > > > > > > > 7904 > > > > > > > cb9a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ba7904b > > > > > > > c172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b6dddba2384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b6ddda6776a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b6ddda97d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2ac523a5d76a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2ac523a8dd42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2adc0125c384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2adc0112176a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2adc01151d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b57d9c67d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b57da34bd5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ac524171d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2ba78f73f384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2ba78f60476a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2ac8d4e18d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ac8d54fcd5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2ad6ea9c1d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ad6eb0a5d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2adc01835d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b6dde17bd5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2ba78f634d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ba78fd18d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b7049374d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b2 > > > > > > > 5221 > > > > > > > 9d9a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b252218 > > > > > > > e172] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2ae > > > > > > > 4cad > > > > > > > 209a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ae4cad1 > > > > > > > 1172] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2ad > > > > > > > 0265 > > > > > > > 099a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ad0264f > > > > > > > a172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2ae4c9f94384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2ae4c9e5976a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b2521411384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b25212d676a] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b7 > > > > > > > f83f > > > > > > > c49a4] > > > > > > > > > > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b7f83fb > > > > > > > 5172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2ae4c9e89d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ae4ca56dd5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0 > > > > > > > x2b4 > > > > > > > c9fa > > > > > > > 019a4] > > > > > > > /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x > > > > > > > 2b4c > > > > > > > 9f9f > > > > > > > 2172] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b4c9ec75384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b4c9eb3a76a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2ad02577d384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2ad02564276a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2ad025672d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ad025d56d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b2521306d42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b25219ead5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_mtl > > > > > > > _psm_add_procs+0x1a4)[0x2b7f83238384] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(ompi_com > > > > > > > m_get_rprocs+0x2fa)[0x2b7f830fd76a] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b7f8312dd42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu- > > > > > 4.9/lib/libmpi.so.1(MPI_Inte > > > > > > > rcomm_create+0x332)[0x2b4c9eb6ad42] > > > > > > > mpi_test_suite[0x46cd00] > > > > > > > mpi_test_suite[0x44434c] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b4c9f24ed5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b7f83811d5d] > > > > > > > mpi_test_suite[0x4058e9] > > > > > > > ------------------------------------------------------- > > > > > > > Primary job terminated normally, but 1 process returned a > > > > > > > non-zero exit code.. Per user-direction, the job has been aborted. > > > > > > > ------------------------------------------------------- > > > > > > > ------------------------------------------------------------ > > > > > > > ---- > > > > > > > ---- > > > > > > > ------ mpirun detected that one or more processes exited > > > > > > > with non-zero status, thus causing the job to be terminated. > > > > > > > The first process to do so was: > > > > > > > > > > > > > > Process name: [[9290,1],0] > > > > > > > Exit code: 1 > > > > > > > ------------------------------------------------------------ > > > > > > > ---- > > > > > > > ---- > > > > > > > ------ > _______________________________________________ > > > > > > > devel mailing list > > > > > > > de...@open-mpi.org > > > > > > > Subscription: > > > > > > > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > > > > Link to this post: > > > > > > > http://www.open- > > > mpi.org/community/lists/devel/2014/10/16093.php > > > > > > > > > > > > _______________________________________________ > > > > > > devel mailing list > > > > > > de...@open-mpi.org > > > > > > Subscription: > > > > > > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > > > Link to this post: > > > > > > http://www.open- > mpi.org/community/lists/devel/2014/10/16099.ph > > > > > > p > > > > > _______________________________________________ > > > > > devel mailing list > > > > > de...@open-mpi.org > > > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > > Link to this post: http://www.open- > > > > > mpi.org/community/lists/devel/2014/10/16100.php > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: http://www.open- > mpi.org/community/lists/devel/2014/11/16280.php
