I'm afraid I don't know much about OpenFoam. You'll likely need to ask assistance from the OpenFoam community.
> On Jun 1, 2016, at 6:13 PM, Megdich Islem <megdich_is...@yahoo.fr> wrote: > > Hi! > > Thank you Jeff. I was able to run a case of OF by setting the absolute path > name for mpiexec. But, when I wanted to run a coupled case in which OF is > coupled with dummyCSM through EMPIRE using these three command lines: > > mpiexec -np 1 Emperor emperorInput.xml > mpiexec -np 1 dummyCSM dummyCSMInput > mpiexec -np 1 pimpleDyMFoam -case OF, > > I was still getting OF not able to connect. In the user guide of EMPIRE, it > is said that emperor (the client) has to recognize the clients which are > dummyCSM and OpenFOAM. For some reasons emperor is not able to recognize > OpenFoam but it recognizes dummyCSM. > > What can be the cause that a server can not recognize a client ? > > Regards, > Islem > > > Le Mercredi 1 juin 2016 17h00, "users-requ...@open-mpi.org" > <users-requ...@open-mpi.org> a écrit : > > > Send users mailing list submissions to > us...@open-mpi.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.open-mpi.org/mailman/listinfo.cgi/users > or, via email, send a message with subject or body 'help' to > users-requ...@open-mpi.org > > You can reach the person managing the list at > users-ow...@open-mpi.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users digest..." > > > Today's Topics: > > 1. Re: Firewall settings for MPI communication > (Jeff Squyres (jsquyres)) > 2. Re: users Digest, Vol 3514, Issue 1 (Jeff Squyres (jsquyres)) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 1 Jun 2016 13:02:22 +0000 > From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> > To: "Open MPI User's List" <us...@open-mpi.org> > Subject: Re: [OMPI users] Firewall settings for MPI communication > Message-ID: <ae97b273-c16b-4bc3-bc30-924d25697...@cisco.com> > Content-Type: text/plain; charset="utf-8" > > In addition, you might want to consider upgrading to Open MPI v1.10.x (v1.6.x > is fairly ancient). > > > On Jun 1, 2016, at 7:46 AM, Gilles Gouaillardet > > <gilles.gouaillar...@gmail.com> wrote: > > > > which network are your VMs using for communications ? > > if this is tcp, then you also have to specify a restricted set of allowed > > ports for the tcp btl > > > > that would be something like > > mpirun --mca btl_tcp_dynamic_ports 49990-50010 ... > > > > please double check the Open MPI 1.6.5 parameter and syntax with > > ompi_info --all > > (or check the archives, I think I posted the correct command line a few > > weeks ago) > > > > Cheers, > > > > Gilles > > > > On Wednesday, June 1, 2016, Ping Wang <ping.w...@asc-s.de> wrote: > > I'm using Open MPI 1.6.5 to run OpenFOAM in parallel on several VMs on a > > cloud. mpirun hangs without any error messages. I think this is a firewall > > issue. Because when I open all the TCP ports(1-65535) in the security group > > of VMs, mpirun works well. However I was suggested to open as less ports as > > possible. So I have to limit MPI to run on a range of ports. I opened the > > port range 49990-50010 for MPI communication. And use command > > > > > > > > mpirun --mca oob_tcp_dynamic_ports 49990-50010 -np 4 --hostfile machines > > simpleFoam ?parallel. > > > > > > > > But it still hangs. How can I specify a port range that OpenMPI will use? I > > appreciate any help you can provide. > > > > > > > > Best, > > > > Ping Wang > > > > > > > > <image001.png> > > > > ------------------------------------------------------ > > > > Ping Wang > > > > Automotive Simulation Center Stuttgart e.V. > > > > Nobelstra?e 15 > > > > D-70569 Stuttgart > > > > Telefon: +49 711 699659-14 > > > > Fax: +49 711 699659-29 > > > > E-Mail: ping.w...@asc-s.de > > > > Web: http://www.asc-s.de > > > > Social Media: <image002.gif>/asc.stuttgart > > > > ------------------------------------------------------ > > > > > > > > > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > Link to this post: > > http://www.open-mpi.org/community/lists/users/2016/06/29340.php > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > ------------------------------ > > Message: 2 > Date: Wed, 1 Jun 2016 13:51:55 +0000 > From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> > To: Megdich Islem <megdich_is...@yahoo.fr>, "Open MPI User's List" > <us...@open-mpi.org> > Subject: Re: [OMPI users] users Digest, Vol 3514, Issue 1 > Message-ID: <c44110cb-7576-4272-bc53-9c274ef1e...@cisco.com> > Content-Type: text/plain; charset="utf-8" > > The example you list below has all MPICH paths -- I don't see any Open MPI > setups in there. > > What I was suggesting was that if you absolutely need to have both Open MPI > and MPICH installed and in your PATH / LD_LIBRARY_PATH / MANPATH, then you > can use the full, absolute path name to each of the Open MPI executables -- > e.g., /path/to/openmpi/install/bin/mpicc, etc. That way, you can use Open > MPI's mpicc without having it in your path. > > Additionally, per > https://www.open-mpi.org/faq/?category=running#mpirun-prefix, if you specify > the absolute path name to mpirun (or mpiexec -- they're identical in Open > MPI) and you're using the rsh/ssh launcher in Open MPI, then Open MPI will > set the right PATH / LD_LIBRARY_PATH on remote servers for you. See the FAQ > link for more detail. > > > > > On Jun 1, 2016, at 8:41 AM, Megdich Islem <megdich_is...@yahoo.fr> wrote: > > > > Hi! > > > > Thank you Jeff for you suggestion. But, I am still not able to understand > > what do you mean by using absolute path names to for > > mpicc/mpifort-mpirun/mpiexec ? > > > > This is how my .bashrc looks like > > > > source /opt/openfoam30/etc/bashrc > > > > export PATH=/home/Desktop/mpich/bin:$PATH > > export LD_LIBRARY_PATH="/home/islem/Desktop/mpich/lib/:$LD_LIBRARY_PATH" > > export MPICH_F90=gfortran > > export MPICH_CC=/opt/intel/bin/icc > > export MPICH_CXX=/opt/intel/bin/icpc > > export MPICH_LINK_CXX="-L/home/Desktop/mpich/lib/ -Wl,-rpath > > -Wl,/home/islem/Desktop/mpich/lib -lmpichcxx -lmpich -lopa -lmpl -lrt > > -lpthread" > > > > export PATH=$PATH:/opt/intel/bin/ > > LD_LIBRARY_PATH="/opt/intel/lib/intel64:$LD_LIBRARY_PATH" > > export LD_LIBRARY_PATH > > source > > /opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/mpivars.sh > > intel64 > > > > alias startEMPIRE=". /home/islem/software/empire/EMPIRE-Core/etc/bashrc.sh > > ICC" > > > > mpirun --version gives mpich 3.0.4 > > > > This is how I run one example that couples 2 clients through the server > > EMPIRE. > > I use three terminals, in each I write one of these command lines > > > > mpiexec -np 1 Emperor emperorInput.xml (I got a message in the terminal > > saying that Empire started) > > > > mpiexec -np 1 dummyCSM dummyCSMInput (I get a message that Emperor > > acknowledged connection) > > mpiexec -np 1 pimpleDyMFoam -case OF (I got no message in the terminal > > which means no connection) > > > > How can I use the mpirun and where to right any modifications ? > > > > Regards, > > Islem > > > > > > Le Vendredi 27 mai 2016 17h00, "users-requ...@open-mpi.org" > > <users-requ...@open-mpi.org> a ?crit : > > > > > > Send users mailing list submissions to > > us...@open-mpi.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://www.open-mpi.org/mailman/listinfo.cgi/users > > or, via email, send a message with subject or body 'help' to > > users-requ...@open-mpi.org > > > > You can reach the person managing the list at > > users-ow...@open-mpi.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of users digest..." > > > > > > Today's Topics: > > > > 1. Re: users Digest, Vol 3510, Issue 2 (Jeff Squyres (jsquyres)) > > 2. Re: segmentation fault for slot-list and openmpi-1.10.3rc2 > > (Siegmar Gross) > > 3. OpenMPI virtualization aware (Marco D'Amico) > > 4. Re: OpenMPI virtualization aware (Ralph Castain) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Thu, 26 May 2016 23:28:17 +0000 > > From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> > > To: Megdich Islem <megdich_is...@yahoo.fr>, "Open MPI User's List" > > <us...@open-mpi.org> > > Cc: Dave Love <d.l...@liverpool.ac.uk> > > Subject: Re: [OMPI users] users Digest, Vol 3510, Issue 2 > > Message-ID: <441f803d-fdbb-443d-82aa-74ff3845a...@cisco.com> > > Content-Type: text/plain; charset="utf-8" > > > > You're still intermingling your Open MPI and MPICH installations. > > > > You need to ensure to use the wrapper compilers and mpirun/mpiexec from the > > same MPI implementation. > > > > For example, if you use mpicc/mpifort from Open MPI to build your program, > > then you must use Open MPI's mpirun/mpiexec. > > > > If you absolutely need to have both MPI implementations in your PATH / > > LD_LIBRARY_PATH, you might want to use absolute path names to for > > mpicc/mpifort/mpirun/mpiexec. > > > > > > > > > On May 26, 2016, at 3:46 PM, Megdich Islem <megdich_is...@yahoo.fr> wrote: > > > > > > Thank you all for your suggestions !! > > > > > > I found an answer to a similar case in Open MPI FAQ (Question 15) > > > FAQ: Running MPI jobs > > > > > > > > > > > > > > > > > > > > > > > > > > > FAQ: Running MPI jobs > > > Table of contents: What pre-requisites are necessary for running an Open > > > MPI job? What ABI guarantees does Open MPI provide? Do I need a common > > > filesystem on a... > > > Afficher sur www.open-mpi.org > > > Aper?u par Yahoo > > > > > > which suggests to use mpirun's prefix command line option or to use the > > > mpirun wrapper. > > > > > > I modified my command to the following > > > mpirun --prefix > > > /opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/Openmpi-system -np 1 > > > pimpleDyMFoam -case OF > > > > > > But, I got an error (see attached picture). Is the syntax correct? How > > > can I solve the problem? That first method seems to be easier than using > > > the mpirun wrapper. > > > > > > Otherwise, how can I use the mpirun wrapper? > > > > > > Regards, > > > islem > > > > > > > > > Le Mercredi 25 mai 2016 16h40, Dave Love <d.l...@liverpool.ac.uk> a ?crit > > > : > > > > > > > > > I wrote: > > > > > > > > > > You could wrap one (set of) program(s) in a script to set the > > > > appropriate environment before invoking the real program. > > > > > > > > > I realize I should have said something like "program invocations", > > > i.e. if you have no control over something invoking mpirun for programs > > > using different MPIs, then an mpirun wrapper needs to check what it's > > > being asked to run. > > > > > > > > > > > > <mpirun-error.png><path-to-open-mpi.png>_______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > > Link to this post: > > > http://www.open-mpi.org/community/lists/users/2016/05/29317.php > > > > > > -- > > Jeff Squyres > > jsquy...@cisco.com > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > ------------------------------ > > > > Message: 2 > > Date: Fri, 27 May 2016 08:16:41 +0200 > > From: Siegmar Gross <siegmar.gr...@informatik.hs-fulda.de> > > To: Open MPI Users <us...@open-mpi.org> > > Subject: Re: [OMPI users] segmentation fault for slot-list and > > openmpi-1.10.3rc2 > > Message-ID: > > <f5653a5c-174f-4569-c730-082a9db82...@informatik.hs-fulda.de> > > Content-Type: text/plain; charset=windows-1252; format=flowed > > > > Hi Ralph, > > > > > > Am 26.05.2016 um 17:38 schrieb Ralph Castain: > > > I?m afraid I honestly can?t make any sense of it. It seems > > > you at least have a simple workaround (use a hostfile instead > > > of -host), yes? > > > > Only the combination "--host" and "--slot-list" breaks. > > Everything else works as expected. One more remark: As you > > can see below, this combination worked using gdb and "next" > > after the breakpoint. The process blocks, if I keep the > > enter-key pressed down and I have to kill simple_spawn in > > another window to get control back in gdb (<Ctrl-c> or > > anything else didn't work). I got this error yesterday > > evening. > > > > ... > > (gdb) > > ompi_mpi_init (argc=0, argv=0x0, requested=0, provided=0x7fffffffbc0c) > > at ../../openmpi-1.10.3rc3/ompi/runtime/ompi_mpi_init.c:738 > > 738 if (OMPI_SUCCESS != (ret = ompi_file_init())) { > > (gdb) > > 744 if (OMPI_SUCCESS != (ret = ompi_win_init())) { > > (gdb) > > 750 if (OMPI_SUCCESS != (ret = ompi_attr_init())) { > > (gdb) > > 758 if (OMPI_SUCCESS != (ret = ompi_proc_complete_init())) { > > (gdb) > > 764 ret = MCA_PML_CALL(enable(true)); > > (gdb) > > 765 if( OMPI_SUCCESS != ret ) { > > (gdb) > > 771 if (NULL == (procs = ompi_proc_world(&nprocs))) { > > (gdb) > > 775 ret = MCA_PML_CALL(add_procs(procs, nprocs)); > > (gdb) > > 776 free(procs); > > (gdb) > > 780 if (OMPI_ERR_UNREACH == ret) { > > (gdb) > > 785 } else if (OMPI_SUCCESS != ret) { > > (gdb) > > 790 MCA_PML_CALL(add_comm(&ompi_mpi_comm_world.comm)); > > (gdb) > > 791 MCA_PML_CALL(add_comm(&ompi_mpi_comm_self.comm)); > > (gdb) > > 796 if (ompi_mpi_show_mca_params) { > > (gdb) > > 803 ompi_rte_wait_for_debugger(); > > (gdb) > > 807 if (ompi_enable_timing && 0 == OMPI_PROC_MY_NAME->vpid) { > > (gdb) > > 817 coll = OBJ_NEW(ompi_rte_collective_t); > > (gdb) > > 818 coll->id = ompi_process_info.peer_init_barrier; > > (gdb) > > 819 coll->active = true; > > (gdb) > > 820 if (OMPI_SUCCESS != (ret = ompi_rte_barrier(coll))) { > > (gdb) > > 825 OMPI_WAIT_FOR_COMPLETION(coll->active); > > (gdb) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Program received signal SIGTERM, Terminated. > > 0x00007ffff7a7acd0 in opal_progress@plt () > > from /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12 > > (gdb) > > Single stepping until exit from function opal_progress@plt, > > which has no line number information. > > [Thread 0x7ffff491b700 (LWP 19602) exited] > > > > Program terminated with signal SIGTERM, Terminated. > > The program no longer exists. > > (gdb) > > The program is not being run. > > (gdb) > > ... > > > > > > > > Kind regards > > > > Siegmar > > > > > > >> On May 26, 2016, at 5:48 AM, Siegmar Gross > > >> <siegmar.gr...@informatik.hs-fulda.de> wrote: > > >> > > >> Hi Ralph and Gilles, > > >> > > >> it's strange that the program works with "--host" and "--slot-list" > > >> in your environment and not in mine. I get the following output, if > > >> I run the program in gdb without a breakpoint. > > >> > > >> > > >> loki spawn 142 gdb /usr/local/openmpi-1.10.3_64_gcc/bin/mpiexec > > >> GNU gdb (GDB; SUSE Linux Enterprise 12) 7.9.1 > > >> ... > > >> (gdb) set args -np 1 --host loki --slot-list 0:0-1,1:0-1 simple_spawn > > >> (gdb) run > > >> Starting program: /usr/local/openmpi-1.10.3_64_gcc/bin/mpiexec -np 1 > > >> --host loki --slot-list 0:0-1,1:0-1 simple_spawn > > >> [Thread debugging using libthread_db enabled] > > >> Using host libthread_db library "/lib64/libthread_db.so.1". > > >> Detaching after fork from child process 18031. > > >> [pid 18031] starting up! > > >> 0 completed MPI_Init > > >> Parent [pid 18031] about to spawn! > > >> Detaching after fork from child process 18033. > > >> Detaching after fork from child process 18034. > > >> [pid 18033] starting up! > > >> [pid 18034] starting up! > > >> [loki:18034] *** Process received signal *** > > >> [loki:18034] Signal: Segmentation fault (11) > > >> ... > > >> > > >> > > >> > > >> I get a different output, if I run the program in gdb with > > >> a breakpoint. > > >> > > >> gdb /usr/local/openmpi-1.10.3_64_gcc/bin/mpiexec > > >> (gdb) set args -np 1 --host loki --slot-list 0:0-1,1:0-1 simple_spawn > > >> (gbd) set follow-fork-mode child > > >> (gdb) break ompi_proc_self > > >> (gdb) run > > >> (gdb) next > > >> > > >> Repeating "next" very often results in the following output. > > >> > > >> ... > > >> Starting program: > > >> /home/fd1026/work/skripte/master/parallel/prog/mpi/spawn/simple_spawn > > >> [Thread debugging using libthread_db enabled] > > >> Using host libthread_db library "/lib64/libthread_db.so.1". > > >> [pid 13277] starting up! > > >> [New Thread 0x7ffff42ef700 (LWP 13289)] > > >> > > >> Breakpoint 1, ompi_proc_self (size=0x7fffffffc060) > > >> at ../../openmpi-1.10.3rc3/ompi/proc/proc.c:413 > > >> 413 ompi_proc_t **procs = (ompi_proc_t**) > > >> malloc(sizeof(ompi_proc_t*)); > > >> (gdb) n > > >> 414 if (NULL == procs) { > > >> (gdb) > > >> 423 OBJ_RETAIN(ompi_proc_local_proc); > > >> (gdb) > > >> 424 *procs = ompi_proc_local_proc; > > >> (gdb) > > >> 425 *size = 1; > > >> (gdb) > > >> 426 return procs; > > >> (gdb) > > >> 427 } > > >> (gdb) > > >> ompi_comm_init () at > > >> ../../openmpi-1.10.3rc3/ompi/communicator/comm_init.c:138 > > >> 138 group->grp_my_rank = 0; > > >> (gdb) > > >> 139 group->grp_proc_count = (int)size; > > >> ... > > >> 193 ompi_comm_reg_init(); > > >> (gdb) > > >> 196 ompi_comm_request_init (); > > >> (gdb) > > >> 198 return OMPI_SUCCESS; > > >> (gdb) > > >> 199 } > > >> (gdb) > > >> ompi_mpi_init (argc=0, argv=0x0, requested=0, provided=0x7fffffffc21c) > > >> at ../../openmpi-1.10.3rc3/ompi/runtime/ompi_mpi_init.c:738 > > >> 738 if (OMPI_SUCCESS != (ret = ompi_file_init())) { > > >> (gdb) > > >> 744 if (OMPI_SUCCESS != (ret = ompi_win_init())) { > > >> (gdb) > > >> 750 if (OMPI_SUCCESS != (ret = ompi_attr_init())) { > > >> ... > > >> 988 ompi_mpi_initialized = true; > > >> (gdb) > > >> 991 if (ompi_enable_timing && 0 == OMPI_PROC_MY_NAME->vpid) { > > >> (gdb) > > >> 999 return MPI_SUCCESS; > > >> (gdb) > > >> 1000 } > > >> (gdb) > > >> PMPI_Init (argc=0x0, argv=0x0) at pinit.c:94 > > >> 94 if (MPI_SUCCESS != err) { > > >> (gdb) > > >> 104 return MPI_SUCCESS; > > >> (gdb) > > >> 105 } > > >> (gdb) > > >> 0x0000000000400d0c in main () > > >> (gdb) > > >> Single stepping until exit from function main, > > >> which has no line number information. > > >> 0 completed MPI_Init > > >> Parent [pid 13277] about to spawn! > > >> [New process 13472] > > >> [Thread debugging using libthread_db enabled] > > >> Using host libthread_db library "/lib64/libthread_db.so.1". > > >> process 13472 is executing new program: > > >> /usr/local/openmpi-1.10.3_64_gcc/bin/orted > > >> [Thread debugging using libthread_db enabled] > > >> Using host libthread_db library "/lib64/libthread_db.so.1". > > >> [New process 13474] > > >> [Thread debugging using libthread_db enabled] > > >> Using host libthread_db library "/lib64/libthread_db.so.1". > > >> process 13474 is executing new program: > > >> /home/fd1026/work/skripte/master/parallel/prog/mpi/spawn/simple_spawn > > >> [pid 13475] starting up! > > >> [pid 13476] starting up! > > >> [Thread debugging using libthread_db enabled] > > >> Using host libthread_db library "/lib64/libthread_db.so.1". > > >> [pid 13474] starting up! > > >> [New Thread 0x7ffff491b700 (LWP 13480)] > > >> [Switching to Thread 0x7ffff7ff1740 (LWP 13474)] > > >> > > >> Breakpoint 1, ompi_proc_self (size=0x7fffffffba30) > > >> at ../../openmpi-1.10.3rc3/ompi/proc/proc.c:413 > > >> 413 ompi_proc_t **procs = (ompi_proc_t**) > > >> malloc(sizeof(ompi_proc_t*)); > > >> (gdb) > > >> 414 if (NULL == procs) { > > >> ... > > >> 426 return procs; > > >> (gdb) > > >> 427 } > > >> (gdb) > > >> ompi_comm_init () at > > >> ../../openmpi-1.10.3rc3/ompi/communicator/comm_init.c:138 > > >> 138 group->grp_my_rank = 0; > > >> (gdb) > > >> 139 group->grp_proc_count = (int)size; > > >> (gdb) > > >> 140 OMPI_GROUP_SET_INTRINSIC (group); > > >> ... > > >> 193 ompi_comm_reg_init(); > > >> (gdb) > > >> 196 ompi_comm_request_init (); > > >> (gdb) > > >> 198 return OMPI_SUCCESS; > > >> (gdb) > > >> 199 } > > >> (gdb) > > >> ompi_mpi_init (argc=0, argv=0x0, requested=0, provided=0x7fffffffbbec) > > >> at ../../openmpi-1.10.3rc3/ompi/runtime/ompi_mpi_init.c:738 > > >> 738 if (OMPI_SUCCESS != (ret = ompi_file_init())) { > > >> (gdb) > > >> 744 if (OMPI_SUCCESS != (ret = ompi_win_init())) { > > >> (gdb) > > >> 750 if (OMPI_SUCCESS != (ret = ompi_attr_init())) { > > >> ... > > >> 863 if (OMPI_SUCCESS != (ret = ompi_pubsub_base_select())) { > > >> (gdb) > > >> 869 if (OMPI_SUCCESS != (ret = > > >> mca_base_framework_open(&ompi_dpm_base_framework, 0))) { > > >> (gdb) > > >> 873 if (OMPI_SUCCESS != (ret = ompi_dpm_base_select())) { > > >> (gdb) > > >> 884 if ( OMPI_SUCCESS != > > >> (gdb) > > >> 894 if (OMPI_SUCCESS != > > >> (gdb) > > >> 900 if (OMPI_SUCCESS != > > >> (gdb) > > >> 911 if (OMPI_SUCCESS != (ret = ompi_dpm.dyn_init())) { > > >> (gdb) > > >> Parent done with spawn > > >> Parent sending message to child > > >> 2 completed MPI_Init > > >> Hello from the child 2 of 3 on host loki pid 13476 > > >> 1 completed MPI_Init > > >> Hello from the child 1 of 3 on host loki pid 13475 > > >> 921 if (OMPI_SUCCESS != (ret = ompi_cr_init())) { > > >> (gdb) > > >> 931 opal_progress_event_users_decrement(); > > >> (gdb) > > >> 934 opal_progress_set_yield_when_idle(ompi_mpi_yield_when_idle); > > >> (gdb) > > >> 937 if (ompi_mpi_event_tick_rate >= 0) { > > >> (gdb) > > >> 946 if (OMPI_SUCCESS != (ret = ompi_mpiext_init())) { > > >> (gdb) > > >> 953 if (ret != OMPI_SUCCESS) { > > >> (gdb) > > >> 972 OBJ_CONSTRUCT(&ompi_registered_datareps, opal_list_t); > > >> (gdb) > > >> 977 OBJ_CONSTRUCT( &ompi_mpi_f90_integer_hashtable, > > >> opal_hash_table_t); > > >> (gdb) > > >> 978 opal_hash_table_init(&ompi_mpi_f90_integer_hashtable, 16 /* > > >> why not? */); > > >> (gdb) > > >> 980 OBJ_CONSTRUCT( &ompi_mpi_f90_real_hashtable, > > >> opal_hash_table_t); > > >> (gdb) > > >> 981 opal_hash_table_init(&ompi_mpi_f90_real_hashtable, > > >> FLT_MAX_10_EXP); > > >> (gdb) > > >> 983 OBJ_CONSTRUCT( &ompi_mpi_f90_complex_hashtable, > > >> opal_hash_table_t); > > >> (gdb) > > >> 984 opal_hash_table_init(&ompi_mpi_f90_complex_hashtable, > > >> FLT_MAX_10_EXP); > > >> (gdb) > > >> 988 ompi_mpi_initialized = true; > > >> (gdb) > > >> 991 if (ompi_enable_timing && 0 == OMPI_PROC_MY_NAME->vpid) { > > >> (gdb) > > >> 999 return MPI_SUCCESS; > > >> (gdb) > > >> 1000 } > > >> (gdb) > > >> PMPI_Init (argc=0x0, argv=0x0) at pinit.c:94 > > >> 94 if (MPI_SUCCESS != err) { > > >> (gdb) > > >> 104 return MPI_SUCCESS; > > >> (gdb) > > >> 105 } > > >> (gdb) > > >> 0x0000000000400d0c in main () > > >> (gdb) > > >> Single stepping until exit from function main, > > >> which has no line number information. > > >> 0 completed MPI_Init > > >> Hello from the child 0 of 3 on host loki pid 13474 > > >> > > >> Child 2 disconnected > > >> Child 1 disconnected > > >> Child 0 received msg: 38 > > >> Parent disconnected > > >> 13277: exiting > > >> > > >> Program received signal SIGTERM, Terminated. > > >> 0x0000000000400f0a in main () > > >> (gdb) > > >> Single stepping until exit from function main, > > >> which has no line number information. > > >> [tcsetpgrp failed in terminal_inferior: No such process] > > >> [Thread 0x7ffff491b700 (LWP 13480) exited] > > >> > > >> Program terminated with signal SIGTERM, Terminated. > > >> The program no longer exists. > > >> (gdb) > > >> The program is not being run. > > >> (gdb) > > >> The program is not being run. > > >> (gdb) info break > > >> Num Type Disp Enb Address What > > >> 1 breakpoint keep y 0x00007ffff7aa35c7 in ompi_proc_self > > >> at > > >> ../../openmpi-1.10.3rc3/ompi/proc/proc.c:413 inf 8, 7, 6, 5, 4, 3, 2, 1 > > >> breakpoint already hit 2 times > > >> (gdb) delete 1 > > >> (gdb) r > > >> Starting program: > > >> /home/fd1026/work/skripte/master/parallel/prog/mpi/spawn/simple_spawn > > >> [Thread debugging using libthread_db enabled] > > >> Using host libthread_db library "/lib64/libthread_db.so.1". > > >> [pid 16708] starting up! > > >> 0 completed MPI_Init > > >> Parent [pid 16708] about to spawn! > > >> [New process 16720] > > >> [Thread debugging using libthread_db enabled] > > >> Using host libthread_db library "/lib64/libthread_db.so.1". > > >> process 16720 is executing new program: > > >> /usr/local/openmpi-1.10.3_64_gcc/bin/orted > > >> [Thread debugging using libthread_db enabled] > > >> Using host libthread_db library "/lib64/libthread_db.so.1". > > >> [New process 16722] > > >> [Thread debugging using libthread_db enabled] > > >> Using host libthread_db library "/lib64/libthread_db.so.1". > > >> process 16722 is executing new program: > > >> /home/fd1026/work/skripte/master/parallel/prog/mpi/spawn/simple_spawn > > >> [pid 16723] starting up! > > >> [pid 16724] starting up! > > >> [Thread debugging using libthread_db enabled] > > >> Using host libthread_db library "/lib64/libthread_db.so.1". > > >> [pid 16722] starting up! > > >> Parent done with spawn > > >> Parent sending message to child > > >> 1 completed MPI_Init > > >> Hello from the child 1 of 3 on host loki pid 16723 > > >> 2 completed MPI_Init > > >> Hello from the child 2 of 3 on host loki pid 16724 > > >> 0 completed MPI_Init > > >> Hello from the child 0 of 3 on host loki pid 16722 > > >> Child 0 received msg: 38 > > >> Child 0 disconnected > > >> Parent disconnected > > >> Child 1 disconnected > > >> Child 2 disconnected > > >> 16708: exiting > > >> 16724: exiting > > >> 16723: exiting > > >> [New Thread 0x7ffff491b700 (LWP 16729)] > > >> > > >> Program received signal SIGTERM, Terminated. > > >> [Switching to Thread 0x7ffff7ff1740 (LWP 16722)] > > >> __GI__dl_debug_state () at dl-debug.c:74 > > >> 74 dl-debug.c: No such file or directory. > > >> (gdb) > > >> -------------------------------------------------------------------------- > > >> WARNING: A process refused to die despite all the efforts! > > >> This process may still be running and/or consuming resources. > > >> > > >> Host: loki > > >> PID: 16722 > > >> > > >> -------------------------------------------------------------------------- > > >> > > >> > > >> The following simple_spawn processes exist now. > > >> > > >> loki spawn 171 ps -aef | grep simple_spawn > > >> fd1026 11079 11053 0 14:00 pts/0 00:00:00 > > >> /usr/local/openmpi-1.10.3_64_gcc/bin/mpiexec -np 1 --host loki > > >> --slot-list 0:0-1,1:0-1 simple_spawn > > >> fd1026 11095 11079 29 14:01 pts/0 00:09:37 [simple_spawn] <defunct> > > >> fd1026 16722 1 0 14:31 ? 00:00:00 [simple_spawn] <defunct> > > >> fd1026 17271 29963 0 14:33 pts/2 00:00:00 grep simple_spawn > > >> loki spawn 172 > > >> > > >> > > >> Is it possible that there is a race condition? How can I help > > >> to get a solution for my problem? > > >> > > >> > > >> Kind regards > > >> > > >> Siegmar > > >> > > >> Am 24.05.2016 um 16:54 schrieb Ralph Castain: > > >>> Works perfectly for me, so I believe this must be an environment issue > > >>> - I am using gcc 6.0.0 on CentOS7 with x86: > > >>> > > >>> $ mpirun -n 1 -host bend001 --slot-list 0:0-1,1:0-1 --report-bindings > > >>> ./simple_spawn > > >>> [bend001:17599] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket > > >>> 0[core 1[hwt 0-1]], socket 1[core 6[hwt 0-1]], socket 1[core 7[hwt > > >>> 0-1]]: [BB/BB/../../../..][BB/BB/../../../..] > > >>> [pid 17601] starting up! > > >>> 0 completed MPI_Init > > >>> Parent [pid 17601] about to spawn! > > >>> [pid 17603] starting up! > > >>> [bend001:17599] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket > > >>> 0[core 1[hwt 0-1]], socket 1[core 6[hwt 0-1]], socket 1[core 7[hwt > > >>> 0-1]]: [BB/BB/../../../..][BB/BB/../../../..] > > >>> [bend001:17599] MCW rank 1 bound to socket 0[core 0[hwt 0-1]], socket > > >>> 0[core 1[hwt 0-1]], socket 1[core 6[hwt 0-1]], socket 1[core 7[hwt > > >>> 0-1]]: [BB/BB/../../../..][BB/BB/../../../..] > > >>> [bend001:17599] MCW rank 2 bound to socket 0[core 0[hwt 0-1]], socket > > >>> 0[core 1[hwt 0-1]], socket 1[core 6[hwt 0-1]], socket 1[core 7[hwt > > >>> 0-1]]: [BB/BB/../../../..][BB/BB/../../../..] > > >>> [pid 17604] starting up! > > >>> [pid 17605] starting up! > > >>> Parent done with spawn > > >>> Parent sending message to child > > >>> 0 completed MPI_Init > > >>> Hello from the child 0 of 3 on host bend001 pid 17603 > > >>> Child 0 received msg: 38 > > >>> 1 completed MPI_Init > > >>> Hello from the child 1 of 3 on host bend001 pid 17604 > > >>> 2 completed MPI_Init > > >>> Hello from the child 2 of 3 on host bend001 pid 17605 > > >>> Child 0 disconnected > > >>> Child 2 disconnected > > >>> Parent disconnected > > >>> Child 1 disconnected > > >>> 17603: exiting > > >>> 17605: exiting > > >>> 17601: exiting > > >>> 17604: exiting > > >>> $ > > >>> > > >>>> On May 24, 2016, at 7:18 AM, Siegmar Gross > > >>>> <siegmar.gr...@informatik.hs-fulda.de> wrote: > > >>>> > > >>>> Hi Ralph and Gilles, > > >>>> > > >>>> the program breaks only, if I combine "--host" and "--slot-list". > > >>>> Perhaps this > > >>>> information is helpful. I use a different machine now, so that you can > > >>>> see that > > >>>> the problem is not restricted to "loki". > > >>>> > > >>>> > > >>>> pc03 spawn 115 ompi_info | grep -e "OPAL repo revision:" -e "C > > >>>> compiler absolute:" > > >>>> OPAL repo revision: v1.10.2-201-gd23dda8 > > >>>> C compiler absolute: /usr/local/gcc-6.1.0/bin/gcc > > >>>> > > >>>> > > >>>> pc03 spawn 116 uname -a > > >>>> Linux pc03 3.12.55-52.42-default #1 SMP Thu Mar 3 10:35:46 UTC 2016 > > >>>> (4354e1d) x86_64 x86_64 x86_64 GNU/Linux > > >>>> > > >>>> > > >>>> pc03 spawn 117 cat host_pc03.openmpi > > >>>> pc03.informatik.hs-fulda.de slots=12 max_slots=12 > > >>>> > > >>>> > > >>>> pc03 spawn 118 mpicc simple_spawn.c > > >>>> > > >>>> > > >>>> pc03 spawn 119 mpiexec -np 1 --report-bindings a.out > > >>>> [pc03:03711] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: > > >>>> [BB/../../../../..][../../../../../..] > > >>>> [pid 3713] starting up! > > >>>> 0 completed MPI_Init > > >>>> Parent [pid 3713] about to spawn! > > >>>> [pc03:03711] MCW rank 0 bound to socket 1[core 6[hwt 0-1]], socket > > >>>> 1[core 7[hwt 0-1]], socket 1[core 8[hwt 0-1]], socket 1[core 9[hwt > > >>>> 0-1]], socket 1[core 10[hwt 0-1]], socket 1[core 11[hwt 0-1]]: > > >>>> [../../../../../..][BB/BB/BB/BB/BB/BB] > > >>>> [pc03:03711] MCW rank 1 bound to socket 0[core 0[hwt 0-1]], socket > > >>>> 0[core 1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt > > >>>> 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]]: > > >>>> [BB/BB/BB/BB/BB/BB][../../../../../..] > > >>>> [pc03:03711] MCW rank 2 bound to socket 1[core 6[hwt 0-1]], socket > > >>>> 1[core 7[hwt 0-1]], socket 1[core 8[hwt 0-1]], socket 1[core 9[hwt > > >>>> 0-1]], socket 1[core 10[hwt 0-1]], socket 1[core 11[hwt 0-1]]: > > >>>> [../../../../../..][BB/BB/BB/BB/BB/BB] > > >>>> [pid 3715] starting up! > > >>>> [pid 3716] starting up! > > >>>> [pid 3717] starting up! > > >>>> Parent done with spawn > > >>>> Parent sending message to child > > >>>> 0 completed MPI_Init > > >>>> Hello from the child 0 of 3 on host pc03 pid 3715 > > >>>> 1 completed MPI_Init > > >>>> Hello from the child 1 of 3 on host pc03 pid 3716 > > >>>> 2 completed MPI_Init > > >>>> Hello from the child 2 of 3 on host pc03 pid 3717 > > >>>> Child 0 received msg: 38 > > >>>> Child 0 disconnected > > >>>> Child 2 disconnected > > >>>> Parent disconnected > > >>>> Child 1 disconnected > > >>>> 3713: exiting > > >>>> 3715: exiting > > >>>> 3716: exiting > > >>>> 3717: exiting > > >>>> > > >>>> > > >>>> pc03 spawn 120 mpiexec -np 1 --hostfile host_pc03.openmpi --slot-list > > >>>> 0:0-1,1:0-1 --report-bindings a.out > > >>>> [pc03:03729] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket > > >>>> 0[core 1[hwt 0-1]], socket 1[core 6[hwt 0-1]], socket 1[core 7[hwt > > >>>> 0-1]]: [BB/BB/../../../..][BB/BB/../../../..] > > >>>> [pid 3731] starting up! > > >>>> 0 completed MPI_Init > > >>>> Parent [pid 3731] about to spawn! > > >>>> [pc03:03729] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket > > >>>> 0[core 1[hwt 0-1]], socket 1[core 6[hwt 0-1]], socket 1[core 7[hwt > > >>>> 0-1]]: [BB/BB/../../../..][BB/BB/../../../..] > > >>>> [pc03:03729] MCW rank 1 bound to socket 0[core 0[hwt 0-1]], socket > > >>>> 0[core 1[hwt 0-1]], socket 1[core 6[hwt 0-1]], socket 1[core 7[hwt > > >>>> 0-1]]: [BB/BB/../../../..][BB/BB/../../../..] > > >>>> [pc03:03729] MCW rank 2 bound to socket 0[core 0[hwt 0-1]], socket > > >>>> 0[core 1[hwt 0-1]], socket 1[core 6[hwt 0-1]], socket 1[core 7[hwt > > >>>> 0-1]]: [BB/BB/../../../..][BB/BB/../../../..] > > >>>> [pid 3733] starting up! > > >>>> [pid 3734] starting up! > > >>>> [pid 3735] starting up! > > >>>> Parent done with spawn > > >>>> Parent sending message to child > > >>>> 2 completed MPI_Init > > >>>> Hello from the child 2 of 3 on host pc03 pid 3735 > > >>>> 1 completed MPI_Init > > >>>> Hello from the child 1 of 3 on host pc03 pid 3734 > > >>>> 0 completed MPI_Init > > >>>> Hello from the child 0 of 3 on host pc03 pid 3733 > > >>>> Child 0 received msg: 38 > > >>>> Child 0 disconnected > > >>>> Child 2 disconnected > > >>>> Child 1 disconnected > > >>>> Parent disconnected > > >>>> 3731: exiting > > >>>> 3734: exiting > > >>>> 3733: exiting > > >>>> 3735: exiting > > >>>> > > >>>> > > >>>> pc03 spawn 121 mpiexec -np 1 --host pc03 --slot-list 0:0-1,1:0-1 > > >>>> --report-bindings a.out > > >>>> [pc03:03744] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket > > >>>> 0[core 1[hwt 0-1]], socket 1[core 6[hwt 0-1]], socket 1[core 7[hwt > > >>>> 0-1]]: [BB/BB/../../../..][BB/BB/../../../..] > > >>>> [pid 3746] starting up! > > >>>> 0 completed MPI_Init > > >>>> Parent [pid 3746] about to spawn! > > >>>> [pc03:03744] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket > > >>>> 0[core 1[hwt 0-1]], socket 1[core 6[hwt 0-1]], socket 1[core 7[hwt > > >>>> 0-1]]: [BB/BB/../../../..][BB/BB/../../../..] > > >>>> [pc03:03744] MCW rank 2 bound to socket 0[core 0[hwt 0-1]], socket > > >>>> 0[core 1[hwt 0-1]], socket 1[core 6[hwt 0-1]], socket 1[core 7[hwt > > >>>> 0-1]]: [BB/BB/../../../..][BB/BB/../../../..] > > >>>> [pid 3748] starting up! > > >>>> [pid 3749] starting up! > > >>>> [pc03:03749] *** Process received signal *** > > >>>> [pc03:03749] Signal: Segmentation fault (11) > > >>>> [pc03:03749] Signal code: Address not mapped (1) > > >>>> [pc03:03749] Failing at address: 0x8 > > >>>> [pc03:03749] [ 0] /lib64/libpthread.so.0(+0xf870)[0x7fe6f0d1f870] > > >>>> [pc03:03749] [ 1] > > >>>> /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12(ompi_proc_self+0x35)[0x7fe6f0f825b0] > > >>>> [pc03:03749] [ 2] > > >>>> /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12(ompi_comm_init+0x68b)[0x7fe6f0f61b08] > > >>>> [pc03:03749] [ 3] > > >>>> /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12(ompi_mpi_init+0xa90)[0x7fe6f0f87e8a] > > >>>> [pc03:03749] [ 4] > > >>>> /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12(MPI_Init+0x1a0)[0x7fe6f0fc42ae] > > >>>> [pc03:03749] [ 5] a.out[0x400d0c] > > >>>> [pc03:03749] [ 6] > > >>>> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fe6f0989b05] > > >>>> [pc03:03749] [ 7] a.out[0x400bf9] > > >>>> [pc03:03749] *** End of error message *** > > >>>> -------------------------------------------------------------------------- > > >>>> mpiexec noticed that process rank 2 with PID 3749 on node pc03 exited > > >>>> on signal 11 (Segmentation fault). > > >>>> -------------------------------------------------------------------------- > > >>>> pc03 spawn 122 > > >>>> > > >>>> > > >>>> > > >>>> Kind regards > > >>>> > > >>>> Siegmar > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 05/24/16 15:44, Ralph Castain wrote: > > >>>>> > > >>>>>> On May 24, 2016, at 6:21 AM, Siegmar Gross > > >>>>>> <siegmar.gr...@informatik.hs-fulda.de> wrote: > > >>>>>> > > >>>>>> Hi Ralph, > > >>>>>> > > >>>>>> I copy the relevant lines to this place, so that it is easier to see > > >>>>>> what > > >>>>>> happens. "a.out" is your program, which I compiled with mpicc. > > >>>>>> > > >>>>>>>> loki spawn 153 ompi_info | grep -e "OPAL repo revision:" -e "C > > >>>>>>>> compiler > > >>>>>>>> absolute:" > > >>>>>>>> OPAL repo revision: v1.10.2-201-gd23dda8 > > >>>>>>>> C compiler absolute: /usr/local/gcc-6.1.0/bin/gcc > > >>>>>>>> loki spawn 154 mpicc simple_spawn.c > > >>>>>> > > >>>>>>>> loki spawn 155 mpiexec -np 1 a.out > > >>>>>>>> [pid 24008] starting up! > > >>>>>>>> 0 completed MPI_Init > > >>>>>> ... > > >>>>>> > > >>>>>> "mpiexec -np 1 a.out" works. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>> I don?t know what ?a.out? is, but it looks like there is some memory > > >>>>>>> corruption there. > > >>>>>> > > >>>>>> "a.out" is still your program. I get the same error on different > > >>>>>> machines, so that it is not very likely, that the (hardware) memory > > >>>>>> is corrupted. > > >>>>>> > > >>>>>> > > >>>>>>>> loki spawn 156 mpiexec -np 1 --host loki --slot-list 0-5 a.out > > >>>>>>>> [pid 24102] starting up! > > >>>>>>>> 0 completed MPI_Init > > >>>>>>>> Parent [pid 24102] about to spawn! > > >>>>>>>> [pid 24104] starting up! > > >>>>>>>> [pid 24105] starting up! > > >>>>>>>> [loki:24105] *** Process received signal *** > > >>>>>>>> [loki:24105] Signal: Segmentation fault (11) > > >>>>>>>> [loki:24105] Signal code: Address not mapped (1) > > >>>>>> ... > > >>>>>> > > >>>>>> "mpiexec -np 1 --host loki --slot-list 0-5 a.out" breaks with a > > >>>>>> segmentation > > >>>>>> faUlt. Can I do something, so that you can find out, what happens? > > >>>>> > > >>>>> I honestly have no idea - perhaps Gilles can help, as I have no > > >>>>> access to that kind of environment. We aren?t seeing such problems > > >>>>> elsewhere, so it is likely something local. > > >>>>> > > >>>>>> > > >>>>>> > > >>>>>> Kind regards > > >>>>>> > > >>>>>> Siegmar > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> On 05/24/16 15:07, Ralph Castain wrote: > > >>>>>>> > > >>>>>>>> On May 24, 2016, at 4:19 AM, Siegmar Gross > > >>>>>>>> <siegmar.gr...@informatik.hs-fulda.de > > >>>>>>>> <mailto:siegmar.gr...@informatik.hs-fulda.de>> wrote: > > >>>>>>>> > > >>>>>>>> Hi Ralph, > > >>>>>>>> > > >>>>>>>> thank you very much for your answer and your example program. > > >>>>>>>> > > >>>>>>>> On 05/23/16 17:45, Ralph Castain wrote: > > >>>>>>>>> I cannot replicate the problem - both scenarios work fine for me. > > >>>>>>>>> I?m not > > >>>>>>>>> convinced your test code is correct, however, as you call > > >>>>>>>>> Comm_free the > > >>>>>>>>> inter-communicator but didn?t call Comm_disconnect. Checkout the > > >>>>>>>>> attached > > >>>>>>>>> for a correct code and see if it works for you. > > >>>>>>>> > > >>>>>>>> I thought that I only need MPI_Comm_Disconnect, if I would have > > >>>>>>>> established a > > >>>>>>>> connection with MPI_Comm_connect before. The man page for > > >>>>>>>> MPI_Comm_free states > > >>>>>>>> > > >>>>>>>> "This operation marks the communicator object for deallocation. > > >>>>>>>> The > > >>>>>>>> handle is set to MPI_COMM_NULL. Any pending operations that use > > >>>>>>>> this > > >>>>>>>> communicator will complete normally; the object is actually > > >>>>>>>> deallocated only > > >>>>>>>> if there are no other active references to it.". > > >>>>>>>> > > >>>>>>>> The man page for MPI_Comm_disconnect states > > >>>>>>>> > > >>>>>>>> "MPI_Comm_disconnect waits for all pending communication on comm > > >>>>>>>> to complete > > >>>>>>>> internally, deallocates the communicator object, and sets the > > >>>>>>>> handle to > > >>>>>>>> MPI_COMM_NULL. It is a collective operation.". > > >>>>>>>> > > >>>>>>>> I don't see a difference for my spawned processes, because both > > >>>>>>>> functions will > > >>>>>>>> "wait" until all pending operations have finished, before the > > >>>>>>>> object will be > > >>>>>>>> destroyed. Nevertheless, perhaps my small example program worked > > >>>>>>>> all the years > > >>>>>>>> by chance. > > >>>>>>>> > > >>>>>>>> However, I don't understand, why my program works with > > >>>>>>>> "mpiexec -np 1 --host loki,loki,loki,loki,loki spawn_master" and > > >>>>>>>> breaks with > > >>>>>>>> "mpiexec -np 1 --host loki --slot-list 0:0-5,1:0-5 spawn_master". > > >>>>>>>> You are right, > > >>>>>>>> my slot-list is equivalent to "-bind-to none". I could also have > > >>>>>>>> used > > >>>>>>>> "mpiexec -np 1 --host loki --oversubscribe spawn_master" which > > >>>>>>>> works as well. > > >>>>>>> > > >>>>>>> Well, you are only giving us one slot when you specify "-host > > >>>>>>> loki?, and then > > >>>>>>> you are trying to launch multiple processes into it. The > > >>>>>>> ?slot-list? option only > > >>>>>>> tells us what cpus to bind each process to - it doesn?t allocate > > >>>>>>> process slots. > > >>>>>>> So you have to tell us how many processes are allowed to run on > > >>>>>>> this node. > > >>>>>>> > > >>>>>>>> > > >>>>>>>> The program breaks with "There are not enough slots available in > > >>>>>>>> the system > > >>>>>>>> to satisfy ...", if I only use "--host loki" or different host > > >>>>>>>> names, > > >>>>>>>> without mentioning five host names, using "slot-list", or > > >>>>>>>> "oversubscribe", > > >>>>>>>> Unfortunately "--host <host name>:<number of slots>" isn't > > >>>>>>>> available for > > >>>>>>>> openmpi-1.10.3rc2 to specify the number of available slots. > > >>>>>>> > > >>>>>>> Correct - we did not backport the new syntax > > >>>>>>> > > >>>>>>>> > > >>>>>>>> Your program behaves the same way as mine, so that > > >>>>>>>> MPI_Comm_disconnect > > >>>>>>>> will not solve my problem. I had to modify your program in a > > >>>>>>>> negligible way > > >>>>>>>> to get it compiled. > > >>>>>>>> > > >>>>>>>> loki spawn 153 ompi_info | grep -e "OPAL repo revision:" -e "C > > >>>>>>>> compiler absolute:" > > >>>>>>>> OPAL repo revision: v1.10.2-201-gd23dda8 > > >>>>>>>> C compiler absolute: /usr/local/gcc-6.1.0/bin/gcc > > >>>>>>>> loki spawn 154 mpicc simple_spawn.c > > >>>>>>>> loki spawn 155 mpiexec -np 1 a.out > > >>>>>>>> [pid 24008] starting up! > > >>>>>>>> 0 completed MPI_Init > > >>>>>>>> Parent [pid 24008] about to spawn! > > >>>>>>>> [pid 24010] starting up! > > >>>>>>>> [pid 24011] starting up! > > >>>>>>>> [pid 24012] starting up! > > >>>>>>>> Parent done with spawn > > >>>>>>>> Parent sending message to child > > >>>>>>>> 0 completed MPI_Init > > >>>>>>>> Hello from the child 0 of 3 on host loki pid 24010 > > >>>>>>>> 1 completed MPI_Init > > >>>>>>>> Hello from the child 1 of 3 on host loki pid 24011 > > >>>>>>>> 2 completed MPI_Init > > >>>>>>>> Hello from the child 2 of 3 on host loki pid 24012 > > >>>>>>>> Child 0 received msg: 38 > > >>>>>>>> Child 0 disconnected > > >>>>>>>> Child 1 disconnected > > >>>>>>>> Child 2 disconnected > > >>>>>>>> Parent disconnected > > >>>>>>>> 24012: exiting > > >>>>>>>> 24010: exiting > > >>>>>>>> 24008: exiting > > >>>>>>>> 24011: exiting > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Is something wrong with my command line? I didn't use slot-list > > >>>>>>>> before, so > > >>>>>>>> that I'm not sure, if I use it in the intended way. > > >>>>>>> > > >>>>>>> I don?t know what ?a.out? is, but it looks like there is some > > >>>>>>> memory corruption > > >>>>>>> there. > > >>>>>>> > > >>>>>>>> > > >>>>>>>> loki spawn 156 mpiexec -np 1 --host loki --slot-list 0-5 a.out > > >>>>>>>> [pid 24102] starting up! > > >>>>>>>> 0 completed MPI_Init > > >>>>>>>> Parent [pid 24102] about to spawn! > > >>>>>>>> [pid 24104] starting up! > > >>>>>>>> [pid 24105] starting up! > > >>>>>>>> [loki:24105] *** Process received signal *** > > >>>>>>>> [loki:24105] Signal: Segmentation fault (11) > > >>>>>>>> [loki:24105] Signal code: Address not mapped (1) > > >>>>>>>> [loki:24105] Failing at address: 0x8 > > >>>>>>>> [loki:24105] [ 0] /lib64/libpthread.so.0(+0xf870)[0x7f39aa76f870] > > >>>>>>>> [loki:24105] [ 1] > > >>>>>>>> /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12(ompi_proc_self+0x35)[0x7f39aa9d25b0] > > >>>>>>>> [loki:24105] [ 2] > > >>>>>>>> /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12(ompi_comm_init+0x68b)[0x7f39aa9b1b08] > > >>>>>>>> [loki:24105] [ 3] *** An error occurred in MPI_Init > > >>>>>>>> *** on a NULL communicator > > >>>>>>>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now > > >>>>>>>> abort, > > >>>>>>>> *** and potentially your MPI job) > > >>>>>>>> [loki:24104] Local abort before MPI_INIT completed successfully; > > >>>>>>>> not able to > > >>>>>>>> aggregate error messages, and not able to guarantee that all other > > >>>>>>>> processes > > >>>>>>>> were killed! > > >>>>>>>> /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12(ompi_mpi_init+0xa90)[0x7f39aa9d7e8a] > > >>>>>>>> [loki:24105] [ 4] > > >>>>>>>> /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12(MPI_Init+0x1a0)[0x7f39aaa142ae] > > >>>>>>>> [loki:24105] [ 5] a.out[0x400d0c] > > >>>>>>>> [loki:24105] [ 6] > > >>>>>>>> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f39aa3d9b05] > > >>>>>>>> [loki:24105] [ 7] a.out[0x400bf9] > > >>>>>>>> [loki:24105] *** End of error message *** > > >>>>>>>> ------------------------------------------------------- > > >>>>>>>> Child job 2 terminated normally, but 1 process returned > > >>>>>>>> a non-zero exit code.. Per user-direction, the job has been > > >>>>>>>> aborted. > > >>>>>>>> ------------------------------------------------------- > > >>>>>>>> -------------------------------------------------------------------------- > > >>>>>>>> mpiexec detected that one or more processes exited with non-zero > > >>>>>>>> status, thus > > >>>>>>>> causing > > >>>>>>>> the job to be terminated. The first process to do so was: > > >>>>>>>> > > >>>>>>>> Process name: [[49560,2],0] > > >>>>>>>> Exit code: 1 > > >>>>>>>> -------------------------------------------------------------------------- > > >>>>>>>> loki spawn 157 > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Hopefully, you will find out what happens. Please let me know, if > > >>>>>>>> I can > > >>>>>>>> help you in any way. > > >>>>>>>> > > >>>>>>>> Kind regards > > >>>>>>>> > > >>>>>>>> Siegmar > > >>>>>>>> > > >>>>>>>> > > >>>>>>>>> FWIW: I don?t know how many cores you have on your sockets, but > > >>>>>>>>> if you > > >>>>>>>>> have 6 cores/socket, then your slot-list is equivalent to > > >>>>>>>>> ??bind-to none? > > >>>>>>>>> as the slot-list applies to every process being launched > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>> On May 23, 2016, at 6:26 AM, Siegmar Gross > > >>>>>>>>>> <siegmar.gr...@informatik.hs-fulda.de > > >>>>>>>>>> <mailto:siegmar.gr...@informatik.hs-fulda.de>> wrote: > > >>>>>>>>>> > > >>>>>>>>>> Hi, > > >>>>>>>>>> > > >>>>>>>>>> I installed openmpi-1.10.3rc2 on my "SUSE Linux Enterprise Server > > >>>>>>>>>> 12 (x86_64)" with Sun C 5.13 and gcc-6.1.0. Unfortunately I get > > >>>>>>>>>> a segmentation fault for "--slot-list" for one of my small > > >>>>>>>>>> programs. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> loki spawn 119 ompi_info | grep -e "OPAL repo revision:" -e "C > > >>>>>>>>>> compiler > > >>>>>>>>>> absolute:" > > >>>>>>>>>> OPAL repo revision: v1.10.2-201-gd23dda8 > > >>>>>>>>>> C compiler absolute: /usr/local/gcc-6.1.0/bin/gcc > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> loki spawn 120 mpiexec -np 1 --host loki,loki,loki,loki,loki > > >>>>>>>>>> spawn_master > > >>>>>>>>>> > > >>>>>>>>>> Parent process 0 running on loki > > >>>>>>>>>> I create 4 slave processes > > >>>>>>>>>> > > >>>>>>>>>> Parent process 0: tasks in MPI_COMM_WORLD: 1 > > >>>>>>>>>> tasks in COMM_CHILD_PROCESSES local group: 1 > > >>>>>>>>>> tasks in COMM_CHILD_PROCESSES remote group: 4 > > >>>>>>>>>> > > >>>>>>>>>> Slave process 0 of 4 running on loki > > >>>>>>>>>> Slave process 1 of 4 running on loki > > >>>>>>>>>> Slave process 2 of 4 running on loki > > >>>>>>>>>> spawn_slave 2: argv[0]: spawn_slave > > >>>>>>>>>> Slave process 3 of 4 running on loki > > >>>>>>>>>> spawn_slave 0: argv[0]: spawn_slave > > >>>>>>>>>> spawn_slave 1: argv[0]: spawn_slave > > >>>>>>>>>> spawn_slave 3: argv[0]: spawn_slave > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> loki spawn 121 mpiexec -np 1 --host loki --slot-list 0:0-5,1:0-5 > > >>>>>>>>>> spawn_master > > >>>>>>>>>> > > >>>>>>>>>> Parent process 0 running on loki > > >>>>>>>>>> I create 4 slave processes > > >>>>>>>>>> > > >>>>>>>>>> [loki:17326] *** Process received signal *** > > >>>>>>>>>> [loki:17326] Signal: Segmentation fault (11) > > >>>>>>>>>> [loki:17326] Signal code: Address not mapped (1) > > >>>>>>>>>> [loki:17326] Failing at address: 0x8 > > >>>>>>>>>> [loki:17326] [ 0] /lib64/libpthread.so.0(+0xf870)[0x7f4e469b3870] > > >>>>>>>>>> [loki:17326] [ 1] *** An error occurred in MPI_Init > > >>>>>>>>>> *** on a NULL communicator > > >>>>>>>>>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will > > >>>>>>>>>> now abort, > > >>>>>>>>>> *** and potentially your MPI job) > > >>>>>>>>>> [loki:17324] Local abort before MPI_INIT completed successfully; > > >>>>>>>>>> not able to > > >>>>>>>>>> aggregate error messages, and not able to guarantee that all > > >>>>>>>>>> other processes > > >>>>>>>>>> were killed! > > >>>>>>>>>> /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12(ompi_proc_self+0x35)[0x7f4e46c165b0] > > >>>>>>>>>> [loki:17326] [ 2] > > >>>>>>>>>> /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12(ompi_comm_init+0x68b)[0x7f4e46bf5b08] > > >>>>>>>>>> [loki:17326] [ 3] *** An error occurred in MPI_Init > > >>>>>>>>>> *** on a NULL communicator > > >>>>>>>>>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will > > >>>>>>>>>> now abort, > > >>>>>>>>>> *** and potentially your MPI job) > > >>>>>>>>>> [loki:17325] Local abort before MPI_INIT completed successfully; > > >>>>>>>>>> not able to > > >>>>>>>>>> aggregate error messages, and not able to guarantee that all > > >>>>>>>>>> other processes > > >>>>>>>>>> were killed! > > >>>>>>>>>> /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12(ompi_mpi_init+0xa90)[0x7f4e46c1be8a] > > >>>>>>>>>> [loki:17326] [ 4] > > >>>>>>>>>> /usr/local/openmpi-1.10.3_64_gcc/lib64/libmpi.so.12(MPI_Init+0x180)[0x7f4e46c5828e] > > >>>>>>>>>> [loki:17326] [ 5] spawn_slave[0x40097e] > > >>>>>>>>>> [loki:17326] [ 6] > > >>>>>>>>>> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4e4661db05] > > >>>>>>>>>> [loki:17326] [ 7] spawn_slave[0x400a54] > > >>>>>>>>>> [loki:17326] *** End of error message *** > > >>>>>>>>>> ------------------------------------------------------- > > >>>>>>>>>> Child job 2 terminated normally, but 1 process returned > > >>>>>>>>>> a non-zero exit code.. Per user-direction, the job has been > > >>>>>>>>>> aborted. > > >>>>>>>>>> ------------------------------------------------------- > > >>>>>>>>>> -------------------------------------------------------------------------- > > >>>>>>>>>> mpiexec detected that one or more processes exited with non-zero > > >>>>>>>>>> status, > > >>>>>>>>>> thus causing > > >>>>>>>>>> the job to be terminated. The first process to do so was: > > >>>>>>>>>> > > >>>>>>>>>> Process name: [[56340,2],0] > > >>>>>>>>>> Exit code: 1 > > >>>>>>>>>> -------------------------------------------------------------------------- > > >>>>>>>>>> loki spawn 122 > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> I would be grateful, if somebody can fix the problem. Thank you > > >>>>>>>>>> very much for any help in advance. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Kind regards > > >>>>>>>>>> > > >>>>>>>>>> Siegmar > > >>>>>>>>>> _______________________________________________ > > >>>>>>>>>> users mailing list > > >>>>>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org> > > >>>>>>>>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > >>>>>>>>>> Link to this post: > > >>>>>>>>>> http://www.open-mpi.org/community/lists/users/2016/05/29281.php > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> _______________________________________________ > > >>>>>>>>> users mailing list > > >>>>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org> > > >>>>>>>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > >>>>>>>>> Link to this > > >>>>>>>>> post: > > >>>>>>>>> http://www.open-mpi.org/community/lists/users/2016/05/29284.php > > >>>>>>>>> > > >>>>>>>> <simple_spawn_modified.c>_______________________________________________ > > >>>>>>>> users mailing list > > >>>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org> > > >>>>>>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > >>>>>>>> Link to this post: > > >>>>>>>> http://www.open-mpi.org/community/lists/users/2016/05/29300.php > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> _______________________________________________ > > >>>>>>> users mailing list > > >>>>>>> us...@open-mpi.org > > >>>>>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > >>>>>>> Link to this post: > > >>>>>>> http://www.open-mpi.org/community/lists/users/2016/05/29301.php > > >>>>>>> > > >>>>>> _______________________________________________ > > >>>>>> users mailing list > > >>>>>> us...@open-mpi.org > > >>>>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > >>>>>> Link to this post: > > >>>>>> http://www.open-mpi.org/community/lists/users/2016/05/29304.php > > >>>>> > > >>>>> _______________________________________________ > > >>>>> users mailing list > > >>>>> us...@open-mpi.org > > >>>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > >>>>> Link to this post: > > >>>>> http://www.open-mpi.org/community/lists/users/2016/05/29307.php > > >>>>> > > >>>> _______________________________________________ > > >>>> users mailing list > > >>>> us...@open-mpi.org > > >>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > >>>> Link to this post: > > >>>> http://www.open-mpi.org/community/lists/users/2016/05/29308.php > > >>> > > >>> _______________________________________________ > > >>> users mailing list > > >>> us...@open-mpi.org > > >>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > >>> Link to this post: > > >>> http://www.open-mpi.org/community/lists/users/2016/05/29309.php > > >>> > > >> _______________________________________________ > > >> users mailing list > > >> us...@open-mpi.org > > >> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > >> Link to this post: > > >> http://www.open-mpi.org/community/lists/users/2016/05/29315.php > > > > > > _______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > > Link to this post: > > > http://www.open-mpi.org/community/lists/users/2016/05/29316.php > > > > > > > > > > > ------------------------------ > > > > Message: 3 > > Date: Fri, 27 May 2016 09:14:42 +0000 > > From: "Marco D'Amico" <marco.damic...@gmail.com> > > To: us...@open-mpi.org > > Subject: [OMPI users] OpenMPI virtualization aware > > Message-ID: > > <CABi-01XH+vdi2egBD=knen_cyxpecg0j-+3rtvnfnc6mtd+...@mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Hi I'm recently investigating in Virtualization used in HPC field, and I > > found out that MVAPICH has a "Virtualization aware" version, that permit to > > overcome the big latencies problems of using a Virtualization environment > > for HPC. > > > > My question is if there is any similar efforts in OpenMPI, since I would > > eventually contribute in it. > > > > Best regards, > > Marco D'Amico > > -------------- next part -------------- > > HTML attachment scrubbed and removed > > > > ------------------------------ > > > > Message: 4 > > Date: Fri, 27 May 2016 06:45:05 -0700 > > From: Ralph Castain <r...@open-mpi.org> > > To: Open MPI Users <us...@open-mpi.org> > > Subject: Re: [OMPI users] OpenMPI virtualization aware > > Message-ID: <bbeb8e66-40b0-4688-8284-2113252e1...@open-mpi.org> > > Content-Type: text/plain; charset="utf-8" > > > > Hi Marco > > > > OMPI has integrated support for the Singularity container: > > > > http://singularity.lbl.gov/index.html > > <http://singularity.lbl.gov/index.html> > > > > https://groups.google.com/a/lbl.gov/forum/#!forum/singularity > > <https://groups.google.com/a/lbl.gov/forum/#!forum/singularity> > > > > It is in OMPI master now, and an early version is in 2.0 - the full > > integration will be in 2.1. Singularity is undergoing changes for its 2.0 > > release (so we?ll need to do some updating of the OMPI integration), and > > there is still plenty that can be done to further optimize its integration > > - so contributions would be welcome! > > > > Ralph > > > > > > > > > On May 27, 2016, at 2:14 AM, Marco D'Amico <marco.damic...@gmail.com> > > > wrote: > > > > > > Hi I'm recently investigating in Virtualization used in HPC field, and I > > > found out that MVAPICH has a "Virtualization aware" version, that permit > > > to overcome the big latencies problems of using a Virtualization > > > environment for HPC. > > > > > > My question is if there is any similar efforts in OpenMPI, since I would > > > eventually contribute in it. > > > > > > Best regards, > > > Marco D'Amico > > > _______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > > Link to this post: > > > http://www.open-mpi.org/community/lists/users/2016/05/29320.php > > > > -------------- next part -------------- > > HTML attachment scrubbed and removed > > > > ------------------------------ > > > > Subject: Digest Footer > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > https://www.open-mpi.org/mailman/listinfo.cgi/users > > > > ------------------------------ > > > > End of users Digest, Vol 3514, Issue 1 > > ************************************** > > > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > Link to this post: > > http://www.open-mpi.org/community/lists/users/2016/06/29341.php > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > users mailing list > us...@open-mpi.org > https://www.open-mpi.org/mailman/listinfo.cgi/users > > ------------------------------ > > End of users Digest, Vol 3518, Issue 2 > ************************************** > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/06/29344.php -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
