I'm not calling mpirun in this case because this particular calculation doesn't use more than one processor. What I'm doing on my command line is this:
/home/user/myapp/tools/python/bin/python test/regression/regression-test.py test/regression/regression-jobs and internally I check for rank/size. This command is executed in the context of a souped up LD_LIBRARY_PATH. You can see the variable argv in opal_argv_join is ending up with the last argument on my command line. I suppose your question implies that mpirun is mandatory for executing anything compiled with OpenMPI > 1.6 ? On Sep 29, 2014, at 10:28 AM, Ralph Castain <r...@open-mpi.org> wrote: > Can you pass us the actual mpirun command line being executed? Especially > need to see the argv being passed to your application. > > > On Sep 27, 2014, at 7:09 PM, Amos Anderson <amos.ander...@protabit.com> wrote: > >> FWIW, I've confirmed that the segfault also happens with OpenMPI 1.7.5. >> Also, I have some gdb output (from 1.7.5) for your perusal, including a >> printout of some of the variables' values. >> >> >> >> Starting program: /home/user/myapp/tools/python/bin/python >> test/regression/regression-test.py test/regression/regression-jobs >> [Thread debugging using libthread_db enabled] >> >> Program received signal SIGSEGV, Segmentation fault. >> 0x00002aaaabc8df1e in opal_argv_join (argv=0xa39398, delimiter=32) at >> argv.c:299 >> 299 str_len += strlen(*p) + 1; >> (gdb) where >> #0 0x00002aaaabc8df1e in opal_argv_join (argv=0xa39398, delimiter=32) at >> argv.c:299 >> #1 0x00002aaaaab2ce4e in ompi_mpi_init (argc=2, argv=0xa39390, requested=0, >> provided=0x7fffffffba98) at runtime/ompi_mpi_init.c:450 >> #2 0x00002aaaaab63e39 in PMPI_Init (argc=0x7fffffffbb8c, >> argv=0x7fffffffbb80) at pinit.c:84 >> #3 0x00002aaab7b965d6 in boost::mpi::environment::environment >> (this=0xa3a1d0, argc=@0x7fffffffbb8c, argv=@0x7fffffffbb80, >> abort_on_exception=true) >> at ../tools/boost/libs/mpi/src/environment.cpp:98 >> #4 0x00002aaabc7b311d in boost::mpi::python::mpi_init (python_argv=..., >> abort_on_exception=true) at >> ../tools/boost/libs/mpi/src/python/py_environment.cpp:60 >> #5 0x00002aaabc7b33fb in boost::mpi::python::export_environment () at >> ../tools/boost/libs/mpi/src/python/py_environment.cpp:94 >> #6 0x00002aaabc7d5ab5 in boost::mpi::python::init_module_mpi () at >> ../tools/boost/libs/mpi/src/python/module.cpp:44 >> #7 0x00002aaab792a2f2 in >> boost::detail::function::void_function_ref_invoker0<void (*)(), >> void>::invoke (function_obj_ptr=...) >> at ../tools/boost/boost/function/function_template.hpp:188 >> #8 0x00002aaab7929e6b in boost::function0<void>::operator() >> (this=0x7fffffffc110) at >> ../tools/boost/boost/function/function_template.hpp:767 >> #9 0x00002aaab7928f11 in boost::python::handle_exception_impl (f=...) at >> ../tools/boost/libs/python/src/errors.cpp:25 >> #10 0x00002aaab792a54f in boost::python::handle_exception<void (*)()> >> (f=0x2aaabc7d5746 <boost::mpi::python::init_module_mpi()>) at >> ../tools/boost/boost/python/errors.hpp:29 >> #11 0x00002aaab792a1d9 in boost::python::detail::(anonymous >> namespace)::init_module_in_scope (m=0x2aaabc617f68, >> init_function=0x2aaabc7d5746 <boost::mpi::python::init_module_mpi()>) at >> ../tools/boost/libs/python/src/module.cpp:24 >> #12 0x00002aaab792a26c in boost::python::detail::init_module >> (name=0x2aaabc7f7f4d "mpi", init_function=0x2aaabc7d5746 >> <boost::mpi::python::init_module_mpi()>) >> at ../tools/boost/libs/python/src/module.cpp:59 >> #13 0x00002aaabc7d5b2b in boost::mpi::python::initmpi () at >> ../tools/boost/libs/mpi/src/python/module.cpp:34 >> #14 0x00002aaaab27e095 in _PyImport_LoadDynamicModule (name=0xac9435 "mpi", >> pathname=0xb08c60 "/home/user/myapp/lib/mpi.so", fp=0xaca450) at >> ./Python/importdl.c:53 >> #15 0x00002aaaab279fd4 in load_module (name=0xac9435 "mpi", fp=0xaca450, >> pathname=0xb08c60 "/home/user/myapp/lib/mpi.so", type=3, loader=0x0) at >> Python/import.c:1915 >> #16 0x00002aaaab27c2e8 in import_submodule (mod=0x2aaaab533a20, >> subname=0xac9435 "mpi", fullname=0xac9435 "mpi") at Python/import.c:2700 >> #17 0x00002aaaab27b8fa in load_next (mod=0x2aaab0f075a8, >> altmod=0x2aaaab533a20, p_name=0x7fffffffc3f8, buf=0xac9430 "util.mpi", >> p_buflen=0x7fffffffc408) >> at Python/import.c:2519 >> #18 0x00002aaaab27a98d in import_module_level (name=0x0, globals=0xe95a70, >> locals=0xe95a70, fromlist=0x2aaaab533a20, level=-1) at Python/import.c:2224 >> #19 0x00002aaaab27aeda in PyImport_ImportModuleLevel (name=0x2aaab0f00964 >> "mpi", globals=0xe95a70, locals=0xe95a70, fromlist=0x2aaaab533a20, level=-1) >> at Python/import.c:2288 >> #20 0x00002aaaab2419c4 in builtin___import__ (self=0x0, args=0x2aaabc6211f8, >> kwds=0x0) at Python/bltinmodule.c:49 >> #21 0x00002aaaab1b19c7 in PyCFunction_Call (func=0x2aaaabf85510, >> arg=0x2aaabc6211f8, kw=0x0) at Objects/methodobject.c:85 >> #22 0x00002aaaab14d673 in PyObject_Call (func=0x2aaaabf85510, >> arg=0x2aaabc6211f8, kw=0x0) at Objects/abstract.c:2529 >> #23 0x00002aaaab25ad03 in PyEval_CallObjectWithKeywords >> (func=0x2aaaabf85510, arg=0x2aaabc6211f8, kw=0x0) at Python/ceval.c:3890 >> #24 0x00002aaaab2543e5 in PyEval_EvalFrameEx (f=0xe8aef0, throwflag=0) at >> Python/ceval.c:2333 >> #25 0x00002aaaab258b7e in PyEval_EvalCodeEx (co=0x2aaabc61ce00, >> globals=0xe95a70, locals=0xe95a70, args=0x0, argcount=0, kws=0x0, kwcount=0, >> defs=0x0, defcount=0, >> closure=0x0) at Python/ceval.c:3253 >> #26 0x00002aaaab24b5ce in PyEval_EvalCode (co=0x2aaabc61ce00, >> globals=0xe95a70, locals=0xe95a70) at Python/ceval.c:667 >> #27 0x00002aaaab2779e2 in PyImport_ExecCodeModuleEx (name=0xaa9080 >> "util.myappMPI", co=0x2aaabc61ce00, pathname=0xe7d380 >> "/home/user/myapp/src/util/myappMPI.pyc") >> at Python/import.c:709 >> #28 0x00002aaaab278629 in load_source_module (name=0xaa9080 "util.myappMPI", >> pathname=0xe7d380 "/home/user/myapp/src/util/myappMPI.pyc", fp=0x76eb00) >> at Python/import.c:1099 >> #29 0x00002aaaab279fa0 in load_module (name=0xaa9080 "util.myappMPI", >> fp=0x76eb00, pathname=0x80fe00 "/home/user/myapp/src/util/myappMPI.py", >> type=1, loader=0x0) >> at Python/import.c:1906 >> #30 0x00002aaaab27c2e8 in import_submodule (mod=0x2aaab0f075a8, >> subname=0xaa9085 "myappMPI", fullname=0xaa9080 "util.myappMPI") at >> Python/import.c:2700 >> #31 0x00002aaaab27b860 in load_next (mod=0x2aaab0f075a8, >> altmod=0x2aaaab533a20, p_name=0x7fffffffcd98, buf=0xaa9080 "util.myappMPI", >> p_buflen=0x7fffffffcda8) >> at Python/import.c:2515 >> #32 0x00002aaaab27a98d in import_module_level (name=0x0, globals=0x7a3c70, >> locals=0x7a3c70, fromlist=0x2aaaaf6f53e0, level=-1) at Python/import.c:2224 >> #33 0x00002aaaab27aeda in PyImport_ImportModuleLevel (name=0x2aaaaf6e8854 >> "myappMPI", globals=0x7a3c70, locals=0x7a3c70, fromlist=0x2aaaaf6f53e0, >> level=-1) >> at Python/import.c:2288 >> #34 0x00002aaaab2419c4 in builtin___import__ (self=0x0, args=0x2aaab0a9f7d0, >> kwds=0x0) at Python/bltinmodule.c:49 >> #35 0x00002aaaab1b19c7 in PyCFunction_Call (func=0x2aaaabf85510, >> arg=0x2aaab0a9f7d0, kw=0x0) at Objects/methodobject.c:85 >> #36 0x00002aaaab14d673 in PyObject_Call (func=0x2aaaabf85510, >> arg=0x2aaab0a9f7d0, kw=0x0) at Objects/abstract.c:2529 >> #37 0x00002aaaab25ad03 in PyEval_CallObjectWithKeywords >> (func=0x2aaaabf85510, arg=0x2aaab0a9f7d0, kw=0x0) at Python/ceval.c:3890 >> #38 0x00002aaaab2543e5 in PyEval_EvalFrameEx (f=0x79d000, throwflag=0) at >> Python/ceval.c:2333 >> #39 0x00002aaaab258b7e in PyEval_EvalCodeEx (co=0x2aaab0253880, >> globals=0x7a3c70, locals=0x7a3c70, args=0x0, argcount=0, kws=0x0, kwcount=0, >> defs=0x0, defcount=0, >> closure=0x0) at Python/ceval.c:3253 >> #40 0x00002aaaab24b5ce in PyEval_EvalCode (co=0x2aaab0253880, >> globals=0x7a3c70, locals=0x7a3c70) at Python/ceval.c:667 >> #41 0x00002aaaab2779e2 in PyImport_ExecCodeModuleEx (name=0x7153a0 "util", >> co=0x2aaab0253880, pathname=0x7e98a0 >> "/home/user/myapp/src/util/__init__.pyc") >> at Python/import.c:709 >> #42 0x00002aaaab278629 in load_source_module (name=0x7153a0 "util", >> pathname=0x7e98a0 "/home/user/myapp/src/util/__init__.pyc", fp=0x6fe020) at >> Python/import.c:1099 >> #43 0x00002aaaab279fa0 in load_module (name=0x7153a0 "util", fp=0x6fe020, >> pathname=0x755e40 "/home/user/myapp/src/util/__init__.py", type=1, >> loader=0x0) >> at Python/import.c:1906 >> ---Type <return> to continue, or q <return> to quit--- >> #44 0x00002aaaab2788ef in load_package (name=0x7153a0 "util", >> pathname=0x703390 "/home/user/myapp/src/util") at Python/import.c:1166 >> #45 0x00002aaaab279fea in load_module (name=0x7153a0 "util", fp=0x0, >> pathname=0x703390 "/home/user/myapp/src/util", type=5, loader=0x0) at >> Python/import.c:1920 >> #46 0x00002aaaab27c2e8 in import_submodule (mod=0x2aaaab533a20, >> subname=0x7153a0 "util", fullname=0x7153a0 "util") at Python/import.c:2700 >> #47 0x00002aaaab27b860 in load_next (mod=0x2aaaab533a20, >> altmod=0x2aaaab533a20, p_name=0x7fffffffd818, buf=0x7153a0 "util", >> p_buflen=0x7fffffffd828) at Python/import.c:2515 >> #48 0x00002aaaab27a98d in import_module_level (name=0x2aaaac0e0f59 >> "myappSubmission", globals=0x6443c0, locals=0x6443c0, >> fromlist=0x2aaaac0f1760, level=-1) >> at Python/import.c:2224 >> #49 0x00002aaaab27aeda in PyImport_ImportModuleLevel (name=0x2aaaac0e0f54 >> "util.myappSubmission", globals=0x6443c0, locals=0x6443c0, >> fromlist=0x2aaaac0f1760, level=-1) >> at Python/import.c:2288 >> #50 0x00002aaaab2419c4 in builtin___import__ (self=0x0, args=0x2aaaac0dbb00, >> kwds=0x0) at Python/bltinmodule.c:49 >> #51 0x00002aaaab1b19c7 in PyCFunction_Call (func=0x2aaaabf85510, >> arg=0x2aaaac0dbb00, kw=0x0) at Objects/methodobject.c:85 >> #52 0x00002aaaab14d673 in PyObject_Call (func=0x2aaaabf85510, >> arg=0x2aaaac0dbb00, kw=0x0) at Objects/abstract.c:2529 >> #53 0x00002aaaab25ad03 in PyEval_CallObjectWithKeywords >> (func=0x2aaaabf85510, arg=0x2aaaac0dbb00, kw=0x0) at Python/ceval.c:3890 >> #54 0x00002aaaab2543e5 in PyEval_EvalFrameEx (f=0x71a1a0, throwflag=0) at >> Python/ceval.c:2333 >> #55 0x00002aaaab258b7e in PyEval_EvalCodeEx (co=0x2aaaac0dd720, >> globals=0x6443c0, locals=0x6443c0, args=0x0, argcount=0, kws=0x0, kwcount=0, >> defs=0x0, defcount=0, >> closure=0x0) at Python/ceval.c:3253 >> #56 0x00002aaaab24b5ce in PyEval_EvalCode (co=0x2aaaac0dd720, >> globals=0x6443c0, locals=0x6443c0) at Python/ceval.c:667 >> #57 0x00002aaaab28d492 in run_mod (mod=0x720960, filename=0x7fffffffe6bf >> "test/regression/regression-test.py", globals=0x6443c0, locals=0x6443c0, >> flags=0x7fffffffe220, >> arena=0x6707d0) at Python/pythonrun.c:1370 >> #58 0x00002aaaab28d41c in PyRun_FileExFlags (fp=0x6cb7c0, >> filename=0x7fffffffe6bf "test/regression/regression-test.py", start=257, >> globals=0x6443c0, locals=0x6443c0, >> closeit=1, flags=0x7fffffffe220) at Python/pythonrun.c:1356 >> #59 0x00002aaaab28bbfe in PyRun_SimpleFileExFlags (fp=0x6cb7c0, >> filename=0x7fffffffe6bf "test/regression/regression-test.py", closeit=1, >> flags=0x7fffffffe220) >> at Python/pythonrun.c:948 >> #60 0x00002aaaab28b1be in PyRun_AnyFileExFlags (fp=0x6cb7c0, >> filename=0x7fffffffe6bf "test/regression/regression-test.py", closeit=1, >> flags=0x7fffffffe220) >> at Python/pythonrun.c:752 >> #61 0x00002aaaab2a7497 in Py_Main (argc=3, argv=0x7fffffffe3a8) at >> Modules/main.c:640 >> #62 0x00000000004006f3 in main (argc=3, argv=0x7fffffffe3a8) at >> ./Modules/python.c:23 >> (gdb) p argv >> $1 = (char **) 0xa39398 >> (gdb) p *argv >> $2 = 0xa26390 "test/regression/regression-jobs" >> (gdb) p **argv >> $3 = 116 't' >> (gdb) p p >> $4 = (char **) 0xa393a0 >> (gdb) p *p >> $5 = 0x20 <Address 0x20 out of bounds> >> (gdb) p str_len >> $6 = 32 >> >> >> >> >> >> On Sep 26, 2014, at 5:19 PM, Amos Anderson <amos.ander...@protabit.com> >> wrote: >> >>> Hello all -- >>> >>> I'm trying to get a working configuration for my application and I can get >>> OpenMPI 1.6.5 to work, while OpenMPI 1.8.2 segfaults. >>> >>> >>> Here's how I compile OpenMPI: >>> >>> OPENMPI = openmpi-1.8.2 >>> FLAGS = --enable-static >>> cd $(OPENMPI) ; ./configure $(FLAGS) --with-tm=/opt/torque-2.5.9/ >>> --prefix=$(CURDIR) >>> >>> >>> I'm able to compile openmpi successfully, and I use a bjam instruction like >>> this to compile my program (which uses boost python boost_1_55_0): >>> using mpi : ../tools/openmpi/bin/mpic++ ; >>> >>> and I run my program in a Torque pbs script like this: >>> >>> /bin/rm -rf jobname.nodes >>> for i in `cat ${PBS_NODEFILE} | sort -u` >>> do >>> echo $i slots \= `grep $i ${PBS_NODEFILE} | wc -l` >> jobname.nodes >>> done >>> /home/user/myapp/tools/openmpi/bin/mpirun -np 2 -hostfile jobname.nodes >>> /home/user/myapp/myapp.exe >>> >>> >>> >>> which also compiles just fine. But when I run my program I get the segfault >>> I printed below. When I switch to: >>> OPENMPI = openmpi-1.6.5 >>> >>> then everything works as expected. (As a side question, do I need both >>> -hostfile and --with-tm? I asked this question earlier today on this list). >>> That is, I believe that I'm using the exact same setup in both cases, and >>> 1.6.5 works while 1.8.2 fails. Any suggestions what I might be doing wrong? >>> >>> I suppose if I have a working setup I can give up even if it's with an >>> older version... but this could be evidence of something I'll have to >>> confront eventually. >>> >>> Thanks for any advice! >>> Amos. >>> >>> >>> >>> >>> [local:27921] *** Process received signal *** >>> [local:27921] Signal: Segmentation fault (11) >>> [local:27921] Signal code: Address not mapped (1) >>> [local:27921] Failing at address: 0x40 >>> [local:27921] [ 0] /lib64/libpthread.so.0[0x322180e4c0] >>> [local:27921] [ 1] /lib64/libc.so.6(strlen+0x30)[0x3220c78d80] >>> [local:27921] [ 2] >>> /home/user/myapp/tools/openmpi/lib/libopen-pal.so.6(opal_argv_join+0x95)[0x2b87f5c4e175] >>> [local:27921] [ 3] >>> /home/user/myapp/tools/openmpi/lib/libmpi.so.1(ompi_mpi_init+0x82d)[0x2b87f3c9ec0d] >>> [local:27921] [ 4] >>> /home/user/myapp/tools/openmpi/lib/libmpi.so.1(MPI_Init+0xf0)[0x2b87f3cbc310] >>> [local:27921] [ 5] >>> /home/user/myapp/lib/libboost_mpi.so.1.55.0(_ZN5boost3mpi11environmentC1ERiRPPcb+0x36)[0x2b87f3795826] >>> [local:27921] [ 6] >>> /home/user/myapp/lib/mpi.so(_ZN5boost3mpi6python8mpi_initENS_6python4listEb+0x314)[0x2b87f30bc7b4] >>> [local:27921] [ 7] >>> /home/user/myapp/lib/mpi.so(_ZN5boost3mpi6python18export_environmentEv+0xcc6)[0x2b87f30bd5f6] >>> [local:27921] [ 8] >>> /home/user/myapp/lib/mpi.so(_ZN5boost3mpi6python15init_module_mpiEv+0x547)[0x2b87f30d4967] >>> [local:27921] [ 9] >>> /home/user/myapp/lib/libboost_python.so.1.55.0(_ZN5boost6python21handle_exception_implENS_9function0IvEE+0x530)[0x2b87f3558430] >>> [local:27921] [10] >>> /home/user/myapp/lib/libboost_python.so.1.55.0(_ZN5boost6python16handle_exceptionIPFvvEEEbT_+0x38)[0x2b87f3559798] >>> [local:27921] [11] >>> /home/user/myapp/lib/libboost_python.so.1.55.0(_ZN5boost6python6detail11init_moduleEPKcPFvvE+0x63)[0x2b87f3559463] >>> [local:27921] [12] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0(_PyImport_LoadDynamicModule+0xc2)[0x2b87e8c79282] >>> [local:27921] [13] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0[0x2b87e8c771a9] >>> [local:27921] [14] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0[0x2b87e8c776c1] >>> [local:27921] [15] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x1b7)[0x2b87e8c77977] >>> [local:27921] [16] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0[0x2b87e8c57bcd] >>> [local:27921] [17] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b87e8bb7ae8] >>> [local:27921] [18] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x56)[0x2b87e8c58216] >>> [local:27921] [19] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x381c)[0x2b87e8c5c79c] >>> [local:27921] [20] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8c9)[0x2b87e8c60c89] >>> [local:27921] [21] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x2b87e8c60d02] >>> [local:27921] [22] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xc2)[0x2b87e8c74432] >>> [local:27921] [23] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0[0x2b87e8c769f0] >>> [local:27921] [24] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0[0x2b87e8c771a9] >>> [local:27921] [25] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0[0x2b87e8c77642] >>> [local:27921] [26] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x1b7)[0x2b87e8c77977] >>> [local:27921] [27] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0[0x2b87e8c57bcd] >>> [local:27921] [28] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b87e8bb7ae8] >>> [local:27921] [29] >>> /home/user/myapp/tools/python/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x56)[0x2b87e8c58216] >>> [local:27921] *** End of error message *** >>> >>> >>> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/09/25396.php > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/09/25401.php