Scott --
Sorry; something I should have clarified in my original email: I meant you to
run the "ps" command **while mpirun was still hung**. I.e., do it in another
terminal, before you hit ctrl-C to exit mpirun.
I want to see if mpirun has launched the foo.sh or not. Gilles' test is a
different mechanism to give a similar result (i.e., it makes a side effect that
allows you to tell if the child process was actually launched or not). In the
ps test, if there's *2* copies of mpirun, it would be useful to lldb attach to
each of them and get the backtrace from both (you have the parent backtrace
already; I'm really interested to see what the child mpirun's backtrace is --
that would tell us the exact line number where the child is hung.
Gilles' observation about the firewall and IP/hostname stuff is interesting,
too. The weirdness here is that the backtrace you posted earlier implies that
the parent mpirun hadn't even finished its fork/exec sequence (i.e., mpirun
itself is still in the "do_parent()" function, which implies that it didn't
complete the pipe handshake that happens immediately after forking the child
process... which is weird).
--
Jeff Squyres
jsquy...@cisco.com
________________________________________
From: Scott Sayres <ssay...@asu.edu>
Sent: Wednesday, May 4, 2022 4:02 PM
To: Jeff Squyres (jsquyres)
Cc: Open MPI Users
Subject: Re: [OMPI users] mpirun hangs on m1 mac w openmpi-4.1.3
foo.sh is executable, again hangs without output.
I command c x2 to return to shell, then
ps auxwww | egrep 'mpirun|foo.sh'
output shown below
scottsayres@scotts-mbp trouble-shoot % ./foo.sh
Wed May 4 12:59:15 MST 2022
Wed May 4 12:59:16 MST 2022
Wed May 4 12:59:17 MST 2022
Wed May 4 12:59:18 MST 2022
Wed May 4 12:59:19 MST 2022
Wed May 4 12:59:20 MST 2022
Wed May 4 12:59:21 MST 2022
Wed May 4 12:59:22 MST 2022
Wed May 4 12:59:23 MST 2022
Wed May 4 12:59:24 MST 2022
scottsayres@scotts-mbp trouble-shoot % mpirun -np 1 foo.sh
^C^C%
scottsayres@scotts-mbp trouble-shoot % ps auxwww | egrep 'mpirun|foo.sh'
scottsayres 91795 100.0 0.0 409067920 1456 s002 R 12:59PM 0:14.07
mpirun -np 1 foo.sh
scottsayres 91798 0.0 0.0 408628368 1632 s002 S+ 1:00PM 0:00.00
egrep mpirun|foo.sh
scottsayres@scotts-mbp trouble-shoot %
On Wed, May 4, 2022 at 12:42 PM Jeff Squyres (jsquyres)
<jsquy...@cisco.com<mailto:jsquy...@cisco.com>> wrote:
That backtrace seems to imply that the launch may not have completed.
Can you make an executable script foo.sh with:
#!/bin/bash
i=0
while test $i -lt 10; do
date
sleep 1
let i=$i+1
done
Make sure that foo.sh is executable and then run it via:
mpirun -np 1 foo.sh
If you start seeing output, good! If it completes, better!
If it hangs, and/or if you don't see any output at all, do this:
ps auxwww | egrep 'mpirun|foo.sh'
It should show mpirun and 2 copies of foo.sh (and probably a grep). Does it?
--
Jeff Squyres
jsquy...@cisco.com<mailto:jsquy...@cisco.com>
________________________________________
From: Scott Sayres <ssay...@asu.edu<mailto:ssay...@asu.edu>>
Sent: Wednesday, May 4, 2022 2:47 PM
To: Open MPI Users
Cc: Jeff Squyres (jsquyres)
Subject: Re: [OMPI users] mpirun hangs on m1 mac w openmpi-4.1.3
Following Jeff's advice, I have rebuilt open-mpi by hand using the -g option.
This shows more information as below. I am attempting George's advice of how
to track the child but notice that gdb does not support arm64. attempting to
update lldb.
scottsayres@scotts-mbp openmpi-4.1.3 % lldb mpirun -- -np 1 hostname
(lldb) target create "mpirun"
Current executable set to 'mpirun' (arm64).
(lldb) settings set -- target.run-args "-np" "1" "hostname"
(lldb) run
Process 90950 launched: '/usr/local/bin/mpirun' (arm64)
Process 90950 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
frame #0: 0x00000001bde25654 libsystem_kernel.dylib`read + 8
libsystem_kernel.dylib`read:
-> 0x1bde25654 <+8>: b.lo 0x1bde25674 ; <+40>
0x1bde25658 <+12>: pacibsp
0x1bde2565c <+16>: stp x29, x30, [sp, #-0x10]!
0x1bde25660 <+20>: mov x29, sp
Target 0: (mpirun) stopped.
(lldb) ^C
(lldb) thread backtrace
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
* frame #0: 0x00000001bde25654 libsystem_kernel.dylib`read + 8
frame #1: 0x000000010056169c libopen-pal.40.dylib`opal_fd_read(fd=27,
len=20, buffer=0x000000016fdfe90c) at fd.c:51:14
frame #2: 0x00000001027b3388
mca_odls_default.so`do_parent(cd=0x0000600003e00000, read_fd=27) at
odls_default_module.c:495:14
frame #3: 0x00000001027b2d90
mca_odls_default.so`odls_default_fork_local_proc(cdptr=0x0000600003e00000) at
odls_default_module.c:651:12
frame #4: 0x00000001003246f8
libopen-rte.40.dylib`orte_odls_base_spawn_proc(fd=-1, sd=4,
cbdata=0x0000600003e00000) at odls_base_default_fns.c:1046:31
frame #5: 0x000000010057a7a0
libopen-pal.40.dylib`opal_libevent2022_event_base_loop [inlined]
event_process_active_single_queue(base=0x00000001007061c0) at event.c:1370:4
[opt]
frame #6: 0x000000010057a628
libopen-pal.40.dylib`opal_libevent2022_event_base_loop [inlined]
event_process_active(base=0x00000001007061c0) at event.c:1440:8 [opt]
frame #7: 0x000000010057a5ec
libopen-pal.40.dylib`opal_libevent2022_event_base_loop(base=0x00000001007061c0,
flags=<unavailable>) at event.c:1644:12 [opt]
frame #8: 0x0000000100003b04 mpirun`orterun(argc=4,
argv=0x000000016fdff268) at orterun.c:179:9
frame #9: 0x0000000100003904 mpirun`main(argc=4, argv=0x000000016fdff268)
at main.c:13:12
frame #10: 0x0000000100015088 dyld`start + 516
--
Scott G Sayres
Assistant Professor
School of Molecular Sciences (formerly Department of Chemistry & Biochemistry)
Biodesign Center for Applied Structural Discovery
Arizona State University
