Excellent! Yes, we use pipe in several places, including in the run-time during various stages of launch, so that could be a problem.
Also, be aware that other users have reported problems on LDAP-based systems when attempting to launch large jobs. The problem is that the OpenMPI launch system has no rate control in it - and the LDAP's slapd servers get overwhelmed by the launch when we ssh on a large number of nodes. I promised another user to concoct a fix for this problem, but am taking a break from the project for a few months so it may be a little while before a fix is available. When I do get it done, it may or may not make it into an OpenMPI release for some time - I'm not sure how they will decide to schedule the change (is it a "bug", or a new "feature"?). So I may do an interim release as a patch on the OpenRTE site (since that is the run-time underneath OpenMPI). I'll let people know via this mailing list either way. Ralph On 3/18/07 2:06 PM, "David Bronke" <whitel...@gmail.com> wrote: > I just received an email from a friend who is helping me work on > resolving this; he was able to trace the problem back to a pipe() call > in OpenMPI apparently: > >> The problem is with the pipe() system call (which is invoked by the >> MPI_Send() as far as I can tell) by a LDAP authenticated user. Still >> working out where exactly that goes wrong, but the fact is that it isn't >> actually a permissions problem - the reason it works as root is because >> root is a local user and does /etc/passwd normal authentication. > > I had forgotten to mention that we use LDAP for authentication on this > machine; PAM and NSS are set up to use it, but I'm guessing that > either OpenMPI itself or the pipe() system call won't check with them > when needed... We have made some local users on the machine to get > things going, but I'll probably have to find an LDAP mailing list to > get this issue resolved. > > Thanks for all the help so far! > > On 3/16/07, Ralph Castain <r...@lanl.gov> wrote: >> I'm afraid I have zero knowledge or experience with gentoo portage, so I >> can't help you there. I always install our releases from the tarball source >> as it is pretty trivial to do and avoids any issues. >> >> I will have to defer to someone who knows that system to help you from here. >> It sounds like an installation or configuration issue. >> >> Ralph >> >> >> >> On 3/16/07 3:15 PM, "David Bronke" <whitel...@gmail.com> wrote: >> >>> On 3/15/07, Ralph Castain <r...@lanl.gov> wrote: >>>> Hmmm...well, a few thoughts to hopefully help with the debugging. One >>>> initial comment, though - 1.1.2 is quite old. You might want to upgrade to >>>> 1.2 (releasing momentarily - you can use the last release candidate in the >>>> interim as it is identical). >>> >>> Version 1.2 doesn't seem to be in gentoo portage yet, so I may end up >>> having to compile from source... I generally prefer to do everything >>> from portage if possible, because it makes upgrades and maintenance >>> much cleaner. >>> >>>> Meantime, looking at this output, there appear to be a couple of common >>>> possibilities. First, I don't see any of the diagnostic output from after >>>> we >>>> do a local fork (we do this prior to actually launching the daemon). Is it >>>> possible your system doesn't allow you to fork processes (some don't, >>>> though >>>> it's unusual)? >>> >>> I don't see any problems with forking on this system... I'm able to >>> start a dbus daemon as a regular user without any problems. >>> >>>> Second, it could be that the "orted" program isn't being found in your >>>> path. >>>> People often forget that the path in shells started up by programs isn't >>>> necessarily the same as that in their login shell. You might try executing >>>> a >>>> simple shellscript that outputs the results of "which orted" to verify this >>>> is correct. >>> >>> 'which orted' from a shell script gives me '/usr/bin/orted', which >>> seems to be correct. >>> >>>> BTW, I should have asked as well: what are you running this on, and how did >>>> you configure openmpi? >>> >>> I'm running this on two identical machines with 2 dual-core >>> hyperthreading Xeon processors. (EM64T) I simply installed OpenMPI >>> using portage, with the USE flags "debug fortran pbs -threads". (I've >>> also tried it with "-debug fortran pbs threads") >>> >>>> Ralph >>>> >>>> >>>> >>>> On 3/15/07 5:33 PM, "David Bronke" <whitel...@gmail.com> wrote: >>>> >>>>> I'm using OpenMPI version 1.1.2. I installed it using gentoo portage, >>>>> so I think it has the right permissions... I tried doing 'equery f >>>>> openmpi | xargs ls -dl' and inspecting the permissions of each file, >>>>> and I don't see much out of the ordinary; it is all owned by >>>>> root:root, but every file has read permission for user, group, and >>>>> other. (and execute for each as well when appropriate) From the debug >>>>> output, I can tell that mpirun is creating the session tree in /tmp, >>>>> and it does seem to be working fine... Here's the output when using >>>>> --debug-daemons: >>>>> >>>>> $ mpirun -aborted 8 -v -d --debug-daemons -np 8 >>>>> /workspace/bronke/mpi/hello >>>>> [trixie:25228] [0,0,0] setting up session dir with >>>>> [trixie:25228] universe default-universe >>>>> [trixie:25228] user bronke >>>>> [trixie:25228] host trixie >>>>> [trixie:25228] jobid 0 >>>>> [trixie:25228] procid 0 >>>>> [trixie:25228] procdir: >>>>> /tmp/openmpi-sessions-bronke@trixie_0/default-universe/0/0 >>>>> [trixie:25228] jobdir: >>>>> /tmp/openmpi-sessions-bronke@trixie_0/default-universe/0 >>>>> [trixie:25228] unidir: >>>>> /tmp/openmpi-sessions-bronke@trixie_0/default-universe >>>>> [trixie:25228] top: openmpi-sessions-bronke@trixie_0 >>>>> [trixie:25228] tmp: /tmp >>>>> [trixie:25228] [0,0,0] contact_file >>>>> /tmp/openmpi-sessions-bronke@trixie_0/default-universe/universe-setup.txt >>>>> [trixie:25228] [0,0,0] wrote setup file >>>>> [trixie:25228] pls:rsh: local csh: 0, local bash: 1 >>>>> [trixie:25228] pls:rsh: assuming same remote shell as local shell >>>>> [trixie:25228] pls:rsh: remote csh: 0, remote bash: 1 >>>>> [trixie:25228] pls:rsh: final template argv: >>>>> [trixie:25228] pls:rsh: /usr/bin/ssh <template> orted --debug >>>>> --debug-daemons --bootproxy 1 --name <template> --num_procs 2 >>>>> --vpid_start 0 --nodename <template> --universe >>>>> bronke@trixie:default-universe --nsreplica >>>>> "0.0.0;tcp://141.238.31.33:43838" --gprreplica >>>>> "0.0.0;tcp://141.238.31.33:43838" --mpi-call-yield 0 >>>>> [trixie:25228] sess_dir_finalize: proc session dir not empty - leaving >>>>> [trixie:25228] spawn: in job_state_callback(jobid = 1, state = 0x100) >>>>> mpirun noticed that job rank 0 with PID 0 on node "localhost" exited >>>>> on signal 13. >>>>> [trixie:25228] sess_dir_finalize: proc session dir not empty - leaving >>>>> [trixie:25228] sess_dir_finalize: proc session dir not empty - leaving >>>>> [trixie:25228] sess_dir_finalize: proc session dir not empty - leaving >>>>> [trixie:25228] sess_dir_finalize: proc session dir not empty - leaving >>>>> [trixie:25228] sess_dir_finalize: proc session dir not empty - leaving >>>>> [trixie:25228] sess_dir_finalize: proc session dir not empty - leaving >>>>> [trixie:25228] sess_dir_finalize: proc session dir not empty - leaving >>>>> [trixie:25228] spawn: in job_state_callback(jobid = 1, state = 0x80) >>>>> mpirun noticed that job rank 0 with PID 0 on node "localhost" exited >>>>> on signal 13. >>>>> mpirun noticed that job rank 1 with PID 0 on node "localhost" exited >>>>> on signal 13. >>>>> mpirun noticed that job rank 2 with PID 0 on node "localhost" exited >>>>> on signal 13. >>>>> mpirun noticed that job rank 3 with PID 0 on node "localhost" exited >>>>> on signal 13. >>>>> mpirun noticed that job rank 4 with PID 0 on node "localhost" exited >>>>> on signal 13. >>>>> mpirun noticed that job rank 5 with PID 0 on node "localhost" exited >>>>> on signal 13. >>>>> mpirun noticed that job rank 6 with PID 0 on node "localhost" exited >>>>> on signal 13. >>>>> [trixie:25228] ERROR: A daemon on node localhost failed to start as >>>>> expected. >>>>> [trixie:25228] ERROR: There may be more information available from >>>>> [trixie:25228] ERROR: the remote shell (see above). >>>>> [trixie:25228] The daemon received a signal 13. >>>>> 1 additional process aborted (not shown) >>>>> [trixie:25228] sess_dir_finalize: found proc session dir empty - deleting >>>>> [trixie:25228] sess_dir_finalize: found job session dir empty - deleting >>>>> [trixie:25228] sess_dir_finalize: found univ session dir empty - deleting >>>>> [trixie:25228] sess_dir_finalize: found top session dir empty - deleting >>>>> >>>>> On 3/15/07, Ralph H Castain <r...@lanl.gov> wrote: >>>>>> It isn't a /dev issue. The problem is likely that the system lacks >>>>>> sufficient permissions to either: >>>>>> >>>>>> 1. create the Open MPI session directory tree. We create a hierarchy of >>>>>> subdirectories for temporary storage used for things like your shared >>>>>> memory >>>>>> file - the location of the head of that tree can be specified at run >>>>>> time, >>>>>> but has a series of built-in defaults it can search if you don't specify >>>>>> it >>>>>> (we look at your environmental variables - e.g., TMP or TMPDIR - as well >>>>>> as >>>>>> the typical Linux/Unix places). You might check to see what your tmp >>>>>> directory is, and that you have write permission into it. Alternatively, >>>>>> you >>>>>> can specify your own location (where you know you have permissions!) by >>>>>> setting --tmpdir your-dir on the mpirun command line. >>>>>> >>>>>> 2. execute or access the various binaries and/or libraries. This is >>>>>> usually >>>>>> caused when someone installs OpenMPI as root, and then tries to execute >>>>>> as >>>>>> a >>>>>> non-root user. Best thing here is to either run through the installation >>>>>> directory and add the correct permissions (assuming it is a system-level >>>>>> install), or reinstall as the non-root user (if the install is solely for >>>>>> you anyway). >>>>>> >>>>>> You can also set --debug-daemons on the mpirun command line to get more >>>>>> diagnostic output from the daemons and then send that along. >>>>>> >>>>>> BTW: if possible, it helps us to advise you if we know which version of >>>>>> OpenMPI you are using. ;-) >>>>>> >>>>>> Hope that helps. >>>>>> Ralph >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 3/15/07 1:51 PM, "David Bronke" <whitel...@gmail.com> wrote: >>>>>> >>>>>>> Ok, now that I've figured out what the signal means, I'm wondering >>>>>>> exactly what is running into permission problems... the program I'm >>>>>>> running doesn't use any functions except printf, sprintf, and MPI_*... >>>>>>> I was thinking that possibly changes to permissions on certain /dev >>>>>>> entries in newer distros might cause this, but I'm not even sure what >>>>>>> /dev entries would be used by MPI. >>>>>>> >>>>>>> On 3/15/07, McCalla, Mac <macmcca...@hess.com> wrote: >>>>>>>> Hi, >>>>>>>> If the perror command is available on your system it will tell >>>>>>>> you what the message is associated with the signal value. On my system >>>>>>>> RHEL4U3, it is permission denied. >>>>>>>> >>>>>>>> HTH, >>>>>>>> >>>>>>>> mac mccalla >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On >>>>>>>> Behalf Of David Bronke >>>>>>>> Sent: Thursday, March 15, 2007 12:25 PM >>>>>>>> To: us...@open-mpi.org >>>>>>>> Subject: [OMPI users] Signal 13 >>>>>>>> >>>>>>>> I've been trying to get OpenMPI working on two of the computers at a >>>>>>>> lab >>>>>>>> I help administer, and I'm running into a rather large issue. When >>>>>>>> running anything using mpirun as a normal user, I get the following >>>>>>>> output: >>>>>>>> >>>>>>>> >>>>>>>> $ mpirun --no-daemonize --host >>>>>>>> localhost,localhost,localhost,localhost,localhost,localhost,localhost,l>>>>>>>> o >>>>>>>> calhost >>>>>>>> /workspace/bronke/mpi/hello >>>>>>>> mpirun noticed that job rank 0 with PID 0 on node "localhost" exited on >>>>>>>> signal 13. >>>>>>>> [trixie:18104] ERROR: A daemon on node localhost failed to start as >>>>>>>> expected. >>>>>>>> [trixie:18104] ERROR: There may be more information available from >>>>>>>> [trixie:18104] ERROR: the remote shell (see above). >>>>>>>> [trixie:18104] The daemon received a signal 13. >>>>>>>> 8 additional processes aborted (not shown) >>>>>>>> >>>>>>>> >>>>>>>> However, running the same exact command line as root works fine: >>>>>>>> >>>>>>>> >>>>>>>> $ sudo mpirun --no-daemonize --host >>>>>>>> localhost,localhost,localhost,localhost,localhost,localhost,localhost,l>>>>>>>> o >>>>>>>> calhost >>>>>>>> /workspace/bronke/mpi/hello >>>>>>>> Password: >>>>>>>> p is 8, my_rank is 0 >>>>>>>> p is 8, my_rank is 1 >>>>>>>> p is 8, my_rank is 2 >>>>>>>> p is 8, my_rank is 3 >>>>>>>> p is 8, my_rank is 6 >>>>>>>> p is 8, my_rank is 7 >>>>>>>> Greetings from process 1! >>>>>>>> >>>>>>>> Greetings from process 2! >>>>>>>> >>>>>>>> Greetings from process 3! >>>>>>>> >>>>>>>> p is 8, my_rank is 5 >>>>>>>> p is 8, my_rank is 4 >>>>>>>> Greetings from process 4! >>>>>>>> >>>>>>>> Greetings from process 5! >>>>>>>> >>>>>>>> Greetings from process 6! >>>>>>>> >>>>>>>> Greetings from process 7! >>>>>>>> >>>>>>>> >>>>>>>> I've looked up signal 13, and have found that it is apparently SIGPIPE; >>>>>>>> I also found a thread on the LAM-MPI site: >>>>>>>> http://www.lam-mpi.org/MailArchives/lam/2004/08/8486.php >>>>>>>> However, this thread seems to indicate that the problem would be in the >>>>>>>> application, (/workspace/bronke/mpi/hello in this case) but there are >>>>>>>> no >>>>>>>> pipes in use in this app, and the fact that it works as expected as >>>>>>>> root >>>>>>>> doesn't seem to fit either. I have tried running mpirun with --verbose >>>>>>>> and it doesn't show any more output than without it, so I've run into a >>>>>>>> sort of dead-end on this issue. Does anyone know of any way I can >>>>>>>> figure >>>>>>>> out what's going wrong or how I can fix it? >>>>>>>> >>>>>>>> Thanks! >>>>>>>> -- >>>>>>>> David H. Bronke >>>>>>>> Lead Programmer >>>>>>>> G33X Nexus Entertainment >>>>>>>> http://games.g33xnexus.com/precursors/ >>>>>>>> >>>>>>>> v3sw5/7Hhw5/6ln4pr6Ock3ma7u7+8Lw3/7Tm3l6+7Gi2e4t4Mb7Hen5g8+9ORPa22s6MSr>>>>>>>> 7 >>>>>>>> p6 >>>>>>>> hackerkey.com >>>>>>>> Support Web Standards! http://www.webstandards.org/ >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>> >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >
