Re: [OMPI users] Occasional mpirun hang on completion
. The expected output > > > appears on > > > > the screen, but mpirun needs a SIGKILL to return to the console. > > > > > > > > This has been verified with OpenMPI v1.2.4 compiled with both icc > > > 9.1 > > > > 20061101 (aka 9.1.045) and gcc 4.1.0 20060304 (aka Red Hat > > > 4.1.0-3). I > > > > have also tried earlier versions of OpenMPI and found the same bug > > > > (1.1.2 and 1.2.2). > > > > > > > > Using -verbose didn't provide any additional output. I'm happy > > > to help > > > > tracking down whatever is causing this. > > > > > > A couple more data points: > > > > > > mpirun -np 4 -hostfile ~/hostfile --no-daemonize uptime > > > > > > hung twice over 100 runs. Without the --no-daemonize, the command > > > hung > > > 16 times over 100 runs. (This is using the version compiled with > > > icc.) > > > > > > Barry > > > > > > > > > > > Many thanks, > > > > > > > > Barry Rountree > > > > Ph.D. Candidate, Computer Science > > > > University of Georgia > > > > > > > > ___ > > > > users mailing list > > > > us...@open-mpi.org > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > ___ > > > users mailing list > > > us...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > -- > > Jeff Squyres > > Cisco Systems > > > > ___ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > -- - Marco Sbrighi m.sbri...@cineca.it HPC Group CINECA Interuniversity Computing Centre via Magnanelli, 6/3 40033 Casalecchio di Reno (Bo) ITALY tel. 051 6171516
Re: [OMPI users] Bug in oob_tcp_[in|ex]clude?
On Mon, 2007-12-17 at 20:58 -0500, Brian Dobbins wrote: > Hi Marco and Jeff, > > My own knowledge of OpenMPI's internals is limited, but I thought > I'd add my less-than-two-cents... > > > I've found only a way in order to have tcp connections > binded only to > > the eth1 interface, using both the following MCA directives > in the > > command line: > > > > mpirun --mca oob_tcp_include eth1 --mca > oob_tcp_include > > lo,eth0,ib0,ib1 . > > > > This sounds me as bug. > > > Yes, it does. Specifying the MCA same param twice on the > command line > results in undefined behavior -- it will only take one of > them, and I > assume it'll take the first (but I'd have to check the code to > be sure). > > I think that Marco intended to write: > mpirun --mca oob_tcp_include eth1 --mca oob_tcp_exclude > lo,eth0,ib0,ib1 ... no, I intended to write exactly what I wrote. The double statement is reported by --mca mpi_show_mca_params exactly as I write one statement only, as follows: --mca oob_tcp_include eth1,lo,eth0,ib0,ib1 > > Is this correct? So you're not specifying include twice, you're > specifying include and exclude, so each interface is explicitly stated > in one list or the other. I remember encountering this behaviour as > well, in a slightly different format, but I can't seem to reproduce it > now either. notice, the two lists are never intersecting. > That said, with these options, won't the MPI traffic (as opposed to > the OOB traffic) still use the eth1,ib0 and ib1 interfaces? You'd > need to add '-mca btl_tcp_include eth1' in order to say it should only > go over that NIC, I think. Yes I know, in fact -mca btl_tcp_[if]_exclude lo,eth0,ib0,ib1 works fine (seems). I'm using this MCA parameter since open-mpi 1.2.1 and the trouble with oob_tcp_[if]_[in|ex]clude sounded quite strange to me, after all the code used for the parser should be more or less the same . > > As for the 'connection errors', two bizarre things to check are, > first, that all of your nodes using eth1 actually have > correct /etc/hosts mappings to the other nodes. One system I ran on > had this problem when some nodes had an IP address for node002 as one > thing, and another node had node002's IP address as something > different. This should be easy enough by trying to run on one node > first, then two nodes that you're sure have the correct addresses. Yes, I've already verified that. > > .. The second situation is if you're launching an MPMD program. > Here, you need to use '-gmca ' instead of '-mca '. > No, currently I'm using only SPMD ones, and I hope to use them for the rest of the century :-) > Hope some of that is at least a tad useful. :) > Thanks you very much Brian, Marco > Cheers, > - Brian > -- - Marco Sbrighi m.sbri...@cineca.it HPC Group CINECA Interuniversity Computing Centre via Magnanelli, 6/3 40033 Casalecchio di Reno (Bo) ITALY tel. 051 6171516
Re: [OMPI users] Bug in oob_tcp_[in|ex]clude?
On Mon, 2007-12-17 at 17:19 -0500, Jeff Squyres wrote: > On Dec 17, 2007, at 8:35 AM, Marco Sbrighi wrote: > > > I'm using Open MPI 1.2.2 over OFED 1.2 on an 256 nodes, dual Opteron, > > dual core, Linux cluster. Of course, with Infiniband 4x interconnect. > > > > Each cluster node is equipped with 4 (or more) ethernet interface, > > namely 2 gigabit ones plus 2 IPoIB. The two gig are named eth0,eth1, > > while the two IPoIB are named ib0,ib1. > > > > It happens that the eth0 is a management network, with poor > > performances, and furthermore we wouldn't use the ib* to carry MPI's > > traffic (neither OOB or TCP), so we would like the eth1 is used for > > open > > MPI OOB and TCP. > > > > In order to drive the OOB over only eth1 I've tried various > > combinations > > of oob_tcp_[ex|in]clude MCA statements, starting from the obvious > > > > oob_tcp_exclude = lo,eth0,ib0,ib1 > > > > then trying the othe obvious: > > > > oob_tcp_include = eth1 > > This one statement (_include) should be sufficient. I agree with your interpretation, but what I'm experimenting here is "it should" but in fact it doesn't . > > Assumedly this(these) statement(s) are in a config file that is being > read by Open MPI, such as $HOME/.openmpi/mca-params.conf? I've tried many combinations: only in $HOME/.openmpi/mca-params.conf, only in command line and both; but none seems to work correctly. Nevertheless, what I'm expecting is that if something is specified in $HOME/.openmpi/mca-params.conf, then if differently specified in command line, the last should be assumed, I think. > > > and both at the same time. > > > > Next I've tried the following: > > > > oob_tcp_exclude = eth0 > > > > but after the job starts, I still have a lot of tcp connections > > established using eth0 or ib0 or ib1. > > Furthermore It happens the following error: > > > > [node191:03976] [0,1,14]-[0,1,12] mca_oob_tcp_peer_complete_connect: > > connection failed: Connection timed out (110) - retrying > > This is quite odd. :-( > > > I've found only a way in order to have tcp connections binded only to > > the eth1 interface, using both the following MCA directives in the > > command line: > > > > mpirun --mca oob_tcp_include eth1 --mca oob_tcp_include > > lo,eth0,ib0,ib1 . > > > > This sounds me as bug. > > Yes, it does. Specifying the MCA same param twice on the command line > results in undefined behavior -- it will only take one of them, and I > assume it'll take the first (but I'd have to check the code to be sure). OK, I can obtain the same behaviour using only one statement: --mca oob_tcp_include eth1,lo,eth0,ib0,ib1 note that using --mca mpi_show_mca_params what I'm seeing in the report is the same for both statements (twice and single): . [node255:30188] oob_tcp_debug=0 [node255:30188] oob_tcp_include=eth1,lo,eth0,ib0,ib1 [node255:30188] oob_tcp_exclude= ... > > > Is there someone able to reproduce this behaviour? > > If this is a bug, are there fixes? > > > I'm unfortunately unable to reproduce this behavior. I have a test > cluster with 2 IP interfaces: ib0, eth0. I have tried several > combinations of MCA params with 1.2.2: > > --mca oob_tcp_include ib0 > --mca oob_tcp_include ib0,bogus > --mca oob_tcp_include eth0 > --mca oob_tcp_include eth0,bogus > --mca oob_tcp_exclude ib0 > --mca oob_tcp_exclude ib0,bogus > --mca oob_tcp_exclude eth0 > --mca oob_tcp_exclude eth0,bogus > > All do as they are supposed to -- including or excluding ib0 or eth0. > > I do note, however, that the handling of these parameters changed in > 1.2.3 -- as well as their names. The names changed to > "oob_tcp_if_include" and "oob_tcp_if_exclude" to match other MCA > parameter name conventions from other components. > > Could you try with 1.2.3 or 1.2.4 (1.2.4 is the most recent; 1.2.5 is > due out "soon" -- it *may* get out before the holiday break, but no > promises...)? we have 1.2.3 in another cluster and it performs the same behaviour as 1.2.2 (BTW the other cluster has the same eth ifaces) > > If you can't upgrade, let me know and I can provide a debugging patch > that will give us a little more insight into what is happening on your > machines. Thanks. It is quite difficult for us to upgrade the open-mpi now. We have the official CISCO packages installed, and I know the 1.2.2-1 is the only official CISCO's open-mpi distribution today In any case I would like to try your debug patch. Thanks Marco > -- - Marco Sbrighi m.sbri...@cineca.it HPC Group CINECA Interuniversity Computing Centre via Magnanelli, 6/3 40033 Casalecchio di Reno (Bo) ITALY tel. 051 6171516
