Thanks for all your replies and sorry for the delay in getting back to you.
On Tue, Jun 19, 2007 at 01:40:21PM -0400, Jeff Squyres wrote:
> On Jun 19, 2007, at 9:18 AM, Chris Reeves wrote:
>
> > Also attached is a small patch that I wrote to work around some firewall
> > limitations on the nodes (I don't know if there's a better way to do this
> > - suggestions are welcome). The patch may or may not be relevant, but I'm
> > not ruling out network issues and a bit of peer review never goes amiss
> > in case I've done something very silly.
>
> From the looks of the patch, it looks like you just want Open MPI to
> restrict itself to a specific range of ports, right? If that's the
> case, we'd probably do this slightly differently (with MCA parameters
> -- we certainly wouldn't want to force everyone to use a hard-coded
> port range). Brian's also re-working some TCP and OOB issues on a /
> tmp branch right now; we'd want to wait until he's done before
> applying a similar patch.
I thought that would be the 'official', configurable way to do it. But I lack
a thorough enough understanding of how everything fits together to implement
it in that way.
> My first question is: why are you calling MPI_BARRIER? ;-)
Good question. Thinking about it, not all occurences are probably necessary. I
didn't write this code, but I will discuss this with my colleague.
> Clearly, if we're getting stuck in there, it could be a bug. Have
> you run your code through a memory-checking debugger? It's hard to
> say exactly what the problem is without more information -- it could
> be your app, it could be OMPI, it could be the network, ...
>
> It's a good datapoint to run with other MPI implementations, but "it
> worked with MPI X" isn't always an iron-clad indication that the new
> MPI is at fault. I'm not saying we don't have bugs in Open MPI :-)
> -- I'm just saying that I agree with you: more data is necessary.
The code is compiled with debugging turned on (with gcc's -g flag). I believe
that this does a certain degree of memory checking, but I'd have to look it up
to make sure...
Indeed. I'm not necessarily blaming OpenMPI :-p The above was merely, as you
say, an additional datapoint.
> > (gdb) where
> > #0 0x9000121c in sigprocmask ()
> > #1 0x01c46f96 in opal_evsignal_recalc ()
> > #2 0x01c458c2 in opal_event_base_loop ()
> > #3 0x01c45d32 in opal_event_loop ()
> > #4 0x01c3e6f2 in opal_progress ()
> > #5 0x01b6083e in ompi_request_wait_all ()
> > #6 0x01ec68d8 in ompi_coll_tuned_sendrecv_actual ()
> > #7 0x01ecbf64 in ompi_coll_tuned_barrier_intra_bruck ()
> > #8 0x01b75590 in MPI_Barrier ()
>
> Just a quick sanity check: I assume the call stack is the same on all
> processes, right? I.e., ompi_coll_tuned_barrier_intra_bruck () is
> the call right after MPI_BARRIER?
It is similar. Obviously different processes are at different points in the
loop when I attach, but the traces are similar enough. All of them have
ompi_coll_tuned_barrier_intra_bruck in the stack after MPI_Barrier.
> > What if some packets went missing on the network? Surely TCP should take
> > care of this and resend?
>
> What is the topology of the network that you're running on?
9 machines are physically co-located and each have a single connection to one
of two linked switches. The 10th machine is in a different part of the
building, but on the same subnet (off a different switch). All machines can
talk to each other under normal conditions.
> > As implied by my line of questioning, my current thoughts are that some
> > messages between nodes have somehow gone missing. Could this happen? What
> > could cause this? All machines are on the same subnet.
>
> Hmm. On a single subnet, but you need the firewall capability -- are
> they physically remote from each other, or do you just have the local
> firewalling capabilities enabled on each node?
Each node has a local firewall set up by the systems administrator, who was
persuaded to poke a 'small' (1000-port) hole in said firewall for
communication between the nodes. There are no further firewalls between the
nodes. The firewalls are there to stay.
Cheers for all your help so far,
Chris
