Re: [OMPI devel] MTT tests: segv's with sm on large messages
If it would help in tracking this problem to give someone access to Sif, I can probably make that happen. Just let me know. Cheers, Josh On May 5, 2009, at 8:08 PM, Eugene Loh wrote: Jeff Squyres wrote: On May 5, 2009, at 6:01 PM, Eugene Loh wrote: You and Terry saw something that was occurring about 0.01% of the time during MPI_Init during add_procs. That does not seem to be what we are seeing here. Right -- that's what I'm saying. It's different than the MPI_INIT errors. I was trying to say that there are two kinds of MPI_Init errors. One, which you and Terry have seen, is in add_procs and shows up about 0.01% of the time. The other, um, is not and occurs more like 1% of the time. I'm not real sure what "1%" means. It isn't always 1%. But the times I've seen it has been in MTT runs in which there are dozens of failures among thousands of runs. But we have seen failures in 1.3.1 and 1.3.2 that look like the one here. They occur more like 1% of the time and can occur during MPI_Init *OR* later during a collective call. What we're looking at here seems to be related. E.g., see http://www.open-mpi.org/community/lists/devel/2009/03/5768.php Good to see that we're agreeing. Yes, I agree that this is not a new error, but it is worth fixing. Cisco's MTT didn't run last night because there was no new trunk tarball last night. I'll check Cisco's MTT tomorrow morning and see if there are any sm failures of this new flavor, and how frequently they're happening. I just took a stroll down memory lane and these errors seem to be harder to find than I thought. But, got some: http://www.open- mpi.org/mtt/index.php?do_redir=1030 IU, v1.3.1 Ah, and http://www.open-mpi.org/mtt/index.php?do_redir=1031 IU_Sif, v1.3 January 4/9700 failures I'm not sure what to key in on to find these particular errors. Yeah, worth fixing. ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel
[OMPI devel] Revise paffinity method?
We have used several different methods for binding processes over the years. Our current method is based on having the orted pass environmental parameters to the individual processes when it fork/ exec's them, and then having each process bind itself to the specified processor(s) during MPI_Init via a call to the OPAL paffinity framework. It was noted earlier this week on the user mailing list that not binding the process until it calls MPI_Init has some disadvantages - I have added some that we have previously discussed: 1. the process is free to move around -until- it calls MPI_Init, thus possibly conflicting with other processes on the node that have already called MPI_Init and been bound. 2. memory allocated by the process prior to calling MPI_Init may not be local to the eventual processor the process is bound to, thus hurting performance 3. while we support non-MPI applications, our current method will not bind them. This was actually one of the problems that motivated the user list discussion as the user was testing with "hostname" and failing to see it bound. While we can argue that we are Open -MPI-, there is a little issue here with "user surprise". 4. from the user mailing list, it was clear that some users at least expected the process to be bound from start of execution. Eugene did note on one such discussion that he has seen similar behavior (i.e., not bound until MPI_Init) on other MPI implementations, but I think fairly questioned whether or not this was the right way to go. I am sure others can think of more issues - this isn't meant to be an exhaustive list. I should note that we never see these problems in our tests because they always call MPI_Init right away at the beginning of the program. I admit that many of our local applications do the same - however, many of them also do setup memory regions prior to calling MPI_Init, which does reflect Eugene's use-case. Any thoughts on this? Should we change it? If so, who wants to be involved in the re-design? I'm pretty sure it would require some modification of the paffinity framework, plus some minor mods to the odls framework and (since you cannot bind a process other than yourself) addition of a new small "proxy" script that would bind-then-exec each process started by the orted (Eugene posted a candidate on the user list, though we will have to deal with some system-specific issues in it). Ralph
Re: [OMPI devel] Revise paffinity method?
On Wed, 6 May 2009, Ralph Castain wrote: Any thoughts on this? Should we change it? Yes, we should change this (IMHO) :). If so, who wants to be involved in the re-design? I'm pretty sure it would require some modification of the paffinity framework, plus some minor mods to the odls framework and (since you cannot bind a process other than yourself) addition of a new small "proxy" script that would bind-then-exec each process started by the orted (Eugene posted a candidate on the user list, though we will have to deal with some system-specific issues in it). I can't contribute a whole lot of time, but I'd be happy to lurk, offer advice, and write some small bits of code. But I definitely can't lead. Fist offering of opinion from me. I think we can avoid the "proxy" script by doing the binding after the fork but before the exec. This will definitely require minor changes to the odls and probably a bunch of changes to the paffinity framework. This will make things slightly less fragile than a script would, and yet get us what we want. Brian
[OMPI devel] mpi very slow to start on Mac OS X
I am using the system openmpi (1.2.3) on Mac OS X 10.5.6, gcc 4.0.1. I
believe these are all the standard tools.
If I mpirun as:
mpirun -n 2 ./a.out
Using the test code from http://www.boost.org/doc/libs/1_39_0/doc/html/mpi/getting_started.html#mpi.config
(note that at the moment I have not installed, and am not using,
boost)
I get the following output:
~/temp$ time mpirun -n 2 ./mpi-test
[christopher-jeffersons-macbook.local:06005] [0,1,0]-[0,0,0]
mca_oob_tcp_peer_complete_connect: connection failed: Connection
refused (61) - retrying
[christopher-jeffersons-macbook.local:06006] [0,1,1]-[0,0,0]
mca_oob_tcp_peer_complete_connect: connection failed: Connection
refused (61) - retrying
[christopher-jeffersons-macbook.local:06005] [0,1,0]-[0,0,0]
mca_oob_tcp_peer_complete_connect: connection failed: Operation timed
out (60) - retrying
[christopher-jeffersons-macbook.local:06006] [0,1,1]-[0,0,0]
mca_oob_tcp_peer_complete_connect: connection failed: Operation timed
out (60) - retrying
[christopher-jeffersons-macbook.local:06005] [0,1,0]-[0,1,1]
mca_oob_tcp_peer_complete_connect: connection failed: Connection
refused (61) - retrying
[christopher-jeffersons-macbook.local:06005] [0,1,0]-[0,1,1]
mca_oob_tcp_peer_complete_connect: connection failed: Operation timed
out (60) - retrying
Rank 0 OK!
Rank 1 OK!
real3m52.070s
user0m0.019s
sys 0m0.025s
Note in particular the time.
Running just ' int main(void) { }' takes about 1 minute 16 seconds.
Any suggestions very welcome!
Re: [OMPI devel] mpi very slow to start on Mac OS X
Apologies, soon after I sent my mail I managed to solve my own problem. For possible future reference, the problem was that my disconnected ethernet port had somehow decided to take up residence on the same subnet as my wireless. The relevant parts of ifconfig are: lo0: flags=8049 mtu 16384 inet 127.0.0.1 netmask 0xff00 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 gif0: flags=8010 mtu 1280 stf0: flags=0<> mtu 1280 en0: flags=8822 mtu 1500 inet 192.168.2.1 netmask 0xff00 broadcast 192.168.2.255 ether 00:25:00:a9:68:44 media: autoselect status: inactive supported media: none autoselect 10baseT/UTP 10baseT/ UTP 10baseT/UTP 10baseT/UTP 100baseTX 100baseTX duplex> 100baseTX 100baseTX loopback> 1000baseT 1000baseT 1000baseT en1: flags=8863 mtu 1500 inet6 fe80::225:ff:fe3e:2474%en1 prefixlen 64 scopeid 0x5 inet 192.168.2.2 netmask 0xff00 broadcast 192.168.2.255 ether 00:25:00:3e:24:74 media: autoselect status: active supported media: autoselect By disabling en0 with 'sudo ifconfig en0 down', open-mpi performs as expected. Chris
Re: [OMPI devel] MPI Message Communication over TCP/IP
Thanks! Would you mind if I posted this on www.open-mpi.org? On Apr 25, 2009, at 10:05 AM, Timothy Hayes wrote: I uploaded it to http://www.hotshare.net/file/131218-829472246c.html I'm not sure if it's any good or even if it's 100% accurate; but if someone gets any use out of it, that would be good. Tim 2009/4/17 Jeff Squyres On Apr 16, 2009, at 11:38 AM, Timothy Hayes wrote: From what I understand MPI_Send will hit 3 separate layers of code before reaching the socket file descriptors you've found. The PML (Point to Point Messaging Layer) is a layer that bridges the MPI semantics from the underlying point to point communications. The standard PML implementation is called 'ob1' which is what indirectly calls the code you found. MPI_Send should go through pml_isend() or pml_send() which will request a BTL (Byte Transfer Layer) modules from the BML (BTL Management Layer) and invoke the BTL's btl_prepare_src() or btl_alloc() functions before calling the btl_send(). It becomes clearer when you step through it all with a debugger though ;-) If you're interested, I've recently implemented a BTL component of my own and am now writing up a report on the process. It will be ready next week, so if you think it might be useful, just let me know. Ooohh... that would be positively yummy! We can even host/link to that on www.open-mpi.org. :-) -- Jeff Squyres Cisco Systems ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres Cisco Systems
Re: [OMPI devel] MPI Message Communication over TCP/IP
Hi Jeff, As far as I am concerned, I have no problems. It would be much better that everyone knows about the OpenMPI tcp/ip internals. Regards Pranav On Wed, May 6, 2009 at 4:47 PM, Jeff Squyres wrote: > Thanks! > > Would you mind if I posted this on www.open-mpi.org? > > > > On Apr 25, 2009, at 10:05 AM, Timothy Hayes wrote: > > I uploaded it to http://www.hotshare.net/file/131218-829472246c.html >> >> I'm not sure if it's any good or even if it's 100% accurate; but if >> someone gets any use out of it, that would be good. >> >> Tim >> 2009/4/17 Jeff Squyres >> On Apr 16, 2009, at 11:38 AM, Timothy Hayes wrote: >> >> From what I understand MPI_Send will hit 3 separate layers of code before >> reaching the socket file descriptors you've found. The PML (Point to Point >> Messaging Layer) is a layer that bridges the MPI semantics from the >> underlying point to point communications. The standard PML implementation is >> called 'ob1' which is what indirectly calls the code you found. MPI_Send >> should go through pml_isend() or pml_send() which will request a BTL (Byte >> Transfer Layer) modules from the BML (BTL Management Layer) and invoke the >> BTL's btl_prepare_src() or btl_alloc() functions before calling the >> btl_send(). It becomes clearer when you step through it all with a debugger >> though ;-) >> >> If you're interested, I've recently implemented a BTL component of my own >> and am now writing up a report on the process. It will be ready next week, >> so if you think it might be useful, just let me know. >> >> Ooohh... that would be positively yummy! We can even host/link to that on >> www.open-mpi.org. :-) >> >> -- >> Jeff Squyres >> Cisco Systems >> >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> > > > -- > Jeff Squyres > Cisco Systems > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel > -- Pranav Jadhav Graduate Student Computer Science Department Stony Brook University NY-11790.
Re: [OMPI devel] MPI Message Communication over TCP/IP
You wouldn't happen to have a bibtext reference, would you? On May 6, 2009, at 5:04 PM, pranav jadhav wrote: Hi Jeff, As far as I am concerned, I have no problems. It would be much better that everyone knows about the OpenMPI tcp/ip internals. Regards Pranav On Wed, May 6, 2009 at 4:47 PM, Jeff Squyres wrote: Thanks! Would you mind if I posted this on www.open-mpi.org? On Apr 25, 2009, at 10:05 AM, Timothy Hayes wrote: I uploaded it to http://www.hotshare.net/file/131218-829472246c.html I'm not sure if it's any good or even if it's 100% accurate; but if someone gets any use out of it, that would be good. Tim 2009/4/17 Jeff Squyres On Apr 16, 2009, at 11:38 AM, Timothy Hayes wrote: >From what I understand MPI_Send will hit 3 separate layers of code before reaching the socket file descriptors you've found. The PML (Point to Point Messaging Layer) is a layer that bridges the MPI semantics from the underlying point to point communications. The standard PML implementation is called 'ob1' which is what indirectly calls the code you found. MPI_Send should go through pml_isend() or pml_send() which will request a BTL (Byte Transfer Layer) modules from the BML (BTL Management Layer) and invoke the BTL's btl_prepare_src() or btl_alloc() functions before calling the btl_send(). It becomes clearer when you step through it all with a debugger though ;-) If you're interested, I've recently implemented a BTL component of my own and am now writing up a report on the process. It will be ready next week, so if you think it might be useful, just let me know. Ooohh... that would be positively yummy! We can even host/link to that on www.open-mpi.org. :-) -- Jeff Squyres Cisco Systems ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres Cisco Systems ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Pranav Jadhav Graduate Student Computer Science Department Stony Brook University NY-11790. ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres Cisco Systems
Re: [OMPI devel] MPI Message Communication over TCP/IP
Are you talking about my document? If so, that's no problem at all. If there are any mistakes in my facts just let me know and I'll change them. Tim 2009/5/6 Jeff Squyres > Thanks! > > Would you mind if I posted this on www.open-mpi.org? > > > > On Apr 25, 2009, at 10:05 AM, Timothy Hayes wrote: > > I uploaded it to http://www.hotshare.net/file/131218-829472246c.html >> >> I'm not sure if it's any good or even if it's 100% accurate; but if >> someone gets any use out of it, that would be good. >> >> Tim >> 2009/4/17 Jeff Squyres >> On Apr 16, 2009, at 11:38 AM, Timothy Hayes wrote: >> >> From what I understand MPI_Send will hit 3 separate layers of code before >> reaching the socket file descriptors you've found. The PML (Point to Point >> Messaging Layer) is a layer that bridges the MPI semantics from the >> underlying point to point communications. The standard PML implementation is >> called 'ob1' which is what indirectly calls the code you found. MPI_Send >> should go through pml_isend() or pml_send() which will request a BTL (Byte >> Transfer Layer) modules from the BML (BTL Management Layer) and invoke the >> BTL's btl_prepare_src() or btl_alloc() functions before calling the >> btl_send(). It becomes clearer when you step through it all with a debugger >> though ;-) >> >> If you're interested, I've recently implemented a BTL component of my own >> and am now writing up a report on the process. It will be ready next week, >> so if you think it might be useful, just let me know. >> >> Ooohh... that would be positively yummy! We can even host/link to that on >> www.open-mpi.org. :-) >> >> -- >> Jeff Squyres >> Cisco Systems >> >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> > > > -- > Jeff Squyres > Cisco Systems > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel > >
Re: [OMPI devel] MPI Message Communication over TCP/IP
Yes, I'm talking about your document -- sorry -- I just replied to the wrong guy! :-) If you have a bibtex reference for it, that would be most helpful. On May 6, 2009, at 5:20 PM, Timothy Hayes wrote: Are you talking about my document? If so, that's no problem at all. If there are any mistakes in my facts just let me know and I'll change them. Tim 2009/5/6 Jeff Squyres Thanks! Would you mind if I posted this on www.open-mpi.org? On Apr 25, 2009, at 10:05 AM, Timothy Hayes wrote: I uploaded it to http://www.hotshare.net/file/131218-829472246c.html I'm not sure if it's any good or even if it's 100% accurate; but if someone gets any use out of it, that would be good. Tim 2009/4/17 Jeff Squyres On Apr 16, 2009, at 11:38 AM, Timothy Hayes wrote: From what I understand MPI_Send will hit 3 separate layers of code before reaching the socket file descriptors you've found. The PML (Point to Point Messaging Layer) is a layer that bridges the MPI semantics from the underlying point to point communications. The standard PML implementation is called 'ob1' which is what indirectly calls the code you found. MPI_Send should go through pml_isend() or pml_send() which will request a BTL (Byte Transfer Layer) modules from the BML (BTL Management Layer) and invoke the BTL's btl_prepare_src() or btl_alloc() functions before calling the btl_send(). It becomes clearer when you step through it all with a debugger though ;-) If you're interested, I've recently implemented a BTL component of my own and am now writing up a report on the process. It will be ready next week, so if you think it might be useful, just let me know. Ooohh... that would be positively yummy! We can even host/link to that on www.open-mpi.org. :-) -- Jeff Squyres Cisco Systems ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres Cisco Systems ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres Cisco Systems
Re: [OMPI devel] MPI Message Communication over TCP/IP
Would this be okay?
@misc{hayes09_ugrad_xenmpi,
author = {Timothy Hayes},
title = {An Effcient Open MPI Transport System for Virtual Worker Nodes},
howpublished = {Trinity College Dublin Final Year Project},
year = {2009},
note = "Undergraduate Dissertation",
}
It's really just a report for a project I did in college but it wouldn't be
completely incorrect to call it a 'dissertation'. In any case, thanks very
much for putting it up :-)
Tim
2009/5/6 Jeff Squyres
> Yes, I'm talking about your document -- sorry -- I just replied to the
> wrong guy! :-)
>
> If you have a bibtex reference for it, that would be most helpful.
>
>
>
> On May 6, 2009, at 5:20 PM, Timothy Hayes wrote:
>
> Are you talking about my document? If so, that's no problem at all. If
>> there are any mistakes in my facts just let me know and I'll change them.
>>
>> Tim
>>
>> 2009/5/6 Jeff Squyres
>> Thanks!
>>
>> Would you mind if I posted this on www.open-mpi.org?
>>
>>
>>
>> On Apr 25, 2009, at 10:05 AM, Timothy Hayes wrote:
>>
>> I uploaded it to http://www.hotshare.net/file/131218-829472246c.html
>>
>> I'm not sure if it's any good or even if it's 100% accurate; but if
>> someone gets any use out of it, that would be good.
>>
>> Tim
>> 2009/4/17 Jeff Squyres
>> On Apr 16, 2009, at 11:38 AM, Timothy Hayes wrote:
>>
>> From what I understand MPI_Send will hit 3 separate layers of code before
>> reaching the socket file descriptors you've found. The PML (Point to Point
>> Messaging Layer) is a layer that bridges the MPI semantics from the
>> underlying point to point communications. The standard PML implementation is
>> called 'ob1' which is what indirectly calls the code you found. MPI_Send
>> should go through pml_isend() or pml_send() which will request a BTL (Byte
>> Transfer Layer) modules from the BML (BTL Management Layer) and invoke the
>> BTL's btl_prepare_src() or btl_alloc() functions before calling the
>> btl_send(). It becomes clearer when you step through it all with a debugger
>> though ;-)
>>
>> If you're interested, I've recently implemented a BTL component of my own
>> and am now writing up a report on the process. It will be ready next week,
>> so if you think it might be useful, just let me know.
>>
>> Ooohh... that would be positively yummy! We can even host/link to that on
>> www.open-mpi.org. :-)
>>
>> --
>> Jeff Squyres
>> Cisco Systems
>>
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
>> --
>> Jeff Squyres
>> Cisco Systems
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>
>
> --
> Jeff Squyres
> Cisco Systems
>
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
