Re: [MTT users] funclet and quoting
On Tue, Jun/12/2007 04:53:21PM, Jeff Squyres wrote: > On Jun 12, 2007, at 10:50 AM, Ethan Mallove wrote: > > >>> eval $btls = MTT::Functions::if(MTT::Functions::regexp > >>> (MTT::Functions::shell("hostname"), "v440-2|v20z-2"), > >>> MTT::Functions::enumerate("self,sm,tcp"), > >>> MTT::Functions::if("MTT::Functions::regexp > >>> ("MTT::Functions::shell("hostname"), "v440-6"), > >>> MTT::Functions::enumerate("udapl"), > >>> MTT::Functions::enumerate("sm", "tcp", > >>> "sm,tcp", "udapl", "sm,udapl") > >>> ) > >>> ) > >> > >> Ah -- I had to think about this a bit before I figured it out/ > >> remembered. I think we won't get the expansion effect if you do it > >> this way. > > > > No expansion effect? As long as the perl code evaluates to > > an array, couldn't it to work? E.g., > > We'll need to revamp all the current funclets to accept > arrays and return [potentially] arrays of arrays. What > if, as a counterproposal, we have a () funclet? > Then you could do the following: > > > > btls = ("if ($hostname =~ /foo/) { \ > return ("self,sm,tcp"); \ > } elsif ($hostname =~ /bar/) { \ > return ("udapl");\ > } else { \ > return ("sm", "tcp", "sm,tcp", "udapl", "sm,udapl"); \ > }") > > > > (or something like that) > I like that. Is that in anticipation of having (), (), (), (), (), ... ? To reduce some clutter, maybe a param like this would be fun, e.g., default_ini_evaluator = perl So MTT will assume you mean () if there's no leading '&'. Other legal options being ... default_ini_evaluator = python default_ini_evaluator = sh default_ini_evaluator = tcl default_ini_evaluator = ... -Ethan > > > > -- > Jeff Squyres > Cisco Systems > > ___ > mtt-users mailing list > mtt-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
Re: [MTT users] funclet and quoting
On Jun 12, 2007, at 10:50 AM, Ethan Mallove wrote: eval $btls = MTT::Functions::if(MTT::Functions::regexp (MTT::Functions::shell("hostname"), "v440-2|v20z-2"), MTT::Functions::enumerate("self,sm,tcp"), MTT::Functions::if("MTT::Functions::regexp ("MTT::Functions::shell("hostname"), "v440-6"), MTT::Functions::enumerate("udapl"), MTT::Functions::enumerate("sm", "tcp", "sm,tcp", "udapl", "sm,udapl") ) ) Ah -- I had to think about this a bit before I figured it out/ remembered. I think we won't get the expansion effect if you do it this way. No expansion effect? As long as the perl code evaluates to an array, couldn't it to work? E.g., We'll need to revamp all the current funclets to accept arrays and return [potentially] arrays of arrays. What if, as a counterproposal, we have a () funclet? Then you could do the following: btls = ("if ($hostname =~ /foo/) { \ return ("self,sm,tcp"); \ } elsif ($hostname =~ /bar/) { \ return ("udapl");\ } else { \ return ("sm", "tcp", "sm,tcp", "udapl", "sm,udapl"); \ }") (or something like that) -- Jeff Squyres Cisco Systems
Re: [OMPI users] TCP connection errors
On 12/06/07, George Bosilcawrote: Jonathan Underwood wrote: > Presumably switching the two interfaces on the frontend (eth0<->eth1) > would also solve this problem? > If you have root privileges this seems to be a another good approach. I don't, but will explain the issue to sysadmin. Thanks again. Jonathan.
Re: [OMPI users] TCP connection errors
Jonathan Underwood wrote: Presumably switching the two interfaces on the frontend (eth0<->eth1) would also solve this problem? If you have root privileges this seems to be a another good approach. george.
Re: [OMPI users] TCP connection errors
On 12/06/07, George Bosilcawrote: Jonathan, It will be difficult to make it works in this configuration. The problem is that on the head node the network interface that have to be used is eth1 while on the compute nodes is eth0. Therefore, the tcp_if_include will not help ... Now, if you only start processes on the compute nodes you will not have to face this problem. Right now, I think this is the safest approach. We have a patch for this kind of problems, but it's not yet in the trunk. I let you know as soon as we commit it and then you will have to use the unstable version until the patch make its way into a stable version. OK, thanks very much for letting me know. Presumably switching the two interfaces on the frontend (eth0<->eth1) would also solve this problem? Cheers, Jonathan
Re: [OMPI users] f90 support not built with gfortran?
Thanks guys! Setting F77=gfortran did the trick. Jeff F. Pummill Senior Linux Cluster Administrator University of Arkansas Fayetteville, Arkansas 72701 (479) 575 - 4590 http://hpc.uark.edu "A supercomputer is a device for turning compute-bound problems into I/O-bound problems." -Seymour Cray Jeff Squyres wrote: On Jun 12, 2007, at 5:56 AM, Terry Frankcombe wrote: I downloaded and configured v1.2.2 this morning on an Opteron cluster using the following configure directives... ./configure --prefix=/share/apps CC=gcc CXX=g++ F77=g77 FC=gfortran CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 FCFLAGS=-m64 What does config.log say? (Look for 'Fortran 90'.) config.log should be your first port of call when trying to debug build problems in any "configure"-d project. Exactly. OMPI's configure probably determined that it should not build the F90 bindings, so it didn't (hence, mpif90 is non- functional). If I had to guess, it's because you specified both g77 and gfortran. When using gfortran, you should probably use it for both F77 and FC. That will likely fix your problem. If it doesn't, please see this web page for more details on getting help: http://www.open-mpi.org/community/help/ Consider this a compile-time problem (because OMPI decided not to build the F90 bindings) and send all the information listed. Thanks!
Re: [OMPI users] rdma over tcp?
On Jun 12, 2007, at 10:52 AM, Brian Barrett wrote: On Jun 11, 2007, at 9:27 AM, Brock Palen wrote: With openmpi-1.2.0 i ran a: ompi_info --param btl tcp and i see reference to: MCA btl: parameter "btl_tcp_min_rdma_size" (current value: "131072") MCA btl: parameter "btl_tcp_max_rdma_size" (current value: "2147483647") Can TCP support RDMA? I thought you needed fancy hardware to get such support? Light on this subject is highly appreciated. Also if a user using ethernet, is trying to up the limit for 'greedy' messages that the btl_tcp_eager_limit? Is there a problem increasing its size? We will test it with his app of-course, but was wondering if there was a 'gotcha' I was going to walk into. Hi Brock - The "rdma" part of the TCP transport isn't real RDMA, but just which protocol is used by the upper layers to transfer data. In the send/ receive protocol, receives always involve a copy. Using the RDMA protocol (which is pretty simple to fake with a send/receive interface), the TCP BTL header includes the remote address and no copy is involved. So no, we haven't discovered some hidden interface in TCP -- just trying to have as few special cases for various interconnects as possible :). Yes, increasing the btl_tcp_eager_limit is how you increase the "greedy" message size. It's currently 64k, and the only problem with increasing it is memory usage. With TCP, even if you need to send a 4 byte message, 64K will be used on both the sender and receiver during transfer and these fragments are free-listed, so you can very quickly cause Open MPI to use lots and lots of memory if the eager limit is too big. If you start seeing segfauls, bus errors, and failed mallocs, you might have bumped the eager limit too high and run yourself out of memory... Brian Thanks thats exactly what i want to know! (and kind of assumed) Brock ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [MTT users] funclet and quoting
On Tue, Jun/12/2007 08:07:20AM, Jeff Squyres wrote: > On Jun 11, 2007, at 6:02 PM, Ethan Mallove wrote: > > > btls = ("("("hostname")", "v440-2|v20z-2")", \ > >"("self,sm,tcp")", \ > >"("("("hostname")", "v440-6")", \ > >"("udapl")", \ > >"("sm", "tcp", "sm,tcp", "udapl", > > "sm,udapl")"" \ > >) \ > >) > > Yowza! We really need to make this quoting situation better > somehow. :-( > > Don't you have an extra ) in there at the end? > > > How come EvaluateString() doesn't translate this: > > > > btls = ("("("hostname")", "v440-2|v20z-2")", \ > >"("self,sm,tcp")", \ > >"("("("hostname")", "v440-6")", \ > >"("udapl")", \ > >"("sm", "tcp", "sm,tcp", "udapl", > > "sm,udapl")"" \ > >) \ > >) > > > > Into one big eval block like this: > > > > eval $btls = MTT::Functions::if(MTT::Functions::regexp > > (MTT::Functions::shell("hostname"), "v440-2|v20z-2"), > > MTT::Functions::enumerate("self,sm,tcp"), > > MTT::Functions::if("MTT::Functions::regexp > > ("MTT::Functions::shell("hostname"), "v440-6"), > > MTT::Functions::enumerate("udapl"), > > MTT::Functions::enumerate("sm", "tcp", > > "sm,tcp", "udapl", "sm,udapl") > > ) > > ) > > > > So we could put the burden of the nasty recursive > > parenthesis/quote parsing on perl, rather than MTT. That > > would seem to naturally make the INI funclet syntax more > > Perl-like. > > Ah -- I had to think about this a bit before I figured it out/ > remembered. I think we won't get the expansion effect if you do it > this way. No expansion effect? As long as the perl code evaluates to an array, couldn't it to work? E.g., exec = mpirun --mca btl @btls@ ... # Returns an array btls = ("sm", "tcp") So EvaluateString() would do only this: * Substitute all () with MTT::Functions::funcs() * Substitute all $vars with same-INI-section params * Backslash escape @ symbols so they show up as strings in the eval'd result * Perform an "eval" on the final string * Iterate over the result of the above step's string, replacing "@multiplier" strings with the corresponding element of the @multiplier array > Although it might solve some other issues (e.g., https:// > svn.open-mpi.org/trac/mtt/ticket/184)...? > The mess atop this email would simplify to: btls = if ($hostname =~ /foo/) {\ return ("self,sm,tcp"); \ } elsif ($hostname =~ /bar/) { \ return ("udapl");\ } else { \ return ("sm", "tcp", "sm,tcp", "udapl", "sm,udapl"); \ } -Ethan > -- > Jeff Squyres > Cisco Systems > > ___ > mtt-users mailing list > mtt-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
Re: [OMPI users] mpirun hanging when processes started on head node
Hi Sean > [Sean] I'm working through the strace output to follow the progression on the > head node. It looks like mpirun consults '/bpfs/self' and determines that the > request is to be run on the local machine so it fork/execs 'orted' which then > runs 'hostname'. 'mpirun' didn't consult '/bpfs' or utilize 'rsh' after the > determination to run on the local machine was made. When the 'hostname' > command completes, 'orted' receives the SIGCHLD signal, performs some work and > then both 'mpirun' and 'orted' go into what appears to be a poll() waiting for > events. This is the core of the problem - I confess to being blown away that mpirun is fork/exec'ing that local orted. I will have to go through the code and try to figure that one out - we have never seen that behavior. There should be no way at all for that to happen. The problem is that, if the code fork/exec's that local orted, then the bproc code components have no idea it exists. Hence, the system doesn't know it should shutdown when complete because (a) there is still a lingering orted out there, but (b) the dominant component (bproc, in this case) has no earthly idea where it is or how to tell it to go away. FWIW, this problem will vanish in 1.3 due to a major change in the way we handle orteds. However, the idea that we could fork/exec an orted under bproc is something we definitely will have to fix. Sorry for the problem. I'll have to see if there is a fix for 1.2 - it may require too much code change and have to wait for 1.3. I'll advise as soon as I figure this one out. Ralph > > > Hope that helps at least a little. > > [Sean] I appreciate the help. We are running processes on the head node > because the head node is the only node which can access external resources > (storage devices). > > > Ralph > > > > > > On 6/11/07 1:04 PM, "Kelley, Sean"wrote: > >> I forgot to add that we are using 'bproc'. Launching processes on the compute >> nodes using bproc works well, I'm not sure if bproc is involved when >> processes are launched on the local node. >> >> Sean >> >> >> From: users-boun...@open-mpi.org on behalf of Kelley, Sean >> Sent: Mon 6/11/2007 2:07 PM >> To: us...@open-mpi.org >> Subject: [OMPI users] mpirun hanging when processes started on head node >> >> Hi, >> We are running the OFED 1.2rc4 distribution containing openmpi-1.2.2 on >> a RedhatEL4U4 system with Scyld Clusterware 4.1. The hardware configuration >> consists of a DELL 2950 as the headnode and 3 DELL 1950 blades as compute >> nodes using Cisco TopSpin Infiband HCAs and switches for the interconnect. >> >> When we use 'mpirun' from the OFED/Open MPI distribution to start >> processes on the compute nodes, everything works correctly. However, when we >> try to start processes on the head node, the processes appear to run >> correctly but 'mpirun' hangs and does not terminate until killed. The >> attached 'run1.tgz' file contains detailed information from running the >> following command: >> >> mpirun --hostfile hostfile1 --np 1 --byslot --debug-daemons -d hostname >> >> where 'hostfile1' contains the following: >> >> -1 slots=2 max_slots=2 >> >> The 'run.log' is the output of the above line. The 'strace.out.0' is the >> result of 'strace -f' on the mpirun process (and the 'hostname' child process >> since mpirun simply forks the local processes). The child process (pid 23415 >> in this case) runs to completion and exits successfully. The parent process >> (mpirun) doesn't appear to recognize that the child has completed and hangs >> until killed (with a ^c). >> >> Additionally, when we run a set of processes which span the headnode and the >> compute nodes, the processes on the head node complete successfully, but the >> processes on the compute nodes do not appear to start. mpirun again appears >> to hang. >> >> Do I have a configuration error or is there a problem that I have >> encountered? Thank you in advance for your assistance or suggestions >> >> Sean >> >> -- >> Sean M. Kelley >> sean.kel...@solers.com >> >> >> >> >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [MTT users] funclet and quoting
On Jun 11, 2007, at 6:02 PM, Ethan Mallove wrote: btls = ("("("hostname")", "v440-2|v20z-2")", \ "("self,sm,tcp")", \ "("("("hostname")", "v440-6")", \ "("udapl")", \ "("sm", "tcp", "sm,tcp", "udapl", "sm,udapl")"" \ ) \ ) Yowza! We really need to make this quoting situation better somehow. :-( Don't you have an extra ) in there at the end? How come EvaluateString() doesn't translate this: btls = ("("("hostname")", "v440-2|v20z-2")", \ "("self,sm,tcp")", \ "("("("hostname")", "v440-6")", \ "("udapl")", \ "("sm", "tcp", "sm,tcp", "udapl", "sm,udapl")"" \ ) \ ) Into one big eval block like this: eval $btls = MTT::Functions::if(MTT::Functions::regexp (MTT::Functions::shell("hostname"), "v440-2|v20z-2"), MTT::Functions::enumerate("self,sm,tcp"), MTT::Functions::if("MTT::Functions::regexp ("MTT::Functions::shell("hostname"), "v440-6"), MTT::Functions::enumerate("udapl"), MTT::Functions::enumerate("sm", "tcp", "sm,tcp", "udapl", "sm,udapl") ) ) So we could put the burden of the nasty recursive parenthesis/quote parsing on perl, rather than MTT. That would seem to naturally make the INI funclet syntax more Perl-like. Ah -- I had to think about this a bit before I figured it out/ remembered. I think we won't get the expansion effect if you do it this way. Although it might solve some other issues (e.g., https:// svn.open-mpi.org/trac/mtt/ticket/184)...? -- Jeff Squyres Cisco Systems
Re: [OMPI users] f90 support not built with gfortran?
On Jun 12, 2007, at 5:56 AM, Terry Frankcombe wrote: I downloaded and configured v1.2.2 this morning on an Opteron cluster using the following configure directives... ./configure --prefix=/share/apps CC=gcc CXX=g++ F77=g77 FC=gfortran CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 FCFLAGS=-m64 What does config.log say? (Look for 'Fortran 90'.) config.log should be your first port of call when trying to debug build problems in any "configure"-d project. Exactly. OMPI's configure probably determined that it should not build the F90 bindings, so it didn't (hence, mpif90 is non- functional). If I had to guess, it's because you specified both g77 and gfortran. When using gfortran, you should probably use it for both F77 and FC. That will likely fix your problem. If it doesn't, please see this web page for more details on getting help: http://www.open-mpi.org/community/help/ Consider this a compile-time problem (because OMPI decided not to build the F90 bindings) and send all the information listed. Thanks! -- Jeff Squyres Cisco Systems
Re: [OMPI users] f90 support not built with gfortran?
On Mon, 2007-06-11 at 12:10 -0500, Jeff Pummill wrote: > Greetings all, > > I downloaded and configured v1.2.2 this morning on an Opteron cluster > using the following configure directives... > > ./configure --prefix=/share/apps CC=gcc CXX=g++ F77=g77 FC=gfortran > CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 FCFLAGS=-m64 What does config.log say? (Look for 'Fortran 90'.) config.log should be your first port of call when trying to debug build problems in any "configure"-d project.
Re: [OMPI users] TCP connection errors
Jonathan, It will be difficult to make it works in this configuration. The problem is that on the head node the network interface that have to be used is eth1 while on the compute nodes is eth0. Therefore, the tcp_if_include will not help ... Now, if you only start processes on the compute nodes you will not have to face this problem. Right now, I think this is the safest approach. We have a patch for this kind of problems, but it's not yet in the trunk. I let you know as soon as we commit it and then you will have to use the unstable version until the patch make its way into a stable version. Thanks, george. On Mon, 11 Jun 2007, Jonathan Underwood wrote: Hi, I am seeing problems with a small linux cluster when running OpenMPI jobs. The error message I get is: [frontend][0,1,0][btl_tcp_endpoint.c:572:mca_btl_tcp_endpoint_complete_connect] connect() failed with errno=110 Following the FAQ, I looked to see what this error code corresponds to: $ perl -e 'die$!=110' Connection timed out at -e line 1. This error message occurs the first time one of the compute nodes, which are on a private network, attempts to send data to the frontend (from where the job was started with mpirun). In actual fact, it seems that the error occurs the first time a process on the frontend tries to send data to another process on the frontend. I tried to play about with things like --mca btl_tcp_if_exclude lo,eth0, but that didn't help matters. Nothing in the FAQ section on TCP and routing actually seemed to help. Any advice would be very welcome The network configurations are: a) frontend (2 network adapters, eth1 private for the cluster): $ /sbin/ifconfig eth0 Link encap:Ethernet HWaddr 00:E0:81:30:A1:CE inet addr:128.40.5.39 Bcast:128.40.5.255 Mask:255.255.255.0 inet6 addr: fe80::2e0:81ff:fe30:a1ce/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3496038 errors:0 dropped:0 overruns:0 frame:0 TX packets:2833685 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:500939570 (477.7 MiB) TX bytes:671589665 (640.4 MiB) Interrupt:193 eth1 Link encap:Ethernet HWaddr 00:E0:81:30:A1:CF inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::2e0:81ff:fe30:a1cf/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2201778 errors:0 dropped:0 overruns:0 frame:0 TX packets:2046572 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:188615778 (179.8 MiB) TX bytes:247305804 (235.8 MiB) Interrupt:201 loLink encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:1528 errors:0 dropped:0 overruns:0 frame:0 TX packets:1528 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:363101 (354.5 KiB) TX bytes:363101 (354.5 KiB) $ /sbin/route Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 192.168.1.0 * 255.255.255.0 U 0 00 eth1 128.40.5.0 * 255.255.255.0 U 0 00 eth0 default 128.40.5.2450.0.0.0 UG0 00 eth0 b) Compute nodes: $ /sbin/ifconfig eth0 Link encap:Ethernet HWaddr 00:E0:81:30:A0:72 inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::2e0:81ff:fe30:a072/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:189207 errors:0 dropped:0 overruns:0 frame:0 TX packets:203507 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:23075241 (22.0 MiB) TX bytes:17693363 (16.8 MiB) Interrupt:193 loLink encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:185 errors:0 dropped:0 overruns:0 frame:0 TX packets:185 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:12644 (12.3 KiB) TX bytes:12644 (12.3 KiB) $ /sbin/route Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 192.168.1.0 * 255.255.255.0 U 0 00 eth0 default frontend.cluste 0.0.0.0 UG0 00 eth0 TIS Jonathan ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users "We must accept finite disappointment, but we must never lose infinite hope." Martin Luther King
Re: [OMPI users] mixing MX and TCP
Well, as expected this call is not documented ... and I get to it only with some help from Loic. On the version of MX I have tested it is not required to add the 3th and 4th arguments as I don't want to set anything. The NIC is already specified through the mx_btl->mx_endpoint isn't it ? george. On Mon, 11 Jun 2007, Reese Faucette wrote: ! if( (status = mx_get_info( mx_btl->mx_endpoint, MX_LINE_SPEED, !_id, sizeof(nic_id), , sizeof(int))) != MX_SUCCESS ) { yes, a NIC ID is required for this call because a host may have multiple NICs with different linespeeds, e.g. a 2G card and a 10G card. -reese ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users "We must accept finite disappointment, but we must never lose infinite hope." Martin Luther King