Hi again, I changed the subject of my previous posting to reflect a new problem encountered when I changed my strategy to using SSH instead of Xgrid on two mac pros. I've set up a login-less ssh communication between the two macs (connected via direct ethernet, both running openmpi 1.2.8 on OSX 10.6.8) per the instructions on the FAQ. I can type in 'ssh computer-name.local' on either computer and connect without a password prompt. From what I can see, the ssh-agent is up and running - the following is listed in my ENV:
SSH_AUTH_SOCK=/tmp/launch-5FoCc1/Listeners SSH_AGENT_PID=61058 My host file simply lists 'localhost' and 'chrisjones2@allana-welshs-mac-pro.local'. When I run a simple hello_world test, I get what seems like a reasonable output: chris-joness-mac-pro:~ chrisjones$ mpirun -np 8 -hostfile hostfile ./test_hello Hello world from process 0 of 8 Hello world from process 1 of 8 Hello world from process 2 of 8 Hello world from process 3 of 8 Hello world from process 4 of 8 Hello world from process 7 of 8 Hello world from process 5 of 8 Hello world from process 6 of 8 I can also run hostname and get what seems to be an ok response (unless I'm wrong about this): chris-joness-mac-pro:~ chrisjones$ mpirun -np 8 -hostfile hostfile hostname allana-welshs-mac-pro.local allana-welshs-mac-pro.local allana-welshs-mac-pro.local allana-welshs-mac-pro.local quadcore.mikrob.slu.se quadcore.mikrob.slu.se quadcore.mikrob.slu.se quadcore.mikrob.slu.se However, when I run the ring_c test, it freezes: chris-joness-mac-pro:~ chrisjones$ mpirun -np 8 -hostfile hostfile ./ring_c Process 0 sending 10 to 1, tag 201 (8 processes in ring) Process 0 sent to 1 Process 0 decremented value: 9 (I noted that processors on both computers are active). ring_c was compiled separately on each computer, however both have the same version of openmpi and OSX. I've gone through the FAQ and searched the user forum, but I can't quite seems to get this problem unstuck. Many thanks for your time, Chris On Aug 5, 2011, at 6:00 PM, <users-requ...@open-mpi.org> <users-requ...@open-mpi.org> wrote: > Send users mailing list submissions to > us...@open-mpi.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://www.open-mpi.org/mailman/listinfo.cgi/users > or, via email, send a message with subject or body 'help' to > users-requ...@open-mpi.org > > You can reach the person managing the list at > users-ow...@open-mpi.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users digest..." > > > Today's Topics: > > 1. Re: OpenMPI causing WRF to crash (Jeff Squyres) > 2. Re: OpenMPI causing WRF to crash (Anthony Chan) > 3. Re: Program hangs on send when run with nodes on remote > machine (Jeff Squyres) > 4. Re: openmpi 1.2.8 on Xgrid noob issue (Jeff Squyres) > 5. Re: parallel I/O on 64-bit indexed arays (Rob Latham) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 4 Aug 2011 19:18:36 -0400 > From: Jeff Squyres <jsquy...@cisco.com> > Subject: Re: [OMPI users] OpenMPI causing WRF to crash > To: Open MPI Users <us...@open-mpi.org> > Message-ID: <3f0e661f-a74f-4e51-86c0-1f84feb07...@cisco.com> > Content-Type: text/plain; charset=windows-1252 > > Signal 15 is usually SIGTERM on Linux, meaning that some external entity > probably killed the job. > > The OMPI error message you describe is also typical for that kind of scenario > -- i.e., a process exited without calling MPI_Finalize could mean that it > called exit() or some external process killed it. > > > On Aug 3, 2011, at 7:24 AM, BasitAli Khan wrote: > >> I am trying to run a rather heavy wrf simulation with spectral nudging but >> the simulation crashes after 1.8 minutes of integration. >> The simulation has two domains with d01 = 601x601 and d02 = 721x721 and >> 51 vertical levels. I tried this simulation on two different systems but >> result was more or less same. For example >> >> On our Bluegene/P with SUSE Linux Enterprise Server 10 ppc and XLF compiler >> I tried to run wrf on 2048 shared memory nodes (1 compute node = 4 cores , >> 32 bit, 850 Mhz). For the parallel run I used mpixlc, mpixlcxx and mpixlf90. >> I got the following error message in the wrf.err file >> >> <Aug 01 19:50:21.244540> BE_MPI (ERROR): The error message in the job >> record is as follows: >> <Aug 01 19:50:21.244657> BE_MPI (ERROR): "killed with signal 15" >> >> I also tried to run the same simulation on our linux cluster (Linux Red Hat >> Enterprise 5.4m x86_64 and Intel compiler) with 8, 16 and 64 nodes (1 >> compute node=8 cores). For the parallel run I am used >> mpi/openmpi/1.4.2-intel-11. I got the following error message in the error >> log after couple of minutes of integration. >> >> "mpirun has exited due to process rank 45 with PID 19540 on >> node ci118 exiting without calling "finalize". This may >> have caused other processes in the application to be >> terminated by signals sent by mpirun (as reported here)." >> >> I tried many things but nothing seems to be working. However, if I reduce >> grid points below 200, the simulation goes fine. It appears that probably >> OpenMP has problem with large number of grid points but I have no idea how >> to fix it. I will greatly appreciate if you could suggest some solution. >> >> Best regards, >> --- >> Basit A. Khan, Ph.D. >> Postdoctoral Fellow >> Division of Physical Sciences & Engineering >> Office# 3204, Level 3, Building 1, >> King Abdullah University of Science & Technology >> 4700 King Abdullah Blvd, Box 2753, Thuwal 23955 ?6900, >> Kingdom of Saudi Arabia. >> >> Office: +966(0)2 808 0276, Mobile: +966(0)5 9538 7592 >> E-mail: basitali.k...@kaust.edu.sa >> Skype name: basit.a.khan >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > ------------------------------ > > Message: 2 > Date: Thu, 4 Aug 2011 18:59:59 -0500 (CDT) > From: Anthony Chan <c...@mcs.anl.gov> > Subject: Re: [OMPI users] OpenMPI causing WRF to crash > To: Open MPI Users <us...@open-mpi.org> > Message-ID: > <660521091.191111.1312502399225.javamail.r...@zimbra.anl.gov> > Content-Type: text/plain; charset=utf-8 > > > If you want to debug this on BGP, you could set BG_COREDUMPONERROR=1 > and look at the backtrace in the light weight core files > (you probably need to recompile everything with -g). > > A.Chan > > ----- Original Message ----- >> Hi Dmitry, >> Thanks for a prompt and fairly detailed response. I have also >> forwarded >> the email to wrf community in the hope that somebody would have some >> straight forward solution. I will try to debug the error as suggested >> by >> you if I would not have much luck from the wrf forum. >> >> Cheers, >> --- >> >> Basit A. Khan, Ph.D. >> Postdoctoral Fellow >> Division of Physical Sciences & Engineering >> Office# 3204, Level 3, Building 1, >> King Abdullah University of Science & Technology >> 4700 King Abdullah Blvd, Box 2753, Thuwal 23955 ?6900, >> Kingdom of Saudi Arabia. >> >> Office: +966(0)2 808 0276, Mobile: +966(0)5 9538 7592 >> E-mail: basitali.k...@kaust.edu.sa >> Skype name: basit.a.khan >> >> >> >> >> On 8/3/11 2:46 PM, "Dmitry N. Mikushin" <maemar...@gmail.com> wrote: >> >>> 5 apparently means one of the WRF's MPI processes has been >>> unexpectedly terminated, maybe by program decision. No matter, if it >>> is OpenMPI-specifi >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > ------------------------------ > > Message: 3 > Date: Thu, 4 Aug 2011 20:46:16 -0400 > From: Jeff Squyres <jsquy...@cisco.com> > Subject: Re: [OMPI users] Program hangs on send when run with nodes on > remote machine > To: Open MPI Users <us...@open-mpi.org> > Message-ID: <f344f301-ad7b-4e83-b0df-a6e001072...@cisco.com> > Content-Type: text/plain; charset=us-ascii > > I notice that in the worker, you have: > > eth2 Link encap:Ethernet HWaddr 00:1b:21:77:c5:d4 > inet addr:192.168.1.155 Bcast:192.168.1.255 Mask:255.255.255.0 > inet6 addr: fe80::21b:21ff:fe77:c5d4/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:9225846 errors:0 dropped:75175 overruns:0 frame:0 > TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:1336628768 (1.3 GB) TX bytes:552 (552.0 B) > > eth3 Link encap:Ethernet HWaddr 00:1b:21:77:c5:d5 > inet addr:192.168.1.156 Bcast:192.168.1.255 Mask:255.255.255.0 > inet6 addr: fe80::21b:21ff:fe77:c5d5/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:26481809 errors:0 dropped:75059 overruns:0 frame:0 > TX packets:18030236 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:70061260271 (70.0 GB) TX bytes:11844181778 (11.8 GB) > > Two different NICs are on the same subnet -- that doesn't seem like a good > idea...? I think this topic has come up on the users list before, and, IIRC, > the general consensus is "don't do that" because it's not clear as to which > NIC Linux will actually send outgoing traffic across bound for the > 192.168.1.x subnet. > > > > On Aug 4, 2011, at 1:59 PM, Keith Manville wrote: > >> I am having trouble running my MPI program on multiple nodes. I can >> run multiple processes on a single node, and I can spawn processes on >> on remote nodes, but when I call Send from a remote node, the node >> never returns, even though there is an appropriate Recv waiting. I'm >> pretty sure this is an issue with my configuration, not my code. I've >> tried some other sample programs I found and had the same problem of >> hanging on a send from one host to another. >> >> Here's an in depth description: >> >> I wrote a quick test program where each process with rank > 1 sends an >> int to the master (rank 0), and the master receives until it gets >> something from every other process. >> >> My test program works fine when I run multiple processes on a single machine. >> >> either the local node: >> >> $ ./mpirun -n 4 ./mpi-test >> Hi I'm localhost:2 >> Hi I'm localhost:1 >> localhost:1 sending 11... >> localhost:2 sending 12... >> localhost:2 sent 12 >> localhost:1 sent 11 >> Hi I'm localhost:0 >> localhost:0 received 11 from 1 >> localhost:0 received 12 from 2 >> Hi I'm localhost:3 >> localhost:3 sending 13... >> localhost:3 sent 13 >> localhost:0 received 13 from 3 >> all workers checked in! >> >> or a remote one: >> >> $ ./mpirun -np 2 -host remotehost ./mpi-test >> Hi I'm remotehost:0 >> remotehost:0 received 11 from 1 >> all workers checked in! >> Hi I'm remotehost:1 >> remotehost:1 sending 11... >> remotehost:1 sent 11 >> >> But when I try to run the master locally and the worker(s) remotely >> (this is the way I am actually interested in running it), Send never >> returns and it hangs indefinitely. >> >> $ ./mpirun -np 2 -host localhost,remotehost ./mpi-test >> Hi I'm localhost:0 >> Hi I'm remotehost:1 >> remotehost:1 sending 11... >> >> Just to see if it would work, I tried spawning the master on the >> remotehost and the worker on the localhost. >> >> $ ./mpirun -np 2 -host remotehost,localhost ./mpi-test >> Hi I'm localhost:1 >> localhost:1 sending 11... >> localhost:1 sent 11 >> Hi I'm remotehost:0 >> remotehost:0 received 0 from 1 >> all workers checked in! >> >> It doesn't hang on Send, but the wrong value is received. >> >> Any idea what's going on? I've attached my code, my config.log, >> ifconfig output, and ompi_info output. >> >> Thanks, >> Keith >> <mpi.tgz>_______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > ------------------------------ > > Message: 4 > Date: Thu, 4 Aug 2011 20:48:30 -0400 > From: Jeff Squyres <jsquy...@cisco.com> > Subject: Re: [OMPI users] openmpi 1.2.8 on Xgrid noob issue > To: Open MPI Users <us...@open-mpi.org> > Message-ID: <c2ea7fd0-badb-4d05-851c-c444be26f...@cisco.com> > Content-Type: text/plain; charset=us-ascii > > I'm afraid our Xgrid support has lagged, and Apple hasn't show much interest > in MPI + Xgrid support -- much less HPC. :-\ > > Have you see the FAQ items about Xgrid? > > http://www.open-mpi.org/faq/?category=osx#xgrid-howto > > > On Aug 4, 2011, at 4:16 AM, Christopher Jones wrote: > >> Hi there, >> >> I'm currently trying to set up a small xgrid between two mac pros (a single >> quadcore and a 2 duo core), where both are directly connected via an >> ethernet cable. I've set up xgrid using the password authentication (rather >> than the kerberos), and from what I can tell in the Xgrid admin tool it >> seems to be working. However, once I try a simple hello world program, I get >> this error: >> >> chris-joness-mac-pro:~ chrisjones$ mpirun -np 4 ./test_hello >> mpirun noticed that job rank 0 with PID 381 on node xgrid-node-0 exited on >> signal 15 (Terminated). >> 1 additional process aborted (not shown) >> 2011-08-04 10:02:16.329 mpirun[350:903] *** Terminating app due to uncaught >> exception 'NSInvalidArgumentException', reason: '*** >> -[NSKVONotifying_XGConnection<0x1001325a0> finalize]: called when collecting >> not enabled' >> *** Call stack at first throw: >> ( >> 0 CoreFoundation 0x00007fff814237b4 >> __exceptionPreprocess + 180 >> 1 libobjc.A.dylib 0x00007fff84fe8f03 >> objc_exception_throw + 45 >> 2 CoreFoundation 0x00007fff8143e631 >> -[NSObject(NSObject) finalize] + 129 >> 3 mca_pls_xgrid.so 0x00000001002a9ce3 >> -[PlsXGridClient dealloc] + 419 >> 4 mca_pls_xgrid.so 0x00000001002a9837 >> orte_pls_xgrid_finalize + 40 >> 5 libopen-rte.0.dylib 0x000000010002d0f9 >> orte_pls_base_close + 249 >> 6 libopen-rte.0.dylib 0x0000000100012027 >> orte_system_finalize + 119 >> 7 libopen-rte.0.dylib 0x000000010000e968 >> orte_finalize + 40 >> 8 mpirun 0x00000001000011ff orterun + >> 2042 >> 9 mpirun 0x0000000100000a03 main + 27 >> 10 mpirun 0x00000001000009e0 start + 52 >> 11 ??? 0x0000000000000004 0x0 + 4 >> ) >> terminate called after throwing an instance of 'NSException' >> [chris-joness-mac-pro:00350] *** Process received signal *** >> [chris-joness-mac-pro:00350] Signal: Abort trap (6) >> [chris-joness-mac-pro:00350] Signal code: (0) >> [chris-joness-mac-pro:00350] [ 0] 2 libSystem.B.dylib >> 0x00007fff81ca51ba _sigtramp + 26 >> [chris-joness-mac-pro:00350] [ 1] 3 ??? >> 0x00000001000cd400 0x0 + 4295808000 >> [chris-joness-mac-pro:00350] [ 2] 4 libstdc++.6.dylib >> 0x00007fff830965d2 __tcf_0 + 0 >> [chris-joness-mac-pro:00350] [ 3] 5 libobjc.A.dylib >> 0x00007fff84fecb39 _objc_terminate + 100 >> [chris-joness-mac-pro:00350] [ 4] 6 libstdc++.6.dylib >> 0x00007fff83094ae1 _ZN10__cxxabiv111__terminateEPFvvE + 11 >> [chris-joness-mac-pro:00350] [ 5] 7 libstdc++.6.dylib >> 0x00007fff83094b16 _ZN10__cxxabiv112__unexpectedEPFvvE + 0 >> [chris-joness-mac-pro:00350] [ 6] 8 libstdc++.6.dylib >> 0x00007fff83094bfc >> _ZL23__gxx_exception_cleanup19_Unwind_Reason_CodeP17_Unwind_Exception + 0 >> [chris-joness-mac-pro:00350] [ 7] 9 libobjc.A.dylib >> 0x00007fff84fe8fa2 object_getIvar + 0 >> [chris-joness-mac-pro:00350] [ 8] 10 CoreFoundation >> 0x00007fff8143e631 -[NSObject(NSObject) finalize] + 129 >> [chris-joness-mac-pro:00350] [ 9] 11 mca_pls_xgrid.so >> 0x00000001002a9ce3 -[PlsXGridClient dealloc] + 419 >> [chris-joness-mac-pro:00350] [10] 12 mca_pls_xgrid.so >> 0x00000001002a9837 orte_pls_xgrid_finalize + 40 >> [chris-joness-mac-pro:00350] [11] 13 libopen-rte.0.dylib >> 0x000000010002d0f9 orte_pls_base_close + 249 >> [chris-joness-mac-pro:00350] [12] 14 libopen-rte.0.dylib >> 0x0000000100012027 orte_system_finalize + 119 >> [chris-joness-mac-pro:00350] [13] 15 libopen-rte.0.dylib >> 0x000000010000e968 orte_finalize + 40 >> [chris-joness-mac-pro:00350] [14] 16 mpirun >> 0x00000001000011ff orterun + 2042 >> [chris-joness-mac-pro:00350] [15] 17 mpirun >> 0x0000000100000a03 main + 27 >> [chris-joness-mac-pro:00350] [16] 18 mpirun >> 0x00000001000009e0 start + 52 >> [chris-joness-mac-pro:00350] [17] 19 ??? >> 0x0000000000000004 0x0 + 4 >> [chris-joness-mac-pro:00350] *** End of error message *** >> Abort trap >> >> >> I've seen this error in a previous mailing, and it seems that the issue has >> something to do with forcing everything to use kerberos (SSO). However, I >> noticed that in the computer being used as an agent, this option is grayed >> on in the Xgrid sharing configuration (I have no idea why). I would >> therefore ask if it is absolutely necessary to use SSO to get openmpi to run >> with xgrid, or am I missing something with the password setup. Seems that >> the kerberos option is much more complicated, and I may even want to switch >> to just using openmpi with ssh. >> >> Many thanks, >> Chris >> >> >> Chris Jones >> Post-doctoral Research Assistant, >> >> Department of Microbiology >> Swedish University of Agricultural Sciences >> Uppsala, Sweden >> phone: +46 (0)18 67 3222 >> email: chris.jo...@slu.se >> >> Department of Soil and Environmental Microbiology >> National Institute for Agronomic Research >> Dijon, France >> >> >> >> >> >> >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > ------------------------------ > > Message: 5 > Date: Fri, 5 Aug 2011 08:41:58 -0500 > From: Rob Latham <r...@mcs.anl.gov> > Subject: Re: [OMPI users] parallel I/O on 64-bit indexed arays > To: Open MPI Users <us...@open-mpi.org> > Cc: Quincey Koziol <koz...@hdfgroup.org>, Fab Tillier > <ftill...@microsoft.com> > Message-ID: <20110805134158.ga28...@mcs.anl.gov> > Content-Type: text/plain; charset=us-ascii > > On Wed, Jul 27, 2011 at 06:13:05PM +0200, Troels Haugboelle wrote: >> and we get good (+GB/s) performance when writing files from large runs. >> >> Interestingly, an alternative and conceptually simpler option is to >> use MPI_FILE_WRITE_ORDERED, but the performance of that function on >> Blue-Gene/P sucks - 20 MB/s instead of GB/s. I do not know why. > > Ordered mode as implemented in ROMIO is awful. Entirely serialized. > We pass a token from process to process. Each process acquires the > token, updates the shared file pointer, does its i/o, then passes the > token to the next process. > > What we should do, and have done in test branches [1], is use MPI_SCAN > to look at the shared file pointer once, tell all the processors their > offset, then update the shared file pointer while all processes do I/O > in parallel. > > [1]: Robert Latham, Robert Ross, and Rajeev Thakur. "Implementing > MPI-IO Atomic Mode and Shared File Pointers Using MPI One-Sided > Communication". International Journal of High Performance Computing > Applications, 21(2):132-143, 2007 > > Since no one uses the shared file pointers, and even fewer people use > ordered mode, we just haven't seen the need to do so. > > Do you want to rebuild your MPI library on BlueGene? I can pretty > quickly generate and send a patch that will make ordered mode go whip > fast. > > ==rob > >> >> Troels >> >> On 6/7/11 15:04 , Jeff Squyres wrote: >>> On Jun 7, 2011, at 4:53 AM, Troels Haugboelle wrote: >>> >>>> In principle yes, but the problem is we have an unequal amount of >>>> particles on each node, so the length of each array is not guaranteed to >>>> be divisible by 2, 4 or any other number. If I have understood the >>>> definition of MPI_TYPE_CREATE_SUBARRAY correctly the offset can be 64-bit, >>>> but not the global array size, so, optimally, what I am looking for is >>>> something that has unequal size for each thread, simple vector, and with >>>> 64-bit offsets and global array size. >>> It's a bit awkward, but you can still make datatypes to give the offset >>> that you want. E.g., if you need an offset of 2B+31 bytes, you can make >>> datatype A with type contig of N=(2B/sizeof(int)) int's. Then make >>> datatype B with type struct, containing type A and 31 MPI_BYTEs. Then use >>> 1 instance of datatype B to get the offset that you want. >>> >>> You could make utility functions that, given a specific (64 bit) offset, it >>> makes an MPI datatype that matches the offset, and then frees it (and all >>> sub-datatypes). >>> >>> There is a bit of overhead in creating these datatypes, but it should be >>> dwarfed by the amount of data that you're reading/writing, right? >>> >>> It's awkward, but it should work. >>> >>>> Another possible workaround would be to identify subsections that do not >>>> pass 2B elements, make sub communicators, and then let each of them dump >>>> their elements with proper offsets. It may work. The problematic >>>> architecture is a BG/P. On other clusters doing simple I/O, letting all >>>> threads open the file, seek to their position, and then write their chunk >>>> works fine, but somehow on BG/P performance drops dramatically. My guess >>>> is that there is some file locking, or we are overwhelming the I/O nodes.. >>>> >>>>> This ticket for the MPI-3 standard is a first step in the right >>>>> direction, but won't do everything you need (this is more FYI): >>>>> >>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/265 >>>>> >>>>> See the PDF attached to the ticket; it's going up for a "first reading" >>>>> in a month. It'll hopefully be part of the MPI-3 standard by the end of >>>>> the year (Fab Tillier, CC'ed, has been the chief proponent of this ticket >>>>> for the past several months). >>>>> >>>>> Quincey Koziol from the HDF group is going to propose a follow on to this >>>>> ticket, specifically about the case you're referring to -- large counts >>>>> for file functions and datatype constructors. Quincey -- can you expand >>>>> on what you'll be proposing, perchance? >>>> Interesting, I think something along the lines of the note would be very >>>> useful and needed for large applications. >>>> >>>> Thanks a lot for the pointers and your suggestions, >>>> >>>> cheers, >>>> >>>> Troels >>> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > > > ------------------------------ > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > End of users Digest, Vol 1977, Issue 1 > **************************************
