Hi all,
I have a MPI_Isend/MPI_Recv problem in a multi-thread program. In the program: The first machine has one thread does some computation and call MPI_Isend to send buffer to the second machine, and another thread is always trying to MPI_Recv data from the second machine. And the first thread will MPI_Wait its last MPI_Isend to complete before call MPI_Isend again. The second machine does the exact same thing. Then I got the result that : The first machine: Thread 0 : MPI_Isend data to the second machine successfully. but blocked in MPI_Wait because last MPI_Isend did not complete. Thread 1 : try to MPI_Recv data from the second machine, but no data and it blocked. The second machine: Thread 0 : MPI_Isend data to the first machine successfully. but blocked in MPI_Wait because last MPI_Isend did not complete. Thread 1 : try to MPI_Recv data from the first machine, but no data and it blocked. Does anyone have any ideas? I appreciated it very much, because I have tracked the problem for two days but no progress. Eason Zhao At 2012-12-05 01:00:02,users-requ...@open-mpi.org wrote: >Send users mailing list submissions to > us...@open-mpi.org > >To subscribe or unsubscribe via the World Wide Web, visit > http://www.open-mpi.org/mailman/listinfo.cgi/users >or, via email, send a message with subject or body 'help' to > users-requ...@open-mpi.org > >You can reach the person managing the list at > users-ow...@open-mpi.org > >When replying, please edit your Subject line so it is more specific >than "Re: Contents of users digest..." > > >Today's Topics: > > 1. Re: 0xc000007b error exit on 64-bit Windows 7 > (dam...@0x544745.com) > 2. Re: 0xc000007b error exit on 64-bit Windows 7 (Damien) > 3. Windows support for OpenMPI (Damien) > 4. Re: Windows support for OpenMPI (John R. Cary) > 5. Re: Windows support for OpenMPI (Damien) > 6. Romio and OpenMPI builds (Brock Palen) > 7. Re: Romio and OpenMPI builds (Eric Chamberland) > 8. Re: CentOS 6.3 & OpenMPI 1.6.3 (Joseph Farran) > 9. Re: BLCR + Qlogic infiniband (William Hay) > > >---------------------------------------------------------------------- > >Message: 1 >Date: Mon, 3 Dec 2012 10:03:41 -0700 >From: dam...@0x544745.com >Subject: Re: [OMPI users] 0xc000007b error exit on 64-bit Windows 7 >To: Open MPI Users <us...@open-mpi.org> >Message-ID: <233b9754-b3a9-47d7-9dd1-3e6680ba4318.maildroid@localhost> >Content-Type: text/plain; charset="utf-8" > >Ignore what I posted, Hristo is right. On my phone screen the C looked like >a 0. Sorry. > >Damien > >Sent from my android device. > > > >-----Original Message----- >From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> >To: Open MPI Users <us...@open-mpi.org> >Cc: "us...@open-mpi.org" <us...@open-mpi.org> >Sent: Mon, 03 Dec 2012 9:01 AM >Subject: Re: [OMPI users] 0xc000007b error exit on 64-bit Windows 7 > >I'm afraid we've lost the open MPI community windows developer. So I don't >know if you'll get a good answer to this question. > >Sorry! :( > >Sent from my phone. No type good. > >On Dec 3, 2012, at 6:40 AM, "Ilias Miroslav" <miroslav.il...@umb.sk> wrote: > >> Dear experts, >> >> I just installed >> http://www.open-mpi.org/software/ompi/v1.6/downloads/OpenMPI_v1.6.1-1_win64.exe >> on our Intel i7 64-bit Windows 7 system. >> >> >> When I try to run some executable, I am getting error "Application Error >> The application was unable to start correctly (0xc000007b)..." >> >> Any help please ? The "C:\Program Files (x86)\OpenMPI_v1.6.1-x64\bin" string >> is in my %Path% variable. >> >> Yours, Miro >> >> PS: On 32-bit Windows 7 the 32-bit OpenMPI application works fine. >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >_______________________________________________ >users mailing list >us...@open-mpi.org >http://www.open-mpi.org/mailman/listinfo.cgi/users >-------------- next part -------------- >HTML attachment scrubbed and removed > >------------------------------ > >Message: 2 >Date: Mon, 03 Dec 2012 10:19:39 -0700 >From: Damien <dam...@khubla.com> >Subject: Re: [OMPI users] 0xc000007b error exit on 64-bit Windows 7 >To: Open MPI Users <us...@open-mpi.org> >Message-ID: <50bcdf2b.4040...@khubla.com> >Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" > >I just tried it on a clean VM, the 64-bit OpenMPI installer does install >to Program Files (x86). That's not the end of the world, but you have >to watch your paths. > >Miroslav, when you ran the installer did you say yes to adding OpenMPI >to the system path? If you installed both 32 and 64-bit binaries, and >added both to the system path, it will typically just append the paths. >So if you installed 32-bit first, then 64-bit, whenever you run >something it will load the 32-bit OpenMPI runtime first, even running >64-bit, which will cause that bad image error. I think that's why your >32-bit run works and 64-bit doesn't. > >I suggest uninstalling both 32 and 64-bit OpenMPIs, make sure they're >removed from the path, then reinstall them *without* putting them into >the system path, and try again from there. You'll have to set your >paths manually, but you'll be running with the right binaries each time. > >Damien > >On 03/12/2012 9:55 AM, Iliev, Hristo wrote: >> Hi, >> >> 0xC000007B is STATUS_INVALID_IMAGE_FORMAT. It mostly means that some of the >> dynamic link libraries (DLLs) that the executable is linked against are of >> different "bitness", e.g. 32-bit. It could be a packaging error in Open MPI, >> or it could be messed up installation. You could use the Dependency Walker >> tool to examine the list of DLLs that the executable depends upon and see >> which one is the culprit. Dependency Walker is available here: >> >> http://www.dependencywalker.com/ >> >> Which brings me to the question: why the win64 version of Open MPI is >> installed in "Program Files (x86)", where 32-bit things go?! >> >> Hope that helps. >> >> Kind regards, >> Hristo >> >> -- >> Hristo Iliev, Ph.D. -- High Performance Computing >> RWTH Aachen University, Center for Computing and Communication >> Rechen- und Kommunikationszentrum der RWTH Aachen >> Seffenter Weg 23, D 52074 Aachen (Germany) >> >> >>> -----Original Message----- >>> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] >>> On Behalf Of Ilias Miroslav >>> Sent: Monday, December 03, 2012 3:40 PM >>> To: us...@open-mpi.org >>> Subject: [OMPI users] 0xc000007b error exit on 64-bit Windows 7 >>> >>> Dear experts, >>> >>> I just installed http://www.open- >>> mpi.org/software/ompi/v1.6/downloads/OpenMPI_v1.6.1-1_win64.exe on >>> our Intel i7 64-bit Windows 7 system. >>> >>> >>> When I try to run some executable, I am getting error "Application Error >> The >>> application was unable to start correctly (0xc000007b)..." >>> >>> Any help please ? The "C:\Program Files (x86)\OpenMPI_v1.6.1-x64\bin" >>> string is in my %Path% variable. >>> >>> Yours, Miro >>> >>> PS: On 32-bit Windows 7 the 32-bit OpenMPI application works fine. >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >-------------- next part -------------- >HTML attachment scrubbed and removed > >------------------------------ > >Message: 3 >Date: Mon, 03 Dec 2012 10:32:57 -0700 >From: Damien <dam...@khubla.com> >Subject: [OMPI users] Windows support for OpenMPI >To: Open MPI Users <us...@open-mpi.org> >Message-ID: <50bce249.3020...@khubla.com> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > >All, > >I completely missed the message about Shiqing departing as the OpenMPI >Windows maintainer. I'll try and keep Windows builds going for 1.6 at >least, I have 2011 and 2013 Intel licenses and VS2008 and 2012, but not >2010. I see that the 1.6.3 code base already doesn't build on Windows >in VS2012 :-(. > >While I can try and keep builds going, I don't have access to a Windows >cluster right now, and I'm flat out on two other projects. I can test on >my workstation, but that will only go so far. Longer-term, there needs >to be a decision made on whether Windows gets to be a first-class >citizen in OpenMPI or not. Jeff's already told me that 1.7 is lagging >behind on Windows. It would be a shame to have all the work Shiqing put >in gradually decay because it can't be supported enough. If there's any >Microsoft/HPC/Azure folks observing this list, or any other vendors who >run on Windows with OpenMPI, maybe we can see what can be done if you're >interested. > >Damien > > >------------------------------ > >Message: 4 >Date: Mon, 03 Dec 2012 11:59:13 -0700 >From: "John R. Cary" <c...@txcorp.com> >Subject: Re: [OMPI users] Windows support for OpenMPI >To: us...@open-mpi.org >Message-ID: <50bcf681.40...@txcorp.com> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > >Dear OpenMPI community, > >This email is about whether a commercial version of OpenMPI for Windows >could be successful. I hesitated before sending this, but upon asking >some others (notably Jeff) on this list, it seemed appropriate. > >We at Tech-X have been asking whether a commercial/freemium support >model for a Windows >version of OpenMPI would work. We are currently working on this for some >other products, notably PETSc, which is discussed at >http://www.txcorp.com/home/cosml. > >We see some downsides - in particular, with Microsoft's HPC Pack, >Windows users >have free access to an MPI solution. This has to be balanced by some >particular advantages of OpenMPI such that there would be a group of >users who would pay for it for anyone to make this work. > >We would be very interested in hearing from folks on this list who either >(1) help define the competitive advantage of having OpenMPI on Windows or >(2) would be interested in a commercial solution, were it available. > >Naturally, any solution should benefit the OpenMPI community as well to >be a success. > >I would be glad to hear from folks on list or off. > >Thx....John Cary > > > > > > > >On 12/3/2012 10:32 AM, Damien wrote: >> All, >> >> I completely missed the message about Shiqing departing as the OpenMPI >> Windows maintainer. I'll try and keep Windows builds going for 1.6 at >> least, I have 2011 and 2013 Intel licenses and VS2008 and 2012, but >> not 2010. I see that the 1.6.3 code base already doesn't build on >> Windows in VS2012 :-(. >> >> While I can try and keep builds going, I don't have access to a >> Windows cluster right now, and I'm flat out on two other projects. I >> can test on my workstation, but that will only go so far. Longer-term, >> there needs to be a decision made on whether Windows gets to be a >> first-class citizen in OpenMPI or not. Jeff's already told me that >> 1.7 is lagging behind on Windows. It would be a shame to have all the >> work Shiqing put in gradually decay because it can't be supported >> enough. If there's any Microsoft/HPC/Azure folks observing this list, >> or any other vendors who run on Windows with OpenMPI, maybe we can see >> what can be done if you're interested. >> >> Damien >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > > > >------------------------------ > >Message: 5 >Date: Mon, 03 Dec 2012 12:42:23 -0700 >From: Damien <dam...@khubla.com> >Subject: Re: [OMPI users] Windows support for OpenMPI >To: Open MPI Users <us...@open-mpi.org> >Message-ID: <50bd009f.4060...@khubla.com> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > >This is a good start. Stepping up a level and without wanting to start >a bunfight with other MPI implementations, what are the advantages of >OpenMPI over those other various MPI implementations, irrespective of >platform? There must be some advantages, or OpenMPI wouldn't exist. Do >those advantages apply on Windows and would they justify ongoing Windows >support? > >Damien > > >On 03/12/2012 11:59 AM, John R. Cary wrote: >> Dear OpenMPI community, >> >> This email is about whether a commercial version of OpenMPI for Windows >> could be successful. I hesitated before sending this, but upon asking >> some others (notably Jeff) on this list, it seemed appropriate. >> >> We at Tech-X have been asking whether a commercial/freemium support >> model for a Windows >> version of OpenMPI would work. We are currently working on this for some >> other products, notably PETSc, which is discussed at >> http://www.txcorp.com/home/cosml. >> >> We see some downsides - in particular, with Microsoft's HPC Pack, >> Windows users >> have free access to an MPI solution. This has to be balanced by some >> particular advantages of OpenMPI such that there would be a group of >> users who would pay for it for anyone to make this work. >> >> We would be very interested in hearing from folks on this list who either >> (1) help define the competitive advantage of having OpenMPI on Windows or >> (2) would be interested in a commercial solution, were it available. >> >> Naturally, any solution should benefit the OpenMPI community as well to >> be a success. >> >> I would be glad to hear from folks on list or off. >> >> Thx....John Cary >> >> >> >> >> >> >> >> On 12/3/2012 10:32 AM, Damien wrote: >>> All, >>> >>> I completely missed the message about Shiqing departing as the >>> OpenMPI Windows maintainer. I'll try and keep Windows builds going >>> for 1.6 at least, I have 2011 and 2013 Intel licenses and VS2008 and >>> 2012, but not 2010. I see that the 1.6.3 code base already doesn't >>> build on Windows in VS2012 :-(. >>> >>> While I can try and keep builds going, I don't have access to a >>> Windows cluster right now, and I'm flat out on two other projects. I >>> can test on my workstation, but that will only go so far. >>> Longer-term, there needs to be a decision made on whether Windows >>> gets to be a first-class citizen in OpenMPI or not. Jeff's already >>> told me that 1.7 is lagging behind on Windows. It would be a shame to >>> have all the work Shiqing put in gradually decay because it can't be >>> supported enough. If there's any Microsoft/HPC/Azure folks observing >>> this list, or any other vendors who run on Windows with OpenMPI, >>> maybe we can see what can be done if you're interested. >>> >>> Damien >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > >------------------------------ > >Message: 6 >Date: Mon, 3 Dec 2012 17:37:36 -0500 >From: Brock Palen <bro...@umich.edu> >Subject: [OMPI users] Romio and OpenMPI builds >To: Open MPI Users <us...@open-mpi.org> >Message-ID: <8a9d5fa0-5444-4427-8790-1a5ca03bc...@umich.edu> >Content-Type: text/plain; charset=us-ascii > >I was trying to use hints with ROMIO and lustre prompted by another post on >this list. > >I have a simple MPI-IO code and I cannot using the notes I find set the lustre >striping using the config file and setting ROMIO_HINTS. > >Question: > >How can I check which ADIO drivers ROMIO in OPenMPI was built with when I >built it? >Can I make ROMIO go into 'verbose' mode and have it print what it is setting >all its values to? > >Thanks! > >Brock Palen >www.umich.edu/~brockp >CAEN Advanced Computing >bro...@umich.edu >(734)936-1985 > > > > > > >------------------------------ > >Message: 7 >Date: Mon, 03 Dec 2012 19:12:27 -0500 >From: Eric Chamberland <eric.chamberl...@giref.ulaval.ca> >Subject: Re: [OMPI users] Romio and OpenMPI builds >To: us...@open-mpi.org >Message-ID: <50bd3feb.2060...@giref.ulaval.ca> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > >Le 12/03/2012 05:37 PM, Brock Palen a ?crit : >> I was trying to use hints with ROMIO and lustre prompted by another post on >> this list. >> >> I have a simple MPI-IO code and I cannot using the notes I find set the >> lustre striping using the config file and setting ROMIO_HINTS. >> >> Question: >> >> How can I check which ADIO drivers ROMIO in OPenMPI was built with when I >> built it? >> Can I make ROMIO go into 'verbose' mode and have it print what it is setting >> all its values to? >Try "ompi_info -a" and check for lustre in the output: > >ompi_info -a | grep -i romio >... > MCA io: information "io_romio_user_configure_params" >(value: <--with-file-system=testfs+ufs+nfs+lustre>, data source: default >value) > User-specified command line parameters passed >to ROMIO's configure script > MCA io: information >"io_romio_complete_configure_params" (value: ><--with-file-system=testfs+ufs+nfs+lustre CFLAGS='-DNDEBUG -O3 -xHOST >-Wall -finline-functions -fno-strict-aliasing -restrict -pthread' >CPPFLAGS=' >-I/clumeq/src/Open-MPI/1.6.3/intel/openmpi-1.6.3/opal/mca/hwloc/hwloc132/hwloc/include > >-I/clumeq/src/Open-MPI/1.6.3/intel/Build/opal/mca/hwloc/hwloc132/hwloc/include > >-I/usr/include/infiniband -I/usr/include/infiniband' FFLAGS='' LDFLAGS=' >' --enable-shared --enable-static >--with-file-system=testfs+ufs+nfs+lustre >--prefix=/software/MPI/openmpi/1.6.3_intel --with-mpi=open_mpi >--disable-aio>, data source: default value) > Complete set of command line parameters >passed to ROMIO's configure script > >Eric > > > >------------------------------ > >Message: 8 >Date: Mon, 03 Dec 2012 23:20:49 -0800 >From: Joseph Farran <jfar...@uci.edu> >Subject: Re: [OMPI users] CentOS 6.3 & OpenMPI 1.6.3 >To: Mike Dubman <mike.o...@gmail.com> >Cc: Open MPI Users <us...@open-mpi.org> >Message-ID: <50bda451.8060...@uci.edu> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > >Hi Mike. > >Removed the old mxm, downloaded and installed: > >/tmp/mxm/v1.1/per-ofed/1.5.4.1/mxm-1.1.3a5e745-1.x86_64-rhel6u3.rpm > >I am suing OFED 1.5.4.1 and it still fails at the same spot: > >make[2]: Entering directory `/data/apps/sources/openmpi-1.6.3/ompi/mca/mtl/mxm' > CC mtl_mxm.lo > CC mtl_mxm_cancel.lo > CC mtl_mxm_component.lo > CC mtl_mxm_endpoint.lo > CC mtl_mxm_probe.lo > CC mtl_mxm_recv.lo > CC mtl_mxm_send.lo > CCLD mca_mtl_mxm.la >/bin/grep: /usr/local/mofed-inst/1.5.4.1/lib/librdmacm.la: No such file or >directory >/bin/sed: can't read /usr/local/mofed-inst/1.5.4.1/lib/librdmacm.la: No such >file or directory >libtool: link: `/usr/local/mofed-inst/1.5.4.1/lib/librdmacm.la' is not a valid >libtool archive >make[2]: *** [mca_mtl_mxm.la] Error 1 >make[2]: Leaving directory `/data/apps/sources/openmpi-1.6.3/ompi/mca/mtl/mxm' >make[1]: *** [all-recursive] Error 1 >make[1]: Leaving directory `/data/apps/sources/openmpi-1.6.3/ompi' >make: *** [all-recursive] Error 1 > > >On 12/2/2012 10:18 PM, Mike Dubman wrote: >> ohh.. you have MOFED 1.5.4.1, thought it was 1.5.3-3.1.0 >> will provide you a link to mxm package compiled with this MOFED version >> (thanks to no ABI in OFED). >> >> On Sun, Dec 2, 2012 at 10:04 PM, Joseph Farran <jfar...@uci.edu >> <mailto:jfar...@uci.edu>> wrote: >> >> 1.5.4.1 >> >> > > > >------------------------------ > >Message: 9 >Date: Tue, 4 Dec 2012 10:43:05 +0000 >From: William Hay <w....@ucl.ac.uk> >Subject: Re: [OMPI users] BLCR + Qlogic infiniband >To: us...@open-mpi.org >Message-ID: > <caoeauj71xf06ow6oo3tw+_sxm1zqjxnydqil2wg9nqd7zrk...@mail.gmail.com> >Content-Type: text/plain; charset="iso-8859-1" > >On 28 November 2012 11:14, William Hay <w....@ucl.ac.uk> wrote: > >> I'm trying to build openmpi with support for BLCR plus qlogic infiniband >> (plus grid engine). Everything seems to compile OK and checkpoints are >> taken but whenever I try to restore a checkpoint I get the following error: >> - do_mmap(<file>, 00002aaab18c7000, 0000000000001000, ...) failed: >> ffffffffffffffea >> - mmap failed: /dev/ipath >> - thaw_threads returned error, aborting. -22 >> - thaw_threads returned error, aborting. -22 >> Restart failed: Invalid argument >> >> This occurs whether I specify psm or openib as the btl. >> >> This looks like the sort of thing I would expect to be handled by the blcr >> supporting code in openmpi. So I guess I have a couple ofquestions. >> 1)Are Infiniband and BLCR support in openmpi compatible? >> 2)Are there any special tricks necessary to get them working together. >> >> A third question occurred to me that may be relevant. How do I verify >that my openmpi install has blcr support built in? I would have thought >this would mean that either mpiexec or binaries built with mpicc would have >libcr linked in. However running ldd doesn't report this in either case. > I'm setting LD_PRELOAD to point to it but I would have thought openmpi >would need to register a callback with blcr and it would be easier to do >this if the library were linked in rather than trying to detect whether it >has been LD_PRELOADed. I'm building with the following options: >./configure --prefix=/home/ccaawih/openmpi-blcr --with-openib --without-psm >--with-blcr=/usr --with-blcr-libdir=/usr/lib64 --with-ft=cr >--enable-ft-thread --enable-mpi-threads --with-sge >-------------- next part -------------- >HTML attachment scrubbed and removed > >------------------------------ > >_______________________________________________ >users mailing list >us...@open-mpi.org >http://www.open-mpi.org/mailman/listinfo.cgi/users > >End of users Digest, Vol 2428, Issue 1 >**************************************