On May 20, 2011, at 03:25 , David Büttner wrote: > Hello, > > thanks for the quick answer. I am sorry that I forgot to mention this: I did > compile OpenMPI with MPI_THREAD_MULTIPLE support and test if required == > provided after the MPI_Thread_init call. > >> I do not see any mechanism for protecting the accesses to the requests to a >> single thread? What is the thread model you're using? >> > Again I am sorry that this was not clear: In the pseudo code below I wanted > to indicate the access-protection I do by thread-id dependent calls if(0 == > thread-id) and by using the trylock(...) (using pthread-mutexes). In the code > all accesses concerning one MPI_Request (which are pthread-global-pointers in > my case) are protected and called in sequential order, i.e. MPI_Isend/recv is > returns before any thread is allowed to call the corresponding MPI_Test and > no-one can call MPI_Test any more when a thread is allowed to call MPI_Wait.
If all these are true the code is then supposed to work. We have multi-threaded software, that uses a non multi-threaded version of MPI (Open MPI in this instance), for overlapping communications and computations. Basically what we're doing is very similar to what you described above, except we ensure no two threads are accessing __any__ MPI functions in same time. And the code works perfectly. > I did this in the same manner before with other MPI implementations, but also > on the same machine with the same (untouched) OpenMPI implementation, also > using pthreads and MPI in combination, but I used > > MPI_Request req; > > instead of > > MPI_Request* req; > (and later) > req = (MPI_Request*)malloc(sizeof(MPI_Request)); > > > In my recent (problem) code, I also tried not using pointers, but got the > same problem. Also, as I described in the first mail, I tried everything > concerning the memory allocation of the MPI_Request objects. > I tried not calling malloc. This I guessed wouldn't work, but the OpenMPI > documentation says this: > > " Nonblocking calls allocate a communication request object and associate it > with the request handle the argument request). " > [http://www.open-mpi.org/doc/v1.4/man3/MPI_Isend.3.php] and > > " [...] if the communication object was created by a nonblocking send or > receive, then it is deallocated and the request handle is set to > MPI_REQUEST_NULL." [http://www.open-mpi.org/doc/v1.4/man3/MPI_Test.3.php] and > (in slightly different words) > [http://www.open-mpi.org/doc/v1.4/man3/MPI_Wait.3.php] > > So I thought that it might do some kind of optimized memory stuff internally. > > I also tried allocating req (for each used MPI_Request) once before the first > use and deallocation after the last use (which I thought was the way it was > supposed to work), but that crashes also. > > I tried replacing the pointers through global variables > > MPI_Request req; > > which didn't do the job... > > The only thing that seems to work is what I mentioned below: Allocate every > time I am going to need it in the MPI_Isend/recv, use it in MPI_Test/Wait and > after that deallocate it by hand each time. > I don't think that this is supposed to be like this since I have to do a call > to malloc and free so often (for multiple MPI_Request objects in each > iteration) that it will most likely limit performance... I would really recheck the code that make sure that multiple threads cannot complete a request in same time (MPI_Wait and MPI_Test on the same request on two threads). Second, I will declare the MPI_Request as volatile, to forbid the compiler to optimize the accesses to it. > Anyway I still have the same problem and am still unclear on what kind of > memory allocation I should be doing for the MPI_Requests. Is there anything > else (besides MPI_THREAD_MULTIPLE support, thread access control, sequential > order of MPI_Isend/recv, MPI_Test and MPI_Wait for one MPI_Request object) I > need to take care of? If not, what could I do to find the source of my > problem? If what I proposed above doesn't work, I will go for a thread correctness checker: valgrind, the intel thread checker or Thread Sanitizer (http://code.google.com/p/data-race-test/wiki/ThreadSanitizer). george. > > Thanks again for any kind of help! > > Kind regards, > David > > > >> > From an implementation perspective, your code is correct only if you >> > initialize the MPI library with MPI_THREAD_MULTIPLE and if the library >> > accepts. Otherwise, there is an assumption that the application is single >> > threaded, or that the MPI behavior is implementation dependent. Please >> > read the MPI standard regarding to MPI_Init_thread for more details. >> >> Regards, >> george. >> >> On May 19, 2011, at 02:34 , David Büttner wrote: >> >>> Hello, >>> >>> I am working on a hybrid MPI (OpenMPI 1.4.3) and Pthread code. I am using >>> MPI_Isend and MPI_Irecv for communication and MPI_Test/MPI_Wait to check if >>> it is done. I do this repeatedly in the outer loop of my code. The MPI_Test >>> is used in the inner loop to check if some function can be called which >>> depends on the received data. >>> The program regularly crashed (only when not using printf...) and after >>> debugging it I figured out the following problem: >>> >>> In MPI_Isend I have an invalid read of memory. I fixed the problem with not >>> re-using a >>> >>> MPI_Request req_s, req_r; >>> >>> but by using >>> >>> MPI_Request* req_s; >>> MPI_Request* req_r >>> >>> and re-allocating them before the MPI_Isend/recv. >>> >>> The documentation says, that in MPI_Wait and MPI_Test (if successful) the >>> request-objects are deallocated and set to MPI_REQUEST_NULL. >>> It also says, that in MPI_Isend and MPI_Irecv, it allocates the Objects and >>> associates it with the request object. >>> >>> As I understand this, this either means I can use a pointer to MPI_Request >>> which I don't have to initialize for this (it doesn't work but crashes), or >>> that I can use a MPI_Request pointer which I have initialized with >>> malloc(sizeof(MPI_REQUEST)) (or passing the address of a MPI_Request req), >>> which is set and unset in the functions. But this version crashes, too. >>> What works is using a pointer, which I allocate before the MPI_Isend/recv >>> and which I free after MPI_Wait in every iteration. In other words: It only >>> uses if I don't reuse any kind of MPI_Request. Only if I recreate one every >>> time. >>> >>> Is this, what is should be like? I believe that a reuse of the memory would >>> be a lot more efficient (less calls to malloc...). Am I missing something >>> here? Or am I doing something wrong? >>> >>> >>> Let me provide some more detailed information about my problem: >>> >>> I am running the program on a 30 node infiniband cluster. Each node has 4 >>> single core Opteron CPUs. I am running 1 MPI Rank per node and 4 threads >>> per rank (-> one thread per core). >>> I am compiling with mpicc of OpenMPI using gcc below. >>> Some pseudo-code of the program can be found at the end of this e-mail. >>> >>> I was able to reproduce the problem using different amount of nodes and >>> even using one node only. The problem does not arise when I put >>> printf-debugging information into the code. This pointed me into the >>> direction that I have some memory problem, where some write accesses some >>> memory it is not supposed to. >>> I ran the tests using valgrind with --leak-check=full and >>> --show-reachable=yes, which pointed me either to MPI_Isend or MPI_Wait >>> depending on whether I had the threads spin in a loop for MPI_Test to >>> return success or used MPI_Wait respectively. >>> >>> I would appreciate your help with this. Am I missing something important >>> here? Is there a way to re-use the request in the different iterations >>> other than I thought it should work? >>> Or is there a way to re-initialize the allocated memory before the >>> MPI_Isend/recv so that I at least don't have to call free and malloc each >>> time? >>> >>> Thank you very much for your help! >>> Kind regards, >>> David Büttner >>> >>> _____________________ >>> Pseudo-Code of program: >>> >>> MPI_Request* req_s; >>> MPI_Request* req_w; >>> OUTER-LOOP >>> if(0 == threadid) >>> { >>> req_s = malloc(sizeof(MPI_Request)); >>> req_r = malloc(sizeof(MPI_Request)); >>> MPI_Isend(..., req_s) >>> MPI_Irecv(..., req_r) >>> } >>> pthread_barrier >>> INNER-LOOP (while NOT_DONE or RET) >>> if(TRYLOCK&& NOT_DONE) >>> { >>> if(MPI_TEST(req_r)) >>> { >>> Call_Function_A; >>> NOT_DONE = 0; >>> } >>> >>> } >>> RET = Call_Function_B; >>> } >>> pthread_barrier_wait >>> if(0 == threadid) >>> { >>> MPI_WAIT(req_s) >>> MPI_WAIT(req_r) >>> free(req_s); >>> free(req_r); >>> } >>> _____________ >>> >>> >>> -- >>> David Büttner, Informatik, Technische Universität München >>> TUM I-10 - FMI 01.06.059 - Tel. 089 / 289-17676 >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> "To preserve the freedom of the human mind then and freedom of the press, >> every spirit should be ready to devote itself to martyrdom; for as long as >> we may think as we will, and speak as we think, the condition of man will >> proceed in improvement." >> -- Thomas Jefferson, 1799 >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > -- > David Büttner, Informatik, Technische Universität München > TUM I-10 - FMI 01.06.059 - Tel. 089 / 289-17676 > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users George Bosilca Research Assistant Professor Innovative Computing Laboratory Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville http://web.eecs.utk.edu/~bosilca/
