Re: [OMPI users] When is it save to free the buffer after MPI_Isend?
The snippets suggest you were storing a reference to an object on the stack. Stack variables go out of scope when the function returns. Using a reference to them out-of-scope is illegal but often fails nondeterministically. Good compilers will issue a warning about this under the right conditions (ie compiler flags). Jeff On Sat, Aug 10, 2019 at 10:59 AM carlos aguni via users < users@lists.open-mpi.org> wrote: > Hi all, > > Sorry no reply. > > I just figured out the solution. > > The problem was that I had a function that would MPI_Isend a message on > every call to it. Then I'd store its request pointer to a list. > My MPI_Isend snippet: > MPI_Request req; > MPI_Isend(blabla, &req) > task_push(&req); > > From time to time at the beginning of that function I'd call another > function that would iterate over that list MPI_Testing if that message > had completed to then free the buffers used. > The problem was that even if the flag returned from MPI_Test(&req, &flag, > &status) previouly assigned to 0 returned 1 but my guess is that C had > already deallocated it from the heap (idk much about C though..) > Snippet of my clean function: > > int flag = 0; > MPI_Test(req, &flag, &status); > if (flag) { // then free.. > ... > > My solution that worked was previously malloc it before the MPI_Isend call > like: > MPI_Request *rr = (MPI_Request *)malloc(sizeof(MPI_Request)); > MPI_Isend(blabla, rr); > task_push(rr); > > All I know is that it's working now.. > > Thanks to all. > > Regards, > C. > > On Sun, Jul 28, 2019 at 11:53 AM Jeff Squyres (jsquyres) via users < > users@lists.open-mpi.org> wrote: > >> On Jul 27, 2019, at 10:43 PM, Gilles Gouaillardet via users < >> users@lists.open-mpi.org> wrote: >> > >> > MPI_Isend() does not automatically frees the buffer after it sends the >> message. >> > (it simply cannot do it since the buffer might be pointing to a global >> > variable or to the stack). >> >> Gilles is correct: MPI_Isend does not free the buffer. I was wondering >> if you had somehow used that same buffer -- or some subset of that buffer >> -- in other non-blocking MPI API calls, and freeing it triggered Bad Things >> because MPI was still using (some of) that buffer because of other pending >> MPI requests. >> >> > Can you please extract a reproducer from your program ? >> >> Yes, please do this. >> >> > Out of curiosity, what if you insert an (useless) MPI_Wait() like this ? >> > >> > MPI_Test(req, &flag, &status); >> > if (flag){ >> >MPI_Wait(req, MPI_STATUS_IGNORE); >> >free(buffer); >> > } >> >> That should be a no-op, because "req" should have been turned into >> MPI_REQUEST_NULL if flag==true. >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> >> ___ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users >> > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users -- Jeff Hammond jeff.scie...@gmail.com http://jeffhammond.github.io/ ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] When is it save to free the buffer after MPI_Isend?
Hi all, Sorry no reply. I just figured out the solution. The problem was that I had a function that would MPI_Isend a message on every call to it. Then I'd store its request pointer to a list. My MPI_Isend snippet: MPI_Request req; MPI_Isend(blabla, &req) task_push(&req); >From time to time at the beginning of that function I'd call another function that would iterate over that list MPI_Testing if that message had completed to then free the buffers used. The problem was that even if the flag returned from MPI_Test(&req, &flag, &status) previouly assigned to 0 returned 1 but my guess is that C had already deallocated it from the heap (idk much about C though..) Snippet of my clean function: int flag = 0; MPI_Test(req, &flag, &status); if (flag) { // then free.. ... My solution that worked was previously malloc it before the MPI_Isend call like: MPI_Request *rr = (MPI_Request *)malloc(sizeof(MPI_Request)); MPI_Isend(blabla, rr); task_push(rr); All I know is that it's working now.. Thanks to all. Regards, C. On Sun, Jul 28, 2019 at 11:53 AM Jeff Squyres (jsquyres) via users < users@lists.open-mpi.org> wrote: > On Jul 27, 2019, at 10:43 PM, Gilles Gouaillardet via users < > users@lists.open-mpi.org> wrote: > > > > MPI_Isend() does not automatically frees the buffer after it sends the > message. > > (it simply cannot do it since the buffer might be pointing to a global > > variable or to the stack). > > Gilles is correct: MPI_Isend does not free the buffer. I was wondering if > you had somehow used that same buffer -- or some subset of that buffer -- > in other non-blocking MPI API calls, and freeing it triggered Bad Things > because MPI was still using (some of) that buffer because of other pending > MPI requests. > > > Can you please extract a reproducer from your program ? > > Yes, please do this. > > > Out of curiosity, what if you insert an (useless) MPI_Wait() like this ? > > > > MPI_Test(req, &flag, &status); > > if (flag){ > >MPI_Wait(req, MPI_STATUS_IGNORE); > >free(buffer); > > } > > That should be a no-op, because "req" should have been turned into > MPI_REQUEST_NULL if flag==true. > > -- > Jeff Squyres > jsquy...@cisco.com > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users > ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] When is it save to free the buffer after MPI_Isend?
On Jul 27, 2019, at 10:43 PM, Gilles Gouaillardet via users wrote: > > MPI_Isend() does not automatically frees the buffer after it sends the > message. > (it simply cannot do it since the buffer might be pointing to a global > variable or to the stack). Gilles is correct: MPI_Isend does not free the buffer. I was wondering if you had somehow used that same buffer -- or some subset of that buffer -- in other non-blocking MPI API calls, and freeing it triggered Bad Things because MPI was still using (some of) that buffer because of other pending MPI requests. > Can you please extract a reproducer from your program ? Yes, please do this. > Out of curiosity, what if you insert an (useless) MPI_Wait() like this ? > > MPI_Test(req, &flag, &status); > if (flag){ >MPI_Wait(req, MPI_STATUS_IGNORE); >free(buffer); > } That should be a no-op, because "req" should have been turned into MPI_REQUEST_NULL if flag==true. -- Jeff Squyres jsquy...@cisco.com ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] When is it save to free the buffer after MPI_Isend?
Carlos, MPI_Isend() does not automatically frees the buffer after it sends the message. (it simply cannot do it since the buffer might be pointing to a global variable or to the stack). Can you please extract a reproducer from your program ? Out of curiosity, what if you insert an (useless) MPI_Wait() like this ? MPI_Test(req, &flag, &status); if (flag){ MPI_Wait(req, MPI_STATUS_IGNORE); free(buffer); } Cheers, Gilles On Sun, Jul 28, 2019 at 5:45 AM carlos aguni via users wrote: > > Hi Jeff, > > Thank you for your reply. > > If i don't free the program completes but I'm not sure whether MPI_Isend > automatically frees the buffer after it sends the message. Does it? > > I put a long sleep at the end to check the memory used using pmap. > > The pmap command reported I'm using around 2GB which I'm guessing it isn't > freeing it. > > Is there anything I could try? > > Regards, > C. > > On Mon, Jul 22, 2019 at 10:59 AM Jeff Squyres (jsquyres) > wrote: >> >> > On Jul 21, 2019, at 11:31 AM, carlos aguni via users >> > wrote: >> > >> > MPI_Isend() >> > ... some stuff.. >> > flag = 0; >> > MPI_Test(req, &flag, &status); >> > if (flag){ >> > free(buffer); >> > } >> > >> > After the free() i'm getting errors like: >> > [[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send] >> > mca_btl_tcp_frag_send: writev error (0x2b9daf474000, 12800) >> > Bad address(1) >> > [[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send] >> > mca_btl_tcp_frag_send: writev error (0x2b9daf473ee8, 19608) >> > Bad address(1) >> > pml_ob1_sendreq.c:308 FATAL >> >> Do you get the same error if you don't free()? >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] When is it save to free the buffer after MPI_Isend?
Hi Jeff, Thank you for your reply. If i don't free the program completes but I'm not sure whether MPI_Isend automatically frees the buffer after it sends the message. Does it? I put a long sleep at the end to check the memory used using pmap. The pmap command reported I'm using around 2GB which I'm guessing it isn't freeing it. Is there anything I could try? Regards, C. On Mon, Jul 22, 2019 at 10:59 AM Jeff Squyres (jsquyres) wrote: > > On Jul 21, 2019, at 11:31 AM, carlos aguni via users < > users@lists.open-mpi.org> wrote: > > > > MPI_Isend() > > ... some stuff.. > > flag = 0; > > MPI_Test(req, &flag, &status); > > if (flag){ > > free(buffer); > > } > > > > After the free() i'm getting errors like: > > [[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send] > mca_btl_tcp_frag_send: writev error (0x2b9daf474000, 12800) > > Bad address(1) > > [[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send] > mca_btl_tcp_frag_send: writev error (0x2b9daf473ee8, 19608) > > Bad address(1) > > pml_ob1_sendreq.c:308 FATAL > > Do you get the same error if you don't free()? > > -- > Jeff Squyres > jsquy...@cisco.com > > ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] When is it save to free the buffer after MPI_Isend?
> On Jul 21, 2019, at 11:31 AM, carlos aguni via users > wrote: > > MPI_Isend() > ... some stuff.. > flag = 0; > MPI_Test(req, &flag, &status); > if (flag){ > free(buffer); > } > > After the free() i'm getting errors like: > [[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send] > mca_btl_tcp_frag_send: writev error (0x2b9daf474000, 12800) > Bad address(1) > [[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send] > mca_btl_tcp_frag_send: writev error (0x2b9daf473ee8, 19608) > Bad address(1) > pml_ob1_sendreq.c:308 FATAL Do you get the same error if you don't free()? -- Jeff Squyres jsquy...@cisco.com ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users