Re: [OMPI users] When is it save to free the buffer after MPI_Isend?
The snippets suggest you were storing a reference to an object on the
stack. Stack variables go out of scope when the function returns. Using a
reference to them out-of-scope is illegal but often fails
nondeterministically. Good compilers will issue a warning about this under
the right conditions (ie compiler flags).
Jeff
On Sat, Aug 10, 2019 at 10:59 AM carlos aguni via users <
users@lists.open-mpi.org> wrote:
> Hi all,
>
> Sorry no reply.
>
> I just figured out the solution.
>
> The problem was that I had a function that would MPI_Isend a message on
> every call to it. Then I'd store its request pointer to a list.
> My MPI_Isend snippet:
> MPI_Request req;
> MPI_Isend(blabla, )
> task_push();
>
> From time to time at the beginning of that function I'd call another
> function that would iterate over that list MPI_Testing if that message
> had completed to then free the buffers used.
> The problem was that even if the flag returned from MPI_Test(, ,
> ) previouly assigned to 0 returned 1 but my guess is that C had
> already deallocated it from the heap (idk much about C though..)
> Snippet of my clean function:
>
> int flag = 0;
> MPI_Test(req, , );
> if (flag) { // then free..
> ...
>
> My solution that worked was previously malloc it before the MPI_Isend call
> like:
> MPI_Request *rr = (MPI_Request *)malloc(sizeof(MPI_Request));
> MPI_Isend(blabla, rr);
> task_push(rr);
>
> All I know is that it's working now..
>
> Thanks to all.
>
> Regards,
> C.
>
> On Sun, Jul 28, 2019 at 11:53 AM Jeff Squyres (jsquyres) via users <
> users@lists.open-mpi.org> wrote:
>
>> On Jul 27, 2019, at 10:43 PM, Gilles Gouaillardet via users <
>> users@lists.open-mpi.org> wrote:
>> >
>> > MPI_Isend() does not automatically frees the buffer after it sends the
>> message.
>> > (it simply cannot do it since the buffer might be pointing to a global
>> > variable or to the stack).
>>
>> Gilles is correct: MPI_Isend does not free the buffer. I was wondering
>> if you had somehow used that same buffer -- or some subset of that buffer
>> -- in other non-blocking MPI API calls, and freeing it triggered Bad Things
>> because MPI was still using (some of) that buffer because of other pending
>> MPI requests.
>>
>> > Can you please extract a reproducer from your program ?
>>
>> Yes, please do this.
>>
>> > Out of curiosity, what if you insert an (useless) MPI_Wait() like this ?
>> >
>> > MPI_Test(req, , );
>> > if (flag){
>> >MPI_Wait(req, MPI_STATUS_IGNORE);
>> >free(buffer);
>> > }
>>
>> That should be a no-op, because "req" should have been turned into
>> MPI_REQUEST_NULL if flag==true.
>>
>> --
>> Jeff Squyres
>> jsquy...@cisco.com
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
--
Jeff Hammond
jeff.scie...@gmail.com
http://jeffhammond.github.io/
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] When is it save to free the buffer after MPI_Isend?
Hi all,
Sorry no reply.
I just figured out the solution.
The problem was that I had a function that would MPI_Isend a message on
every call to it. Then I'd store its request pointer to a list.
My MPI_Isend snippet:
MPI_Request req;
MPI_Isend(blabla, )
task_push();
>From time to time at the beginning of that function I'd call another
function that would iterate over that list MPI_Testing if that message
had completed to then free the buffers used.
The problem was that even if the flag returned from MPI_Test(, ,
) previouly assigned to 0 returned 1 but my guess is that C had
already deallocated it from the heap (idk much about C though..)
Snippet of my clean function:
int flag = 0;
MPI_Test(req, , );
if (flag) { // then free..
...
My solution that worked was previously malloc it before the MPI_Isend call
like:
MPI_Request *rr = (MPI_Request *)malloc(sizeof(MPI_Request));
MPI_Isend(blabla, rr);
task_push(rr);
All I know is that it's working now..
Thanks to all.
Regards,
C.
On Sun, Jul 28, 2019 at 11:53 AM Jeff Squyres (jsquyres) via users <
users@lists.open-mpi.org> wrote:
> On Jul 27, 2019, at 10:43 PM, Gilles Gouaillardet via users <
> users@lists.open-mpi.org> wrote:
> >
> > MPI_Isend() does not automatically frees the buffer after it sends the
> message.
> > (it simply cannot do it since the buffer might be pointing to a global
> > variable or to the stack).
>
> Gilles is correct: MPI_Isend does not free the buffer. I was wondering if
> you had somehow used that same buffer -- or some subset of that buffer --
> in other non-blocking MPI API calls, and freeing it triggered Bad Things
> because MPI was still using (some of) that buffer because of other pending
> MPI requests.
>
> > Can you please extract a reproducer from your program ?
>
> Yes, please do this.
>
> > Out of curiosity, what if you insert an (useless) MPI_Wait() like this ?
> >
> > MPI_Test(req, , );
> > if (flag){
> >MPI_Wait(req, MPI_STATUS_IGNORE);
> >free(buffer);
> > }
>
> That should be a no-op, because "req" should have been turned into
> MPI_REQUEST_NULL if flag==true.
>
> --
> Jeff Squyres
> jsquy...@cisco.com
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] When is it save to free the buffer after MPI_Isend?
On Jul 27, 2019, at 10:43 PM, Gilles Gouaillardet via users
wrote:
>
> MPI_Isend() does not automatically frees the buffer after it sends the
> message.
> (it simply cannot do it since the buffer might be pointing to a global
> variable or to the stack).
Gilles is correct: MPI_Isend does not free the buffer. I was wondering if you
had somehow used that same buffer -- or some subset of that buffer -- in other
non-blocking MPI API calls, and freeing it triggered Bad Things because MPI was
still using (some of) that buffer because of other pending MPI requests.
> Can you please extract a reproducer from your program ?
Yes, please do this.
> Out of curiosity, what if you insert an (useless) MPI_Wait() like this ?
>
> MPI_Test(req, , );
> if (flag){
>MPI_Wait(req, MPI_STATUS_IGNORE);
>free(buffer);
> }
That should be a no-op, because "req" should have been turned into
MPI_REQUEST_NULL if flag==true.
--
Jeff Squyres
jsquy...@cisco.com
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] When is it save to free the buffer after MPI_Isend?
Carlos,
MPI_Isend() does not automatically frees the buffer after it sends the message.
(it simply cannot do it since the buffer might be pointing to a global
variable or to the stack).
Can you please extract a reproducer from your program ?
Out of curiosity, what if you insert an (useless) MPI_Wait() like this ?
MPI_Test(req, , );
if (flag){
MPI_Wait(req, MPI_STATUS_IGNORE);
free(buffer);
}
Cheers,
Gilles
On Sun, Jul 28, 2019 at 5:45 AM carlos aguni via users
wrote:
>
> Hi Jeff,
>
> Thank you for your reply.
>
> If i don't free the program completes but I'm not sure whether MPI_Isend
> automatically frees the buffer after it sends the message. Does it?
>
> I put a long sleep at the end to check the memory used using pmap.
>
> The pmap command reported I'm using around 2GB which I'm guessing it isn't
> freeing it.
>
> Is there anything I could try?
>
> Regards,
> C.
>
> On Mon, Jul 22, 2019 at 10:59 AM Jeff Squyres (jsquyres)
> wrote:
>>
>> > On Jul 21, 2019, at 11:31 AM, carlos aguni via users
>> > wrote:
>> >
>> > MPI_Isend()
>> > ... some stuff..
>> > flag = 0;
>> > MPI_Test(req, , );
>> > if (flag){
>> > free(buffer);
>> > }
>> >
>> > After the free() i'm getting errors like:
>> > [[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send]
>> > mca_btl_tcp_frag_send: writev error (0x2b9daf474000, 12800)
>> > Bad address(1)
>> > [[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send]
>> > mca_btl_tcp_frag_send: writev error (0x2b9daf473ee8, 19608)
>> > Bad address(1)
>> > pml_ob1_sendreq.c:308 FATAL
>>
>> Do you get the same error if you don't free()?
>>
>> --
>> Jeff Squyres
>> jsquy...@cisco.com
>>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] When is it save to free the buffer after MPI_Isend?
Hi Jeff,
Thank you for your reply.
If i don't free the program completes but I'm not sure whether MPI_Isend
automatically frees the buffer after it sends the message. Does it?
I put a long sleep at the end to check the memory used using pmap.
The pmap command reported I'm using around 2GB which I'm guessing it isn't
freeing it.
Is there anything I could try?
Regards,
C.
On Mon, Jul 22, 2019 at 10:59 AM Jeff Squyres (jsquyres)
wrote:
> > On Jul 21, 2019, at 11:31 AM, carlos aguni via users <
> users@lists.open-mpi.org> wrote:
> >
> > MPI_Isend()
> > ... some stuff..
> > flag = 0;
> > MPI_Test(req, , );
> > if (flag){
> > free(buffer);
> > }
> >
> > After the free() i'm getting errors like:
> > [[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send]
> mca_btl_tcp_frag_send: writev error (0x2b9daf474000, 12800)
> > Bad address(1)
> > [[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send]
> mca_btl_tcp_frag_send: writev error (0x2b9daf473ee8, 19608)
> > Bad address(1)
> > pml_ob1_sendreq.c:308 FATAL
>
> Do you get the same error if you don't free()?
>
> --
> Jeff Squyres
> jsquy...@cisco.com
>
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] When is it save to free the buffer after MPI_Isend?
> On Jul 21, 2019, at 11:31 AM, carlos aguni via users
> wrote:
>
> MPI_Isend()
> ... some stuff..
> flag = 0;
> MPI_Test(req, , );
> if (flag){
> free(buffer);
> }
>
> After the free() i'm getting errors like:
> [[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send]
> mca_btl_tcp_frag_send: writev error (0x2b9daf474000, 12800)
> Bad address(1)
> [[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send]
> mca_btl_tcp_frag_send: writev error (0x2b9daf473ee8, 19608)
> Bad address(1)
> pml_ob1_sendreq.c:308 FATAL
Do you get the same error if you don't free()?
--
Jeff Squyres
jsquy...@cisco.com
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
[OMPI users] When is it save to free the buffer after MPI_Isend?
Hi all,
I've got a code where I MPI_Isend at a time and later I get the result from
MPI_Test flag to see whether it has completed or not.
So the code is like:
MPI_Isend()
... some stuff..
flag = 0;
MPI_Test(req, , );
if (flag){
free(buffer);
}
After the free() i'm getting errors like:
[[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send]
mca_btl_tcp_frag_send: writev error (0x2b9daf474000, 12800)
Bad address(1)
[[58327,1],0][btl_tcp_frag.c:130:mca_btl_tcp_frag_send]
mca_btl_tcp_frag_send: writev error (0x2b9daf473ee8, 19608)
Bad address(1)
pml_ob1_sendreq.c:308 FATAL
I've no idea what might be the cause.
I'm using openmpi3/3.1.4 from openhpc.
Regards,
Carlos.
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
