+1 on Rolf's explanation. Additionally, note that you don't have to do it this way. You can implement yours in whatever style you want; this is just the style we used for the TCP BTL.
On Apr 4, 2012, at 10:18 AM, Rolf vandeVaart wrote: > Here is my explanation. The call to MCA_BTL_TCP_FRAG_ALLOC_EAGER or > MCA_BTL_TCP_FRAG_ALLOC_MAX allocate a chunk of memory that has space for both > the fragment as well as any payload. So, when we do the frag+1, we are > setting the pointer in the frag to point where the payload of the message > lives. This payload contains the PML header information and potentially the > user's buffer. So, that allocation is actually returning something like 64K > for the eager allocation and 128K for the max allocation. If you look at > btl_tcp_component.c in the function mca_btl_tcp_component_init() you can see > where the eager and max free lists are initialized. > > In the case of TCP, there are two segments. The first segment will contain > the PML header information. If the buffer being sent (or received) is > contiguous, then the rest of the space allocated is not used. Rather, the > second segment will point to the user's buffer as there is no need to first > copy it into a buffer. If the buffer being sent (or received) is > non-contiguous, then the data is first copied into the allocated space as it > needs to be packed. > > Does that make sense? > > >> -----Original Message----- >> From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org] >> On Behalf Of Alex Margolin >> Sent: Wednesday, April 04, 2012 9:23 AM >> To: Open MPI Developers >> Subject: [OMPI devel] mca_btl_tcp_alloc >> >> Hi, >> >> As I'm working out the bugs in my component I used TCP as reference and >> came across the following: >> In mca_btl_tcp_alloc (openmpi-trunk/ompi/mca/btl/tcp/btl_tcp.c:188) the >> first segment is initialized to point to "frag + 1". >> I don't get it... how/when is this location allocated? Isn't it just after >> the >> mca_btl_tcp_frag_t structure ends? >> >> Thanks, >> Alex >> >> mca_btl_base_descriptor_t* mca_btl_tcp_alloc( >> struct mca_btl_base_module_t* btl, >> struct mca_btl_base_endpoint_t* endpoint, >> uint8_t order, >> size_t size, >> uint32_t flags) >> { >> mca_btl_tcp_frag_t* frag = NULL; >> int rc; >> >> if(size <= btl->btl_eager_limit) { >> MCA_BTL_TCP_FRAG_ALLOC_EAGER(frag, rc); >> } else if (size <= btl->btl_max_send_size) { >> MCA_BTL_TCP_FRAG_ALLOC_MAX(frag, rc); >> } >> if( OPAL_UNLIKELY(NULL == frag) ) { >> return NULL; >> } >> >> frag->segments[0].seg_len = size; >> frag->segments[0].seg_addr.pval = frag+1; >> >> frag->base.des_src = frag->segments; >> frag->base.des_src_cnt = 1; >> frag->base.des_dst = NULL; >> frag->base.des_dst_cnt = 0; >> frag->base.des_flags = flags; >> frag->base.order = MCA_BTL_NO_ORDER; >> frag->btl = (mca_btl_tcp_module_t*)btl; >> return (mca_btl_base_descriptor_t*)frag; } >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel > ----------------------------------------------------------------------------------- > This email message is for the sole use of the intended recipient(s) and may > contain > confidential information. Any unauthorized review, use, disclosure or > distribution > is prohibited. If you are not the intended recipient, please contact the > sender by > reply email and destroy all copies of the original message. > ----------------------------------------------------------------------------------- > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
