On Jun 12, 2007, at 10:52 AM, Brian Barrett wrote:
On Jun 11, 2007, at 9:27 AM, Brock Palen wrote:
With openmpi-1.2.0
i ran a: ompi_info --param btl tcp
and i see reference to:
MCA btl: parameter "btl_tcp_min_rdma_size" (current value: "131072")
MCA btl: parameter "btl_tcp_max_rdma_size" (current value:
"2147483647")
Can TCP support RDMA? I thought you needed fancy hardware to get
such support? Light on this subject is highly appreciated.
Also if a user using ethernet, is trying to up the limit for
'greedy' messages that the btl_tcp_eager_limit? Is there a problem
increasing its size? We will test it with his app of-course, but
was wondering if there was a 'gotcha' I was going to walk into.
Hi Brock -
The "rdma" part of the TCP transport isn't real RDMA, but just which
protocol is used by the upper layers to transfer data. In the send/
receive protocol, receives always involve a copy. Using the RDMA
protocol (which is pretty simple to fake with a send/receive
interface), the TCP BTL header includes the remote address and no
copy is involved. So no, we haven't discovered some hidden interface
in TCP -- just trying to have as few special cases for various
interconnects as possible :).
Yes, increasing the btl_tcp_eager_limit is how you increase the
"greedy" message size. It's currently 64k, and the only problem with
increasing it is memory usage. With TCP, even if you need to send a
4 byte message, 64K will be used on both the sender and receiver
during transfer and these fragments are free-listed, so you can very
quickly cause Open MPI to use lots and lots of memory if the eager
limit is too big. If you start seeing segfauls, bus errors, and
failed mallocs, you might have bumped the eager limit too high and
run yourself out of memory...
Brian
Thanks thats exactly what i want to know! (and kind of assumed)
Brock
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
