Introduction
============

The libiwarp aims at facilitating the devolopment of iWARP/RDMA-based
applications, particularly for people who are not familiar with all the details
of the OFED verbs. The library is implemented as a set of wrappers around
libibverbs and librdmacm. As libiwarp does not replace libibverbs, it does not
impose any restrictions with regard to the original functionality. Whenever
there is a feature missing in libiwarp, you can always use the interface
provided by libibverbs.

In particular, the library significantly simplifies the connection management
and event handling. CM events as well as event from the Completion Channel
are dispatched and handled.

Within libiwarp, all the RDMA programming objects like PD, QP, CQ, etc. are
grouped together into a connection context (struct iw_ctx_conn). Think of
it as an abstraction for your iWARP connection.


Installing libiwarp
===================

Befor you can use libiwarp, make sure to install the OFED environment. In
particular, the libibverbs as well as the librdmacm have to be in place.

Check the path to librdmacm.so in iwarp.h before compiling!

After compilation, copy libiwarp.so to your default library location
(e.g., /usr/local/lib/) and iwarp.h as well as iwarp_objs.h to your default
include location (e.g., /usr/local/include).


Using libiwarp
==============

To build an application based on the libiwarp, you have to include the iwarp.h
header file into your source code by adding the following line:

#include <iwarp.h>

Also, you have to make sure to link your binary against the libiwarp.so. This
is done by adding '-liwarp' to your Makefile.


Connection Establishment (Initiator)
------------------------------------

/* 1: allocate iWARP connection context */
iw_ctx_alloc(remote_addr, remote_port, qp_capacity, shared_pd, iw_ctx);

/* 2: connect to responder */
iw_connect(priv_data, priv_data_len, iw_ctx);

Connection Establishment (Responder)
------------------------------------

/* 1: prepare to listen for inbound connection requests */
iw_open(listen_addr, listen_port, backlog, listen_iw_ctx);

/* 2: listen for inbound connection requests */
iw_listen(listen_iw_ctx, cli_iw_ctx);

/* 3: accept the connection request */
iw_accept(priv_data, priv_data_len, cli_iw_ctx);


Buffers (Memory Region) Management
----------------------------------

We distinguish between local (iw_lmr) and remote buffers (iw_rmr).

/* allocate new buffer and register it as MR */
iw_lmr_create(length, access_flags, iw_lmr_out, iw_ctx);

/* register existing buffer as MR */
iw_lmr_register(addr, length, access_flags, iw_lmr_out, iw_ctx);

/* deregister (and optionally free) buffer */
iw_lmr_destroy(iw_lmr, free_buf_flag);

/* advertise a buffer */
iw_post_send_adv(iw_lmr, iw_ctx);

/* pre-post receive for inbound buffer advertisement (non-blocking) */
iw_post_recv_adv(iw_ctx);

/* wait for inbound buffer advertisement (blocking) */
iw_wait_recv_adv(iw_rmr_out, iw_ctx);


Issuing RDMA Operations
-----------------------

These calls simplify RDMA-style data exchanges and are either based
on a single MR or on a scatter/gather list (SGL).

/* Receive */
post_recv_sgl(iw_sgl_dst, iw_ctx);
post_recv_lmr(iw_lmr_dst, offset, length, iw_ctx);

/* Send */
post_send_sgl(iw_sgl_src, send_flags, iw_ctx);
post_send_lmr(iw_lmr_src, offset, length, send_flags, iw_ctx);

/* RDMA Write */
post_write_sgl(iw_sgl_src, iw_rmr_dst, offset_dst, send_flags, iw_ctx);
post_write_lmr(iw_lmr_src, offset_src, iw_rmr_dst, offset_dst, length,
               send_flags, iw_ctx);

/* RDMA Read */
post_read_sgl(iw_sgl_dst, iw_rmr_src, offset_src, send_flags, iw_ctx);
post_read_lmr(iw_lmr_dst, offset_dst, iw_rmr_src, offset_src, length,
              send_flags, iw_ctx);

All of the above operations immediately return after having posted the
work request. If the signaled flag was used in send_flags, the following
function can be used to wait for the completion.

/* await work completion */
await_completions(cq_type, num_wcs, wc_list_out, iw_ctx);

Where the cq_type is either IW_SCQ (Send Completion Queue) or IW_RCQ
(Receive Completion Queue).


Connection Teardown
-------------------

/* active connection teardown */
iw_disconnect(iw_ctx);

/* passive connection teardown */
iw_await_disconnect(iw_ctx);


ToDo
====

- setting the path to librdmacm in iwarp.h is ugly
- use same build system as libibverbs
- the library has only been tested on x86_64 (resolve x86 warnings)
