On 11/26/2010 1:56 PM, Walukiewicz, Miroslaw wrote:
Form the trace it looks like the __up_read() - 11% wastes most of time. It is
called from idr_read_qp when a put_uobj_read is called. if
(copy_from_user(&cmd, buf, sizeof cmd)) - 5% it is called twice from
ib_uverbs_post_send() for IMA and once in ib_uverbs_write() per each frame... and
__kmalloc/kfree - 5% is the third function that has a big meaning. It is called
twice for each frame transmitted. It is about 20% of performance loss comparing to
nes_ud_sksq path which we miss when we use a OFED path.
What I can modify is a kmalloc/kfree optimization - it is possible to make
allocation only at start and use pre-allocated buffers. I don't see any way for
optimalization of idr_read_qp usage or copy_user. In current approach we use a
shared page and a separate nes_ud_sksq handle for each created QP so there is
no need for any user space data copy or QP lookup.
As was mentioned earlier on this thread, and repeated here, the
kmalloc/kfree can be removed, as or the 2nd copy_from_user, I don't see
why the ib uverbs flow (BTW - the data path has nothing to do with the
rdma_cm, you're working with /dev/infiniband/uverbsX), can't be enhanced
e.g to support shared-page which is allocated && mmaped from uverbs to
user space and used in the same manner your implementation does. The 1st
copy_from_user should add pretty nothing and if it does, it can be
replaced with different user/kernel IPC mechanism which costs less. So
we're basically remained with the idr_read_qp, I wonder what other
people think if/how this can be optimized?
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
