RE: IB atomic operations

[Bharath Ramesh]

> -----Original Message-----
> From: frank zago [mailto:fz...@systemfabricworks.com]
> 
> On 02/25/2011 10:04 AM, Bharath Ramesh wrote:
> >> -----Original Message-----
> >> From: frank zago [mailto:fz...@systemfabricworks.com]
> >>
> >> On 02/25/2011 01:30 AM, Bharath Ramesh wrote:
> >>> Hi,
> >>>   I am trying to use IB atomic operations, specifically fetch and
> >> add.
> >>> I tried a small example but I am unable to get it to work. The WC
> >> status is
> >>> IBV_WC_REM_INV_REQ_ERR. My sample code is as follows:
> >>
> >> Did you register your memory with IBV_ACCESS_REMOTE_ATOMIC ?
> >
> > I have registered both the remote and local data with
> > IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_ATOMIC |
> IBV_ACCESS_REMOTE_READ |
> > IBV_ACCESS_REMOTE_WRITE
> >
> 
> Do you create your QPs with RDMA CM or do you transition them by hand ?
> In the later case there is a couple flags to set.

I transition them by hand, my QP creation is done as follows:

memset (&init_attr, 0, sizeof (struct ibv_qp_init_attr));
init_attr.recv_cq = ib_cq;
init_attr.send_cq = ib_cq;
init_attr.cap.max_recv_wr = IB_MAX_WR;
init_attr.cap.max_send_wr = IB_MAX_WR;
init_attr.cap.max_recv_sge = IB_MAX_SGE;
init_attr.cap.max_send_sge = IB_MAX_SGE;
init_attr.qp_type = IBV_QPT_RC;
qp = ibv_create_qp (ib_pd, &init_attr);
if (qp == NULL) {
        // QP creation error.
}

I transition the QP as follows:

// Transition QP to INIT state.
attr.qp_state = IBV_QPS_INIT;
attr.pkey_index = 0;
attr.port_num = IB_PORT;
attr.qp_access_flags = IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_ATOMIC |
IBV_ACCESS_REMOTE_READ
        | IBV_ACCESS_REMOTE_WRITE;
if (ibv_modify_qp (qp, &attr, IBV_QP_STATE | IBV_QP_PKEY_INDEX |
                IBV_QP_PORT | IBV_QP_ACCESS_FLAGS) != 0) {
        // QP modify error.
}

memset (&attr, 0, sizeof (struct ibv_qp_attr));
attr.qp_state = IBV_QPS_RTR;
attr.path_mtu = ib_mtu;
attr.dest_qp_num = qpn;
attr.rq_psn = 0;
attr.max_dest_rd_atomic = IB_MAX_RD_ATOMIC;
attr.min_rnr_timer = IB_RNR_TIMER;
attr.ah_attr.is_global = 0;
attr.ah_attr.dlid = lid;
attr.ah_attr.sl = 0;
attr.ah_attr.src_path_bits = 0;
attr.ah_attr.port_num = IB_PORT;
if (ibv_modify_qp (qp, &attr, IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU |
IBV_QP_DEST_QPN |
                IBV_QP_RQ_PSN | IBV_QP_MAX_DEST_RD_ATOMIC |
IBV_QP_MIN_RNR_TIMER) != 0) {
        // QP modify error.
}

// Transition the QP to RTS.
attr.timeout = IB_TIMEOUT;
attr.retry_cnt = IB_RETRY_CNT;
attr.rnr_retry = IB_RNR_RETRY;
attr.qp_state = IBV_QPS_RTS;
attr.sq_psn = 0;
attr.max_rd_atomic = IB_MAX_RD_ATOMIC;
if (ibv_modify_qp (qp, &attr, IBV_QP_STATE | IBV_QP_SQ_PSN |
IBV_QP_MAX_QP_RD_ATOMIC |
                IBV_QP_RETRY_CNT | IBV_QP_RNR_RETRY | IBV_QP_TIMEOUT) != 0)
{
        // QP modify error.
}

Regards,

Bharath

smime.p7s
Description: S/MIME cryptographic signature