[openib-general] How to get NFS with RDMA on OFED-1.1?
I know this might end up as an RTFM but before reading the wrong manual I'd rather ask We are currently using OFED-1.1 on SLES-10 doing RDMA via libibverbs and lnow ike to utilize NFS with RDMA over IB for another project. Any help, RTFM etc. welcome Thanks Thomas Bub Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com <http://www.grassvalley.com> ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] How to increase the waiting period/timeout for the completion of an RDMA operation?
Oops, I forgot to mention that the user paramter that fills this local_ack_timeout is called packet_life_time out of the ib_sa_path_record handed into the ib_cm_send_req. Maybe someone can put this into the upcoming man pages, if not already done. Thomas > -Original Message- > From: Bub Thomas > Sent: Saturday, November 18, 2006 12:41 PM > To: '[EMAIL PROTECTED]'; Roland Dreier > Cc: Thomas Bub; 'Erez Cohen'; openib-general@openib.org > Subject: RE: [openib-general] How to increase the waiting period/timeout > for the completion of an RDMA operation? > > Thanks that helped. > Thomas > > > > -Original Message- > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > > Sent: Saturday, November 18, 2006 6:55 AM > > To: Roland Dreier > > Cc: Thomas Bub; 'Erez Cohen'; Bub Thomas; openib-general@openib.org > > Subject: Re: [openib-general] How to increase the waiting period/timeout > > for the completion of an RDMA operation? > > > > > I think the attribute to look at is struct ibv_qp_attr.timeout. I'm > > > not really sure if the CM sets it or not. > > > > > > - R. > > > > The CM sets this attribute and it called "local_ack_timeout" in the CM > > structures. > > > > Dotan > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] How to increase the waiting period/timeout for the completion of an RDMA operation?
Thanks that helped. Thomas > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > Sent: Saturday, November 18, 2006 6:55 AM > To: Roland Dreier > Cc: Thomas Bub; 'Erez Cohen'; Bub Thomas; openib-general@openib.org > Subject: Re: [openib-general] How to increase the waiting period/timeout > for the completion of an RDMA operation? > > > I think the attribute to look at is struct ibv_qp_attr.timeout. I'm > > not really sure if the CM sets it or not. > > > > - R. > > The CM sets this attribute and it called "local_ack_timeout" in the CM > structures. > > Dotan > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] What could prevent a gen2 x86 client qp from doing RDMA_READ on a gen1 PowerPC client?
Dotan and all others. I could isolate the problem with hardware measurement a little more today I just opened a new thread on the problem naming: "How to increase the waiting period/timeout for the completion of an RDMA operation?" Thanks for the support so far. Thomas ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] How to increase the waiting period/timeout for the completion of an RDMA operation?
Yesterday I complained about a gen2 x86 client RDMA read not working on a gen1 PowerPC server. Today I got my hardware guy measuring the PCI-Express interface on the PowerPC gen1 server. It seems as if the RDMA read is at least started. So all ideas about endianess problems To explain the problem I have to give a little more details on our application: We are using the InfiniBand interface between a PowerPC gen1 server and a gen1/gen 2 x86 client for high speed film image transport from a scanner (server) to a workstation (client). In order to get the fastest response and performance the x86 client reads the images out of the scanner by RDMA on a hardware FIFO which is registered as physical memory. Unfortunately the scanner does not always have an image ready to deliver. So to avoid time consuming connected, rdma and disconnect for each and every image I'm holding the connection up and the scanner FIFO hardware delays the responses to the memory reads up to 500 msec. And here is now the problem: It seems as if the gen2 stack does not like that long delay on the rdma. Since a gen1 stack client can live with that long delay I think the gen2 stack should be able to do the same on the same hardware? The question now is: How to increase the waiting period/timeout for the completion of an RDMA operation? Things are getting clearer. Thomas ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] What could prevent a gen2 x86 client qp from doing RDMA_READ on a gen1 PowerPC client?
Endian-ness issues? Yes that is the first thought. But where? Since my gen2 x86 rdma code can do an rdma read from a gen1 and gen2 x86 server I think the only values in the ibv_send_wr that can be wrong talking to a PowerPC server can be remote_addr and rkey right I already swapped both but without success. Are there other places in the ibv_send_wr or the underlying code that might be endian-ness fooled? Since I can do a VAPI_SEND (non RDMA) from the gen2 x86 client to the gen1 PowerPC server I think the qp should be OK? Is there something RDMA READ specific in the qp that still might not be right after my CM connection from gen2 to gen1? Don't forget the RDMA WRITE from the gen1 PowerPC server to the gen2 x86 client on the same qp works just before the RDMA READ from gen2 x86 client on the gen1 PowerPC server fails. Still confusing. Thomas ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] What could prevent a gen2 x86 client qp from doing RDMA_READ on a gen1 PowerPC client?
Here is my next and hopefully last problem.
As described earlier I'm connecting a gen2 x86 clients to a gen1 PowerPC
server
After having sorted out the trouble with the CM parameters I'm now
having trouble with RDMA read from client on the server.
What works is:
- gen2 x86 client doing a VAPI_SEND to gen1 PowerPC server. (this
wasn't working last time)
- RDMA write from gen1 PowerPC server to gen2 x86 client
What is not working is:
- RDMA read from gen2 x86 client on gen1 PowerPC server. I'm
getting a vendor_error 0x81 VAPI_RETRY_EXC_ERR in the send completion
queue.
The RDMA start address, length and key have been exchanged and look
identical on both sides.
Doing connections and transfers between x86 only gen1 server x86 gen2
client works in all directions. (Send and receive as well as RDMA read
and write)
So a gen2 client can do a RDMA read from a gen1 server!
Having a gen1 PowerPC server and a gen1 x86 client works as well.
So a gen1 PowerPC server can be RDMA read from an x86 client!
I'm again a little puzzled what can the gen2 server do wrong in a RDMA
read on a PowerPC server when it can do the same operation a x86
server?
Any ideas, thoughts, help are more then welcome
Thanks
Thomas
Here is my code I'm using to do RDMA
I'm always having only a single segment to be transmitted!
rdma(ibv_sge *sgList,
int sgListlen,
int size,
bool write)
{
struct ibv_send_wr wr;
struct ibv_send_wr *bad_wr;
int res;
int localErrno = 0;
uint64_tremainingBytes =
ntohl(_remoteBufferInfo->totalSize);
sgList[0].length = remainingBytes;
memset(&wr, 0, sizeof(wr));
wr.next= NULL;
wr.wr_id = 1;
wr.opcode = write ? IBV_WR_RDMA_WRITE :
IBV_WR_RDMA_READ;
wr.send_flags = IBV_SEND_SIGNALED;
wr.sg_list = sgList;
wr.num_sge = 1;
wr.wr.rdma.remote_addr = ntohll(_remoteBufferInfo->sgList[0].addr);
wr.wr.rdma.rkey= ntohl (_remoteBufferInfo->sgList[0].lkey);
cleanCq(_sCq);
res = ibv_post_send(_dataQp, &wr, &bad_wr);
if (res != 0)
{
DEBUG1("Error in RDMA operation scheduling: %s\n",
strerror(res));
sv2BreakConnection();
localErrno = ENOTCONN;
return 0;
}
else
{
if (waitOnCq(_sCq))
{
localErrno = -1;
}
}
}
Thomas Bub
Grass Valley Germany GmbH
Brunnenweg 9
64331 Weiterstadt, Germany
Tel: +49 6150 104 147
Fax: +49 6150 104 656
Email: [EMAIL PROTECTED]
www.GrassValley.com
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Unusable QP's on CM established connections from gen2 client to gen1 server.
I don't have a svn repository. I'm living on OFED release will this work on the version in OFED-1.1? Thomas > -Original Message- > From: Sasha Khapyorsky [mailto:[EMAIL PROTECTED] > Sent: Monday, November 13, 2006 9:12 PM > To: Hal Rosenstock; Bub Thomas > Cc: Sean Hefty; Erez Cohen; openib-general@openib.org > Subject: Re: [openib-general] Unusable QP's on CM established connections > from gen2 client to gen1 server. > > On 19:49 Mon 13 Nov , Hal Rosenstock wrote: > > There may be an OpenSM bug with setting hop limit in the path record > response. I'm looking at it now. > > Looks like it is - OpenSM returns the same hop_limit value as was in the > request. > > Bub, could you try the patch below? > > Thanks, > Sasha > > > diff --git a/osm/opensm/osm_sa_path_record.c > b/osm/opensm/osm_sa_path_record.c > index 560e385..72a89d0 100644 > --- a/osm/opensm/osm_sa_path_record.c > +++ b/osm/opensm/osm_sa_path_record.c > @@ -709,6 +710,8 @@ __osm_pr_rcv_build_pr( >p_pr->dlid = cl_hton16( dest_lid_ho ); >p_pr->slid = cl_hton16( src_lid_ho ); > > + p_pr->hop_flow_raw &= cl_hton32(1<<31); > + >p_pr->pkey = p_parms->pkey; >p_pr->sl = cl_hton16(p_parms->sl); >p_pr->mtu = (uint8_t)(p_parms->mtu | 0x80); > > > > > > -- Hal > > > > > > > > From: [EMAIL PROTECTED] on behalf of Sean Hefty > > Sent: Mon 11/13/2006 11:26 AM > > To: Bub Thomas > > Cc: Erez Cohen; openib-general@openib.org > > Subject: Re: [openib-general] Unusable QP's on CM established > connections from gen2 client to gen1 server. > > > > > > > > Bub Thomas wrote: > > > Setting the hop_limit from 64 down to 0 or 1 solved the problem. :-) > > > Don't ask me where I got that hop_limit from, it must have been an > > > example I found somewhere. > > > Can you explain why that hop_limit/is_global makes a difference in > > > communication between gen1 and gen2? Does the counterpart need to have > > > the same hop_limit? > > > > The gen2 stack uses a hop_limit > 0 to indicate that global routing is > being > > used. If the hop_limit is > 0, then the global routing information must > be valid. > > > > > The path record values I use are queried from the OSM using a > > > SERVICE_RECORD query followed by a path record query. > > > I'm not using any alternate path record values, is this critical? > > > > Everything is supposed to work with the path records returned from the > SM. I > > was wondering if you were querying for the path record, modifying the > returned > > value, or creating a path record yourself. > > > > > path_record.packet_life_time = 0; > > > > I would set this higher (maybe between 12-20, with: 18 = 1 second, 19 = > 2 > > seconds, 20 = 4 seconds, etc.) > > > > ___ > > openib-general mailing list > > openib-general@openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- > general > > > > > > > > > > ___ > > openib-general mailing list > > openib-general@openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- > general > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Unusable QP's on CM established connections from gen2 client to gen1 server.
Sean, you got it! Setting the hop_limit from 64 down to 0 or 1 solved the problem. :-) Don't ask me where I got that hop_limit from, it must have been an example I found somewhere. Can you explain why that hop_limit/is_global makes a difference in communication between gen1 and gen2? Does the counterpart need to have the same hop_limit? The path record values I use are queried from the OSM using a SERVICE_RECORD query followed by a path record query. I'm not using any alternate path record values, is this critical? In addition I enclose the values I path into the ib_cm_sned_req call. Can you pls have a look if you find something alse looking abnormal. Thanks Thomas Bub req_param.qp_type= IBV_QPT_RC; req_param.qp_num = _dataQpNum; req_param.starting_psn = _dataQpNum;; req_param.service_id = htonll(SERVICE_ID); req_param.primary_path = &path_record; req_param.alternate_path = NULL; req_param.private_data = NULL; req_param.private_data_len = 0; req_param.responder_resources= 4; req_param.initiator_depth= 4; req_param.remote_cm_response_timeout = 20; req_param.local_cm_response_timeout = 20; req_param.retry_count= 7; req_param.rnr_retry_count= 7; req_param.max_cm_retries = 5; path_record.sgid = _localGid; path_record.dgid = _remoteGid; path_record.slid = htons(_localLID); path_record.dlid = htons(_remoteLID); path_record.flow_label = 0; path_record.hop_limit= 0; path_record.traffic_class= 0; path_record.pkey = 0x; path_record.sl = 0; path_record.rate = IBV_RATE_10_GBPS; path_record.packet_life_time = 0; path_record.mtu = IBV_MTU_2048; ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Unusable QP's on CM established connections from gen2 client to gen1 server.
Sean, I had a bug in my debug printout so that the value at the end where showing up wrong. Find enclosed the values after corrections in the printout. I see all your questionable values in a good shape now. I tried to increase the retry_count but without success. In fact the maximum value I could set was 7. The only thing that worries me is the timeout of 1 for the gen2 stack which is 12 for the gen1 stack. Is there a way to increase this? Question about psn: In my gen1 application I have no place where I explicitly the psn's. The psn's are either set by qp creation or the cm kernel code (I don't know and care it works!) In my gen2 code I copied an example where the ib_cm_req_param.starting_psn is explicitly set to the qp_num and the qp_attr.rq_psn is set to qp_num in the transition from Rts to Rtr. Whithout that last setting in Rts to Rtr even gen2 to gen2 does not work. Is there something that I'm missing? Thomas Gen1 Bad Server (0xb75f5a80) sv.c:2168 getQpAttributes qp_state: 3 en_sqd_asyn_notif: 0 sq_draining: 0 qp_num: 1442826 remote_atomic_flags: 7 qkey: 0 path_mtu: 4 path_mig_state: 0 rq_psn: 8693334 sq_psn: 3867654 qp_ous_rd_atom: 4 ous_dst_rd_atom: 4 min_rnr_timer: 27 cap.max_oust_wr_sq: 200 cap.max_oust_wr_rq: 200 cap.max_sg_size_sq: 28 cap.max_sg_size_rq: 28 cap.max_inline_data_sq: 460 dest_qp_num: 3867654 sched_queue: 0 pkey_ix: 0 port: 1 av.subnetId: 0 av.guid: 0 av.sl: 0 av.dlid: 3 av.src_path_bits: 105 av.static_rate: 0 av.grh_flag: 0 av.traffic_class: 0 av.hop_limit: 0 av.flow_label: 0 av.sgid_index: 0 av.port: 0 timeout: 12 retry_count: 6 rnr_retry: 6 alt_pkey_ix: 0 alt_port: 0 av.subnetId: 0 av.guid: 0 av.sl: 0 alt_av.dlid: 0 alt_av.src_path_bits: 0 alt_av.static_rate: 0 alt_av.grh_flag: 0 alt_av.traffic_class: 0 alt_av.hop_limit: 0 alt_av.flow_label: 0 alt_av.sgid_index: 0 alt_av.port: 0 alt_timeout: 0 Gen2 Bad Client (0x41001940) sv2.c:1520 queryQpState qp_num: 3867654 qp_state: 3 cur_qp_state: 3 path_mtu: 4 path_mig_state: 0 qkey: 131079 rq_psn: 3867654 sq_psn: 8693334 dest_qp_num: 1442826 qp_access_flags: 14 cap.max_send_wr: 256 cap.max_recv_wr: 256 cap.max_send_sge: 29 cap.max_recv_sge: 30 cap.max_inline_data: 460 ah_attr.grh.dgid.subnet_prefix: 80fe ah_attr.grh.dgid.interface_id: d17960304f10800 ah_attr.grh.flow_label: 0 ah.attr.grh.sgid_index: 0 ah.attr.grh.hop_limit: 64 ah.attr.grh.traffic_class: 0 ah_attr.dlid: 105 ah_attr.sl: 0 ah_attr.src_path_bits: 3 ah_attr.static_rate: 3 ah_attr.is_global: 1 ah_attr.port_num: 1 alt_ah.a
[openib-general] Unusable QP's on CM established connections from gen2 client to gen1 server.
Title: Unusable QP's on CM established connections from gen2 client to gen1 server. As written before I have to connect a gen2 client with a gen1 server using CM. The connection is established fine and both sides go into the connected state. However I can’t send any data from none of the two sides. As soon as I do so I get a 0x81 VAPI_RETRY_EXC_ERR when trying to send from the gen1 or a vendor_err 0x81 when trying to send from the gen2 side. What works so far with my gen1 and gen2 code is: Connecting gen1 client and gen1 server is no problem. Connecting gen2 client and gen2 server is no problem. Connecting gen1 client and gen2 server is no problem Unfortunately the only usage scenario I have is to connect gen2 client and gen1 server. L Is there a way to diagnose the reason for trouble when a VAPI_SEND / IBV_WR_SEND returns that 0x81 VAPI_RETRY_EXC_ERR? It seems as if the receiving side which is sure in the receive mode does not get any notification as if the sender does not even start sending. I already printed out the qp_state of both the gen2 client and the gen1 server before the send fails. Which look like: Gen2 client QP state. qp_state: 3 cur_qp_state: 3 path_mtu: 4 path_mig_state: 0 qkey: 16391 rq_psn: 5571589 sq_psn: 15025863 dest_qp_num: 10814473 qp_access_flags: 14 cap: 256 ah_attr: 256 alt_ah_attr: 29 pkey_index: 3 alt_pkey_index: 460 en_sqd_async_notify: 33022 sq_draining: 0 max_rd_atomic: 82905088 max_dest_rd_atomic: 219649539 min_rnr_timer: 0 port_num: 16384 timeout: 50331753 retry_cnt: 65795 rnr_retry: 0 alt_port_num: 0 alt_timeout: 0 Gen1 server QP state. qp_state: 3 en_sqd_asyn_notif: 0 sq_draining: 0 qp_num: 10814473 remote_atomic_flags: 7 qkey: 0 path_mtu: 4 path_mig_state: 0 rq_psn: 15025863 sq_psn: 5571589 qp_ous_rd_atom: 4 ous_dst_rd_atom: 4 min_rnr_timer: 27 cap: 200 dest_qp_num: 200 sched_queue: 28 pkey_ix: 28 port: 460 av: 5571589 timeout: 0 retry_count: 0 rnr_retry: 1 alt_pkey_ix: 0 alt_port: 0 alt_av: 0 alt_timeout: 0 In the good cases described above the qp_state look similar to the ones described here Any help welcome. Thomas Bub Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] How to register, query and delete a service_id?
Title: How to register, query and delete a service_id? Hi, as I’m porting my gen1 application to gen2 my last task is to port the service_id registration, query and deletion to gen2. With the help of Mellanox I got it running under gen1 using read/write of mad messages on the device “/dev/ts_ua0”. I browsed though the ofed sources and got lost in there. Is there some good and simple example that can help me out of my blind? I assume I have to use the ibmad and/or ibumad library? Thanks Thomas Bub Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] OFED for SLES10
Thierry, the support for SLES10 has been introduced after the OFED-1.0 release had been done. Best thing is you download the OFED-1.1 Release Candidate 6 from: https://openib.org/svn/gen2/branches/1.1/ofed/releases/OFED-1.1-rc6.tgz You should have the kernel sources of the SLES10 kernel installed. Unpack the tgz and run the install script found in the directory OFED-1.1-rc6. It does everything you need to get the drivers build and installed. That should be all whats needed. Thomas Bub Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com > -Original Message- > From: [EMAIL PROTECTED] [mailto:openib-general- > [EMAIL PROTECTED] On Behalf Of Thierry Delaitre > Sent: Friday, September 22, 2006 4:00 PM > To: openib-general@openib.org > Subject: [openib-general] OFED for SLES10 > > > Hi, > > Could someone point me in the right direction on how to compile ofed for > sles10 ? > > Do i need to recompile modules for the kernel ? > Do i need to patch the kernel before compiling the module ? > Do i need to delete the existing infiniband directory in the kernel and > replace it with ofed ? > Does the ofed distri include the kernel modules ? > > Thanks, > > Thierry. > > > Dr Thierry DELAITRE > Systems and Services Manager, CSCS > University of Westminster > 115 New Cavendish Street, London W1W 6UW > > Tel: 020 7911 5000 ext: 3586 > Fax: 020 7911 5089 > Mobile short dial code 1788 > > http://www.cscs.wmin.ac.uk/~delaitt > > > This e-mail and its attachments are intended for the above named only > and may be confidential. If they have come to you in error you must > not copy or show them to anyone, nor should you take any action based > on them, other than to notify the error by replying to the sender. > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- > general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] What can be the reason for VAPI_WR_FLUSH_ERR when sending from gen2 to gen1
Title: What can be the reason for VAPI_WR_FLUSH_ERR when sending from gen2 to gen1 This seems to be the very last little bug in my journey migrating from gen1 client and server to gen2 client and gen1 server. While I came over all CM issues I had so far (thanks to Sean Heafty) I’m now in the situation that I have a gen2 client connected to a gen1 server via CM. Unfortunately the first IBV_WR_SEND causes a: (syndrome=0xf9=VAPI_WR_FLUSH_ERR , opcode=6=VAPI_CQE_RQ_SEND_DATA) error in the receive completion queue of the server. Doing the CM connection and the first send in the opposite direction from gen1 client to gen2 server it is OK. Needless to say that connection and send between gen1 <-> gen1 and gen2<->gen2 is OK as well. I copied Erez Cohen from Mellanox as well. Maybe someone can explain me in more detail what the error is about and how to avoid it. Thanks Thomas ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Any chance to get 32-Bit libraries on SLES9 x86_64?
Title: Any chance to get 32-Bit libraries on SLES9 x86_64? Is there any chance/trick to get 32-Bit Libraries build and usable on SLES9 x86_64? When I installed OFED-1.1-rc4 I get: WARNING: sysfsutils 32-bit version is required to build 32-bit libibverbs package. WARNING: Skiping build of 32-bit libraries. I googled around and didn’t find any sysfsutils 32-bit for SLES9. I now that tit is working under SLES10 b ut our customer base is on SLES9 and very conservative when it comes down to using the latest and greates Os/distribution. Thomas Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Different byte order between gen1 CM and gen2 CM ->RE: How to connect gen2 CM to gen1 IBGD CM?
Title: Different byte order between gen1 CM and gen2 CM ->RE: How to connect gen2 CM to gen1 IBGD CM? Sean, I should have checked this earlier after you told me last time that the LID is taken in network order by the gen2 CM instead of host order in gen1. This time it was the service_id I stumbeled over. After putting my service_id into network order I could at least get a REQ_RECEIVED. The rest must be fine tuning from here onwards. Do you know rany other Verbs or CM parameter that does have a different byte order between gen1 and gen2? Thanks Thomas P.S.: Maybe someone should put a big “Warning” sign somewhere so that others don’t stumple into that pit again. ;-) _ From: Bub Thomas Sent: Wednesday, September 13, 2006 4:11 PM To: 'Sean Hefty'; '[EMAIL PROTECTED]' Cc: openib-general@openib.org Subject: How to connect gen2 CM to gen1 IBGD CM? Sean, with your patience, the cmpost.c example and the OFED 1.1-rc4 on all machines I finally got a gen2 connection under SLES10 even with a 32-Bit executable on a x86_64 machine. Cool! Now the last part on my journey is standing out. It’s a gen2 client connecting to a gen1 IBGD server. I have to do this since my gen1 server is running a 2.4 Montavista RT Linux on a PowerPC that I can’t upgrade to gen2. L BTW.: Our application is a high speed film image transfer in the film postproduction industry leveraging the benefits of the high speed IB RDMA transport. While I have gen1 to gen1 and gen2 to gen2 running the only thing that is missing is the gen2 connecting to gen1. Just tried this with my test-executables but I did not get anything to the gen1 server. The gen1 userspace application does not even receive the IB_CM_REQ. So since your cmpost example did help me a lot on gen2 the question is: Do you have a cmpost for gen1 IBGD I can use to connect from gen2 to gen1? Or is there any other trick to play here? Thanks in advance for your assistance Thomas Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] OFED can't compile against sa.h under SLES10 x86_64
Jack et all, I have to apologize my -I. include path pointed to OFED-1.0.1 includes where the ibv_sa_path_record was not defined yet. Doing it right it works Thanks to all for the sudden support Thomas (humbling backwards) ;-) > -Original Message- > From: [EMAIL PROTECTED] [mailto:openib-general- > [EMAIL PROTECTED] On Behalf Of Jack Morgenstein > Sent: Thursday, September 14, 2006 1:12 PM > To: Bub Thomas > Cc: openib-general@openib.org > Subject: [openib-general] OFED can't compile against sa.h under SLES10 > x86_64 > > I was unable to reproduce the problem you describe, under SLES10 x86_64. > Here, your cmpost.c file compiled and linked without any problems. > I used a slightly different gcc command line (given below). > > I took the cmpost.c file you provided, placed it under > /usr/local/ofed/src/openib-1.1/src/userspace/libibcm/examples > (under an OFED 1.1-rc5 prerelease candidate installation). > I then did the following: > > cd libibcm/examples > > gcc -ggdb -Wall -O0 -I/usr/local/ofed/include -D__x86_64__ > /usr/local/ofed/lib64/libibcommon.so > /usr/local/ofed/lib64/librdmacm.so > /usr/local/ofed/lib64/libibcm.so -o cmpost cmpost.c > > (the above gcc command is broken up into several lines for easy reading) > > The compilation was successful. I did not experience any compilation or > linkage problems. > I was able to run the resulting "cmpost" executable file. > > gcc version: gcc (GCC) 4.1.0 (SUSE Linux) > Linux distribution: (from file /etc/SuSE-release): > SUSE Linux Enterprise Server 10 (x86_64) > VERSION = 10 > > Kernel version (uname -a): > Linux 2.6.16.21-0.8-smp #1 SMP Mon Jul 3 18:25:39 UTC 2006 x86_64 > x86_64 x86_64 GNU/Linux > > I then retried everything using OFED 1.1 RC4, and also succeeded in > compiling and running cmpost.c. > > The following is the list of OFED packages that I installed for the above > experiment: > ib_ipoib > ib_mthca > ib_verbs > kernel-ib > kernel-ib-devel > libibcm > libibcm-devel > libibcommon > libibcommon-devel > libibmad > libibmad-devel > libibumad > libibumad-devel > libibverbs > libibverbs-devel > libibverbs-utils > libmthca > libmthca-devel > librdmacm > librdmacm-devel > librdmacm-utils > ofed-scripts > > - Jack > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- > general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] OFED can't compile against sa.h under SLES10 x86_64
Michael and Bryan, find the libibcm example cmpost.c that fails to compile enclosed. My compiler output looks like: cc -ggdb -Wall -O0 -I. -I./usr/include -I./oibfix -I/usr/local/ofed/include -I/usr/src/linux/drivers/infiniband/include -D__x86_64__ -c -o cmpost.o cmpost.c cmpost.c: In function 'query_for_path': cmpost.c:658: error: invalid use of undefined type 'struct ibv_sa_path_rec' cmpost.c:658: error: dereferencing pointer to incomplete type cmpost.c: In function 'run_client': cmpost.c:679: warning: assignment from incompatible pointer type make: *** [cmpost.o] Error 1 It's OK under SLES9 but fails under SLES10. Thanks Thomas cmpost.c Description: cmpost.c ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] How to connect gen2 CM to gen1 IBGD CM?
Title: How to connect gen2 CM to gen1 IBGD CM? Sean, with your patience, the cmpost.c example and the OFED 1.1-rc4 on all machines I finally got a gen2 connection under SLES10 even with a 32-Bit executable on a x86_64 machine. Cool! Now the last part on my journey is standing out. It’s a gen2 client connecting to a gen1 IBGD server. I have to do this since my gen1 server is running a 2.4 Montavista RT Linux on a PowerPC that I can’t upgrade to gen2. L BTW.: Our application is a high speed film image transfer in the film postproduction industry leveraging the benefits of the high speed IB RDMA transport. While I have gen1 to gen1 and gen2 to gen2 running the only thing that is missing is the gen2 connecting to gen1. Just tried this with my test-executables but I did not get anything to the gen1 server. The gen1 userspace application does not even receive the IB_CM_REQ. So since your cmpost example did help me a lot on gen2 the question is: Do you have a cmpost for gen1 IBGD I can use to connect from gen2 to gen1? Or is there any other trick to play here? Thanks in advance for your assistance Thomas Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] OFED can't compile against sa.h under SLES10 x86_64
Title: OFED can't compile against sa.h under SLES10 x86_64 Michael, this is another little issue using OFED under SLES10. In sa.h there is the definition of a struct ibv_sa_path_record that gets re-defined against ib_sa_path_record in the same header file. While the gcc 3.3.3 compile of SLES 9 is OK with this the gcc 4.1 comiple of SLEs 10 does not like this. Thomas Bub ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] cmpost establisehd connections are very fragile!?
Title: cmpost establisehd connections are very fragile!? Sean, got my libibverbs/libibcm code working on SLES9 x86_64 after following all the tricks in cmpost.c What I don’t understand why the local_cm_response_timeout set to 254 instead of 20 can block IBV_WR_SEND from client to server while the opposite direction from server to client works!? You don’t have a more detailed description to the libibcm parameters? There are a lot more that I don’t understand. ;-) After having a running gen2 example I moved to my final distribution which is SLES 10 x86_64. I have to do this since I have to use a 32 Bit executable for 32-Bit and 64 Bit machines and this is supported in OFED from SLES10 onwards. Coming back to the fragile connection I encountered the same issue where the client can’t do an IBV_WR_SEND to the server. This time both your cmpost example and my code failed. I tried to reduce the local_cm_response_timeout to 10 but thuis did not help at all. All above is done for 64-Bit executables. Interesting enough the 32 Bit executable of cmpost and my own build on a x86 SLES9 machine did not have the IBV_WR_SEND trouble. Thanks in advance for enlighten me. ;-) Thomas Bub ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Trouble installing OFED-1.1-rc3 on a x86_64 SLES 10 machine
Michael, I don't understand what you mean on the iser trouble. I'm only a "comsumer" and not actively developing in the openIB world. I'm having enough trouble with my own application connecting a PowerPC gen1 from an x86_64 PC gen2 using verbs and cm. ;-) Thus I haven't installed SVN and can't work on this. I wanted to let the people know that there are some issues. Thanks Thomas -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Michael S. Tsirkin Sent: Tuesday, September 12, 2006 2:55 PM To: Bub Thomas Cc: openib-general@openib.org Subject: Re: [openib-general] Trouble installing OFED-1.1-rc3 on a x86_64 SLES 10 machine Quoting r. Bub Thomas <[EMAIL PROTECTED]>: > Subject: Trouble installing OFED-1.1-rc3 on a x86_64 SLES 10 machine > > Just migrated from SLES 9 x86_64 to SLES 10 x86_64 in order to get > 32-Bit support. > Stumbled over some installation problems. > First I tried "All packages" then "Basic install". Both failed to build > at different places. > Only a "customizied" installation worked. > Find the details blow. > > Thomas Bub > > An All packages fails at: > > gcc > -Wp,-MD,/var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/.i > scsi_iser.o.d -nostdinc -isystem > /usr/lib64/gcc/x86_64-suse-linux/4.1.0/include -D__KERNEL__ > -I/var/tmp/OFEDRPM/BUILD/openib-1.1/include > -I/var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/include > -Iinclude -Iinclude2 -I/usr/src/linux-2.6.16.21-0.8/include > -I/var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser -Wall > -Wundef -Wstrict-prototypes -Wno-trigraphs > -Werror-implicit-function-declaration -fno-strict-aliasing -fno-common > -ffreestanding -Os -fomit-frame-pointer -mtune=generic -m64 > -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks > -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time > -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Wdeclaration-after-statement > -Wno-pointer-sign -I/var/tmp/OFEDRPM/BUILD/openib-1.1/include > -I/var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/include > -I/var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/ipoib > -I/var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/debug -DMODULE > -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(iscsi_iser)" > -D"KBUILD_MODNAME=KBUILD_STR(ib_iser)" -c -o > /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/.tmp_iscsi > _iser.o > /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser > .c > /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser > .c: In function 'iscsi_iser_set_param': > /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser > .c:478: error: implicit declaration of function 'iscsi_set_param' > /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser > .c: At top level: > /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser > .c:612: warning: initialization from incompatible pointer type > /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser > .c:613: error: 'iscsi_conn_get_param' undeclared here (not in a > function) > /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser > .c:614: error: 'iscsi_session_get_param' undeclared here (not in a > function) Or - could you check this please? AFAIK iser should work on this kernel. > A Basic install fails at: > > gcc -DHAVE_CONFIG_H -I. -I. -I. -I../libibverbs/include -Wall > -D_GNU_SOURCE -g -O2 -MT src_ipathverbs_la-ipathverbs.lo -MD -MP -MF > .deps/src_ipathverbs_la-ipathverbs.Tpo -c src/ipathverbs.c -fPIC -DPIC > -o .libs/src_ipathverbs_la-ipa > thverbs.o > In file included from src/ipathverbs.c:45: > src/ipathverbs.h: In function 'to_ictx': > src/ipathverbs.h:72: warning: implicit declaration of function > 'offsetof' > src/ipathverbs.h:72: error: expected expression before 'struct'ib_mthca Looks like ipthverbs.h uses offsetof without including stddef.h Please post fix for trunk and OFED branch. > My customized installation that works: > > ib_verbs > kernel-ib > kernel-ib-devel > libibcm > libibcm-devel > libibverbs > libibverbs-devel > libibverbs-utils > libmthca > libmthca-devel -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Trouble installing OFED-1.1-rc3 on a x86_64 SLES 10 machine
Just migrated from SLES 9 x86_64 to SLES 10 x86_64 in order to get 32-Bit support. Stumbled over some installation problems. First I tried "All packages" then "Basic install". Both failed to build at different places. Only a "customizied" installation worked. Find the details blow. Thomas Bub An All packages fails at: gcc -Wp,-MD,/var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/.i scsi_iser.o.d -nostdinc -isystem /usr/lib64/gcc/x86_64-suse-linux/4.1.0/include -D__KERNEL__ -I/var/tmp/OFEDRPM/BUILD/openib-1.1/include -I/var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/include -Iinclude -Iinclude2 -I/usr/src/linux-2.6.16.21-0.8/include -I/var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -Werror-implicit-function-declaration -fno-strict-aliasing -fno-common -ffreestanding -Os -fomit-frame-pointer -mtune=generic -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Wdeclaration-after-statement -Wno-pointer-sign -I/var/tmp/OFEDRPM/BUILD/openib-1.1/include -I/var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/include -I/var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/ipoib -I/var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/debug -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(iscsi_iser)" -D"KBUILD_MODNAME=KBUILD_STR(ib_iser)" -c -o /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/.tmp_iscsi _iser.o /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser .c /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser .c: In function 'iscsi_iser_set_param': /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser .c:478: error: implicit declaration of function 'iscsi_set_param' /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser .c: At top level: /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser .c:612: warning: initialization from incompatible pointer type /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser .c:613: error: 'iscsi_conn_get_param' undeclared here (not in a function) /var/tmp/OFEDRPM/BUILD/openib-1.1/drivers/infiniband/ulp/iser/iscsi_iser .c:614: error: 'iscsi_session_get_param' undeclared here (not in a function) A Basic install fails at: gcc -DHAVE_CONFIG_H -I. -I. -I. -I../libibverbs/include -Wall -D_GNU_SOURCE -g -O2 -MT src_ipathverbs_la-ipathverbs.lo -MD -MP -MF .deps/src_ipathverbs_la-ipathverbs.Tpo -c src/ipathverbs.c -fPIC -DPIC -o .libs/src_ipathverbs_la-ipa thverbs.o In file included from src/ipathverbs.c:45: src/ipathverbs.h: In function 'to_ictx': src/ipathverbs.h:72: warning: implicit declaration of function 'offsetof' src/ipathverbs.h:72: error: expected expression before 'struct'ib_mthca My customized installation that works: ib_verbs kernel-ib kernel-ib-devel libibcm libibcm-devel libibverbs libibverbs-devel libibverbs-utils libmthca libmthca-devel ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] ibv_poll_cq after ibv_post_send does not work
Dortan Barak wrote: If you are using RC QP: the reason for not getting any completion in the CQ is that Did you post any RR (Receive Request) at the listener side? Dotan, with the cmpost.c example I now get a cm connection even with another machine. However I don't get the cq event, on the sender side, when the IBV_WR_SEND is done. Is this correct? Is this what you are saying below? If it is correct this is different from gen1 drivers where I got a VAPI_SUCCESS cq event. Is there a way to get this back? On the receiver side I get an cq event for the receive request. Thanks Thomas ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Wrong byte order in lid of struct ibv_port_attr reported by ibv_query port!?
Title: Wrong byte order in lid of struct ibv_port_attr reported by ibv_query port!? Sean, with the help of your modified cmpost.c example I found out that the byte order in the lid your query_for_path in cmpost.c is getting into the ib_sa_path_rec is the opposite to the one reported by ibv_query_port. Since I’m doing the connection establishment based on lid, GUID and subnetID this explains why I can’t connect to a client to antoher machine. Can you tell me which one is wrong or am I doing something wrong here? Thomas ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] ibv_poll_cq after ibv_post_send does not work
Dotan, Find my answers inline. Since I could get the cmpost example from Sean compiled and running I will try to compare cmpost.c with my code and find the bugs in my code this way. I will keep your connect_qp example for the case that I can't find the problems the other way. Thanks Thomas Which QP do you use (RC/UC/UD)? [Bub] Rc do you get any completion in the connector side? [Bub] Only the the errors If you are using RC QP: the reason for not getting any completion in the CQ is that Did you post any RR (Receive Request) at the listener side? [Bub] yes rnr_retry =7 means that in case of RNR retry there will be infinite retries [Bub] rnr_retry is at 4 if the timeout = 0 and the remote QP is not ready then there won't be any retransmition. [Bub] I have the timeout at 254 Dotan ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] libibcm can't connect/talk to libicm on other machine.
Sean, Finally I could compile the cmpost example. The solution was: 1.) Use the OFED-1.1-rc3 instead of OFED-1.0.1 This removed some missing DEFINES. As an "End-User" ;-) I'm not following the SVN tree but installing releases. 2.) Add "#include " to cmpost.c. Now I can compile and use the example at least on one machine. The issues with client and server on one machine, that I reported yesterday, are not visible as well. So I'm able now to debug my connection establishment and the initial data exchange. Next week I can debug cmpost on two different machines, my second machine has been stolen by a developer colleague till mid of next week. ;-) I would suggest that the cmpost.c example, including the missing include from above, might be integrated into the next OFED-release? Thanks Thomas -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Sean Hefty Sent: Tuesday, September 05, 2006 8:15 PM To: Bub Thomas Cc: openib-general@openib.org Subject: Re: [openib-general] libibcm can't connect/talk to libicm on other machine. Bub Thomas wrote: > Dotan, > the ibv_rc_pingpong example works for me so I can exclude the > architecture. > I never got the libibcm example compiled. > Which is your example and which architecture x86 vs. x86_64 did you > compile it for? > Can you share your libibcm the example code? (if it is not the standard > that I can't get compiled) > Thomas Did you try applying the following patch? http://openib.org/pipermail/openib-general/2006-August/025005.html I should also mention that I have a version of cmpost that works with the new libibsa, but I am waiting for the review of the kernel sa_query changes before committing. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] ibv_poll_cq after ibv_post_send does not work
Title: ibv_poll_cq after ibv_post_send does not work I’m still in the process of porting my gen1 code to gen2. As mentioned yesterday I can connect to a listener on the same machine using libibcm. Doing this I have to do the ibv_modify_qp by myself to get the qp’s from INIT via RTR to RTS on both sides. At least the ibv_modify_qp doesn not complain when having done the connection via the libibcm. So my assumption is I have my two qp’s successfully connected. First action after the connection is the listener to wait on it’s receive cq for an IBV_WR_SEND done by the connector. Here is now the problem: · The listener never gets a completion · The connector doing the IBV_WR_SEND does get error on the send cq like opcode=0x7f status=0x5 vendor_err=129 for the first IBV_WR_SEND and opcode=0x7f status=0xc vendor_err=129 for all sub-sequent attempt to send the data Is there anyone out there who can help me out to understand the error codes and or to understand what is wrong? Thanks in advance from Germany Thomas Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] libibcm can't connect/talk to libicm on other machine.
Dotan, the ibv_rc_pingpong example works for me so I can exclude the architecture. I never got the libibcm example compiled. Which is your example and which architecture x86 vs. x86_64 did you compile it for? Can you share your libibcm the example code? (if it is not the standard that I can't get compiled) Thomas -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dotan Barak Sent: Tuesday, September 05, 2006 5:12 PM To: Bub Thomas Cc: openib-general@openib.org Subject: Re: [openib-general] libibcm can't connect/talk to libicm on other machine. Hi bub. Bub Thomas wrote: > > I'm still in the process of migrating my gen1 application to gen2. > > Actually I CAN connect a gen2 application to a gen2 listener > application on the same machine but NOT to a gen 2 listener on another > machine. > > Any hints where to look at? > > Is there anything in the architecture that might prevent a libibcm > connection to another machine? > > I'm using an old Voltaire switch to connect the machines. Can this be > the reason? > > The switch didn't cause problems using gen1 clients. > What is the problem that you see? there are some examples that comes with the libibcm that can show you how to use the library. there can be several reasons for your problem: 1) side A send a req when side B is not ready and there is a timeout failure 2) only in side A the ib_ucm kernel module enabled 3) SM is not working (well) 4) host A cannot be reached to host B using IB 5) endianess issues? i tried to use the libibcm and i don't have any problem (but i don't have any Voltaire switch, so i can't check your scenario). Dotan ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] libibcm can't connect/talk to libicm on other machine.
Title: libibcm can't connect/talk to libicm on other machine. I’m still in the process of migrating my gen1 application to gen2. Actually I CAN connect a gen2 application to a gen2 listener application on the same machine but NOT to a gen 2 listener on another machine. Any hints where to look at? Is there anything in the architecture that might prevent a libibcm connection to another machine? I’m using an old Voltaire switch to connect the machines. Can this be the reason? The switch didn’t cause problems using gen1 clients. Thanks Thomas Bub ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] libibcm can't open /dev/infiniband/ucm0
Made some progress today: The file OFED-1.0.1/SOURCES/openib-1.0.1/src/userspace/libibcm/README talks about "mknod /dev/infiniband/ucm0 c 231 255" This is not working for me. I'm getting the device /dev/infiniband/ucm0 created but the libibcm still could not open the device Looked into the openIB kernel sources and found that the minor number seems to be wrong in the README file. With a minor number "224" and the creation like: "mknod /dev/infiniband/ucm0 c 231 224" I got the device created andtthe libibcm could open the device. So I assume this to be a bug at least in the README file. I'm not the kernel expert to tell where else this has to be changed. Should I open a bug on this? I got a first SLES10 system eaelier this week, there I'm using the ofed-1.0-sles10-rpms_x86_64.tar.gz instead of the OFED-1.0.1.tgz. Interesting enough the /dev/infiniband/ucm0 got created and usable automatically. Thomas -Original Message- From: Sean Hefty [mailto:[EMAIL PROTECTED] Sent: Tuesday, August 22, 2006 5:47 PM To: Bub Thomas; Sean Hefty Cc: openib-general@openib.org Subject: RE: [openib-general] libibcm can't open /dev/infiniband/ucm0 >https://openib.org/tiki/tiki-index.php?page=Install+OpenIB+for+Chelsio+ T >3&highlight=udev The udev information for this link looks correct. >Or is there another way/description? You can run mknod to manually create the file. (See the README file in the libibcm directory.) >Additionally I didn't find the udev rules for the already exsisting >/dev/infinibadn devices in the /etc/udev/udev.rules file. Is there any chance these were created manually at some point? What files are in the directory? (I would expect something like issm*, umad*, ucm*, uverbs*.) - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] libibcm can't open /dev/infiniband/ucm0
Not quite sure what you are talking about? Found the Wiki entry: https://openib.org/tiki/tiki-index.php?page=Install+OpenIB+for+Chelsio+T 3&highlight=udev As I'm not running the referenced Kernel version I'm not sure if I should only apply the udev part of the link. Or is there another way/description? Additionally I didn't find the udev rules for the already exsisting /dev/infinibadn devices in the /etc/udev/udev.rules file. Thanks Thomas -Original Message- From: Sean Hefty [mailto:[EMAIL PROTECTED] Sent: Monday, August 21, 2006 6:48 PM To: Bub Thomas Cc: openib-general@openib.org Subject: Re: [openib-general] libibcm can't open /dev/infiniband/ucm0 Bub Thomas wrote: > Here is the list of all loaded ib modules and their dependencies: > > ib_rds 37656 0 > ib_ucm 21512 0 Did you update udev rules to create the device? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] libibcm can't open /dev/infiniband/ucm0
Sean, ib_ ucm is already loaded!? Here is the list of all loaded ib modules and their dependencies: ib_rds 37656 0 ib_ucm 21512 0 ib_srp 33924 0 ib_sdp 45468 0 rdma_cm26760 3 rdma_ucm,ib_rds,ib_sdp ib_addr10504 1 rdma_cm ib_cm 39952 3 ib_ucm,ib_srp,rdma_cm ib_local_sa14232 2 rdma_ucm,rdma_cm findex 6784 1 ib_local_sa ib_ipoib 59800 0 ib_sa 18196 4 ib_srp,rdma_cm,ib_local_sa,ib_ipoib ib_uverbs 47408 1 rdma_ucm ib_umad20272 0 ib_ipath 70424 0 ipath_core179524 1 ib_ipath ib_mthca 140336 0 ib_mad 43304 5 ib_cm,ib_local_sa,ib_sa,ib_umad,ib_mthca ib_core59520 14 ib_rds,ib_ucm,ib_srp,ib_sdp,rdma_cm,ib_cm,ib_local_sa,ib_ipoib,ib_sa,ib_ uverbs,ib_umad,ib_ipath,ib_mthca,ib_mad scsi_mod 140177 6 ib_srp,qla2xxx,scsi_transport_fc,libata,mptscsih,sd_mod Thanks Thomas -Original Message- From: Sean Hefty [mailto:[EMAIL PROTECTED] Sent: Friday, August 18, 2006 6:06 PM To: Bub Thomas Cc: openib-general@openib.org Subject: Re: [openib-general] libibcm can't open /dev/infiniband/ucm0 Bub Thomas wrote: > It seems as if the problem I had there was not in my code but the > libibcm not being able to open the device /dev/infiniband/ucm0. You will need to load ib_ucm, which exports the IB CM to userspace. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] cmpost: allow cmpost to build with latest RDMA CM
Sean, as I understand cmpost.c and simple.c where originally pure libibcm examples. Is there any other test code utilizing the libibcm? Thanks Thomas -Original Message- From: Sean Hefty [mailto:[EMAIL PROTECTED] Sent: Friday, August 18, 2006 6:09 PM To: Bub Thomas Cc: Sean Hefty; openib-general@openib.org; Erez Cohen Subject: Re: [openib-general] [PATCH] cmpost: allow cmpost to build with latest RDMA CM Bub Thomas wrote: > Can I still use the LID, GUID and SubnetID for connection establishment > then? Then Gen1 counterpart has no IP over IB running. If IPoIB is not running, then you will need to use the IB CM directly. The RDMA CM uses ARP to resolve IP addresses to GIDs. > I'm using OFED-1.0.1. > Do you have a quick link where to find the latest Headers? > (Sorry for the dumb question) https://openfabrics.org/svn/gen2/trunk/src/ https://openfabrics.org/svn/gen2/trunk/src/userspace/libibcm https://openfabrics.org/svn/gen2/trunk/src/userspace/librdmacm https://openfabrics.org/svn/gen2/trunk/src/linux-kernel/infiniband/ - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] libibcm can't open /dev/infiniband/ucm0
Title: libibcm can't open /dev/infiniband/ucm0 I looked into my original libibcm problem today, that made me looking into cmpost, discussed in a sperate thread It seems as if the problem I had there was not in my code but the libibcm not being able to open the device /dev/infiniband/ucm0. All IB modules are installed and the modules loaded the /dev/infiniband devices available are: crw--- 1 root root 231, 64 Aug 17 15:14 issm0 crw-rw-rw- 1 root root 10, 62 Aug 17 15:14 rdma_cm crw--- 1 root root 231, 0 Aug 17 15:14 umad0 crw-rw-rw- 1 root root 231, 192 Aug 17 15:14 uverbs0 The exact error message is: libibcm: error <-1:2> opening device I’m running OFED-1.0.1 on an HP xw9300 running SLES9 SP3 in the x86_64 flawor. Thanks Thomas Bub .... Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] cmpost: allow cmpost to build with latest RDMA CM
OK Can I still use the LID, GUID and SubnetID for connection establishment then? Then Gen1 counterpart has no IP over IB running. I'm using OFED-1.0.1. Do you have a quick link where to find the latest Headers? (Sorry for the dumb question) Thanks Thomas -Original Message- From: Sean Hefty [mailto:[EMAIL PROTECTED] Sent: Thursday, August 17, 2006 6:50 PM To: Bub Thomas Cc: Sean Hefty; openib-general@openib.org; Erez Cohen Subject: Re: [openib-general] [PATCH] cmpost: allow cmpost to build with latest RDMA CM Bub Thomas wrote: > I'm getting a little puzzled. > For me it seems as if we are moving in the wrong direction. > I don't have a RDMA CM on the Gen1 counterpart that my gen2 application > is talking too. The RDMA CM is only used on the local (active or client) side to obtain a path record, which is needed by the libibcm. Using the librdmacm allows cmpost to get a path record given only the remote IP address or host name. The connection is established using the IB CM through libibcm. > If yes you have to explain me what the two different versions: > rdma_cm.h > and > rdma_cma.h rdma_cm.h defines the kernel interface to the RDMA CM. rdma_cma.h defines the userspace interface. > The cmpost.c was using rdma_cma.h up to now but the missing defines are > located in rdma_cm.h Can you verify that you have the latest version of rdma_cma.h? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] cmpost: allow cmpost to build with latest RDMA CM
Sean,
Thanks for the quick reply.
Some errors got resolved some new added.
Some of the errors but not all are caused by the picky gcc 3.4.4 that I
could avoid by some casting.
I left them all in to give you the whole picture.
Here is the actual output:
cmpost.c:65: error: field `path_rec' has incomplete type
cmpost.c: In function `int modify_to_rtr(cmtest_node*)':
cmpost.c:130: error: invalid conversion from `int' to `ibv_qp_attr_mask'
cmpost.c:130: error: initializing argument 3 of `int
ibv_modify_qp(ibv_qp*, ibv_qp_attr*, ibv_qp_attr_mask)'
cmpost.c:142: error: invalid conversion from `int' to `ibv_qp_attr_mask'
cmpost.c:142: error: initializing argument 3 of `int
ibv_modify_qp(ibv_qp*, ibv_qp_attr*, ibv_qp_attr_mask)'
cmpost.c: In function `int modify_to_rts(cmtest_node*)':
cmpost.c:161: error: invalid conversion from `int' to `ibv_qp_attr_mask'
cmpost.c:161: error: initializing argument 3 of `int
ibv_modify_qp(ibv_qp*, ibv_qp_attr*, ibv_qp_attr_mask)'
cmpost.c: In function `void cm_handler(ib_cm_id*, ib_cm_event*)':
cmpost.c:267: error: invalid conversion from `void*' to `cmtest_node*'
cmpost.c: In function `int create_nodes()':
cmpost.c:353: error: invalid conversion from `void*' to `cmtest_node*'
cmpost.c: In function `int query_for_path(char*)':
cmpost.c:631: error: `RDMA_PS_TCP' undeclared (first use this function)
cmpost.c:631: error: (Each undeclared identifier is reported only once
for each function it appears in.)
cmpost.c:657: error: 'struct cmtest' has no member named 'path_rec'
cmpost.c:657: error: invalid use of undefined type `struct
ibv_sa_path_rec'
/vob/pkgs/sys/linux-include/rdma/rdma_cma.h:84: error: forward
declaration of `struct ibv_sa_path_rec'
cmpost.c: In function `void run_client(char*)':
cmpost.c:678: error: 'struct cmtest' has no member named 'path_rec'
*** Error code 1
clearmake: Error: Build script failed for "cmpost.o"
Thomas
-Original Message-
From: Sean Hefty [mailto:[EMAIL PROTECTED]
Sent: Tuesday, August 15, 2006 6:11 PM
To: Bub Thomas; openib-general@openib.org
Cc: Erez Cohen
Subject: [PATCH] cmpost: allow cmpost to build with latest RDMA CM
Can you see if this patch lets you build compost?
Signed-off-by: Sean Hefty <[EMAIL PROTECTED]>
---
Index: examples/cmpost.c
===
--- examples/cmpost.c (revision 8215)
+++ examples/cmpost.c (working copy)
@@ -614,6 +614,7 @@ out:
static int query_for_path(char *dst)
{
+ struct rdma_event_channel *channel;
struct rdma_cm_id *id;
struct sockaddr_in addr_in;
struct rdma_cm_event *event;
@@ -623,15 +624,19 @@ static int query_for_path(char *dst)
if (ret)
return ret;
- ret = rdma_create_id(&id, NULL);
+ channel = rdma_create_event_channel();
+ if (!channel)
+ return -1;
+
+ ret = rdma_create_id(channel, &id, NULL, RDMA_PS_TCP);
if (ret)
- return ret;
+ goto destroy_channel;
ret = rdma_resolve_addr(id, NULL, (struct sockaddr *) &addr_in,
2000);
if (ret)
goto out;
- ret = rdma_get_cm_event(&event);
+ ret = rdma_get_cm_event(channel, &event);
if (!ret && event->event != RDMA_CM_EVENT_ADDR_RESOLVED)
ret = event->status;
rdma_ack_cm_event(event);
@@ -642,7 +647,7 @@ static int query_for_path(char *dst)
if (ret)
goto out;
- ret = rdma_get_cm_event(&event);
+ ret = rdma_get_cm_event(channel, &event);
if (!ret && event->event != RDMA_CM_EVENT_ROUTE_RESOLVED)
ret = event->status;
rdma_ack_cm_event(event);
@@ -652,6 +657,8 @@ static int query_for_path(char *dst)
test.path_rec = id->route.path_rec[0];
out:
rdma_destroy_id(id);
+destroy_channel:
+ rdma_destroy_event_channel(channel);
return ret;
}
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Where are running examples using libibcm in OFED-1.0.1?
Hi, I'm just trying to port my IB gen1 application to gen2 in order to get it running on a SLES 9 SP3 x86_64 Linux workstation. I got stuck in porting the usage of the libcm in gen1 to libibcm in gen2. My code fails on using the ib_cm_create_id call. Looked around for some running examples using the libibcm but could not find them. Got a pointer to: https://openib.org/svn/gen2/branches/1.0/src/userspace/libibcm/examples/ but those wont compile. Can someone help me out here? Here is some background about my application: Out application is running between a PowerPC Linux gen1 IB source and an x86 Linux PC data destination. All is running fine with gen1 drivers from Mellanox IBGD-1.8.2. In order to get our data destination application running on 64-Bit Workstations with SLES 9 SP3 and above as well I have to port it gen2. Thanks in advance. Thomas Bub Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] gen1 drivers as rpm or tgz?
Title: gen1 drivers as rpm or tgz? Hi, we are planning to replace a HighSpeed GSN (Gigabyte System Network) data interface with IB. We’d like to to use the verbs layer with the Access Layer on top. Unfortunately Mellanox, our Hardware provider, does not supply an Access layer with their IBGold 1.8 stack. Yes I know openIB currently supports only gen2 drivers with 2.6er kernels Unfortunately all our environment is still 2.4er kernels under RedHat EL 3 Update 6. Before my whole development and target environment is on 2.6er kernels I have to wait for another 2 month. In order to speed up things for me as an developer I’d like to start with gen1 openIB drivers. Is there soothing like rpm’s or tgz’s I can use which supplies wole IB stack for 2.4er kernels or an Acess layer extension for Mellanox IBGold stack. I’d like to avoid to download via cvs, which I’m not familiar with yet. Thanks Thomas Bub Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
