Re: [ofa-general] Building IB SAN with Linux without switch

2009-09-21 Thread Dotan Barak
Hi. On Fri, Sep 18, 2009 at 11:41 PM, Roberto Fichera ro...@tekno-soft.it wrote: Hi All in the list, I would like to know if it's possible to configure a linux server with 2 or 3 HCAs, with 2 ports each, so that I can connect 4 or 6 nodes without using any switch in the middle. If possible,

Re: [ofa-general] Building IB SAN with Linux without switch

2009-09-21 Thread Dotan Barak
Hi. Do you have any specific answer you wish to ask or everything is fine now? Dotan ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit

Re: [ofa-general] change mtu

2009-08-17 Thread Dotan Barak
Hi. Which MTU do you try to change? (of the IB link?) Dotan On Mon, Aug 17, 2009 at 3:16 PM, Michael Di Domenicomdidomeni...@gmail.com wrote: How do I change the MTU of an MT23108 card?  I have an AMD 8131 chipset server that needs this turned down below 1500, or atleast that's what's

Re: [ofa-general] change mtu

2009-08-17 Thread Dotan Barak
On Mon, Aug 17, 2009 at 3:52 PM, Michael Di Domenicomdidomeni...@gmail.com wrote: Yes, for the IB card itself, not IPoIB Why do you want to do it? (Do you know that even if the MTU of the link is 2K you can connect the QPs to use MTU of 1K between them?) Dotan

Re: [ofa-general] RE: Running more than 894 processes doing rdma_listen

2009-07-23 Thread Dotan Barak
On Wed, Jul 22, 2009 at 9:05 PM, Sean Heftysean.he...@intel.com wrote: Is there an explicit limit on the number of ports that can be listening using rdma_cm? There's no inherent limit built into the code. Maybe this limitation is being caused by the ulimit (ulimit -n produced 1024 for my

Re: [ofa-general] createQP failed when attr-cap.max_inline_data 470

2009-06-22 Thread Dotan Barak
You are welcome to create a QP with maximum number of sg enties and check the max_inline_data value of the newly created QP. The same library (libmthca) is being used for several HCAs with different capabilities. Dotan On Tue, Jun 23, 2009 at 7:41 AM, Tharindu Rukshan

Re: [ofa-general] ofa_1_5_kernel 20090616-0200 daily build status

2009-06-16 Thread Dotan Barak
Hi Vlad. I think that it will be useful if the email subject will indicate if there was a failure or everything passed. thanks Dotan On Tue, Jun 16, 2009 at 12:22 PM, Vladimir Sokolovsky (Mellanox)v...@lists.openfabrics.org wrote: This email was generated automatically, please do not reply

Re: [ofa-general] Can I assign custom LIDs to a port ?

2009-06-14 Thread Dotan Barak
Hi. LMC is the most similar thing to this question. Dotan On Mon, Jun 15, 2009 at 5:19 AM, Jordanaccept...@gmail.com wrote: Now I need to assign a port with custom LIDs, for example , I want to assign the number 1, 2, 3 to a port . Is there some methods to do so ? It seems that opensm

[ofa-general] There is a dead link in the page: http://www.openfabrics.org/downloads.htm

2009-06-11 Thread Dotan Barak
Hi. When i tried to press the link to The previous version of the Downloads page can be found here, I got the following error page: Not Found The requested URL /OldDownloads.htm was not found on this server. Thanks Dotan ___ general mailing list

Re: [ofa-general] Sending two integers via RDMA_WRITE

2009-05-25 Thread Dotan Barak
Hi. Why do you use ntohl() on the rkey/remote_addr? Which QP type is it? (RC or UC). Did you poll for a completion and check that the status is good? Dotan Zafar Gilani wrote: Hi, I am trying to send two integers (essentially a buf array of type uint32_t) to the server via RDMA_WRITE

Re: [ofa-general] Sending two integers via RDMA_WRITE

2009-05-25 Thread Dotan Barak
Zafar Gilani wrote: Thanks for the reply. I read your name under the Author for may be all the IBV structs/operations at linux.die.net http://linux.die.net. So I am highly impressed by the work (can only dream of it myself). :) thanks ... 1. I don't know why the original author (Roland

Re: [ofa-general] Sending two integers via RDMA_WRITE

2009-05-25 Thread Dotan Barak
RDMA Write doesn't produce any completion in the receiver side. Dotan On Mon, May 25, 2009 at 10:55 PM, Zafar Gilani zafargil...@gmail.com wrote: I am attaching the code files (client.c and server.c). I hope I am not bugging you that much! Is there any guide for IBV/RDMA CM? I am using IBA

Re: [ofa-general] How to establish IB communcation m ore effectively?

2009-05-11 Thread Dotan Barak
You can't find such samples in the verbs library; It can be found in the rdma cma library, you should search for rping or ucmatose. Dotan 2009/5/12 zhouyong...@ict.ac.cn: Hi all,    I'm using libibverbs to build a cluster memory pool, and using TCP/IP handshake to exchange memory information

Re: [ofa-general] Queue pair state for multicast group attachment

2009-05-05 Thread Dotan Barak
I believe that the right QP state to attach it to a multicast group is in INIT state, since it this state you can post receive request too. As soon as you will modify the QP state to RTR the multicast messages will be received by this QP. Dotan On Tue, May 5, 2009 at 2:23 PM, Stefan Roscher

***SPAM*** Re: [ofa-general] ***SPAM*** Will this brick my switch?

2009-04-20 Thread Dotan Barak
On Sun, Apr 19, 2009 at 9:45 PM, Chris Worley worl...@gmail.com wrote: I've got a QDR switch from Mellanox that's a few months old... and has no markings, but looks like the only QDR switch described on their web pages. I went to burn the firmware, but got the message that I shouldn't: #

***SPAM*** Re: [ofa-general] Concurrent RDMA Write w/imm between hosts and bandwidth/latency

2009-03-16 Thread Dotan Barak
Hi. Why won't you connect the QPs to different CQs? This way, you will be able to pull the urgent CQ first, and only if it is empty, poll the non-urgent CQ. Dotan 2009/3/16 Diego Guella diego.gue...@sircomtech.com: Hi all, I am facing this problem: I have 3 hosts A, B, C. Hosts A and B

***SPAM*** Re: [ofa-general] ibv_get_sysfs_path()

2009-03-11 Thread Dotan Barak
Don Kerr wrote: Is the functionality provided by ibv_get_sysfs_path() and ibv_read_sysfs_file() defined in the IB Spec? I don't see any meaning for sysfs other than Linux/Unix. So the answer is No, this is an implementation enhancement/feature. Dotan

Re: [ofa-general] what uses UC in the OFED stack?

2009-03-10 Thread Dotan Barak
Hi. I think that there isn't any ULP in the OFED stack that uses UC. Dotan On Tue, Mar 10, 2009 at 6:45 PM, Ted H. Kim ted@sun.com wrote: Folks, Does anything in the OFED stack use the IB Unreliable Connected (UC) transport type? I thought maybe IPonIB-CM would because of the

[ofa-general] ***SPAM*** Re: mlx4_core 0000:c3:00.0: SW2HW_MPT failed (-16) (dmesg)

2009-03-07 Thread Dotan Barak
Please make sure that you have the latest firmware for the HCA that has the failure. Dotan Phillip Wilson wrote: This is related to the thread I stared on Feb 24 Re: [ofa-general] ***SPAM*** Mellanox ibv_reg_mr (memory region) function call fails under load when using the mlx4 driver So

Re: [ofa-general] puzzled by ATOMIC operation

2009-03-04 Thread Dotan Barak
zhouyong...@ict.ac.cn wrote: Hi, all these days I'm puzzled by a weird mistake in atomic operation, when I use FETCH_AND_ADD, the remote buffer only changed for 9 times at most, aka if the initial value is 0, when I execute 10 times FETCH_AND_ADD operations, the value is 9, and no more

***SPAM*** Re: [ofa-general] ***SPAM*** Mellanox ibv_reg_mr (memory region) function call fails under load when using the mlx4 driver

2009-02-24 Thread Dotan Barak
Do you execute your program under the root user or under any other user? (maybe you fail because of the ulimit value of memory which can be pinned) Dotan On Wed, Feb 25, 2009 at 7:51 AM, Phillip Wilson phillipw...@gmail.com wrote: The “ibv_reg_mr()” function call fails with HCA (DID=0x634A)

***SPAM*** Re: [ofa-general] ***SPAM*** troubleshooting with infinband

2009-02-14 Thread Dotan Barak
Vittorio wrote: thanks for the suggestion, but i can't understand which kind of address i should put for the two commands i tried ibping with the server (like suggested) and it works with -G port or with lid but what should i put as argument of ibv_rc_pingpong and rping? thanks a lot

***SPAM*** Re: [ofa-general] ***SPAM*** troubleshooting with infinband

2009-02-13 Thread Dotan Barak
Vittorio wrote: Hello! This is my first message on the list so i hope that i'm not going to ask silly or already answered question i'm a student and i'm porting an electromagnetic field simulator to a parallel and distributed linux cluster for final thesis; i'm using both OpenMP and MPI

***SPAM*** Re: [ofa-general] non zero lkey in send(), write() with num_sge 1?

2009-02-13 Thread Dotan Barak
Bill N wrote: Can stack pass num_sge 1, and lkey !=0 as part of sg_list[] elements, in post_send() call? What are you trying to achieve? [Bill] I just wanted to confirm, that even when Stag !=0, (a) there can be multiple SGEs in the list with different lkey and TO.

Re: [ofa-general] ib_create_qp and ib_get_err_str weirdness

2009-02-13 Thread Dotan Barak
Hi. Diego Guella wrote: Hello, I am using Mellanox WinOF 2.0.0 with a MHES14-XTC SDR single-port card. I noticed a strange behavior of ib_create_qp function: - memset(qp_create, 0, sizeof(qp_create)); qp_create.qp_type = IB_QPT_RELIABLE_CONN; // Reliable Connected qp_create.sq_depth =

***SPAM*** Re: [ofa-general] ***SPAM*** non zero lkey in send(), write() with num_sge 1?

2009-02-09 Thread Dotan Barak
Ofed User wrote: Hi, Can stack pass num_sge 1, and lkey !=0 as part of sg_list[] elements, in post_send() call? What are you trying to achieve? If num_sge 1 = the HCA will try to read the blocks pointed by the sg_list one by one and validate that the address + size is inside a valid

***SPAM*** Re: [ofa-general] ***SPAM*** Byte_Cnt field in the MTHCA_CQE structure

2009-01-27 Thread Dotan Barak
On Mon, Jan 26, 2009 at 11:08 PM, Adit Ranadive adit@gmail.com wrote: Hello, I have been looking at doing some low level work with the OFED library 1.1 in terms of figuring out how many bytes have been sent by an IB application. If you have the source code, you can add a counter when

Re: [ofa-general] Does ib0 always map to port1?

2009-01-22 Thread Dotan Barak
On Thu, Jan 22, 2009 at 9:45 AM, Or Gerlitz ogerl...@voltaire.com wrote: Sumeet Lahorani wrote: I see that ib0 always maps to port1 and ib1 always maps to port2 on the HCA. I'm trying to find out if this will always be the case and if so which script ensures this mapping? Yes, on a dual

Re: [ofa-general] Does ib0 always map to port1?

2009-01-22 Thread Dotan Barak
0x1e). Is there another command (apart from ifconfig) I can use to get a better match? - Sumeet David Dillow wrote: On Thu, 2009-01-22 at 10:18 +0200, Dotan Barak wrote: On Thu, Jan 22, 2009 at 9:45 AM, Or Gerlitz ogerl...@voltaire.com wrote: Sumeet Lahorani wrote: I see that ib0

***SPAM*** Re: [ofa-general] local DMA transfer?

2009-01-17 Thread Dotan Barak
Yicheng Jia wrote: Hi Folks, Is it possible to do local DMA transfer using QPs in a single port HCA? Thanks! Best, Hi. If the question is can i use DMA from local QPs (both QPs are in the same HCA), then the answer is yes. The QPs can be located everywhere, even in the same HCA, or

***SPAM*** Re: [ofa-general] local DMA transfer?

2009-01-17 Thread Dotan Barak
On Sat, Jan 17, 2009 at 9:38 PM, Yicheng Jia y...@tmriusa.com wrote: Hi Dotan, Does HCA provide any internal route for local DMA so that local data transfer doesn't has to go out of the HCA port as regular QPs do? In another word, it's not efficient to use QPs for local DMA transfer, is it

Re: [ofa-general] ***SPAM*** server/client userspace application ibv_alloc_pd seg. fault

2009-01-12 Thread Dotan Barak
Maybe you are using the wrong device context... Did you open the device? You are welcome to send all of the code before calling ibv_alloc_pd, i think this will help understand what went wrong ... Dotan On Tue, Jan 13, 2009 at 6:41 AM, अनुज anu...@gmail.com wrote: Hi I am trying to run

Re: [ofa-general] ibv_post_send returns -1

2008-12-29 Thread Dotan Barak
On Sat, Dec 27, 2008 at 10:17 PM, Bharath Ramesh bram...@vt.edu wrote: Thanks for the reply Dotan, I am posting what I am doing currently, probably I am missing something which you could point out to me. 1) I don't have completion notifications or completion channel associated with my send

Re: [ofa-general] ibv_post_send returns -1

2008-12-26 Thread Dotan Barak
Bharath Ramesh wrote: I using OFED-1.2 for my research project. I am facing an issue where when I try to post IBV_WR_RDMA_WRITE work request to the QP ibv_post_send returns with -1. This I presume means lack of resources to complete the operation. To give a brief summary of my communication

***SPAM*** Re: [ofa-general] ***SPAM*** libibverb-utils.

2008-12-06 Thread Dotan Barak
On Fri, Dec 5, 2008 at 6:20 PM, Jaiwant Mulik [EMAIL PROTECTED] wrote: Hi all, Do the ibv_rc_pingpong, ibv_srq_pingpong, ibv_uc-pingpong and ibv_ud_pingpong utilities work only for IB HCAs and not for iWARP cards? Since those examples do not connect using the CMA, they are not expected to

***SPAM*** Re: [ofa-general] set up QPs with different transfer rate

2008-11-28 Thread Dotan Barak
Yicheng Jia wrote: Hi Folks, I have two applications which require different IB transfer rates. I am using Mellanox 25204 HCA. Can I achieve it by setting up two QPs with different service levels? Can I set SL field in QP context, or it is controlled by SM? Thanks! You can set the SL value

Re: [ofa-general] infiniband multicast (libibverbs)

2008-11-04 Thread Dotan Barak
Kelly Burkhart wrote: I'm experimenting with multicast and am having an interesting issue. The setup is ripped mostly from ib_send_lat.c. I have a client which sends and a server which reads. All sends/recieves use a 2048 byte message. The client can send any number of messages at any message

Re: [ofa-general] Question about ibv_asyncwatch

2008-10-28 Thread Dotan Barak
Karthik Gopalakrishnan wrote: Hi Folks. I have written a standalone program that calls 'ibv_get_async_event()'. I want to know if that program can get async events about errors on QPs (IBV_EVENT_PATH_MIG_ERR for example) that are created by a different process (say some MPI Program). I also

Re: [ofa-general] Bug with SDP on IA64

2008-10-27 Thread Dotan Barak
On Mon, Oct 27, 2008 at 11:09 AM, Nicolas Morey Chaisemartin [EMAIL PROTECTED] wrote: Amir Vadai a écrit : I asked our IB expert Jack for hints and he told me this: From Section 11.6.2 (COMPLETION RETURN STATUS0 of the IB Spec volume 1, revision 1.2.1 * Local Length Error - ... Generated

Re: [ofa-general] ibverbs help

2008-10-24 Thread Dotan Barak
Kelly Burkhart wrote: Hello, I have written a simple program as my first foray into verbs programming. The intent is to create a connection and do a single send. The logic (ripped primarily from rc_pingpong) is this: get device ibv_open_device ibv_alloc_pd allocate buffer ibv_reg_mr

Re: [ofa-general] ibverbs help

2008-10-24 Thread Dotan Barak
Since I've only been at this a few days, I'm quite certain you're right. Unfortunately I don't know what to look for to diagnose the problem. I'll continue to bang my head against this, in the mean time, if any of you could suggest possible places for me to look it would be much appreciated.

***SPAM*** Re: [ofa-general] Querying the number of open queue pairs

2008-10-17 Thread Dotan Barak
Krishnamoorthy, Sriram wrote: I am trying to figure out how many more queue pairs can be created on a device at some point (say after MPI has been initialized). ibv_query_device() returns, among other things, the maximum number of queue pairs that can be created. In OFED 1.3, is there a way

[ofa-general] ***SPAM*** [PATCH] libibverbs: Update Dotan's email in all of the files

2008-10-17 Thread Dotan Barak
Update Dotan's email in all of the files. Signed-off-by: Dotan Barak [EMAIL PROTECTED] --- diff --git a/AUTHORS b/AUTHORS index e2dc298..fcea350 100644 --- a/AUTHORS +++ b/AUTHORS @@ -1,4 +1,4 @@ Roland Dreier [EMAIL PROTECTED] -Dotan Barak[EMAIL PROTECTED] +Dotan Barak

***SPAM*** Re: [ofa-general] ***SPAM*** cleanup resources using uverbs

2008-10-13 Thread Dotan Barak
अनुज wrote: Hi I have used user space server and client programs for simple data transfer (rdma write, send and receive) by using uverbs (libibverbs). is it required to cleanup all the resources (pd, qp, cq, mr etc. ) allocated with in these programs explicitly by using uverbs

***SPAM*** Re: [ofa-general] ***SPAM*** Synchronous access of RDMA memory

2008-09-26 Thread Dotan Barak
Ganesh Sadasivan wrote: On a memory region setup for RDMA, is it possible for the local CPU also to write this memory in a synchronous way e,g by doing ibv_post_send? CPU can access the memory locally, so it don't need any IB access. If the question was: can an HCA write to a local

***SPAM*** Re: [ofa-general] atomic operations on ppc64

2008-09-25 Thread Dotan Barak
The problem that you describes is pretty basic and even an RC shouldn't have this issue. Agreed! I think that you should upgrade the HCA's Firmware. as Ronni suggested. But I'm not sure about the fw version. As I mentioned, on that Mellanox page the latest firwmare for the

Re: [ofa-general] atomic operations on ppc64

2008-09-22 Thread Dotan Barak
: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rui Machado Sent: ו 19 ספטמבר 2008 17:39 To: Dotan Barak Cc: general@lists.openfabrics.org Subject: Re: [ofa-general] atomic operations on ppc64 2008/9/19 Dotan Barak [EMAIL PROTECTED]: It seems that you didn't check the same HCA

***SPAM*** Re: [ofa-general] atomic operations on ppc64

2008-09-19 Thread Dotan Barak
Bugs can be anywhere: application / Driver / HW ... Can you try to use server in x86 and client in PPC64 and then server in PPC64 and client in x86? Which OFED version do you use? Can you send the output of ibv_devinfo? Dotan I tried the combination ppc64-x86 and x86-ppc64. The

***SPAM*** Re: [ofa-general] atomic operations on ppc64

2008-09-19 Thread Dotan Barak
It seems that you didn't check the same HCA in each arch. Can you try to use the mthca device in PPC64 and check the results? (Anyway, i would have suggest you to upgrade the OFED package that you are using) I apologize for my newbieness here but I do not understand what you mean. The

Re: [ofa-general] atomic operations on ppc64

2008-09-18 Thread Dotan Barak
Rui Machado wrote: 2008/9/17 Rui Machado [EMAIL PROTECTED]: From: Rui Machado [EMAIL PROTECTED] Date: 2008/9/17 Subject: Re: [ofa-general] atomic operations on ppc64 To: Dotan Barak [EMAIL PROTECTED] 2008/9/17 Dotan Barak [EMAIL PROTECTED]: On Wed, Sep 17, 2008 at 5:54 PM, Rui

Re: [ofa-general] atomic operations on ppc64

2008-09-17 Thread Dotan Barak
On Wed, Sep 17, 2008 at 5:12 PM, Rui Machado [EMAIL PROTECTED] wrote: Hi list, does anyone have experienced problems using IB atomic operations (fetch and add) on a ppc64 platform? I tried a small example (using fetch and add) on x86 and ppc64 and on x86 worked fine while on ppc64 didn't.

Re: [ofa-general] ***SPAM*** Interrupt RDMA Read

2008-09-04 Thread Dotan Barak
, Dotan Barak [EMAIL PROTECTED] wrote: As much as i know, once you posted a WR, you can not cancel it. The only thing that you can do is flush the whole QP by changing the QP state to ERROR (which flushes the work Queues and produces completion for every WR) or to RESET, which cleans the Queues

Re: [ofa-general] ***SPAM*** Interrupt RDMA Read

2008-09-01 Thread Dotan Barak
As much as i know, once you posted a WR, you can not cancel it. The only thing that you can do is flush the whole QP by changing the QP state to ERROR (which flushes the work Queues and produces completion for every WR) or to RESET, which cleans the Queues from the WRs. Dotan On Sun, Aug 31,

Re: [ofa-general] Efficient management of many connections

2008-08-28 Thread Dotan Barak
On Thu, Aug 28, 2008 at 6:52 PM, Adrien Guillon [EMAIL PROTECTED] wrote: Hi, I am writing some code for a cluster, and I'm using RDMA. I would like each node to be able to access memory of every other node, which requires N-1 connections for N nodes on each. What can I do to implement this

Re: [ofa-general] ***SPAM*** ibv_qp_cap- sensible settings for the parameters

2008-08-28 Thread Dotan Barak
On Thu, Aug 28, 2008 at 6:48 PM, Adrien Guillon [EMAIL PROTECTED] wrote: Hi, I don't quite understand the parameters inside struct ibv_qp_init_attr. In particular the ibv_qp_cap structure... what are sensible defaults? How do these parameters affect overall functioning of the application?

Re: [ofa-general] ***SPAM*** Thread Safety and Infiniband Verbs and RDMA

2008-08-28 Thread Dotan Barak
On Thu, Aug 28, 2008 at 6:45 PM, Adrien Guillon [EMAIL PROTECTED] wrote: Hey all, I'm working on my first RDMA application in C++ (hooray). Are functions from librdmacm and infiniband verbs thread safe, i.e. can many threads be posting work on the same connection at the same time, or do I

***SPAM*** Re: [ofa-general] ***SPAM*** ibv_qp_cap- sensible settings for the parameters

2008-08-28 Thread Dotan Barak
On Thu, Aug 28, 2008 at 10:11 PM, Adrien Guillon [EMAIL PROTECTED] wrote: What happens if any of these values are too low? If you'll try to post a SR/RR with more scatter/gather entries than QP was created with, you'll get an immediate error (QP state won't be changed). If you'll try to post

***SPAM*** Re: [ofa-general] Efficient management of many connections

2008-08-28 Thread Dotan Barak
On Thu, Aug 28, 2008 at 10:13 PM, Adrien Guillon [EMAIL PROTECTED] wrote: I still allocate a separate CQ for each QP right? You may use a separate CQ for each QP, if this is what you want (or you can use one CQ for all of those QPs). Also, I read http://www.hpcwire.com/features/17886984.html

***SPAM*** Re: [ofa-general] Efficient management of many connections

2008-08-28 Thread Dotan Barak
You may get CQ overrun if you are not carfull enough ... Using one CQ can make your life easier, but i suggest to use different CQ for RQ and SQ. (The SQ is the only queue that you can control it's message rate...) Dotan On Thu, Aug 28, 2008 at 10:21 PM, Adrien Guillon [EMAIL PROTECTED] wrote:

Re: [ofa-general] ***SPAM*** RDMA Operations and Endian-ness

2008-08-21 Thread Dotan Barak
On Thu, Aug 21, 2008 at 6:43 AM, Adrien Guillon [EMAIL PROTECTED] wrote: Hey all, When I do RDMA operations between two nodes, I notice that atomic operations will automatically translate the values for endian-ness according to the spec. Is the same true with other RDMA operations, or do I

***SPAM*** Re: [ofa-general] minimum sw components requirement for driver/opensm in a single unmanaged switch network

2008-08-21 Thread Dotan Barak
On Thu, Aug 21, 2008 at 10:16 PM, Yicheng Jia [EMAIL PROTECTED] wrote: Hi Hal, Can opensm just run once? When the subnet is up, it can exit assume that no change will be made in the subnet. Yes, depend on the serives that you will need/use. For example: if you use operations that requires

***SPAM*** Re: [ofa-general] minimum sw components requirement for driver/opensm in a single unmanaged switch network

2008-08-21 Thread Dotan Barak
was down... If you'll connect the QPs using the sockets and you won't depend on other ULP (such as IPoIB, SDP or any other) you will be fine .. Dotan Thanks! Yicheng *Dotan Barak [EMAIL PROTECTED]* 08/21/2008 02:33 PM To Yicheng Jia [EMAIL PROTECTED] cc Hal Rosenstock

Re: [ofa-general] ***SPAM*** Error returned by ibv_poll_cq()

2008-08-18 Thread Dotan Barak
Hi. Did you try to perform (unsuccessful) RDMA operation to this side? Dotan On Mon, Aug 18, 2008 at 6:48 PM, Karthik Gopalakrishnan [EMAIL PROTECTED] wrote: Hello. ibv_poll_cq() returns IBV_WC_REM_ACCESS_ERR or IBV_WC_WR_FLUSH_ERR in wc.status and IBV_WC_SEND in wc.opcode. The async event

Re: [ofa-general] ***SPAM*** Error returned by ibv_poll_cq()

2008-08-18 Thread Dotan Barak
Karthik Gopalakrishnan wrote: Yes. There was an unsuccessful RDMA operation to this side. You should try to check that the r_key + address + size of the send request that you posted is matching a valid MR with the right permissions and size ... Dotan On Mon, Aug 18, 2008 at 2:22 PM, Dotan

Re: [ofa-general] ***SPAM*** Accessing RDMA Memory Locally

2008-08-15 Thread Dotan Barak
Adrien Guillon wrote: If I allocate memory to be accessible by others using RDMA operations, my understanding is that I use RDMA operations myself to access that memory locally. Is that correct? No, the memory is local memory for your process which you can allocate using malloc (or even

Re: [ofa-general] RDMA n00b: Remote Memory Access and Connection Setup Help

2008-08-07 Thread Dotan Barak
Adrien Guillon wrote: Hello List, I've been reading the Infiniband Architecture specification, and I'm ready to write my first RDMA-enabled C++ library. I have looked at some sample code provided by librdmacm, and I'm having a problem going from the big picture of what I want to do to the

Re: [ofa-general] limit on memory registration

2008-08-07 Thread Dotan Barak
Rui Machado wrote: I have a feeling that you refer to the value of max_mr (am i right?) :) yep sorry. The value for max_mr_size is 18446744073709551615 (can this one be ? ) Again, how do I decode this? Mellanox ca type:25218 (vendor_part_id) fw_version : 5.1.400 (fw_ver)

Re: [ofa-general] RDMA n00b: Remote Memory Access and Connection Setup Help

2008-08-07 Thread Dotan Barak
Adrien Guillon wrote: Could you give me the verb that is used to write to remote memory? I just don't seem to be able to find it, but each time I skim through the spec I learn a bit more :-) Yes, the verb is : ibv_post_send, you should use the RDMA Write opcode. (it add a job to the work

Re: [ofa-general] limit on memory registration

2008-08-07 Thread Dotan Barak
Rui Machado wrote: Withouth wanting to abuse on your generosity :) : :) yep sorry. The value for max_mr_size is 18446744073709551615 (can this one be ? ) Again, how do I decode this? What about the huge number above? It is much bigger than 16GB (my current limit). Does this mean

Re: [ofa-general] RDMA n00b: Remote Memory Access and Connection Setup Help

2008-08-07 Thread Dotan Barak
AJ Guillon wrote: Yes, the verb is : ibv_post_send, you should use the RDMA Write opcode. (it add a job to the work queue Thanks. When I use RDMA read/write on a remote system, does that remote system have to do anything (like poll for events) or are the RDMA operations truly transparent

Re: [ofa-general] RDMA Write Error

2008-08-06 Thread Dotan Barak
Philip Frey1 wrote: Hi, I am trying to figure out how efficient MR registration followed by an RDMA write is. For that matter I am running the following loop: // create MR of size 64KB for (i = 0; i max_writes; i++) { // destroy old MR // create MR of size 64KB // RDMA write

Re: [ofa-general] limit on memory registration

2008-08-06 Thread Dotan Barak
Rui Machado wrote: Hi all, is there any limitation on the size that can be registered (ibv_reg_mr) for communication? I seem to be limited to 16GB (on 32GB 64bit x86 machine). Is this normal? Can someone tell me why and/or if there is a workaround? In the device attributes there is an

Re: [ofa-general] limit on memory registration

2008-08-06 Thread Dotan Barak
Rui Machado wrote: Hi Dotan is there any limitation on the size that can be registered (ibv_reg_mr) for communication? I seem to be limited to 16GB (on 32GB 64bit x86 machine). Is this normal? Can someone tell me why and/or if there is a workaround? In the device attributes there

Re: [ofa-general] limit on memory registration

2008-08-06 Thread Dotan Barak
The module parameter num_mtt control the size of the above described table. The default value is (1 20), you might try some higher value than this ... Dotan ___ general mailing list general@lists.openfabrics.org

Re: ***SPAM*** Re: [ofa-general] ***SPAM*** OFED-1.3 RDMA CM, IB_ACCESS_LOCAL_WRITE flag missing

2008-08-03 Thread Dotan Barak
On Sat, Aug 2, 2008 at 1:02 PM, Devesh Sharma [EMAIL PROTECTED] wrote: Thanks for replying, Can you explain me in a bit more detail, because if QP dose not have a IB_ACCESS_LOCAL_WRITE permission, according to IB spec, HCA should generate Local Protection Error while processing the WRs. Is it

Re: [ofa-general] Non-Signaled RDMA Read

2008-07-29 Thread Dotan Barak
Philip Frey1 wrote: Hello, I am trying to post a non-signaled RDMA Read WR. Somehow OFED 1.3 running on a T3 Chelsio Adapter seems not to care what I set for 'enum ibv_send_flags' in my 'struct ibv_send_wr'. I always get a work completion with status set to IBV_WC_SUCCESS and opcode

***SPAM*** Re: [ofa-general] Terrible TCP Qperf Performance with Centos 5.2 and default settings

2008-07-25 Thread Dotan Barak
Robert Dunkley wrote: I see very low qperf TCP performance of IPOIB on Centos 5.2 by default (RDMA performance is fine). Which HW do you use IB or IWarp? thanks Dotan Centos sets the maximum MTU for ib0 on boot automatically. For CM centos sets the maximum MTU that is just slightly less

Re: [ofa-general] Non-blocking send

2008-07-22 Thread Dotan Barak
Hi. On Tue, Jul 22, 2008 at 1:14 PM, Antonino Castelfranco [EMAIL PROTECTED] wrote: Hi. I'm trying to understand how to use a non blocking send. Does someone know the simplest way to do it? The scenario is the following: Node A send 10 messages to Node B in a non-blocking way. Sometimes

Re: [ofa-general] Howto identify master SM

2008-07-19 Thread Dotan Barak
Yevgeny Kliteynik wrote: Matthias Blankenhaus wrote: Howdy ! I was not able to find a tool that lets me identify the master or standby on a fabric. I have used ibsysstat, smpquery and smpdump, but that did not get me any closer. How can I identify a master / standby SM from CLI ? sminfo

[ofa-general] ***SPAM*** [PATCH] core: Fixed several coding style issues according to checkpatch.pl

2008-07-18 Thread Dotan Barak
place in a switch statement Signed-off-by: Dotan Barak [EMAIL PROTECTED] --- diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c index 09a2bec..0a81fac 100644 --- a/drivers/infiniband/core/addr.c +++ b/drivers/infiniband/core/addr.c @@ -364,9 +364,8 @@ static int

[ofa-general] ***SPAM*** [PATCH] core/iwcm: remove IB_ACCESS_LOCAL_WRITE in remote QP attributes

2008-07-18 Thread Dotan Barak
IB_ACCESS_LOCAL_WRITE was removed for qp.qp_access_flags because this attribute is relevant only for remote permissions. Signed-off-by: Dotan Barak [EMAIL PROTECTED] --- diff --git a/drivers/infiniband/core/iwcm.c b/drivers/infiniband/core/iwcm.c index 81c9195..53f53fc 100644 --- a/drivers

Re: [ofa-general] IB verbs client/server programming...

2008-07-15 Thread Dotan Barak
Hi. On Mon, Jul 14, 2008 at 7:51 PM, Peter W. Morreale [EMAIL PROTECTED] wrote: Hello all, I'm starting a new client/server project using RDMA and I'm a little stuck at the moment. I've seen Roland Dreier's slides/client-server example as well as the cmatose.c example in the OFED 1.3

Re: [ofa-general] IB verbs client/server programming...

2008-07-15 Thread Dotan Barak
On Tue, Jul 15, 2008 at 4:07 PM, Peter W. Morreale [EMAIL PROTECTED] wrote: On Tue, 2008-07-15 at 09:39 +0300, Dotan Barak wrote: In other words, what interfaces do I have to look at to implement the blocking portion equivalent of a recv(2)? Do you want to block until the data

***SPAM*** Re: [ofa-general] Infinband and DRBD

2008-07-15 Thread Dotan Barak
(which is a little bit better than 1.3). BUT I think that OFED 1.3 is good enough for your needs ... Dotan Rob -Original Message- From: Dotan Barak [mailto:[EMAIL PROTECTED] Sent: 13 July 2008 07:25 To: Robert Dunkley Cc: general@lists.openfabrics.org Subject: Re: [ofa-general

Re: [ofa-general] IB verbs client/server programming...

2008-07-15 Thread Dotan Barak
Peter W. Morreale wrote: On Tue, 2008-07-15 at 17:07 +0300, Dotan Barak wrote: On Tue, Jul 15, 2008 at 4:07 PM, Peter W. Morreale [EMAIL PROTECTED] wrote: On Tue, 2008-07-15 at 09:39 +0300, Dotan Barak wrote: In other words, what interfaces do I have to look at to implement

***SPAM*** Re: [ofa-general] Infinband and DRBD

2008-07-13 Thread Dotan Barak
I don't know which OFED version is installed in the Distribution version that you are using, but i think (as a general rule, at least at this stage): users should install the latest OFED release if this version number is higher than the one that originally being installed with your distribution.

Re: ***SPAM*** Re: [ofa-general] Getting Started with RDMA

2008-07-11 Thread Dotan Barak
Adrien Guillon wrote: Alright, thanks for helping the n00b everyone. How about IB CM? The page says: Use of the IB CM requires knowledge the IB CM protocols defined in Chapter 12 of the IB Architecture Spec Release 1.2. How difficult is it in reality to use this library? I assume that

Re: [ofa-general] Getting Started with RDMA

2008-07-10 Thread Dotan Barak
Adrien Guillon wrote: On Tue, Jul 8, 2008 at 7:02 AM, Or Gerlitz [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Please note that you have to have a netdevice up and running for the IP subnet addresses you would be using with the rdma-cm. In the IB case, this means IPoIB and in

Re: [ofa-general] Mellanox HCA issue (Dell)

2008-07-04 Thread Dotan Barak
Hi. [EMAIL PROTECTED] wrote: port: 1 state: PORT_INIT (2) phys_state: LINK_UP (5) port: 2 state: PORT_DOWN (1)

Re: [ofa-general] Re: [RFC PATCH] - scripts/CodingStyle_sizeof - standardize sizeof and [PATCH] drivers/infiniband - standardize sizeof(foo)

2008-07-01 Thread Dotan Barak
Roland Dreier wrote: This seems like needless churn to me. I personally prefer to see sizeof foo instead of sizeof(foo), since sizeof is not a function, and in any case I don't see much value in making this type of trivial change. In the IB stack i noticed all of the variations of sizeof:

Re: [ofa-general] IBV_WC_RETRY_EXC_ERR causes

2008-06-21 Thread Dotan Barak
Krishnamoorthy, Sriram wrote: IBV_WC_RETRY_EXE_ERR means that there wasn't any ack by the receiver after 4.096*(2 power 18) * 7 usec. Does an ack from the receiver require the process/thread to be awake? I have been trying to get a small test case, and sleeping without posting

Re: [ofa-general] maximum IB/iWARP max_inline_data size

2008-06-21 Thread Dotan Barak
Jeff Squyres wrote: If you ibv_create_qp() with max_inline_data==0: - Mellanox ConnectX reports a number that is a formula based on several values, such as max_send_sge and max_recv_sge. But its actual max inline value (I think Pasha told me) is 1k. - Chelsio T3 always reports a fixed

Re: [ofa-general] IBV_WC_RETRY_EXC_ERR causes

2008-06-20 Thread Dotan Barak
Krishnamoorthy, Sriram wrote: Can someone please explain what can cause IBV_WC_RETRY_EXC_ERR? I am using a combination of send-receive and RDMA. I have the reliable connection queue pairs initialized as: IBV_WC_RETRY_EXE_ERR means that there wasn't any ack by the receiver after 4.096*(2

Re: [ofa-general] SDP and epoll vs select()

2008-06-08 Thread Dotan Barak
Hi. Bradley Kite wrote: Hi all, Currently my application uses the Linux kernel's epoll interface for socket event notifications. From what I've read it looks like the SDP library only works with select()/poll() - is this actually the case or will epoll work too? I reviewed the libsdp code

Re: [ofa-general] IB/ehca: Reject send WRs only for RESET, INIT and RTR state

2008-06-06 Thread Dotan Barak
Hi. I checked the code in the ehca driver and noticed that post RR to a QP is being accepted in any state (including the RESET state). thanks Dotan Joachim Fenkes wrote: Signed-off-by: Joachim Fenkes [EMAIL PROTECTED] --- drivers/infiniband/hw/ehca/ehca_reqs.c |6 -- 1 files changed,

***SPAM*** Re: [ofa-general] RNIC resource limits

2008-06-04 Thread Dotan Barak
Hi. Philip Frey1 wrote: Hello, I have asked this question about RNIC resource limits before: snip Could you give me some insight in what the limits of the Chelsio RNIC are? (Max MRs, QPs, PDs etc) Many thanks and kind regards, Philip snip Try running ibv_devinfo -v to see

Re: [ofa-general] Length of inbound RDMA send

2008-05-30 Thread Dotan Barak
Hi. Philip Frey wrote: Hello, I was wondering if a receive work completion tells me how many bytes had been placed. Is 'byte_len' the field indicating that value? The various fields of 'struct ibv_wc' are not quite clear to me. Can you point me to a document where this is described? But you

Re: [ofa-general] MLX HCA: CQ request notification for multiple completions not implemented?

2008-05-28 Thread Dotan Barak
size and blocks are the same, 1k, 1k, 2k, 1k Do you have a good solution for such kind of problem? How many QPs do you use? (and how outstanding WR from every QP?) Dotan Best, Yicheng *Dotan Barak [EMAIL PROTECTED]* 05/23/2008 01:27 PM To Yicheng Jia [EMAIL PROTECTED] cc

Re: [ofa-general] Multicast Performance

2008-05-28 Thread Dotan Barak
Marcel Heinz wrote: Marcel Heinz wrote: Dotan Barak wrote: Do you use the latest released FW for this device? The HCAs all use Mallanox' latest released FW version 1.2.0. I'll have a look at the switch later. The Switch is Mellanox MT47396 based and uses FW version 1.0.0

Re: [ofa-general] MLX HCA: CQ request notification for multiple completions not implemented?

2008-05-28 Thread Dotan Barak
on what you are trying to do ... You don't have to ask for completion for any SR that you post, this way you can do some optimization.. Dotan Best, Yicheng *Dotan Barak [EMAIL PROTECTED]* 05/28/2008 12:24 PM To Yicheng Jia [EMAIL PROTECTED] cc general

  1   2   3   >