Re: [ewg] [PATCHv8 07/11] ib_core: Add API to support IBoE from userspace
Address mapping policies depend on the address type. This patch only defines a policy for mapping link-local addresses, and we should indeed take care not to change it (if possible). Later on, we can add more policies for other address types (e.g., normal IPv6 addresses, mapped IPv4 addresses, etc.) Or maybe it is device-specific, and we could wrap it up into the create AH uverbs call we already have? Interesting idea. Not sure what is better here: a seperate ABI call or some additional 'u8 ctx[32]' field in struct ib_uverbs_create_ah_resp that will be interpreted by the hw-specific user-space driver. I don't understand why mapping remote addresses should be driver specific. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] [PATCHv8 07/11] ib_core: Add API to support IBoE from userspace
In this case it will not be just mapping remote addresses but creating
the required AH information which is unique to each device.
I understand that the AH is per device. What I don't get is why we would want
each device to perform the mapping. We don't expect the device to map GIDs to
LIDs when creating an AH. The user must have done the mapping beforehand. The
mapping does not depend on which device I'm using, it should be the same.
Basically, what I want to understand is why does this change make sense?
@@ -1139,6 +1139,10 @@ struct ib_device {
struct ib_grh *in_grh,
struct ib_mad *in_mad,
struct ib_mad *out_mad);
+ int(*get_eth_l2_addr)(struct ib_device *device,
u8 port,
+ union ib_gid *dgid, int
sgid_idx,
+ u8 *mac, u16 *vlan_id, u8
*tagged);
+
- Sean
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] [ANNOUNCE] OFED 1.5.2 beta1 is available
Main changes from 1.5.1: === 1. Updated packages: - Management Using latest daily builds from http://www.openfabrics.org/downloads/management/daily - Updated libnes libnes-1.0.1-0.1.g89ea0ee.tar.gz - Updated libsdp libsdp-1.1.101-0.3.gc767eee.tar.gz - Updated perftest perftest-1.2.4-0.15.g82b7e29.tar.gz - Updated mpitests mpitests-3.2-923.src.rpm 2. Added RHEL5.5 support 3. Use files under /etc/modprobe.d/ instead of /etc/modprobe.conf 4. Bug fixes I would like to add support for IB ACM path resolution (i.e. user space path record caching) into the 1.5.2 release. Vlad pulled in the necessary kernel patch from 2.6.33 a few days ago. All other changes are in user space -- an updated librdmacm release and a release of ibacm -- and must be enabled to be used. I hope to complete testing this week. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] broken support for UC in rdma_cm
I just noticed that Vlad already opened a bugzilla bug (1874) on this. I quote Sean's response: RDMA CM supports UD and RC QPs (port spaces UDP/TCP) only. Support for UC QPs should come from another port space. This makes sense to me. Still we need to address the issues I raised below. Sean, are you going to fix this? This is really a new feature request, which I may not have time to address. It depends on the scope of the changes that are needed. If both sides try using UC QPs, do you know what specific setting causes the connection to fail? - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] [PATCH] [for-2.6.33] rdma/cm: revert associating an RDMA device when binding to loopback
OK, I'm planning on sending this upstream later today. Looks very small and simple, and then we can figure our what if anything we want to do for 2.6.34. Make sense for everyone? yes - thanks ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] rdma/cm: disallow loopback address for iwarp devices
Sean, can you try openmpi? It fails for me, and yet ucmatose succeeds.
I don't understand the difference yet...
I believe the issue is that rdma_bind_addr succeeds (returns 0), but no device
is assigned to the rdma_cm_id (verbs field is NULL).
This was a change from commit 6f8372b69c3198e06cecb1df2cb9682d0c55e657:
The defined behavior of rdma_bind_addr is to associate an RDMA
device with an rdma_cm_id, as long as the user specified a non-
zero address. (ie they weren't just trying to reserve a port)
Currently, if the loopback address is passed to rdma_bind_addr,
no device is associated with the rdma_cm_id. Fix this.
There are two places where rdma_bind_addr() is called in the openmpi source code
(based on a tarball download of the trunk). One is btl_openib_iwarp.c:
rc = rdma_bind_addr(cm_id, ipaddr);
if (rc || !cm_id-verbs) {
rc = OMPI_SUCCESS;
goto out3;
}
The other is btl_openib_connect_rdmacm.c, but that deals with listening. I
can't quickly determine if btl_openib_iwarp.c is usually used for IB or not.
So, to fully keep the behavior of 2.6.32, rdma_bind_addr for 127.0.0.1 should
succeed, but not assign a device. I think this was the change from commit
..c55e657 that changed the behavior:
@@ -2089,7 +2096,9 @@ int rdma_bind_addr(struct rdma_cm_id *id, struct sockaddr
*addr)
if (!cma_comp_exch(id_priv, CMA_IDLE, CMA_ADDR_BOUND))
return -EINVAL;
- if (!cma_any_addr(addr)) {
+ if (cma_loopback_addr(addr)) {
+ ret = cma_bind_loopback(id_priv);
+ } else if (!cma_zero_addr(addr)) {
ret = rdma_translate_ip(addr, id-route.addr.dev_addr);
if (ret)
goto err1;
I'll see if reverting this gives the desired(?) behavior.
- Sean
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] bug 1918 - openmpi broken due to rdma-cm changes
Also note that trying to bind rdma cm to all interface ip addresses was the way that we were advised by openfabrics to figure out which devices are rdma- capable. As such, it is highly desirable to get the fix transparently in rdmacm and preserve the old semantic. More specifically, it seems undesirable to change this semantic in a minor ofed point release. I think the issue is larger than just the rdma_cm. First, it sounds like openmpi tries to bind to 127.0.0.1, which now works. If opemmpi uses shared memory for connections on the same machine, I'm not sure why this is a problem, unless it is passing that address to another machine to use for a connection. If this is the case, then that is a bug in openmpi. Second, I still don't understand whether iwarp is limited to 'loopback' connections that are not bound to 127.0.0.1. For instance, if the RDMA device is associated with 192.168.0.1, then can it handle a connection from 192.168.0.1 - 192.168.0.1? If it can't, then the rdma_cm can't help in this case when bind is called. The failure has to come during connect, which sounds like the behavior that's seen today with 127.0.0.1. So, while the rdma_cm can fail binds to 127.0.0.1 if the RDMA device doesn't support loopback, I'm still not sure how much of a fix this is. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED-1.5.1 failure over iWarp
We should work to get this 'correct' when merging upstream. Following the spirit of the current code, it is probably cma_acquire_dev()'s job to fill in the missing ibdev type information after matching the netdev to an ibdev. This makes sense to me. P.S. - I really wish that we had a cleaner way to match an ibdev to a netdev without overloading the gid table entries. Basically, it should be the job of the entity that created the netdev to make this association, and stuff a pointer in the netdev. Do you have a specific idea here? So far, we've tried to keep the mapping the responsibility of the rdma_cm module. With rocee, we may need to re-architect the solution and have the ib_device driver make this association. Even if it's unlikely, we need to make sure that we don't make the wrong match. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] bug 1918 - openmpi broken due to rdma-cm changes
But how can you determine _which_ rdma device should be used if and app binds to 127.0.0.1? I think this is busted... The code just picks the first rdma device available. To me, this is preferable than simply disallowing the loopback device from working at all. I personally use it all the time, so I don't have to figure out what the ip address is of the system that I'm trying to test on. Loopback support has always been in the rdma_cm and was intended to work; it just didn't work very well... ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] bug 1918 - openmpi broken due to rdma-cm changes
Well then the rdma-cm needs to know which devices support hw loopback. Cuz on a T3-only system, no hwloop... The problem sounds like it's more than just whether 127.0.0.1 is usable. That check may fix openmpi, but it sounds more like the app needs to know whether the device can actually support loopback, regardless of what addresses are used. Is this correct? What would openmpi do if there were two addresses assigned to the T3 device? Does openmpi simply bypass RDMA for all connections on the local machine? Basically, I'm not sure that this is *just* an rdma_cm issue. Although it definitely appears that some sort of change needs to be made to the rdma_cm. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] bug 1918 - openmpi broken due to rdma-cm changes
This solution would work. Will you code it up? I can do that. I just want to make sure that we address the full scope of the problem. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED-1.5.1 failure over iWarp
Sean could this be a result of the new Ipv6 rdma_cm patches that were added to OFED-1.5.1 ? I guess it's possible. I thought DAPL only used ipv4 addresses though. Can you try running some simpler tests, like ucmatose or rping? on server side run: ucmatose [-b optional_local_ip_addres] on client side run: ucmatose -s server_ip_address If you can, run on the server side with and without the -b option. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED-1.5.1 failure over iWarp
[wo...@det-16 ~]$ ucmatose -s 192.168.2.17 cmatose: starting client Btw - this should be 'ucmatose -s 192.168.0.17', and needs to start after the server is running. But, this isn't going to work since... [wo...@det-17 src]$ ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:04:23:AF:8E:CE inet addr:192.168.0.17 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::204:23ff:feaf:8ece/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1272 errors:0 dropped:0 overruns:0 frame:0 TX packets:909 errors:0 dropped:0 overruns:0 carrier:0 collisions:41 txqueuelen:100 RX bytes:309724 (302.4 KiB) TX bytes:123706 (120.8 KiB) Base address:0xdc00 Memory:f9fa-f9fc [wo...@det-17 src]$ ucmatose -b 192.168.0.17 cmatose: starting server cmatose: bind address failed: No such file or directory the server is failing. Can anyone test ucmatose over iwarp on the latest kernel.org kernel? - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [PATCH] uDAPL release note update for OFED 1.5
+ 2. SCM - uses sockets to exchange QP information. IPoIB, ARP, and SA queries NOT required. This is only true if all nodes are connected with a second networking fabric. + 3. UCM - use's IB UD QP to exchange QP info. Sockets, ARP, IPoIB, and SA queries NOT required. Same as above. Requires out of band (i.e. socket) connections to exchange UD QP information. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] Re: [PATCH] rdmaoe/libibverbs: handle binary compatibility
@@ -306,7 +314,7 @@ static struct pingpong_context *pp_init_ctx(struct
ibv_device *ib_dev, int size,
return NULL;
}
- memset(ctx-buf, 0, size);
+ memset(ctx-buf, 0x7b + is_server, size);
ctx-context = ibv_open_device(ib_dev);
if (!ctx-context) {
@@ -481,6 +489,7 @@ static void usage(const char *argv0)
printf( -n, --iters=itersnumber of exchanges (default 1000)\n);
printf( -l, --sl=sl service level value\n);
printf( -e, --events sleep on CQ events (default poll)\n);
+ printf( -g, --gid=remote gid gid of the other port\n);
It seems easier for the user if the tests discovered its local GID and exchanged
it with the remote side like it does with the LID and QPN.
diff --git a/include/infiniband/kern-abi.h b/include/infiniband/kern-abi.h
index 0db083a..8ef8844 100644
/*
@@ -223,7 +224,8 @@ struct ibv_query_port_resp {
__u8 active_width;
__u8 active_speed;
__u8 phys_state;
- __u8 reserved[3];
+ __u8 link_layer;
+ __u8 reserved[2];
};
Should we define the network layer too? Right now we have IB transport, which
assumes IB network and link; iWarp transport, which almost assumes IP network
and Ethernet; and RDMAoE, which may or may not have a network (but requires the
L3 address) and Ethernet (or is it DCB) link.
- Sean
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] Re: [PATCH 7/9] rdma/cm: fix loopback address support
Sean, is the patch series (9 patches) you sent earlier still current? Jason, do you have any updates? Roland added at least one change to fix the build when ipv6 is not enabled. I would check his for-next branch to see what he actually has queued. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [PATCH 7/9] rdma/cm: fix loopback address support
So what will this new librdmacm package will let cover which wasn't possible so far? do you refer to ipv6 support in mckey? anything else? Changes were your changes to mckey, plus changes Dave added to cmatose to support IPv6. The actual library itself hasn't been modified. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [PATCH 7/9] rdma/cm: fix loopback address support
okay, got it. I was under the impression that mckey still misses an option to get from the user an ipv6 multicast address which isn't all zeros nor unmapped, correct? or the -m option will work with both ipv4 and ipv6 addresses? The -m and -b options work with IPv4 and IPv6 addresses. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: SC'09 BOF - Meeting notes and Final Slides
On the topic of scalability and possible future enhancements for scalability, one person asked for verbs extensions to allow asynchronous QP create and modify calls WinOF has asynchronous interfaces for modify QP, and limited testing has shown that it can improve connection times. QP transitions are probably the second largest component of connection setup after the SA. Since the RDMA CM already provides an asynchronous interface, even existing apps may be able to be able to gain some benefit from this. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] RE: SC'09 BOF - Meeting notes and Final Slides
can't anyone get async modify QP today on any platform by just doing the operation in another thread (or thread pool)? It seems that the operations themselves are heavy enough that thread dispatch, locking etc is going to be significant overhead. On WinOF this is basically how things are implemented anyway. The lower level drivers are ports of the Linux drivers and do everything synchronously. The kernel uverbs equivalent just queues the modify calls to a kernel thread pool. It's not as efficient as it could be, since it ends up adding two context switches, but works. I could try adding threads to the librdmacm, move the modify calls to those threads, and see if there's any difference. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] re: [PATCH] librdmacm/cmatose: add support for ipv6
This patch adds ipv6 support to ucmatose. Signed-off-by: David Wilder dwil...@us.ibm.com Thanks. I pulled this patch into my local git tree with just a couple of minor cleanups. What other patches, if any, did you use to test with it? - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [PATCH] librdmacm/cmatose: add support for ipv6
From: David J. Wilder [dwil...@us.ibm.com]
Signed-off-by: David Wilder dwil...@us.ibm.com
Signed-off-by: Sean Hefty sean.he...@intel.com
---
We need to update struct cmatest to allow storing an IPv6 address,
or we can overrun the buffer.
Running with this patch, the client causes a kernel bug on 2.6.31 in
addr_resolve_local, so we'll still fixes for that. It also meant that
I couldn't test this patch.
examples/cmatose.c | 53
1 files changed, 29 insertions(+), 24 deletions(-)
diff --git a/examples/cmatose.c b/examples/cmatose.c
index 8c12347..84831ec 100644
--- a/examples/cmatose.c
+++ b/examples/cmatose.c
@@ -75,10 +75,7 @@ struct cmatest {
int connects_left;
int disconnects_left;
- struct sockaddr_in dst_in;
- struct sockaddr *dst_addr;
- struct sockaddr_in src_in;
- struct sockaddr *src_addr;
+ struct rdma_addraddr;
};
static struct cmatest test;
@@ -505,7 +502,7 @@ static int migrate_channel(struct rdma_cm_id *listen_id)
return ret;
}
-static int get_addr(char *dst, struct sockaddr_in *addr)
+static int get_addr(char *dst, struct sockaddr *addr)
{
struct addrinfo *res;
int ret;
@@ -516,15 +513,15 @@ static int get_addr(char *dst, struct sockaddr_in *addr)
return ret;
}
- if (res-ai_family != PF_INET) {
+ if (res-ai_family == PF_INET)
+ memcpy(addr, res-ai_addr, sizeof(struct sockaddr_in));
+ else if (res-ai_family == PF_INET6)
+ memcpy(addr, res-ai_addr, sizeof(struct sockaddr_in6));
+ else
ret = -1;
- goto out;
- }
- *addr = *(struct sockaddr_in *) res-ai_addr;
-out:
- freeaddrinfo(res);
- return ret;
+freeaddrinfo(res);
+return ret;
}
static int run_server(void)
@@ -540,14 +537,21 @@ static int run_server(void)
}
if (src_addr) {
- ret = get_addr(src_addr, test.src_in);
+ ret = get_addr(src_addr, test.addr.src_addr);
if (ret)
goto out;
- } else
- test.src_in.sin_family = PF_INET;
+ if (test.addr.src_addr.sa_family == AF_INET)
+ ((struct sockaddr_in *) test.addr.src_addr)-sin_port
= port;
+ else
+ ((struct sockaddr_in6 *)
test.addr.src_addr)-sin6_port = port;
+
+ } else {
+ test.addr.src_addr.sa_family = PF_INET;
+ ((struct sockaddr_in *) test.addr.src_addr)-sin_port = port;
+ }
+
+ ret = rdma_bind_addr(listen_id, test.addr.src_addr);
- test.src_in.sin_port = port;
- ret = rdma_bind_addr(listen_id, test.src_addr);
if (ret) {
perror(cmatose: bind address failed);
goto out;
@@ -614,22 +618,25 @@ static int run_client(void)
printf(cmatose: starting client\n);
if (src_addr) {
- ret = get_addr(src_addr, test.src_in);
+ ret = get_addr(src_addr, test.addr.src_addr);
if (ret)
return ret;
}
- ret = get_addr(dst_addr, test.dst_in);
+ ret = get_addr(dst_addr, test.addr.dst_addr);
if (ret)
return ret;
- test.dst_in.sin_port = port;
+ if (test.addr.dst_addr.sa_family == AF_INET)
+ ((struct sockaddr_in *) test.addr.dst_addr)-sin_port = port;
+ else
+ ((struct sockaddr_in6 *) test.addr.dst_addr)-sin6_port = port;
printf(cmatose: connecting\n);
for (i = 0; i connections; i++) {
ret = rdma_resolve_addr(test.nodes[i].cma_id,
- src_addr ? test.src_addr : NULL,
- test.dst_addr, 2000);
+ src_addr ? test.addr.src_addr : NULL,
+ test.addr.dst_addr, 2000);
if (ret) {
perror(cmatose: failure getting addr);
connect_error();
@@ -717,8 +724,6 @@ int main(int argc, char **argv)
}
}
- test.dst_addr = (struct sockaddr *) test.dst_in;
- test.src_addr = (struct sockaddr *) test.src_in;
test.connects_left = connections;
test.disconnects_left = connections;
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [PATCH] link-local address fix for rdma_resolve_addr
For ipv6 I ran what I described previously. What I do need to do is add the option to rping to specify a source address and run it with various address. Any help you can give defining what exactly needs to be tested would be appreciated. You can also test with ucmatose to verify ipv4 still works. Use the -b option to bind to a specific address. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [PATCH] link-local address fix for rdma_resolve_addr
This patch, as Jason's suggested, moves the function of addr_resolve_local()
into addr4_resolve_remote()
and addr6_resolve_remote(). It eliminates the need for addr_resolve_local().
One quick comment, remove '_remote' from function names:
addr4_resolve_remote, addr6_resolve_remote, and addr_resolve_remote
drivers/infiniband/core/addr.c | 99 +++
1 files changed, 18 insertions(+), 81 deletions(-)
diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c
index bd07803..f7a5861 100644
--- a/drivers/infiniband/core/addr.c
+++ b/drivers/infiniband/core/addr.c
@@ -233,6 +233,14 @@ static int addr4_resolve_remote(struct sockaddr_in
*src_in,
if (ret)
goto out;
+ if (rt-idev-dev == init_net.loopback_dev){
+ ret = rdma_translate_ip((struct sockaddr *)dst_in, addr);
+ if (!ret)
+ memcpy(addr-dst_dev_addr, addr-src_dev_addr,
+ MAX_ADDR_LEN);
+ goto put;
+ }
This doesn't end up doing the same thing as what resolve_local did. It only
matches up with the 'else if' portion below:
-static int addr_resolve_local(struct sockaddr *src_in,
-struct sockaddr *dst_in,
-struct rdma_dev_addr *addr)
-{
- struct net_device *dev;
- int ret;
-
- switch (dst_in-sa_family) {
- case AF_INET:
- {
- __be32 src_ip = ((struct sockaddr_in *)
src_in)-sin_addr.s_addr;
- __be32 dst_ip = ((struct sockaddr_in *)
dst_in)-sin_addr.s_addr;
-
- dev = ip_dev_find(init_net, dst_ip);
- if (!dev)
- return -EADDRNOTAVAIL;
-
- if (ipv4_is_zeronet(src_ip)) {
- src_in-sa_family = dst_in-sa_family;
- ((struct sockaddr_in *) src_in)-sin_addr.s_addr =
dst_ip;
- ret = rdma_copy_addr(addr, dev, dev-dev_addr);
- } else if (ipv4_is_loopback(src_ip)) {
- ret = rdma_translate_ip(dst_in, addr);
- if (!ret)
- memcpy(addr-dst_dev_addr, dev-dev_addr,
MAX_ADDR_LEN);
- } else {
- ret = rdma_translate_ip(src_in, addr);
- if (!ret)
- memcpy(addr-dst_dev_addr, dev-dev_addr,
MAX_ADDR_LEN);
- }
We need to handle the case where the source address is not given and provide
one. We also need to handle the case where the source address is given, but may
not match the destination address, or even use the same RDMA device.
I didn't look at the ipv6 changes yet.
- Sean
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [PATCH] link-local address fix for rdma_resolve_addr
$ ip route get 10.0.0.11 from 192.168.122.1 local 10.0.0.11 from 192.168.122.1 dev lo cache local mtu 16436 advmss 16396 hoplimit 64 (192.168.122.1 is bound to a different device on my system than 10.0.0.11) The new case trips the if == loopback and does rdma_translate_ip(10.0.0.11) The old case does rdma_translate_ip(192.168.122.1) I don't think this is a significant difference, both behaviors are reasonable choices and the code/complexity savings are worth it, IMHO. [Alternatively, I suppose one could call rdma_translate_ip(rt-rt_src) and if that fails then do rdma_translate_ip(dst_in) but why bother?] I agree this usage case is weird, but I'm still not sure about the patch. If the destination service is listening on 10.0.0.11, then the listen is bound to a different gid than the source gid that we're trying to connect from. The src_dev_addr and dst_dev_addr need to reflect this, so that a PR gives us a path that routes the CM REQ correctly. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [PATCH] link-local address fix for rdma_resolve_addr
The local loopback case uses PRs? Yes - the rdma cm makes no distinction when resolve route is called. It does a PR query. Even so, it still seems OK to me: Path: addr4_resolve_remote $ ip route get 10.0.0.11 from 192.168.122.1 local 10.0.0.11 from 192.168.122.1 dev lo srcIP = 192.168.122.1 rdma_translate_ip(dst_ip = 10.0.0.11) rdma_copy_addr(eth0); src_dev_addr = eth0.dev_addr (ie GID of 10.0.0.11) memcpy(dst_dev_addr = src_dev_addr) (ie GID of 10.0.0.11) So everthing is bound to the GID of 10.0.0.11 which matches the listen of 10.0.0.11, which seems OK. The source could have called rdma_bind_addr(192.168.122.1) prior to calling rdma_resolve_addr(). (DAPL does this.) This would have returned a different RDMA device than binding to 10.0.0.11. The client app could have allocated resources on that device, but the CM REQ will carry the gid/lid of the other device. The endpoints won't be able to communicate. Yes, it's weird, and may not be optimal, but if a source address is explicitly given, then its mapping to a specific RDMA device should be honored. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [PATCH] link-local address fix for rdma_resolve_addr
That is very difficult to fit into the semantics the IP routing model uses :( And it looks like an API problem in DAPL :( This depends on if your view is that the rdma cm is trying to match IP routing, trying to use IP addresses as convenient names for RDMA ports, or something in between that may lean more one way or the other depending on the device type. IMO, the primary reason for using IP addressing over IB is more for convenience, than compliance. So, I see now, you are proposing that in this case the connection attempt to be routed through the network and not looped back.. I actually have a big problem with that, ignoring a 'lo' entry in a routing table is very much not IP like and not a good idea. That should be respected.. I hesitate comparing RDMA versus IP too closely. Sigh. Anyhow, lets not get side tracked. It seems to me, the easy way out for David's approach is to simply check if the device is already bound via rdma_bind() and if so force it to that device no matter what the routing table lookup returns. Can you suggest a reliable way to make that check? I'm not entirely sure of the best way to test this. Checking the src_dev_addr is my first thought. [What happens now if I do this: rdma_bind(10.0.0.11) rdma_resolve_addr(src = 192.168.122.1 dst = 10.0.0.11) Does the cma_bind path check that it is already bound and give out an error? too late for me to check] rdma_resolve_addr only calls bind if the rdma id is not already bound. The src_addr simply gets ignored in this case, and the bound address is used instead. Once the cma_bind for rdma_resolve_addr is moved into the addr_resolve_remote function then people using the API without calling bind on the client path will get sane IP-like behavior. I like the approach, I'm just concerned about the case where the app has requested a specific source address, which today means that a specific RDMA device should be used. Remember, on Linux the IP is *not* attached to a device, it is part of the host itself. So the idea that a source address somehow specifies a RDMA device does not fit into the Linux IP networking model. I would say we're extending the linux networking model to incorporate this concept, while staying in our imposed little RDMA sand box world. Even iWarp has these issues. Truth be told, to fit the Linux IP model, the RDMA CM should have provided exactly only two ways to bind a cm_id to a specific device - rdma_accept and rdma_resolve_addr. I think this is more restrictive than things need to be. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] Re: Possible process deadlock in RMPP flow
Thanks Or. This one is already in OFED 1.4.2 but apparently this is a
different problem. Once I have information whether the patch Roland
posted fixed it I will update the list.
Eli, did you find a commit that fixes the problem you reported on?
Or.
Not yet :-(
I can't find anything off in the code for this. It's odd, since
unregister_mad_agent() does:
flush_workqueue(port_priv-wq);
ib_cancel_rmpp_recvs(mad_agent_priv);
and ib_cancel_rmpp_recvs() does:
spin_lock_irqsave(agent-lock, flags);
list_for_each_entry(rmpp_recv, agent-rmpp_list, list) {
cancel_delayed_work(rmpp_recv-timeout_work);
cancel_delayed_work(rmpp_recv-cleanup_work);
}
spin_unlock_irqrestore(agent-lock, flags);
flush_workqueue(agent-qp_info-port_priv-wq);
which basically just flushes the same work queue.
I haven't been able to reproduce the problem, but I'm running the latest kernel
- not sure that matters in this case. Does ibnetdiscover just hang forever at
the end of the test when this occurs? Is there any more information available?
- Sean
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [PATCH] link-local address fix for rdma_resolve_addr
@@ -393,7 +393,7 @@ static int addr_resolve_local(struct sockaddr *src_in, for_each_netdev(init_net, dev) if (ipv6_chk_addr(init_net, - ((struct sockaddr_in6 *) addr)- sin6_addr, + ((struct sockaddr_in6 *) dst_in)- sin6_addr, dev, 1)) break; I can believe it fixes the case you describe (ie loopback) but matching the *dest* IP against the local interface's IP list cannot possibly be right. The intent is to see if the destination address is local. A source address may not be given. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofw] [OPENSM] add Windows syntax for varags in a macro
+#ifdef __WIN__
+#define OSM_LOG(log, level, fmt, ...) \
+do { \
+ if (osm_log_is_active(log, (level))) \
+ osm_log(log, level, %s: fmt, __func__, ## __VA_ARGS__); \
__VA_ARGS__ should work on any platform. libibmad : mad.h uses this for windows
and linux.
+} while (0)
+#else
#define OSM_LOG(log, level, fmt, arg...) do { \
if (osm_log_is_active(log, (level))) \
osm_log(log, level, %s: fmt, __func__, ##arg); \
} while (0)
+#endif
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofw] [OPENSM] add Windows syntax for varags in a macro
+#ifdef __WIN__
+#define OSM_LOG(log, level, fmt, ...) \
+do { \
+ if (osm_log_is_active(log, (level))) \
+ osm_log(log, level, %s: fmt, __func__, ## __VA_ARGS__); \
__VA_ARGS__ should work on any platform. libibmad : mad.h uses this
for windows and linux.
+} while (0)
+#else
#define OSM_LOG(log, level, fmt, arg...) do { \
if (osm_log_is_active(log, (level))) \
osm_log(log, level, %s: fmt, __func__, ##arg); \
} while
(0)
+#endif
As coded it would not compile.
I'm suggesting to remove the #ifdef __WIN__ and remove the #else code.
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: Possible process deadlock in RMPP flow
Thanks Or. This one is already in OFED 1.4.2 but apparently this is a different problem. Once I have information whether the patch Roland posted fixed it I will update the list. If ibnetdiscover doesn't use RMPP as Hal indicated, I don't think Roland's patch will help. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: Possible process deadlock in RMPP flow
ibnetdiscover D 80149b8d 0 26968 26544 (L-TLB) 8102c900bd88 0046 81037e8e 81037e8e02e8 8102c900bd78 000a 8102c5b50820 81038a929820 011837bf6105 0ede 8102c5b50a08 0001 Call Trace: [80064207] wait_for_completion+0x79/0xa2 [8008b4cc] default_wake_function+0x0/0xe [882271d9] :ib_mad:ib_cancel_rmpp_recvs+0x87/0xde [88224485] :ib_mad:ib_unregister_mad_agent+0x30d/0x424 [883983e9] :ib_umad:ib_umad_close+0x9d/0xd6 [80012e22] __fput+0xae/0x198 [80023de6] filp_close+0x5c/0x64 [800393df] put_files_struct+0x63/0xae [80015b26] do_exit+0x31c/0x911 [8004971a] cpuset_exit+0x0/0x6c [8005e116] system_call+0x7e/0x83 From the dump it seems that the process is waits on the call to flush_workqueue() in ib_cancel_rmpp_recvs(). The package they use is OFED 1.4.2. Roland just submitted a patch in this area yesterday. I don't know if the patch would fix their issue, but it may be worth trying. What kernel does 1.4.2 map to? What RMPP messages does ibnetdiscover use? If the program is completing successfully, there may be a different race with the rmpp cleanup. I'll see if anything else stands out in that area. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: APM and the CM
In the case of a failure I would like APM to move my connection from one subnet to the other. My question is this, assuming that there is a failure in one of the switches, will the CM still work, or it will also fail. Obviously if it fails, I need to find a different solution. This should work if the Windows CM implements apm correctly. Based on a quick look at the windows code, it looks like everything is there, but I've never tested apm on windows. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: APM and the CM
Should I accept the CM to send the mads (that are used to manage the connection) on a different subnet every time there is an APM migration? Does this code work on Linux? This should work on Linux. I didn't write the Windows CM, so I'm not positive that it does this as well, but looking at the code, at the very least the framework is there. If you look at the kernel al_cm_cep.c file, kcep_t maintains 2 address vectors and uses an index to toggle between which one is primary and used to send MADs. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofa-general] [PATCH] ib_send_bw -b can hang due to too few CQ entries
- ctx-cq = ibv_create_cq(ctx-context, ctx-rx_depth, NULL, ctx-channel, 0); + ctx-cq = ibv_create_cq(ctx-context, ctx-tx_depth + ctx-rx_depth, + NULL, ctx-channel, 0); I'm looking at a windows port of this test, but at least there, rx_depth is set to rx_depth + tx_depth. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofa-general] [PATCH] ib_send_bw -b can hang due to too few CQ entries
Sure. Just above the call to ibv_create_cq(), ctx-rx_depth is set to ctx-rx_depth = rx_depth + tx_depth but the rest of the code does ibv_post_send() and ibv_post_recv() based on ctx-tx_depth and ctx-rx_depth which means the CQ needs to be ctx-tx_depth + ctx-rx_depth big. If the tx_depth is the same on both sides, why would there ever be more than the initial tx_depth and rx_depth completions on the CQ? How many receive completions can there be on the CQ, and what throttles the sender? - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofa-general] [PATCH] ib_send_bw -b can hang due to too few CQ entries
Remember that this fix only affects the bi-directional test. Both client and sever are going to post ctx-rx_depth receives and ctx-tx_depth sends and then check for completions. It won't post more sends or receives until the completions are seen. Okay - I think I understand what's happening. The maximum number of outstanding sends is limited to tx_depth / 2. After posting that many sends, the code waits for completions. Once some sends complete, additional sends may be posted, up to the iteration count. There's nothing that coordinates posting the sends with completing receives on the remote side. (This is what I was missing.) Eventually, all posted receives could be complete and generate CQ entries. The send side is basically throttled by RNR NACKs. Now I don't understand the purpose behind doubling the rx_depth... - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] Wish to remove local sa patches from OFED 1.5
Since Qlogic are using some of the APIs in these files it was decided not to remove them in 1.5 However Qlogic were requested to approach Sean and see if they can move their implementation to the new SA API he is developing now Has this new SA API been proposed to the list as yet (and I missed it :-() ? Um... I think there's been some miscommunication somewhere. I'm not developing an SA API, at least in my waking hours. I am working on a userspace app that should help with scaling for some topologies, but I doubt it will work for all routing algorithms. I'm at least a couple weeks away from posting anything. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet
RDMA over Ethernet (RDMAoE) allows running the IB transport protocol over Ethernet, providing IB capabilities for Ethernet fabrics. The packets are standard Ethernet frames with an Ethertype, an IB GRH, unmodified IB transport headers and payload. HCA RDMAoE ports are no different than regular IB ports from the RDMA stack perspective. I would refer to this as IBoE, not RDMAoE. The RDMA stack should see these ports different than regular IB HCA ports. There are a lot of differences that should not simply be hidden or incorrectly assumed: QP0, QoS, multiple paths, routing(?), no SA, etc. IB subnet management and SA services are not required for RDMAoE operation; Then I would not try to emulate it at all. As Hal mentioned in a separate post, there are too many ways to interact with the SA that an emulation won't cover. Ethernet management practices are used instead. In Ethernet, nodes are commonly referred to by applications by means of an IP address. RDMAoE treats IP addresses that were assigned to the corresponding Ethernet port as GIDs, and makes use of the IP stack to bind a destination address to the corresponding netdevice (just as the CMA does today for IB and iWARP) and to obtain its L2 MAC addresses. Is the actual L3 address an IP address, or just an encoded IP address in an IBoE L3 address? What L3 protocol is being used and will it interoperate with some peer L3 protocol (IP or IB)? - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofa-general] [PATCH 2/9] ib_core: kernel API for GID -- MAC translations
A few support functions are added to allow the translation from GID to MAC which is required by hw drivers supporting RDMAoE. Why not just use IP to MAC calls? Or use the MAC as the GUID? Do the GIDs follow the IB GID format? ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofa-general] [PATCH 3/9] ib_core: RDMAoE support only QP1
Since RDMAoE is using Ethernet there is no need for QP0. This patch will create only QP1 for RDMAoE ports. Which modules will use QP1 and for what purpose? I see sa_query/multicast, but there's not an actual SA. I'm guessing that the ib_cm works without changes. To clarify, do all IBoE packets carry a GRH? ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofa-general] [PATCH 4/9] ib_core: Add RDMAoE SA support
diff --git a/drivers/infiniband/core/multicast.c
b/drivers/infiniband/core/multicast.c
index 107f170..2417f6b 100644
--- a/drivers/infiniband/core/multicast.c
+++ b/drivers/infiniband/core/multicast.c
@@ -488,6 +488,36 @@ retest:
}
}
+struct eth_work {
+ struct work_struct work;
+ struct mcast_member *member;
+ struct ib_device*device;
+ u8 port_num;
+};
+
+static void eth_mcast_work_handler(struct work_struct *work)
+{
+ struct eth_work *w = container_of(work, struct eth_work, work);
+ int err;
+ struct ib_port_attr port_attr;
+ int status = 0;
+
+ err = ib_query_port(w-device, w-port_num, port_attr);
+ if (err)
+ status = err;
+ else if (port_attr.state != IB_PORT_ACTIVE)
+ status = -EAGAIN;
+
+ w-member-multicast.rec.qkey = cpu_to_be32(0xc2c);
How can a user control this? An app needs the same qkey for unicast traffic.
+ atomic_inc(w-member-refcount);
This needs to be moved below...
+ err = w-member-multicast.callback(status, w-member-multicast);
+ deref_member(w-member);
+ if (err)
+ ib_sa_free_multicast(w-member-multicast);
+
+ kfree(w);
+}
+
/*
* Fail a join request if it is still active - at the head of the pending
queue.
*/
@@ -586,21 +616,14 @@ found:
return group;
}
-/*
- * We serialize all join requests to a single group to make our lives much
- * easier. Otherwise, two users could try to join the same group
- * simultaneously, with different configurations, one could leave while the
- * join is in progress, etc., which makes locking around error recovery
- * difficult.
- */
-struct ib_sa_multicast *
-ib_sa_join_multicast(struct ib_sa_client *client,
- struct ib_device *device, u8 port_num,
- struct ib_sa_mcmember_rec *rec,
- ib_sa_comp_mask comp_mask, gfp_t gfp_mask,
- int (*callback)(int status,
- struct ib_sa_multicast *multicast),
- void *context)
+static struct ib_sa_multicast *
+ib_join_multicast(struct ib_sa_client *client,
+struct ib_device *device, u8 port_num,
+struct ib_sa_mcmember_rec *rec,
+ib_sa_comp_mask comp_mask, gfp_t gfp_mask,
+int (*callback)(int status,
+struct ib_sa_multicast *multicast),
+void *context)
{
struct mcast_device *dev;
struct mcast_member *member;
@@ -647,9 +670,81 @@ err:
kfree(member);
return ERR_PTR(ret);
}
+
+static struct ib_sa_multicast *
+eth_join_multicast(struct ib_sa_client *client,
+ struct ib_device *device, u8 port_num,
+ struct ib_sa_mcmember_rec *rec,
+ ib_sa_comp_mask comp_mask, gfp_t gfp_mask,
+ int (*callback)(int status,
+ struct ib_sa_multicast *multicast),
+ void *context)
+{
+ struct mcast_device *dev;
+ struct eth_work *w;
+ struct mcast_member *member;
+ int err;
+
+ dev = ib_get_client_data(device, mcast_client);
+ if (!dev)
+ return ERR_PTR(-ENODEV);
+
+ member = kzalloc(sizeof *member, gfp_mask);
+ if (!member)
+ return ERR_PTR(-ENOMEM);
+
+ w = kzalloc(sizeof *w, gfp_mask);
+ if (!w) {
+ err = -ENOMEM;
+ goto out1;
+ }
+ w-member = member;
+ w-device = device;
+ w-port_num = port_num;
+
+ member-multicast.context = context;
+ member-multicast.callback = callback;
+ member-client = client;
+ member-multicast.rec.mgid = rec-mgid;
+ init_completion(member-comp);
+ atomic_set(member-refcount, 1);
+
+ ib_sa_client_get(client);
+ INIT_WORK(w-work, eth_mcast_work_handler);
+ queue_work(mcast_wq, w-work);
+
+ return member-multicast;
The user could leave/destroy the multicast join request before the queued work
item runs. We need to hold an additional reference on the member until the work
item completes.
+
+out1:
+ kfree(member);
+ return ERR_PTR(err);
+}
+
+/*
+ * We serialize all join requests to a single group to make our lives much
+ * easier. Otherwise, two users could try to join the same group
+ * simultaneously, with different configurations, one could leave while the
+ * join is in progress, etc., which makes locking around error recovery
+ * difficult.
+ */
+struct ib_sa_multicast *
+ib_sa_join_multicast(struct ib_sa_client *client,
+ struct ib_device *device, u8 port_num,
+ struct ib_sa_mcmember_rec *rec,
+ ib_sa_comp_mask comp_mask, gfp_t gfp_mask,
+ int (*callback)(int status,
+ struct ib_sa_multicast *multicast),
+ void *context)
+{
+ return
RE: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet-- some procedural questions
What does it mean to have a standard body associated with a proposal? Does associated mean that the proposal/API is a published standard in that standards body? Or some weaker definition? There's a significant difference between an API and a wire protocol, and even most wire protocols shouldn't be subject to any standards body. We need to be realistic here. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofa-general] [PATCH 1/9] ib_core: Add API to query port link type
This allows to get the type of a port to be either Ethernet or IB which is required by following patches for implementing RDMA over Ethernet - RDMAoE. I don't know if this makes more sense without studying the changes in more detail, but was there a reason why node_type just wasn't extended instead? ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: Do you still need the local SA in OFED 1.5?
Subject: Do you still need the local SA in OFED 1.5? The RDMA/IB CMs do not scale without PR caching or hard-coding PR parameters. I'm personally fine removing it from OFED. MPI and other applications are working around SA scaling issues by connecting over sockets anyway. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofa-general] [PATCH] ib_mad: Fix RMPP header RRespTime manipulation
Fix ib_set_rmpp_flags() to use the correct bit mask for RRespTime. In the 8-bit field of the RMPP header, the first 5 bits are RRespTime and next 3 bits are RMPPFlags. Hence to retain the first 5 bits, the mask should be 0xF8 instead of 0xF1. Signed-off-by: Ramachandra K ramachandra.kuchiman...@qlogic.com Good catch. Acked-by: Sean Hefty sean.he...@intel.com ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: OFED Jan 5, 2009 meeting minutes on OFED plans
* Mellanox suggested to add IB over Eth - this is similar to iWARP but more like IB (e.g. including UD), and can work over ConnectX. A concern was raised by Intel (Dave Sommers) since it is not a standard transport. Decision: This request will be raised in the MWG, and they should decide if OFA can support it. Just is just my opinion, but in the past, OFED has included non-standard features, like extended connected mode, that are still not part of the IBTA spec. Do we know if such a feature would be accepted into the Linux kernel? I think OFED should base their decision more on the answer to that question than IBTA approval. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: OFA BOF slides for SC'08 (Revision 0.8)
Slide 18 Add WinVerbs, WinMad, and 'OFED libraries on Windows' with myself as maintainer. A maintainer may be needed for the OFED diagnostics tools if we cannot agree to share a common code-base. Slide 20 Under WinVerbs, add: Supports OFED libibverbs Slide 21 Under WinVerbs, add: Support OFED librdmacm For discussion points: Slide 7 As we provide the OFED diagnostics on Windows, we need to rethink having libraries that can 'change without notice'. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] Re: [ofa-general] OFED meeting agenda for today (July 14, 2008)
- Question about IBCM for OFED v1.4: - Any estimate on the following ticket: https://bugs.openfabrics.org/show_bug.cgi?id=963 - Any estimate on the ibcm_listen() sometimes failing issue? (I don't think there's a ticket filed against this yet) I don't think anything was done here - Sean? I have been working with Jeff and the ompi developers directly on a different thread, but I don't think that ofa general is copied on it. There's a possibility that ompi is colliding with the SDP listen space, and we're exploring that option. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofa-general] Re: [PATCH] Request For Comments:
The requirement is mostly driven from the receiving side. For cxgb3 it is anyway... Maybe you can help me understand the spec here. If we ignore this feature for a minute, then the side that calls rdma_connect() must instead issue the first 'send' request to the server. Can the first 'send' be a 0B rdma write or read? Why wouldn't the target of that request not have to transition to connected? Is the issue that there's no way for the receiving FW/driver to know that this has occurred so that it can signal that the connection has been established? I.e. a client that does this must signal the server that things are ready through some out of band means. server sends MPA Start response with lets do RTR and send me X where X could be 0B write, 0B read request or 0B send. Are there any restrictions where a client may not be able to issue what the server requests? E.g. the hardware doesn't issue 0B writes. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [ANNOUNCE] librdmacm release 1.0.7
I've published librdmacm release 1.0.7. It contains the following additions over 1.0.6: Add support for rdma_migrate_id(). Kernel support will be in 2.6.25. Several build fixes from Roland. Set reject status correctly. Please pull this package for OFED 1.3.1. The reject status fix has been requested for DAPL. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH] [OFED 1.3 docs] librdmacm: update release notes for OFED 1.3
Signed-off-by: Sean Hefty [EMAIL PROTECTED] --- rdma_cm_release_notes.txt | 154 + 1 files changed, 86 insertions(+), 68 deletions(-) diff --git a/rdma_cm_release_notes.txt b/rdma_cm_release_notes.txt index 24a07af..8060205 100644 --- a/rdma_cm_release_notes.txt +++ b/rdma_cm_release_notes.txt @@ -1,68 +1,86 @@ -Open Fabrics Enterprise Distribution (OFED) - RDMA CM in OFED 1.3 Release Notes - - February 2008 - - -=== -Table of Contents -=== -1. Overview -2. New Features -3. Known Issues - -=== -1. Overview -=== -The RDMA CM is a communication manager used to setup reliable, connected -and unreliable datagram data transfers. It provides an RDMA transport -neutral interface for establishing connections. The API is based on sockets, -but adapted for queue pair (QP) based semantics: communication must be -over a specific RDMA device, and data transfers are message based. - -The RDMA CM only provides the communication management (connection setup / -teardown) portion of an RDMA API. It works in conjunction with the verbs -API for data transfers. - -=== -2. New Features -=== -The RDMA CM now supports connected, datagram, and multicast data transfers. - -When used over Infiniband, the RDMA CM will make use of a local path record -cache, if it is enabled. On large fabrics, use of the local cache can greatly -reduce connection time. Use of a cache is not necessary for iWarp. - -Man pages have been created to describe the various interfaces and test -programs available. For a full list, users should refer to the rdma_cm.7 man -page. - -=== -3. Known Issues -=== -The RDMA CM relies on the operating system's network configuration tables to -map IP addresses to RDMA devices. Incorrectly configured network -configurations can result in the RDMA CM being unable to locate the correct -RDMA device. Currently, the RDMA CM only supports IPv4 addressing. - -All RDMA interfaces must provide a way to map IP addresses to an RDMA device. -For Infiniband, this is done using IPoIB, and requires correctly configured -IPoIB device interfaces sharing the same multicast domain. For details on -configuring IPoIB, refer to ipoib_release_notes.txt. For RDMA devices to -communicate, they must support the same underlying network and data link -layers. - -If you experience problems using the RDMA CM, you may want to check the -following: - - * Verify that you have IP connectivity over the RDMA devices. For example, - ping between iWarp or IPoIB devices. - - * Ensure that IP network addresses assigned to RDMA devices do not - overlap with IP network addresses assigned to standard Ethernet devices. - - * For multicast issues, either bind directly to a specific RDMA device, or - configure the IP routing tables to route multicast traffic over an RDMA - device's IP address. - - +Open Fabrics Enterprise Distribution (OFED) + RDMA CM in OFED 1.3 Release Notes + + February 2008 + + +=== +Table of Contents +=== +1. Overview +2. New Features +3. Known Issues + +=== +1. Overview +=== +The RDMA CM is a communication manager used to setup reliable, connected +and unreliable datagram data transfers. It provides an RDMA transport +neutral interface for establishing connections. The API is based on sockets, +but adapted for queue pair (QP) based semantics: communication must be +over a specific RDMA device, and data transfers are message based. + + +The RDMA CM only provides the communication management (connection setup / +teardown) portion of an RDMA API. It works in conjunction with the verbs +API for data transfers. + +=== +2. New Features +=== +for OFED 1.3: +Added support for valgrind checks. + +Added quality of service support. Quality of service (QoS
[ewg] new releases of libibcm and librdmacm libraries
I've pushed out new releases: libibcm 1.0.2 librdmacm 1.0.6 to my git tree, and the OFA download pages. Please pull both packages into OFED 1.3. Major changes from previous release: libibcm - removes obsolete simple.c test program librdmacm - updates to build, fix setting QP attributes for RDMA reads - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] new releases of libibcm and librdmacm libraries
Can you put a tag with the name ofed_1_3 on these git trees too I have tags of v1.0.2 and v1.0.6. Can these just be used instead? ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] new releases of libibcm and librdmacm libraries
Its just easier for us when we do diffs to have also the ofed_1_3 as all other git trees After some time pass its hard to remember which version was part of which release :-( I've added ofed_1_3 tags. Note that if there's a need to update the libraries again, the tag will need to be deleted and re-created to move it forward. I don't anticipate other changes for OFED 1.3 however. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] Re: [ofa-general] OFED Jan 28 meeting summary onRC3readiness
In all fairness, the kernel portion of all of this, and the process of getting things into Linus' kernel, has *always* been a case of staging things in Roland's tree and then merging upstream. So, at least for the kernel, that's mostly true as OFED is pretty close to Roland's tree generally speaking. As for the user space packages though, you guys *are* the upstream. There's no one to merge upstream to and very little oversight by anyone. So, it's entirely up to all of you just how much your package seems to be a feature of the day change-athon versus a solid, stable program. I don't believe that this is the model actually in use. OFED has accepted kernel features that have not been submitted for upstream inclusion, or, in some cases, that were, but were rejected. (For examples, see local SA, SA event subscription, XRC, SDP, and some of the previous incarnations of IPoIB CM.) There are thousands of lines of code difference between OFED and the kernel upon which it's based. (To be clear, I'm not objecting to any changes, just the sheer volume.) The OFED releases of the userspace libraries are not identical to those provided by the maintainers. (See libibverbs.) Whose version of libibverbs does RedHat plan on using? How do you manage the differences between OFED and Roland's libibverbs libraries? And I'm really not trying to come across harsh here, but if the distros are willing to pull the OFED code, why should OFA bother trying to merge anything upstream? - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] Re: [ofa-general] OFED Jan 28 meeting summaryonRC3readiness
It's currently in, although that isn't written in stone either. Individual changes to existing components, like xrc, can slip past me easier than whole unsubmitted subsystems like RDS. I think for RedHat it would end up being in the kernel, but Roland's userspace library doesn't support it, so it ends up being unused code. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] Re: [ofa-general] OFED Jan 28 meeting summary on RC3readiness
The main reason is not the bugs but the features supported by IBM - CM support for non SRQ and 4K MTU These are entirely my opinions, but... OFED isn't even at RC1 if it's not at feature freeze... OFED has moved well beyond trying to provide an enterprise distribution to simply providing an experimental code base more concerned with including the latest and greatest features. It's become the staging area for getting the code into shape for merging upstream, which wasn't what I thought was the purpose of OFED. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] RE: [ofa-general] OFED Jan 14 meeting summary on RC2readiness
When we started OFED we decided to enable new features that can be in lower stability level, in case they do not harm the overall stability of the OFED release. I think XRC fulfill this criteria. XRC changes the verbs interfaces and code. It increases the risk of instability. Changes to IPoIB increase the risk of instability. Something like SDP doesn't, since it's an entirely separate module. From my view, OFED is enabling new features over stability. I would rather see OFED pull code from upstream with patches added on only for backports and fixes. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [ANNOUNCE] librdmacm release 1.0.5
I've pushed out release 1.0.5 of the librdmacm. It adds some additional documentation to the man pages only. Please update OFED 1.3 to use this version. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: sean_cm_limit_mra_timeout.patch
but it's currently solidly part of both OFED 1.3 and 1.2.5. Should it then be ? IMO - any patch which has been rejected for upstream submission or is considered experimental should be yanked from OFED. Is there some other approach to the specific problem this patch was attempting to fix ? A fix could be provided to the SRP target, which is where the real problem is. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: sean_cm_limit_mra_timeout.patch
The patch has:
+ if (cm_id_priv-timeout_ms cm_convert_to_ms(max_timeout)) {
+ printk(KERN_WARNING PFX req timeout_ms %d %d, decreasing\n,
+ cm_id_priv-timeout_ms, cm_convert_to_ms(max_timeout));
+ cm_id_priv-timeout_ms = cm_convert_to_ms(max_timeout);
+ }
whereas cm.c has:
cm_id_priv-timeout_ms = cm_convert_to_ms(
param-primary_path-packet_life_time) * 2
+
cm_convert_to_ms(
param-remote_cm_response_timeout);
This forces max_timeout to be set one higher than really needed. Should
the comparison in the patch also account for the packet life time ?
The intent is that the max_timeout is the absolute max, and does not adjust
upward based on packet lifetime. Otherwise there's no way to recover from a
packet lifetime that was incorrectly set to an extremely large value.
- Sean
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] Feedback on Developer's Summit
11-12: SA cache session 12-1: IPoIB stateless offload issues Sean, Roland, Dror - can you make it? I should be able to make this, but as soon as you start pushing sessions before noon, time should probably be made for lunch. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] to be discussed at the developer conference
Sean - SA-caching - 45m I think 30 minutes for this should be sufficient. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] to be discussed at the developer conference
1) the long time and endless threads related to the SA caching thing need to be there. Sean - I saw that you prepare a session, correct? will you presenting few possible designs? I was asked to prepare a session and will mention some of the general scalability issues that we've seen with Intel MPI. 3) QoS - Sean, Dror, generally speaking, what where you thinking to discuss? We plan on discussing what was added to the stack and opensm. Keep in mind that both of these are only 20 minutes. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ofa-general] Re: [ewg] to be discussed at the developer conference
7) the inform info code. Sean - you have implemented and attempted to push it through the sa caching push, but since the cache was rejected so did the inform info code. So the questions here - how do we make this push happen? are there any open issues, etc There either needs to be an in kernel user, or we need to reach agreement on the best way to expose this to userspace. Neither this, nor the multicast code are directly exported. I have seen e-mails on the list that event subscription is used by userspace apps, but it is done via the MAD layer directly. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] librdmacm 1.0.4 release
I've pushed out a release 1.0.4 of librdmacm that addresses some of the feedback from Doug. Patches were posted previously to the list, with a small update based on that feedback. Please pull this release into OFED 1.3. Changes from 1.0.3: librdmacm/cma: provide wrapper functions to extract src/dst addresses librdmacm/cma: provide sanity checks for max outstanding rdma ops librdmacm/man: update man pages to clarify connection request params Thanks, - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: Toward OFED-1.3-beta release - prepare ofed_1_3 branch for every package
Vladimir Sokolovsky wrote: Hello, Toward OFED-1.3 beta release we want to prepare git trees with ofed_1_3 branch under git://git.openfabrics.org/ofed_1_3 for every userspace package in the OFED. All maintainers of the user space package please create ofed_1_3 branch in your git trees if this is not done yet on the OpenFabrics server. Rather than pulling from a git branch, why doesn't OFED simply use a specific release from the maintainer? - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH] librdmacm: provide wrapper functions to extract src/dst addresses
Provide wrapper functions to retrieve the source and destination
addresses. This is based on feedback from Doug Ledford.
Signed-off-by: Sean Hefty [EMAIL PROTECTED]
---
If there are no objections, I would like to include this change in the next
release of librdmacm, and request that it go into OFED 1.3.
Makefile.am |2 ++
include/rdma/rdma_cma.h | 10 ++
man/rdma_bind_addr.3|3 ++-
man/rdma_cm.7 |1 +
man/rdma_get_dst_addr.3 | 16
man/rdma_get_dst_port.3 |3 ++-
man/rdma_get_src_addr.3 | 17 +
man/rdma_get_src_port.3 |3 ++-
man/rdma_resolve_addr.3 |3 ++-
9 files changed, 54 insertions(+), 4 deletions(-)
diff --git a/Makefile.am b/Makefile.am
index 1195bd9..c688283 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -49,6 +49,8 @@ man_MANS = \
man/rdma_get_devices.3 \
man/rdma_get_src_port.3 \
man/rdma_get_dst_port.3 \
+ man/rdma_get_src_addr.3 \
+ man/rdma_get_dst_addr.3 \
man/rdma_join_multicast.3 \
man/rdma_leave_multicast.3 \
man/rdma_listen.3 \
diff --git a/include/rdma/rdma_cma.h b/include/rdma/rdma_cma.h
index b0848d5..8ebcaf6 100644
--- a/include/rdma/rdma_cma.h
+++ b/include/rdma/rdma_cma.h
@@ -494,6 +494,16 @@ static inline uint16_t rdma_get_dst_port(struct rdma_cm_id
*id)
((struct sockaddr_in *) id-route.addr.dst_addr)-sin_port;
}
+static inline struct sockaddr *rdma_get_src_addr(struct rdma_cm_id *id)
+{
+ return id-route.addr.src_addr;
+}
+
+static inline struct sockaddr *rdma_get_dst_addr(struct rdma_cm_id *id)
+{
+ return id-route.addr.dst_addr;
+}
+
/**
* rdma_get_devices - Get list of RDMA devices currently available.
* @num_devices: If non-NULL, set to the number of devices returned.
diff --git a/man/rdma_bind_addr.3 b/man/rdma_bind_addr.3
index bed7f0b..dc7a868 100644
--- a/man/rdma_bind_addr.3
+++ b/man/rdma_bind_addr.3
@@ -25,4 +25,5 @@ address.
If used to bind to port 0, the rdma_cm will select an available port
and return it to the user.
.SH SEE ALSO
-rdma_create_id(3), rdma_listen(3), rdma_resolve_addr(3), rdma_create_qp(3)
+rdma_create_id(3), rdma_listen(3), rdma_resolve_addr(3), rdma_create_qp(3),
+rdma_get_src_addr(3), rdma_get_src_port(3)
diff --git a/man/rdma_cm.7 b/man/rdma_cm.7
index bfb3493..2e07706 100644
--- a/man/rdma_cm.7
+++ b/man/rdma_cm.7
@@ -110,5 +110,6 @@ rdma_resolve_route(3), rdma_connect(3), rdma_listen(3),
rdma_accept(3),
rdma_reject(3), rdma_join_multicast(3), rdma_leave_multicast(3),
rdma_notify(3),
rdma_ack_cm_event(3), rdma_disconnect(3), rdma_destroy_qp(3),
rdma_destroy_id(3),
rdma_destroy_event_channel(3), rdma_get_devices(3), rdma_free_devices(3),
+rdma_get_dst_addr(3), rdma_get_src_addr(3),
rdma_get_dst_port(3), rdma_get_src_port(3), rdma_set_option(3)
ucmatose(1), udaddy(1), mckey(1), rping(1)
diff --git a/man/rdma_get_dst_addr.3 b/man/rdma_get_dst_addr.3
new file mode 100644
index 000..054445f
--- /dev/null
+++ b/man/rdma_get_dst_addr.3
@@ -0,0 +1,16 @@
+.TH RDMA_GET_DST_ADDR 3 2007-05-15 librdmacm Librdmacm Programmer's
Manual librdmacm
+.SH NAME
+rdma_get_dst_addr \- Returns the remote IP address of a bound rdma_cm_id.
+.SH SYNOPSIS
+.B #include rdma/rdma_cma.h
+.P
+.B struct sockaddr * rdma_get_dst_addr
+.BI (struct rdma_cm_id * id );
+.SH ARGUMENTS
+.IP id 12
+RDMA identifier.
+.SH DESCRIPTION
+Returns the remote IP address associated with an rdma_cm_id.
+.SH SEE ALSO
+rdma_resolve_addr(3), rdma_get_src_port(3), rdma_get_dst_port(3),
+rdma_get_src_addr(3)
diff --git a/man/rdma_get_dst_port.3 b/man/rdma_get_dst_port.3
index 88e6ec2..658c9f7 100644
--- a/man/rdma_get_dst_port.3
+++ b/man/rdma_get_dst_port.3
@@ -13,4 +13,5 @@ RDMA identifier.
Returns the remote port number for an rdma_cm_id that has been bound to
a remote address.
.SH SEE ALSO
-rdma_connect(3), rdma_accept(3), rdma_get_cm_event(3), rdma_get_src_port(3)
+rdma_connect(3), rdma_accept(3), rdma_get_cm_event(3), rdma_get_src_port(3),
+rdma_get_src_addr(3), rdma_get_dst_addr(3)
diff --git a/man/rdma_get_src_addr.3 b/man/rdma_get_src_addr.3
new file mode 100644
index 000..fa9b256
--- /dev/null
+++ b/man/rdma_get_src_addr.3
@@ -0,0 +1,17 @@
+.TH RDMA_GET_SRC_ADDR 3 2007-05-15 librdmacm Librdmacm Programmer's
Manual librdmacm
+.SH NAME
+rdma_get_src_addr \- Returns the local IP address of a bound rdma_cm_id.
+.SH SYNOPSIS
+.B #include rdma/rdma_cma.h
+.P
+.B struct sockaddr * rdma_get_src_addr
+.BI (struct rdma_cm_id * id );
+.SH ARGUMENTS
+.IP id 12
+RDMA identifier.
+.SH DESCRIPTION
+Returns the local IP address for an rdma_cm_id that has been bound to
+a local device.
+.SH SEE ALSO
+rdma_bind_addr(3), rdma_resolve_addr(3), rdma_get_src_port(3),
+rdma_get_dst_port(3), rdma_get_dst_addr(3)
diff --git a/man/rdma_get_src_port.3 b/man/rdma_get_src_port.3
index 63ee564..88f0920 100644
--- a/man/rdma_get_src_port.3
+++ b/man/rdma_get_src_port.3
@@ -13,4 +13,5 @@ RDMA identifier
[ewg] RE: rdma retry number
During failover test, we found the iscsi over iser reconnected to the iscs target after 100 seconds due to the default max timeout (8sec) and retry number (15). The max timeout was adjustable with the module parameter, max_timeout, of ib_cm.ko, but the retry number wasn't. Can we add the retry number as module parameter of rdma_cm.ko? I added the patch below based on the ofed version, OFED-1.2-20070626-0917. Note that you can abort a connection operation by destroying the corresponding rdma_cm_id. Does iser try to re-establish a connection over the same path on failover? I'm wondering why it tried to connect over the failed path first. +static int cma_max_cm_retries = CMA_MAX_CM_RETRIES; +module_param_named(cma_max_cm_retries, cma_max_cm_retries, int, 0644); +MODULE_PARM_DESC(cma_max_cm_retries, the number of retry); This must be a value between 0-15. I need to see if there's a better way to support users that want smaller connection timeouts. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: librdmacm library for OFED 1.3
This is the correct tree. You want the latest release, 1.0.3. Likewise for libibcm - release 1.0.1. _ From: Tziporet Koren [mailto:[EMAIL PROTECTED] Sent: Monday, October 08, 2007 1:13 PM To: Hefty, Sean; Vladimir Sokolovsky Cc: Woodruff, Robert J; ewg@lists.openfabrics.org Subject: librdmacm library for OFED 1.3 Hi Sean, In OFED 1.3 we take the librdmacm library from: git://git.openfabrics.org/~shefty/librdmacm.git master Please approve if this is the correct tree we should take, or do you have any branch that should be used Thanks, Tziporet Koren Software Director Mellanox Technologies mailto: mailto:[EMAIL PROTECTED] [EMAIL PROTECTED] Tel +972-4-9097200, ext 380 ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] [PATCH] libibcm: add valgrind support to the libibcm
Thanks - applied. I also created a release 1.0.1 of libibcm and added it to my public html file and downloads directory. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] [PATH librdmacm] rping - persistent server support
Thanks - I've added this to my patch list for the librdmacm. I plan on releasing a new version of the librdmacm next week, pending the acceptance of the kernel quality of service changes, which I'll ask Roland to pull for 2.6.24 after he returns. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [ofa-general] Re: ofa_1_2_c_kernel 20070802-0201 daily build status
If I clone from my local system over the net, I _get_ all the branches! Anybody know why local clones on the ofa build server are not pulling all the branches? Maybe I'm abusing git? It sounds like a different between git versions. Older git versions brought in remote branches such that 'git branch' would show them. (This causes problems if a remote branch conflicts with a local branch.) With the newer git version (what's on the ofa server), you need to use 'git branch -r' to see all of the actual branches. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] RE: OFA website edits
http://www.openfabrics.org/downloads/mpi/mvapich (pasha) http://www.openfabrics.org/downloads/mpi/mvapich2 (rowland) http://www.openfabrics.org/downloads/mpi/openmpi (jsquyres) Are all of these MPI versions distributed by OFA? If they have other official sites, should we instead direct users to that site? Or will this be automated enough that people can provide their own links? - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofa-general] Re: RFC OFED-1.3 installation
I think everyone would be better served by a process where individual maintainers were responsible for releasing tarballs of their packages, with schedules coordinated toward an overall openfabrics release For what it's worth, I agree with this approach. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
