[ewg] [PATCH ofed-1.5.2 2/2] cxgb3: Turn off RX coalescing for iwarp connections.
This drops the connect latency from 200ms to 16. Needed for large NP clusters. Signed-off-by: Steve Wise --- .../fixes/iw_cxgb3_1300_rx_coalesce_off.patch | 24 1 files changed, 24 insertions(+), 0 deletions(-) create mode 100644 kernel_patches/fixes/iw_cxgb3_1300_rx_coalesce_off.patch diff --git a/kernel_patches/fixes/iw_cxgb3_1300_rx_coalesce_off.patch b/kernel_patches/fixes/iw_cxgb3_1300_rx_coalesce_off.patch new file mode 100644 index 000..958b3b5 --- /dev/null +++ b/kernel_patches/fixes/iw_cxgb3_1300_rx_coalesce_off.patch @@ -0,0 +1,24 @@ +diff --git a/drivers/infiniband/hw/cxgb3/iwch_cm.c b/drivers/infiniband/hw/cxgb3/iwch_cm.c +index 52d7bb0..687a647 100644 +--- a/drivers/infiniband/hw/cxgb3/iwch_cm.c b/drivers/infiniband/hw/cxgb3/iwch_cm.c +@@ -486,7 +486,8 @@ static int send_connect(struct iwch_ep *ep) + V_MSS_IDX(mtu_idx) | + V_L2T_IDX(ep->l2t->idx) | V_TX_CHANNEL(ep->l2t->smt_idx); + opt0l = V_TOS((ep->tos >> 2) & M_TOS) | V_RCV_BUFSIZ(rcv_win>>10); +- opt2 = V_FLAVORS_VALID(1) | V_CONG_CONTROL_FLAVOR(cong_flavor); ++ opt2 = F_RX_COALESCE_VALID | V_RX_COALESCE(0) | V_FLAVORS_VALID(1) | ++ V_CONG_CONTROL_FLAVOR(cong_flavor); + skb->priority = CPL_PRIORITY_SETUP; + set_arp_failure_handler(skb, act_open_req_arp_failure); + +@@ -1302,7 +1303,8 @@ static void accept_cr(struct iwch_ep *ep, __be32 peer_ip, struct sk_buff *skb) + V_MSS_IDX(mtu_idx) | + V_L2T_IDX(ep->l2t->idx) | V_TX_CHANNEL(ep->l2t->smt_idx); + opt0l = V_TOS((ep->tos >> 2) & M_TOS) | V_RCV_BUFSIZ(rcv_win>>10); +- opt2 = V_FLAVORS_VALID(1) | V_CONG_CONTROL_FLAVOR(cong_flavor); ++ opt2 = F_RX_COALESCE_VALID | V_RX_COALESCE(0) | V_FLAVORS_VALID(1) | ++ V_CONG_CONTROL_FLAVOR(cong_flavor); + + rpl = cplhdr(skb); + rpl->wr.wr_hi = htonl(V_WR_OP(FW_WROPCODE_FORWARD)); ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH ofed-1.5.2 1/2] rdma_ucm: Up the max backlog to 1024.
This is needed for NP256 sized iwarp clusters.
Signed-off-by: Steve Wise
---
.../fixes/core_0590_ucma_backlog_1024.patch| 13 +
1 files changed, 13 insertions(+), 0 deletions(-)
create mode 100644 kernel_patches/fixes/core_0590_ucma_backlog_1024.patch
diff --git a/kernel_patches/fixes/core_0590_ucma_backlog_1024.patch
b/kernel_patches/fixes/core_0590_ucma_backlog_1024.patch
new file mode 100644
index 000..7e3361f
--- /dev/null
+++ b/kernel_patches/fixes/core_0590_ucma_backlog_1024.patch
@@ -0,0 +1,13 @@
+diff --git a/drivers/infiniband/core/ucma.c b/drivers/infiniband/core/ucma.c
+index 4346a24..f9b8d36 100644
+--- a/drivers/infiniband/core/ucma.c
b/drivers/infiniband/core/ucma.c
+@@ -48,7 +48,7 @@ MODULE_DESCRIPTION("RDMA Userspace Connection Manager
Access");
+ MODULE_LICENSE("Dual BSD/GPL");
+
+ enum {
+- UCMA_MAX_BACKLOG= 128
++ UCMA_MAX_BACKLOG= 1024
+ };
+
+ struct ucma_file {
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH ofed-1.5.2 0/2] Critical bug fixes for ofed-1.5.2
Here are the fixes for bugs 2131 and 2132. You can pull these from: ssh://v...@sofa.openfabrics.org/~swise/scm/ofed_kernel ofed_1_5 It would be great if these can get into ofed-1.5.2. Thanks, Steve. Steve Wise (2): cxgb3: Turn off RX coalescing for iwarp connections. rdma_ucm: Up the max backlog to 1024. .../fixes/core_0590_ucma_backlog_1024.patch| 13 +++ .../fixes/iw_cxgb3_1300_rx_coalesce_off.patch | 24 2 files changed, 37 insertions(+), 0 deletions(-) create mode 100644 kernel_patches/fixes/core_0590_ucma_backlog_1024.patch create mode 100644 kernel_patches/fixes/iw_cxgb3_1300_rx_coalesce_off.patch ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED 1.5.2 schedule
I just opened 2 critical bugs that I'd like to get into ofed-1.5.2. They are needed for larger MPI cluster using iWARP. The bugs are 2131 and 2132. The fix for 2131 hits iw_cxgb3 only and is a small change to turn off rx coalescing for iwarp connections. The fix for 2132 is a 1 liner to change the default ucma backlog from 128 to 1024. These are low impact fixes, and I propose we fold them into ofed-1.5.2, even if this slips GA by one week. I'll be posting patches soon. Steve. On 9/16/2010 11:14 AM, Tziporet Koren wrote: > Just to update all on OFED 1.5.2 schedule > > RC7 should be out today > GA is planned for Tuesday 21-Sep 2010 > > > Tziporet& Vlad > > ___ > ewg mailing list > ewg@lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] ofa_1_5_kernel 20100918-0200 daily build status
This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16.60-0.54.5-smp Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-164.el5 Passed on x86_64 with linux-2.6.18-194.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.27.19-5-smp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-89.ELsmp Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.26 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.19 Failed: ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [Q] NFS/RDMA on RHEL 5.5 x86_64
Dear Open Fabrics members. I'm newbie about Infiniband and Open Fabrics. I'm trying to setup NFS/RDMA on RHEL 5.5 x86_64 I'm reading this document /usr/share/doc/ofed-docs-1.5.2/nfs-rdma.release-notes.txt 1) My environment RHEL-5.5 x86_64 OFED-1.5.2rc7 I executed the following command to install OFED # cd OFED-1.5.2rc7 # ./install.pl OFED Distribution Software Installation Menu 1) Basic (OFED modules and basic user level libraries) 2) HPC (OFED modules and libraries, MPI and diagnostic tools) 3) All packages (all of Basic, HPC) 4) Customize Q) Exit Select Option [1-4]:3 Please choose an implementation of MVAPICH2: 1) OFA (IB and iWARP) 2) uDAPL Implementation [1]: Enable ROMIO support [Y/n]: Enable shared library support [Y/n]: Enable Checkpoint-Restart support [y/N]: Below is the list of OFED packages that you have chosen (some may have been added by the installer due to package dependencies): and reboot IPoIB work fine 2) Question I setuped NFS/RMDA step-by-setp to read nfs-rdma.release-notes.txt But it does not work correctly. I can't know what is wrong. 1) mount.rnfs proto option does not work nfs-rdma.release-notes.txt mount.rnfs :/ /mnt -o proto=rdma,port=20049 I executed mount.rnfs, the following error has occured. mount.rnfs 192.168.100.231:/nfstest /mnt -o port=20049,proto=rdma mount.rnfs: Bad nfs mount parameter: proto # /sbin/mount.rnfs -V mount.rnfs (linux nfs-utils 1.1.5) 2) How to check rmda port ?? echo rdma 20049 > /proc/fs/nfsd/portlist Could you tell me how to check it ( like netstat ) I can't find 20049 port netstat -na | grep 20049 Thank you for your information . Sincerely -- Hiroyuki Sato # /sbin/lsmod | egrep '(ib_mthca|ib_ipoib|svcrdma)' svcrdma70208 1 rdma_cm73492 4 xprtrdma,svcrdma,rdma_ucm,ib_sdp ib_ipoib 114016 0 ib_cm 71592 2 rdma_cm,ib_ipoib ib_sa 75784 4 rdma_ucm,rdma_cm,ib_ipoib,ib_cm ib_mthca 157220 0 ib_mad 72100 6 ib_cm,ib_sa,ib_umad,ib_qib,mlx4_ib,ib_mthca ib_core 109440 17 xprtrdma,svcrdma,rdma_ucm,ib_sdp,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_sa,ib_uverbs,ib_umad,iw_nes,iw_cxgb3,ib_qib,mlx4_ib,ib_mthca,ib_mad sunrpc245672 17 xprtrdma,svcrdma,nfsd,nfs,nfs_acl,auth_rpcgss,lockd ipoib_helper 35728 2 ib_ipoib ipv6 435361 35 nfs,rdma_cm,ib_addr,ib_ipoib,lockd # cat /sys/class/infiniband/mthca0/ports/1/state 4: ACTIVE # /sbin/ifconfig ib0 ib0 Link encap:InfiniBand HWaddr 80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:192.168.100.231 Bcast:192.168.100.255 Mask:255.255.255.0 inet6 addr: fe80::202:c902:22:eb9/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:45 errors:0 dropped:0 overruns:0 frame:0 TX packets:33 errors:0 dropped:6 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:6989 (6.8 KiB) TX bytes:2860 (2.7 KiB) # /sbin/service nfs status rpc.mountd (pid 5280) is running... nfsd (pid 5277 5276 5275 5274 5273 5272 5271 5270) is running... rpc.rquotad (pid 5205) is running... # cat /proc/fs/nfsd/portlist rdma 20049 tcp 2049 udp 2049 # netstat -na -p | grep 20049 no output # netstat -na -p | grep 2049 tcp0 0 0.0.0.0:20490.0.0.0:* LISTEN - udp0 0 0.0.0.0:20490.0.0.0:* - # # # NFS/RDMA README # # # Author: NetApp and Open Grid Computing Adapted for OFED 1.5.1 (from linux-2.6.30/Documentation/filesystems/nfs-rdma.txt) by Jon Mason Table of Contents ~ - Overview - OFED 1.5.1 limitations - Getting Help - Installation - Check RDMA and NFS Setup - NFS/RDMA Setup Overview This document describes how to install and setup the Linux NFS/RDMA client and server software. The NFS/RDMA client was first included in Linux 2.6.24. The NFS/RDMA server was first included in the following release, Linux 2.6.25. In our testing, we have obtained excellent performance results (full 10Gbit wire bandwidth at minimal client CPU) under many workloads. The code passes the full Connectathon test suite and operates over both Infiniband and iWARP RDMA adapters. OFED 1.5.1 limitations: ~ NFS-RDMA is supported for the following releases: - Redhat Enterprise Linux (RHEL) version 5.2 - Redhat Enterprise Linux (RHEL) version 5.3 - Redhat Enterprise Linux (RHEL) version 5.4
[ewg] [PATCH] OpenSM Release Notes update
OpenSM Release Notes update.
Signed-off-by: Sasha Khapyorsky
---
On 10:35 Tue 14 Sep , Vladimir Sokolovsky wrote:
> Please update release notes and other relevant documentation for the packages
> that you own.
>
> URL: git://git.openfabrics.org/~tziporet/docs.git
> Branch: ofed_1_5
opensm_release_notes.txt | 81 -
1 files changed, 79 insertions(+), 2 deletions(-)
diff --git a/opensm_release_notes.txt b/opensm_release_notes.txt
index 9b5de67..bb7c589 100644
--- a/opensm_release_notes.txt
+++ b/opensm_release_notes.txt
@@ -3,14 +3,14 @@
Version: OpenSM 3.3.x
Repo:git://git.openfabrics.org/~sashak/management.git
-Date:Dec 2009
+Date:Sep 2010
1 Overview
--
This document describes the contents of the OpenSM 3.3 release.
OpenSM is an InfiniBand compliant Subnet Manager and Administration,
and runs on top of OpenIB. The OpenSM version for this release
-is opensm-3.3.5.
+is opensm-3.3.7.
This document includes the following sections:
1 This Overview section (describing new features and software
@@ -127,6 +127,13 @@ f02f40e opensm: implement 'connect_roots' option in
fat-tree routing
748d41e opensm SA DB dump/restore: added option to dump SA DB on every sweep
b03a95e complib/cl_fleximap: add cl_fmap_match() function
b7a8a87 opensm/include/iba/ib_types.h: adding Congestion Control definitions
+fa356f8 opensm: Add support for optimized SLtoVLMappingTable programming
+8aaae91 Dimension port order file support
+7662eec opensm: Add option to specify prefix to syslog messages
+2382cf3 opensm: Add update_desc command to opensm console
+7cbe193 opensm: toggle sweeping V3
+6f61d8f opensm/osmeventplugin: added new events to monitor SM
+84cf603 opensm/main.c: force stdout to be line-buffered
1.3 Library API Changes
@@ -247,6 +254,11 @@ fa90512 opensm/osm_vendor_*_sa: fix incompatibility with
QLogic SM
55f9772 opensm: Return single PathRecord for SubnAdmGet with DGID/SGID wild
carded
5ec0b5f opensm: compress IPV6 SNM groups to use a single MLID
+26e7e83 opensm/osm_lid_mgr: fix couple of duplicate LIDs bugs
+fedc419 opensm: Multicast root switch calculation
+6772fdb opensm: Fix sl2vl configuration
+dd3470f complib/cl_timer.c: fixing cl_timer calculation
+ae1bcdd opensm/osm_lid_mgr.c: Allow switch lids to be non LMC aligned
4.2 Other Bug Fixes
@@ -507,6 +519,70 @@ c6b4d4a opensm/osm_vendor_ibumad.c: Add transaction ID to
osm_vendor_send
520af84 opensm/osm_sa_path_record.c: don't set dgid pointer for local subnet
4a878fb opensm/osm_mcast_mgr.c: fix osm_mcast_mgr_compute_max_hops for
managed switch
+7c48590 opensm/osm_log.c: add OSM_LOG_SYS to default flags
+89f7cb6 opensm/osm_lid_mgr: use 'first_time_master_sweep' flag
+0cb7fab opensm: conversion to osm_get_port_by_lid()
+9d14fc0 opensm/osm_lid_mgr.c: fix memory leak
+c364aa1 opensm/opensm.init.in: fix install warning on SLES11
+1010c9c opensm/osm_sa_path_record.c: livelock in pr_rcv_get_path_parms
+4b2cd5e opensm/vendor: fix portguids array size
+52bf5b2 opensm/osm_subnet.c: fixing some options to not "hot-swappable"
+8900da0 opensm/osm_subnet.{c,h}: passing options to the event plugins
+051c57f Delete port only after GID OUT trap was sent
+d4ebf7e opensm/complib/cl_passivelock.h: remove unneeded casting
+8fdb17c opensm/complib/cl_types.h: convert cl_status_t to int
+fd7fb1e opensm/osm_mcast_mgr.c: preserve root switch calculation functionality
+fcb0f3a opensm/osm_mcast_mgr.c: code simplifications
+444f559 opensm/osm_mcast_mgr.c: fix bug in MC root switch calculation
+041ebcb opensm/osm_mcast_mgr.c: remove redundant casting
+3717f53 opensm/osm_sa_pkey_record.c: optimize port selection logic
+48352be opensm/osm_mcast_mgr.c: fix memory leak
+f3cf83f opensm/complib/cl_ptr_vector.c: fix bug/compiler warning
+27c8ebd opensm/osm_subnet.h: remove redundant function definition
+f296938 opensm/osm_vl_arb_rcv.c: fix double mutex release bug
+00bc48e opensm/osm_port_info_rcv.c: fix compilation warning
+8823800 opensm/osm_sa.{c,h}: osm_sa_db_file_dump() return values
+f4581f3 opensm/osm_qos.c: Fix typo in OSM_LOG message
+e3c790a opensm/osm_update_node_desc(): minor prototype improvement
+3cc68cb opensm/osm_vl_arb_rcv.c: Dump table after validating block number
+7dbb96e opensm SA DB: dump only if modified
+fa2106d opensm/osm_sa_infrominfo.c: fixes and simplifications in lid range
check
+051a1dd opensm/osm_qos.c: split switch external and end ports setup
+a6c0189 opensm/osm_qos.c: merge SL2VL mapping capability check
+3fe8efe opensm/osm_slvl_map_rcv.c: verify port number values received from
+ network
+88c372c opensm/osm_slvl_map_rcv.c: fix mutex double release bug
+d282093 opensm/osm_slvl_map_rcv.c: fix port parsing on BE machine
+8e9dbd3 osm_sa_path_record.c: use PR DGID by reference
+7c9d375 osm_sa_path_record.c: separate mutlicast processing code
+cb2d18e opensm/osm_sa_path_record.c: MGID must be specified explicitly
+bd3932b opensm/osm_mcast_mgr.c: str
