Hi,
I am pleased to announce that OFED-1.5.2 GA release is done
Notes:
The tarball is available on:
http://www.openfabrics.org/builds/ofed-1.5.2/release/OFED-1.5.2.tgz
To get BUILD_ID run ofed_info
Please report any issues in bugzilla https://bugs.openfabrics.org/ for
OFED 1.5.2
Vladimir & Tziporet
========================================================================
Release information:
--------------------
Linux Operating Systems:
- RedHat EL4 up7 2.6.9-78.ELsmp * (last release of OFED
supporting this OS)
- RedHat EL4 up8 2.6.9-89.ELsmp * (last release of OFED
supporting this OS)
- RedHat EL5 up3 2.6.18-128.el5 * (last release of OFED
supporting this OS)
- RedHat EL5 up4 2.6.18-164.el5
- RedHat EL5 up5 2.6.18-194.el5
- SLES10 SP2 2.6.16.60-0.21-smp * (last release of OFED
supporting this OS)
- SLES10 SP3 2.6.16.60-0.54-smp
- SLES11 2.6.27.19-5-default
- SLES11 SP1 2.6.32.12-0.7-default
- OEL 4 up7 2.6.9-78.ELsmp * (last release of OFED
supporting this OS)
- OEL 4 up8 2.6.9-89.ELsmp * (last release of OFED
supporting this OS)
- CentOS5.3 2.6.18-128.el5 * (last release of OFED
supporting this OS)
- CentOS5.4 2.6.18-164.el5
- CentOS5.5 2.6.18-194.el5
- Fedora Core12 2.6.31.5-127.fc12 *
- OpenSuSE 11.2 2.6.31.5-0.1-default *
- kernel.org 2.6.29, 2.6.30,
2.6.31, 2.6.32 and 2.6.33 *
* Minimal QA for these versions
Systems:
* x86_64
* x86
* ia64
* ppc64
Main Changes from OFED 1.5.1
============================
1. Added RHEL5.5, SLES11 SP1 and kernel 2.6.33 support
2. /etc/modprobe.conf deprecated
ib_ipoib use /etc/modprobe.d/ib_ipoib.conf
ib_sdp use /etc/modprobe.d/ib_sdp.conf
mlx4_en use /etc/modprobe.d/mlx4_en.conf
3. Added RAW Ethernet QP support in libibverbs
4. Added multicast support in perftest
5. Added new package: ibacm. See ibacm_release_notes.txt
6. Added new package infinipath-psm.
7. Updated packages:
libibverbs to rev 1.1.4-0.14.gb6c138b
libehca to rev 1.2.2-0.1.g69e1a88
libmlx4 to rev 1.0-0.13.g4e5c43f
libnes to rev 1.1.0
librdmacm to rev 1.0.13
dapl to rev 2.0.30
compat-depl to rev 1.2.19
libsdp to rev 1.1.103
mstflint to rev 1.4-0.6.g63179f4
perftest to rev 1.3.0-0.27
rds-tools to rev 2.0.4
Management:
opensm to rev 3.3.7
libibmad to rev 1.3.6
libibumad to rev 1.3.6
infiniband-diags to rev 1.5.7
MPI:
openmpi to rev 1.4.2-1
mvapich2 to rev 1.5.1-2.p1
mpitests to rev 3.2-923
8. Bug fixes
See attached.
Known issues:
- ib_sdp fails to be loaded on 32-bit OSes (will be fixed in OFED-1.5.2-1)
Open Fabrics Enterprise Distribution (OFED)
Version 1.5.2
Release Notes
September 2010
===============================================================================
Table of Contents
===============================================================================
1. Overview, which includes:
- OFED Distribution Rev 1.5.2 Contents
- Supported Platforms and Operating Systems
- Supported HCA and RNIC Adapter Cards and Firmware Versions
- Tested Switch Platforms
- Third party Test Packages
- OFED sources
2. Main Changes from OFED 1.4.2
3. Main Changes from OFED 1.5
4. Main Changes from OFED 1.5.1
5. Known Issues
===============================================================================
1. Overview
===============================================================================
These are the release notes of OpenFabrics Enterprise Distribution (OFED)
release 1.5.2. The OFED software package is composed of several software
modules,
and is intended for use on a computer cluster constructed as an InfiniBand
subnet or iWARP network.
Note: If you plan to upgrade the OFED package on your cluster, please upgrade
all of its nodes to this new version.
1.1 OFED 1.5.2 Contents
-----------------------
The OFED package contains the following components:
- OpenFabrics core and ULPs:
- IB HCA drivers (mthca, mlx4, qib, ehca)
- iWARP RNIC driver (cxgb3, nes)
- core
- Upper Layer Protocols: IPoIB, SDP, SRP Initiator and target, iSER
Initiator and target, RDS, uDAPL, qlgc_vnic and NFS-RDMA.
- OpenFabrics utilities:
- OpenSM (OSM): InfiniBand Subnet Manager
- Diagnostic tools
- Performance tests
- MPI:
- OSU MPI stack supporting the InfiniBand and iWARP interface
- Open MPI stack supporting the InfiniBand and iWARP interface
- OSU MVAPICH2 stack supporting the InfiniBand and iWARP interface
- MPI benchmark tests (OSU benchmarks, Intel MPI benchmarks, Presta)
- Extra packages:
- open-iscsi: open-iscsi initiator with iSER support
- ib-bonding: Bonding driver for IPoIB interface
- Sources of all software modules (under conditions mentioned in the modules'
LICENSE files)
- Documentation
Notes:
1. iSER Target and NFS-RDMA are of Beta quality.
2. All other OFED components are of production quality.
3. See release notes for each package in the docs directory.
4. Any Topspin copyright belongs to Cisco Systems, Inc.
1.2 Supported Platforms and Operating Systems
---------------------------------------------
o CPU architectures:
- x86_64
- x86
- ppc64
- ia64
o Linux Operating Systems:
- RedHat EL4 up7 2.6.9-78.ELsmp * (last release of OFED
supporting this OS)
- RedHat EL4 up8 2.6.9-89.ELsmp * (last release of OFED
supporting this OS)
- RedHat EL5 up3 2.6.18-128.el5 * (last release of OFED
supporting this OS)
- RedHat EL5 up4 2.6.18-164.el5
- RedHat EL5 up5 2.6.18-194.el5
- SLES10 SP2 2.6.16.60-0.21-smp * (last release of OFED
supporting this OS)
- SLES10 SP3 2.6.16.60-0.54-smp
- SLES11 2.6.27.19-5-default
- SLES11 SP1 2.6.32.12-0.7-default
- OEL 4 up7 2.6.9-78.ELsmp * (last release of OFED
supporting this OS)
- OEL 4 up8 2.6.9-89.ELsmp * (last release of OFED
supporting this OS)
- CentOS5.3 2.6.18-128.el5 * (last release of OFED
supporting this OS)
- CentOS5.4 2.6.18-164.el5
- CentOS5.5 2.6.18-194.el5
- Fedora Core12 2.6.31.5-127.fc12 *
- OpenSuSE 11.2 2.6.31.5-0.1-default *
- kernel.org 2.6.29, 2.6.30,
2.6.31, 2.6.32 and 2.6.33 *
* Minimal QA for these versions
1.3 HCAs and RNICs Supported
----------------------------
This release supports IB HCAs by Mellanox Technologies, Qlogic and IBM as
well as iWARP RNICs by Chelsio Communications and Intel.
o Mellanox Technologies HCAs (SDR, DDR and QDR Modes are Supported):
- InfiniHost (fw-23108 Rev 3.5.000)
- InfiniHost III Ex (MemFree: fw-25218 Rev 5.3.000
with memory: fw-25208 Rev 4.8.200)
- InfiniHost III Lx (fw-25204 Rev 1.2.000)
- ConnectX IB (fw-25408 Rev 2.7.700)
For official firmware versions please see:
http://www.mellanox.com/content/pages.php?pg=firmware_download
o Qlogic HCAs:
- QHT7140 QLogic InfiniPath SDR HTX HCA
- QLE7140 QLogic InfiniPath SDR PCIe HCA
- QLE7240 QLogic InfiniPath DDR x8 PCIe HCA
- QLE7280 QLogic IniniPath DDR x16 PCIe HCA
o IBM HCAs:
- GX Dual-port SDR 4x IB HCA
- GX Dual-port SDR 12x IB HCA
- GX Dual-port DDR 4x IB HCA
- GX Dual-port DDR 12x IB HCA
o Chelsio RNICs:
- S310/S320 10GbE Storage Accelerators
- R310/R320 10GbE iWARP Adapters
o Intel RNICs:
- NE020 10Gb iWARP Adapter
1.4 Switches Supported
----------------------
This release was tested with switches and gateways provided by the following
companies:
- Voltaire
- Qlogic
- Flextronics
- Oracle (Sun)
- Mellanox
1.5 Third Party Packages
------------------------
The following third party packages have been tested with OFED 1.5.2:
- Intel MPI, Version 3.2.2
- Intel MPI, Version 4.0 beta
1.6 OFED Sources
----------------
All sources are located under git://git.openfabrics.org/
Kernel sources: git://git.openfabrics.org/ofed_1_5/linux-2.6.git ofed_kernel_1_5
User level Sources are downloaded from http://www.openfabrics.org/downloads/
as written in the BUILD_ID
The kernel sources are based on Linux 2.6.30 mainline kernel. Its patches
are included in the OFED sources directory.
For details see HOWTO.build_ofed.
===============================================================================
2. Main Changes from OFED 1.4.2
===============================================================================
Note: For details regarding the various changes, please see the release notes
for each package in the docs directory.
2.1 General changes
o Kernel code based on 2.6.30
o libraries location - all userspace libraries can be downloaded from
http://www.openfabrics.org/downloads/
See BUILD_ID for exact locations
o Qlogic moved the low level driver from ipath to qib.
2.2 SDP
o Performance improvements
o Zero copy in beta level
2.3 uDAPL
o New UCM provider (ofa-v2-mlx4_0-1u) with IB UD-based CM per process.
It is more scalable than rdma_cm (cma) or socket cm (scm).
o Common code base with WinOF 2.1
o Bug fixes
2.4 perftest
o Renamed tests:
ib_rdma_bw -> rdma_bw
ib_rdma_lat -> rdma_lat
2.5 Management
o OpenSM
- Support for Mesh Analysis for LASH routing algorithm
- Reloadable OpenSM configuration (preliminary implementation)
- Routing paths sorted balancing (for UpDown and MinHops)
- Weighted LID matrices calculation (for UpDown, MinHop and DOR)
- I/O nodes connectivity (for FatTree)
2.6 MPI:
a. OSU MVAPICH 1.2.0
b. Open MPI 1.4
c. OSU MVAPICH2 1.4
d. MPI tests 3.2
2.7 iSER:
o Available on:
- kernel.org: 2.6.30 and higher
- RHEL5.4
2.8 NFS-RDMA
o Added support for RHEL5.4, SLES10 SP3, kernel.org 2.6.25 and 2.6.30.
Kernels 2.6.26 and 2.6.27 are not supported
o NFS-RDMA is in Beta level
===============================================================================
3. Main Changes from OFED 1.5
===============================================================================
1. Added RoCEE support - see RoCEE_README.txt
2. Added enhanced atomic operations to ConnectX (kernel only).
See mlx4_release_notes.txt.
3. Updated Open MPI to rev 1.4.1-2ofed
4. Updated MVAPICH2 to rev 1.4.1
5. Updated DAPL to rev 2.0.27
6. Updated libnes to rev 1.0.1
7. Updated librdmacm to rev 1.0.11
8. Removed tvflash RPM
9. NFS-RDMA is not supported on SLES10 SP3
10. Fixed IPv6 support and IPv4 routing corner cases for RDMA CM
11. Bug fixes
===============================================================================
4. Main Changes from OFED 1.5.1
===============================================================================
1. Added RHEL5.5, SLES11 SP1 and kernel 2.6.33 support
2. /etc/modprobe.conf deprecated
ib_ipoib use /etc/modprobe.d/ib_ipoib.conf
ib_sdp use /etc/modprobe.d/ib_sdp.conf
mlx4_en use /etc/modprobe.d/mlx4_en.conf
3. Added RAW Ethernet QP support in libibverbs
4. Added multicast support in perftest
5. Added new package: ibacm. See ibacm_release_notes.txt
6. Added new package infinipath-psm.
7. Updated packages:
libibverbs to rev 1.1.4-0.14.gb6c138b
libehca to rev 1.2.2-0.1.g69e1a88
libmlx4 to rev 1.0-0.13.g4e5c43f
libnes to rev 1.1.0
librdmacm to rev 1.0.13
dapl to rev 2.0.30
compat-depl to rev 1.2.19
libsdp to rev 1.1.103
mstflint to rev 1.4-0.6.g63179f4
perftest to rev 1.3.0-0.27
rds-tools to rev 2.0.4
Management:
opensm to rev 3.3.7
libibmad to rev 1.3.6
libibumad to rev 1.3.6
infiniband-diags to rev 1.5.7
MPI:
openmpi to rev 1.4.2-1
mvapich2 to rev 1.5.1-2.p1
mpitests to rev 3.2-923
===============================================================================
5. Known Issues
===============================================================================
The following is a list of general limitations and known issues of the various
components of the OFED 1.5.2 release.
1. When upgrading from an earlier OFED version, the installation script does
not stop the earlier OFED version prior to uninstalling it.
Workaround: Stop the old OFED stack (/etc/init.d/openibd stop) before
upgrading to OFED 1.5.2 or reboot the server after OFED installation.
2. Memory registration by the user is limited according to administrator
setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for
system configuration.
3. Fork support from kernel 2.6.12 and above is available provided
that applications do not use threads. fork() is supported as long as the
parent process does not run before the child exits or calls exec().
The former can be achieved by calling wait(childpid), and the latter can be
achieved by application specific means. The Posix system() call is
supported.
4. The qib driver is supported only on 64-bit platforms.
5. When installing OFED on OpenSuse or Ubuntu, use the --without-depcheck
option of the install.pl script.
6. IPoIB: brctl utilities do not work on IPoIB interfaces. The reason for that
is that these utilities support devices of type Ethernet only.
7. "openibd stop" can sometime fail with the error:
Unloading ib_cm [FAILED]
ERROR: Module ib_cm is in use by ib_ipoib
Workaround: run "openibd stop" again or remove ib_ipoib aliases from
/etc/modprobe.d/ib_ipoib.conf.
8. When working with ISCSI over IPoIB or mlx4_en, you must disable LRO (even
if IPoIB is set to connected mode). This is due to a bug in older kernels
which causes a kernel panic.
9. On SLES11, and in case uninstall is failing, check the error log and remove
the remaining RPMs manually using 'rpm -e <rpms list>'.
10. On SLES11, set allow_unsupported_modules parameter to 1 in file:
/etc/modprobe.d/unsupported-modules. Without this the modules will not
load.
11. iSER is supported on kernel.org 2.6.30, 2.6.31 and 2.6.32 only.
OFED-1.5.2 will not install iSER on other kernels, and the original iSER
module that comes with the Linux distribution will stop working due to a
mismatch in the symbols version.
12. On SLES10 SP3, the kernel 2.6.16.60-0.54.5 should be updated to
2.6.16.60-0.59.1 or later. The original kernel may cause kernel panic
during '/etc/init.d/openibd restart'.
Note: See the release notes of each component for additional issues.
bug_id,"bug_severity","priority","op_sys","assigned_to","bug_status","resolution","short_short_desc"
592,"normal","P3","Other","[email protected]","CLOSED","FIXED","libsdp memory leak"
767,"major","P3","Other","[email protected]","RESOLVED","FIXED","Non backport Kernels that don't build in genalloc cause compile errors for cxgb3"
833,"normal","P3","Other","[email protected]","CLOSED","FIXED","IPoib hangout while running sdp with multiple"
977,"normal","P3","All","[email protected]","CLOSED","FIXED","binding 2 sockets to the same address fails with the wrong errno"
1403,"minor","P3","RHEL 5","[email protected]","RESOLVED","FIXED","bad sock_put(RESET) while aborting polygraph"
1422,"normal","P3","All","[email protected]","RESOLVED","FIXED","Running net_perf after other SDP tests results in Badness in sdp_reset_sk and sdp_destroy_sk"
1427,"normal","P3","All","[email protected]","CLOSED","FIXED","running netperf on ppc64, results in preload error and causes the client machine to hang"
1502,"normal","P3","Other","[email protected]","CLOSED","FIXED","2.6.16.46-0.12-SLERT-10-15: scheduling while atomic"
1612,"normal","P3","RHEL 5","[email protected]","CLOSED","FIXED","killing stress_connect test results in Kernel BUG on ppc64 machine"
1682,"normal","P3","Other","[email protected]","CLOSED","FIXED","Low performance on ofed-1.5"
1727,"normal","P3","SLES 11","[email protected]","RESOLVED","FIXED","Select fails after connect to invalid address"
1730,"normal","P3","SLES 11","[email protected]","RESOLVED","FIXED","getsockopt reports error after recv was succeed and the client socket was closed."
1847,"normal","P3","All","[email protected]","RESOLVED","FIXED","zcopy send with iovlen > 1 is not supported"
1872,"normal","P3","All","[email protected]","RESOLVED","FIXED","failed to send packets less than 4080 byte by zcopy"
1888,"normal","P2","All","[email protected]","RESOLVED","FIXED","the server succeed to send packets after filling client's buffers."
1906,"normal","P3","Other","[email protected]","RESOLVED","FIXED","Crash while running iperf"
1909,"normal","P3","All","[email protected]","RESOLVED","FIXED","Enable a trivial way to include sdp_socket.h"
1923,"normal","P4","All","[email protected]","RESOLVED","FIXED","Send & receive out-of-band data failed"
1958,"normal","P3","Other","[email protected]","RESOLVED","FIXED","[OFED-1.5.1] - Unable to recompile OFA kernel -1.5.1 after applying krping patch on 2.6.32 kernel"
1965,"major","P3","RHEL 5","[email protected]","VERIFIED","FIXED","Bonding mlx4_en: ping does not resume after failover between 10G ports"
1977,"normal","P3","RHEL 4","[email protected]","RESOLVED","FIXED","SDP memory leak"
1984,"normal","P3","All","[email protected]","RESOLVED","FIXED","Bug in accept()"
1988,"major","P2","All","[email protected]","RESOLVED","FIXED","[OFED-1.5.1-rc4] - Softirq seen on one of the nodes after ~48 hours while running 2 instances each of IMB-MPI1+ OSU+Presta test together on 64 bit platforms"
1999,"normal","P3","Other","[email protected]","RESOLVED","FIXED","ib_modify_cq is called after socket is closed"
2015,"normal","P3","Other","[email protected]","RESOLVED","FIXED","rds-stress intermittently fails to connect with EADDRNOTAVAIL"
2017,"normal","P3","RHEL 4","[email protected]","RESOLVED","FIXED","so_error_test case 10 causes errors in /var/log/messages"
2021,"normal","P3","Other","[email protected]","RESOLVED","FIXED","Kernel crash due to soft lockup while stopping openibd"
2026,"major","P2","SLES 10","[email protected]","CLOSED","FIXED","What's the current target release date for OFED 1.5.2 ?"
2027,"normal","P3","RHEL 5","[email protected]","CLOSED","FIXED","SDP not respecting # SGEs as reported from HW"
2035,"normal","P3","CentOS 5","[email protected]","RESOLVED","FIXED","Intermittent fabric errors triggers bugs in SDP w/zero-copy enabled"
2038,"major","P3","All","[email protected]","RESOLVED","FIXED","Running 3 instances of ttcp benchmark on PPC64 machines, results in client machine hang"
2040,"major","P3","All","[email protected]","RESOLVED","FIXED","Running 2 instances of ttcp benchmark on PPC64 machines, results in client machine hang"
2041,"normal","P3","Other","[email protected]","RESOLVED","FIXED","QIB compilation fails on RHEL5.5"
2043,"normal","P4","RHEL 5","[email protected]","CLOSED","FIXED","OFED 1.5.1 crashes after running Mcast over RoCE (caused by send_lat benchmark)"
2044,"blocker","P3","Other","[email protected]","CLOSED","FIXED","ofa_kernel doesn't build on RHEL 6 due to spec file problem (patch included)"
2049,"critical","P1","RHEL 5","[email protected]","RESOLVED","FIXED","MPI applications that use fork() hang"
2052,"normal","P3","Other","[email protected]","VERIFIED","FIXED","Very high latency with netperf SDP with msg size 65k"
2053,"blocker","P1","SLES 10","[email protected]","RESOLVED","FIXED","FMODE_EXEC redefined"
2054,"critical","P3","RHEL 4","[email protected]","CLOSED","FIXED","Compilation of OFED 1.5.2 rc1 failed on RH4 up 7"
2056,"major","P3","All","[email protected]","RESOLVED","FIXED","Sdp sockets doesn't close after applications end"
2058,"major","P1","RHEL 5","[email protected]","RESOLVED","FIXED","Kernel Panic when running bidirectional netperf"
2059,"critical","P3","RHEL 5","[email protected]","VERIFIED","FIXED","Compilation on OFED 1.5.2-rc1 failed on RH5 up5 (All packages)"
2063,"major","P3","All","[email protected]","VERIFIED","FIXED","mlx4_en: When disabling both nic ports, and enable one - the link is oscillate"
2068,"normal","P3","Other","[email protected]","RESOLVED","FIXED","rds-stress version 1.5 is missing data on the averages line"
2070,"critical","P3","SLES 11","[email protected]","VERIFIED","FIXED","Failed to compile OFED1.5.2-RC2 on sles11 sp1 - open-iscsi failed"
2073,"blocker","P3","RHEL 5","[email protected]","RESOLVED","FIXED","OFED-1.5.2-20100705-0600 : Mvapich2 rpm fails to install with ofed-1.5.2 packages."
2081,"major","P1","SLES 10","[email protected]","RESOLVED","FIXED","infinipath-psm RPM installation fails on SLES10 SP2"
2082,"major","P1","RHEL 5","[email protected]","RESOLVED","FIXED","SDP module unloads with orphan count message"
2085,"normal","P3","Other","[email protected]","RESOLVED","FIXED","OFED-1.5.2-20100711-0600- MVAPICH2 mpi not installing along with OFED packages in SLES11-SP1-RC4(64 bit) & RHEL6.0-Beta(64 bit)."
2086,"minor","P3","Other","[email protected]","RESOLVED","FIXED","OFED-1.5.2-20100715-600 : OFED kernel fails to compile for rdma_krping module on RHEL6.0-BETA(64 bit ) & SLES11-SP1-RC4(64 bit ) platform."
2090,"major","P1","RHEL 5","[email protected]","RESOLVED","FIXED","SDP connection fails with kernel assertion errros"
2092,"major","P1","Other","[email protected]","RESOLVED","FIXED","Kernel Panic under long run iperf stress"
2095,"normal","P3","All","[email protected]","VERIFIED","FIXED","Error message when restarting the stack - OFED 1.5.2 RC3"
2096,"major","P3","Other","[email protected]","RESOLVED","FIXED","ib-send_* and ib_write_* on ia64 does not work on OFED1.5.2 RC3"
2100,"normal","P3","RHEL 5","[email protected]","RESOLVED","FIXED","Very low bw over SDP with msg higher then 65k"
2101,"major","P3","Other","[email protected]","VERIFIED","FIXED","""ifconfig eth2 down"" command does not remove a NES device from ibv_devinfo"
2104,"major","P3","SLES 11","[email protected]","VERIFIED","FIXED","bonding doesn't detect port failure with NetEffect"
2105,"major","P3","RHEL 5","[email protected]","RESOLVED","FIXED","OFED-1.5.2-20100823-0600 fails to install under RedHat5.4x86"
2116,"normal","P3","Other","[email protected]","RESOLVED","FIXED","mcraw with pause frame disabled cause to machine to hung"
2118,"normal","P3","RedHat Fedora Core 5","[email protected]","RESOLVED","FIXED","Failed to build libmlx4 RPM for OFED-1.5.2 on RHEL"
2126,"blocker","P3","RHEL 5","[email protected]","RESOLVED","FIXED","Failed to build mpitests RPM on RHEL5.5"
2128,"normal","P3","Other","[email protected]","RESOLVED","FIXED","UD inline packets with messae size > 884B fails (and does not announce failure)."
