|
I am happy to announce that OFED 1.0 Official Release is now available. The release can be found under: https://openib.org/svn/gen2/branches/1.0/ofed/releases/
And later today it will be on the OpenFabrics download page: http://www.openfabrics.org/downloads.html.
This is the first release that was done in a joint effort of the following companies:
I wish to thank all who contributed to the success of this release.
Tziporet ===============================================================================
Release summary: The OFED software package is composed of several software modules intended for use on a computer cluster constructed as an InfiniBand network.
OFED package contains the following components: o OpenFabrics core and ULPs: - HCA drivers (mthca, ipath) - core - Upper Layer Protocols: IPoIB, SDP, SRP Initiator, iSER Host, RDS and uDAPL o OpenFabrics utilities: - OpenSM: InfiniBand Subnet Manager - Diagnostic tools - Performance tests o MPI: - OSU MPI stack supporting the InfiniBand interface - Open MPI stack supporting the InfiniBand interface - MPI benchmark tests (OSU BW/LAT, Pallas, Presta) o Sources of all software modules (under conditions mentioned in the modules' LICENSE files) o Documentation
Notes: 1. SDP and RDS are in technology preview state. 2. The SRP Initiator and Open MPI are in beta state. 3. All other OFED components are in production state.
Supported Platforms and Operating Systems CPU architectures: * x86_64 * x86 * ia64 * ppc64
Linux Operating Systems: * RedHat EL4 up2: 2.6.9-22.ELsmp * RedHat EL4 up3: 2.6.9-34.ELsmp * Fedora C4: 2.6.11-1.1369_FC4 * SLES10 RC2: 2.6.16.16-1.6-smp (or RC 2.5 2.6.16.14-6-smp) * SLES10 RC1: 2.6.16.14-6-smp * SUSE 10 Pro: 2.6.13-15-smp * kernel.org: 2.6.16.x
HCAs Supported
Mellanox HCAs: - InfiniHost - InfiniHost III Ex (both modes: with memory and MemFree) - InfiniHost III Lx Both SDR and DDR mode of the InfiniHost III family are supported.
For official FW versions please see: http://www.mellanox.com/support/firmware_table.php
Qlogic HCAs: - QHT6040 (PathScale InfiniPath HT-460) - QHT6140 (PathScale InfiniPath HT-465) - QLE6140 (PathScale InfiniPath PE-880)
Switches Supported This release was tested with switches and gateways provided by the following companies: - Cisco - Voltaire - SilverStorm - Flextronics
Attached are the release notes
Tziporet Koren Software Director Mellanox Technologies mailto: [EMAIL PROTECTED]
|
Open Fabrics Enterprise Distribution
Release Notes
Rev 1.0
June 2006
1. Overview
===========
These are the release notes of Open Fabrics Enterprise Distribution (OFED)
release 1.0. The OFED software package is composed of several software modules
intended for use on a computer cluster constructed as an InfiniBand network.
Note: If you plan to upgrade the OFED package on your cluster, please upgrade
all its nodes to this new version.
This document is organized into the following sections:
* Overview, which includes:
- OFED Distribution Rev 1.0 Contents.
- Supported Platforms and Operating Systems.
- Supported HCA Adapter Cards and Firmware Versions.
- Tested Switch Platforms.
* OFED Utilities.
* Debug HOWTOs.
* User Memory Pages for Pinning.
* Known Issues.
1.1 OFED 1.0 Contents
---------------------
OFED package contains the following components:
o OpenFabrics core and ULPs:
- HCA drivers (mthca, ipath)
- core
- Upper Layer Protocols: IPoIB, SDP, SRP Initiator, iSER Host, RDS and
uDAPL
o OpenFabrics utilities:
- OpenSM: InfiniBand Subnet Manager
- Diagnostic tools
- Performance tests
o MPI:
- OSU MPI stack supporting the InfiniBand interface
- Open MPI stack supporting the InfiniBand interface
- MPI benchmark tests (OSU BW/LAT, Pallas, Presta)
o Sources of all software modules (under conditions mentioned in the
modules'
LICENSE files)
o Documentation
Notes:
1. SDP, RDS and Open MPI are in technology preview state.
2. The SRP Initiator is in beta state. All other OFED components are in
production state.
3. See release notes for each package in the docs directory.
4. Any Topspin copyright belongs to Cisco Systems, Inc.
1.2 Supported Platforms and Operating Systems
---------------------------------------------
CPU architectures:
* x86_64
* x86
* ia64
* ppc64
Linux Operating Systems:
* RedHat EL4 up2: 2.6.9-22.ELsmp
* RedHat EL4 up3: 2.6.9-34.ELsmp
* Fedora C4: 2.6.11-1.1369_FC4
* SLES10 RC2: 2.6.16.16-1.6-smp (or RC 2.5 2.6.16.14-6-smp)
* SLES10 RC1: 2.6.16.14-6-smp
* SUSE 10 Pro: 2.6.13-15-smp
* kernel.org: 2.6.16.x
1.3 HCAs Supported
------------------
This release supports HCAs from Mellanox Technologies and Qlogic.
Mellanox HCAs:
- InfiniHost
- InfiniHost III Ex (both modes: with memory and MemFree)
- InfiniHost III Lx
Both SDR and DDR mode of the InfiniHost III family are supported.
For official FW versions please see:
http://www.mellanox.com/support/firmware_table.php
Qlogic HCAs:
- QHT6040 (PathScale InfiniPath HT-460)
- QHT6140 (PathScale InfiniPath HT-465)
- QLE6140 (PathScale InfiniPath PE-880)
1.4 Switches Supported
----------------------
This release was tested with switches and gateways provided by the following
companies:
- Cisco
- Voltaire
- SilverStorm
- Flextronics
2. OFED Utilities
=================
The OFED package includes utilities under <prefix>/bin (prefix stands for the
OFED installation path).
Notes:
1. In this document we describe only a few utilities. Other utilities are useful
too. Use the --help flag to learn about each utility.
2. The sources for all utilities are not part of the RPM installation. However,
all sources exist in the openib-1.0.tgz tarball.
2.1 Device Information
----------------------
Device information can be obtained using several utilities:
a. ibv_devinfo
usage:
ibv_devinfo print the ca attributes
Options:
-d, --ib-dev=<dev> use IB device <dev> (default first device found)
-i, --ib-port=<port> use port <port> of IB device (default all ports)
-l, --list print only the IB devices names
-v, --verbose print all the attributes of the IB device(s)
b. ibstat
usage:
ibstat -d(ebug) -l(ist_of_cas) -s(hort) -p(port_list)] <ca_name>
[portnum]
Examples:
ibstat -l # list all IB devices
ibstat mthca0 2 # stat port 2 of mthca0
c. Using sysfs file system
The driver supports the sysfs file system under: sys/class/infiniband
Examples:
$> ls /sys/class/infiniband/mthca0/
board_id device fw_ver hca_type hw_rev node_desc node_guid node_type
ports
sys_image_guid
$> cat /sys/class/infiniband/mthca0/board_id
MT_0200000001
$> ls /sys/class/infiniband/mthca0/ports/1/
cap_mask counters gids lid lid_mask_count phys_state pkeys rate sm_lid
sm_sl state
$> cat /sys/class/infiniband/mthca0/ports/1/state
4: ACTIVE
2.2 Performance Tests
---------------------
The following performance tests are provided with the OFED release:
1. Latency tests:
- ib_read_lat: RDMA read
- ib_write_lat: RDMA write
- ib_send_lat: UD, UC and RC (default) send
2. Bandwidth tests:
- ib_read_bw: RDMA read
- ib_write_bw: RDMA write
- ib_send_bw: UD, UC and RC (default) send
Usage:
Server: <test name> <options>
Client: <test name> <options> <server IP address>
--help lists the available <options>. The same options must be
passed to both server and client.
<server address> is an Ethernet or IPoIB address.
Example: ib_send_bw
Usage:
ib_send_bw start a server and wait for connection
ib_send_bw <host> connect to server at <host>
options:
-p, --port=<port> listen on/connect to port <port>
(default 18515)
-d, --ib-dev=<dev> use IB device <dev> (default first
device found)
-i, --ib-port=<port> use port <port> of IB device (default 1)
-c, --connection=<RC/UC/UD> connection type RC/UC/UD (default RC)
-m, --mtu=<mtu> mtu size (default 1024)
-s, --size=<size> size of message to exchange (default
65536)
-a, --all run sizes from 2 up to 2^23
-t, --tx-depth=<dep> size of tx queue (default 300)
-n, --iters=<iters> number of exchanges (at least 2,
default 1000)
-b, --bidirectional measure bidirectional bandwidth
(default is unidirectional)
-V, --version display version number
2.3 Ping-pong Example Tests
---------------------------
The ping-pong example tests provide basic connectivity tests. Each test
has a help message (-h).
- ibv_ud_pingpong
- ibv_rc_pingpong
- ibv_srq_pingpong
- ibv_uc_pingpong
Example: ibv_ud_pingpong --h
Usage:
ibv_ud_pingpong start a server and wait for connection
ibv_ud_pingpong <host> connect to server at <host>
options:
-p, --port=<port> listen on/connect to port <port>
(default 18515)
-d, --ib-dev=<dev> use IB device <dev> (default first
device found)
-i, --ib-port=<port> use port <port> of IB device (default 1)
-s, --size=<size> size of message to exchange (default
2048)
-r, --rx-depth=<dep> number of receives to post at a time
(default 500)
-n, --iters=<iters> number of exchanges (default 1000)
-e, --events sleep on CQ events (default poll)
3. Debug HOWTOs
===============
3.1 What is Installed on the Machine
------------------------------------
3.1.1 OFED Components
The text file <prefix>/BUILD_ID provides data on all OFED components (whether
installed or not). For example:
$> cat /usr/local/ofed/BUILD_ID
OFED-1.0
openib-1.0 (REV=7925)
# User space
https://openib.org/svn/gen2/branches/1.0/src/userspace
# Kernel space
https://openib.org/svn/gen2/branches/1.0/ofed/tags/1.0/linux-kernel
Git:
ref: refs/heads/for-2.6.17
commit d9ec5ad24ce80b7ef69a0717363db661d13aada5
# MPI
mpi_osu-0.9.7-mlx2.1.0.tgz
openmpi-1.1b1-1.src.rpm
mpitests-1.0-0.src.rpm
3.1.2 Installed OFED Components
-------------------------------
The script /etc/infiniband/info provides data on the specific OFED installation
on this machine. For example:
$> /etc/infiniband/info
prefix=/usr/local/ofed
Kernel=2.6.9-22.ELsmp
MODULES: CONFIG_INFINIBAND=m CONFIG_INFINIBAND_USER_MAD=m
CONFIG_INFINIBAND_USER_ACCESS=m CONFIG_INFINIBAND_ADDR_TRANS=y
CONFIG_INFINIBAND_MTHCA=m CONFIG_IPATH_CORE=m CONFIG_INFINIBAND_IPATH=m
CONFIG_INFINIBAND_IPOIB=m
User level: --kernel-version 2.6.9-22.ELsmp --kernel-sources
/lib/modules/2.6.9-22.ELsmp/build --with-libibcm --with-libibverbs
--with-libipathverbs --with-libmthca --with-mstflint --with-perftest
3.2 Install Driver in Debug Mode
--------------------------------
You can install the driver with debug messages enabled. To do this, enter prior
to running install.sh:
export OPENIB_PARAMS="--debug"
Alternatively, you can place this export command in the file ofed.conf.
4. User Memory Pages for Pinning
================================
Locking memory is managed by the kernel on a per user basis. Regular users (as
opposed to root) have a limited number of pages which they may pin, where the
limit is pre-set by the administrator. Registering memory for IB verbs
requires pinning memory, thus an application cannot register more memory than
it is allowed to pin. The user can change the system setting as described below.
Note: The steps described next will allow any user in the system to lock as
much as all the memory set in the configuration files.
- Edit the file /etc/security/limits.conf and add the following two lines:
soft memlock <number>
hard memlock <number>
where <number> denotes the number of KBytes that may be locked by a process.
In some systems it may be possible to use "unlimited" for the size to disable
this limit.
Note: The file /etc/security/limits.conf contains further documentation.
5. Known Issues
===============
The following is a list of major limitations and known issues of the various
components of the OFED 1.0 release.
Note: Refer to the component-specific release notes for more details.
1. Memory registration by user is limited according to the administrator
setting. See "User Memory Pages for Pinning" for system configuration.
2. The OFED installation does not support installing lib32 on 64-bit systems
Only libsdp is available in both lib32 and lib64.
3. On RedHat EL4 up2 and Fedora Core 4 the driver may not load properly if
SELINUX is enforced.
Workaround: Change the value of the parameter SELINUX in
/etc/sysconfig/selinux from "enforcing" to "permissive" or "disabled".
4. Registration of huge page memory buffers is not supported. On some systems
attempting to register huge pages may cause unexpected results.
5. Fork support from kernel 2.6.12 and above is available provided that
applications do not use threads. Only system() or fork() and immediate
exec() are supported.
6. MVAPICH and Open MPI do not work on ppc64.
7. On x86_64 systems with Fedora Core 4, if the sysfsutils-devel i386 version
of the package is present, then a build of libibverbs will fail.
Note: See the release notes of each component for additional issues.
_______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
