Hi Check,

You are correct that OFED 4.x does not support any 6.x RHEL based distros
(see attached readme).

I will bring up this topic in tomorrow's EWG meeting.



-----Original Message-----
From: Chuck Lever [mailto:chuck.le...@oracle.com] 
Sent: Friday, August 04, 2017 11:29 AM
To: RSD@SFI <rsda...@soft-forge.com>
Cc: Jeff Becker <jeffrey.c.bec...@nasa.gov>; ewg@lists.openfabrics.org
Subject: Re: [ewg] NFS/RDMA question

Thanks Rupert.

I'm thinking out loud here, but my interest is with the many many
deployments that are forced to remain on EL6-based distributions
(Scientific, CentOS) due to application certification constraints.
This includes many in the HPC community, and those who are doing chip CAD,
for example.

OFED / MOFED seems to be the only way to deliver working NFS/RDMA in that
class of environment, as it replaces the NFS/RDMA code that is in Red Hat's
EL6 kernel with newer 3.18 code.

As I understand it, OFED 4 is not supported on EL6-based systems.

> On Aug 3, 2017, at 5:57 PM, RSD@SFI <rsda...@soft-forge.com> wrote:
> OFED Bugzilla is located here: http://bugs.openfabrics.org/
> We are now working on OFED 4.8-1 and hope to quickly move to OFED 4.14 so
I think it is unlikely that the EWG would spin a new release for OFED 3.18
but we would be glad to know of the issue so we can resolve it if it still
exist in OFED 4.8-1.
> Rupert
> -----Original Message-----
> From: ewg [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Jeff 
> Becker
> Sent: Thursday, August 03, 2017 5:44 PM
> To: Chuck Lever <chuck.le...@oracle.com>; ewg@lists.openfabrics.org
> Subject: Re: [ewg] NFS/RDMA question
> I think, e.g., 3.18.3 is still supported - forwarding to EWG to make sure.
> On 08/03/2017 01:15 PM, Chuck Lever wrote:
>> I had one more question about OFED 3.x. Is it still "supported" -- 
>> meaning, if an admin applies OFED 3 to, say, CentOS 6, and encounters 
>> a bug, who does she contact for help?
>> --
>> Chuck Lever
> _______________________________________________
> ewg mailing list
> ewg@lists.openfabrics.org
> http://lists.openfabrics.org/mailman/listinfo/ewg

Chuck Lever

            Open Fabrics Enterprise Distribution (OFED)
                  Version 4.8 Release Notes
                           July 2017

Table of Contents
1. Overview, which includes:
        - OFED Distribution Rev 4.8 Contents
        - Supported Platforms and Operating Systems
        - Supported HCA and RNIC Adapter Cards and Firmware Versions
        - Tested Switch Platforms
        - Third party Test Packages
        - OFED sources
2. Change log
3. Known Issues

1. Overview
These are the release notes of OpenFabrics Enterprise Distribution (OFED)
release 4.8. The OFED software package is composed of several software modules,
and is intended for use on a computer cluster constructed as an InfiniBand
Fabric, an iWARP Network or a RoCE Fabric.

Note: If you plan to upgrade the OFED package on your cluster, please upgrade
all of its nodes to this new version.

1.1 OFED 4.8 Contents
The OFED package contains the following components:
 - OpenFabrics core and ULPs:
        - IB HCA drivers (mthca, mlx4, mlx5, qib, ehca)
        - iWARP RNIC driver (cxgb3, cxgb4, nes)
        - RoCE drivers (mlx4, ocrdma)
 - IB core
 - Upper Layer Protocols: IPoIB, SRP Initiator and target, RDS,
   uDAPL, qlgc_vnic and NFS-RDMA*.
 - OpenFabrics utilities:
        - OpenSM (OSM): InfiniBand Subnet Manager
        - Diagnostic tools
        - Performance tests
 - Extra packages:
        - infinipath-psm: Performance-Scaled Messaging API, an accelerated
          interface to Intel(R) HCAs
        - Packages for Intel(R) Xeon Phi(TM) coprocessor systems (libibscif, 
        - IBSCIF Driver (ibscif)
        - libfbric - library that exports interfaces for fabric services to 
        - CCL-Direct host-side drivers for Intel(R) Xeon Phi(TM) coprocessor 
        - HCA proxy (ibp_server)
        - Connection Manager proxy (ibp_cm_server)
        - Subnet Administrator proxy (ibp_sa_server)
        - Sources of all software modules (under conditions mentioned in the 
                LICENSE files)
        - Documentation
        - VMware Paravirtual RDMA Driver (vmw_pvrdma) as Tech Preview.

1.2 Supported Platforms and Operating Systems
  o   CPU architectures:
        - x86_64
        - x86
        - ppc64

  o   Linux Operating Systems:
        - RedHat EL7.0          3.10.0-123.el7
        - RedHat EL7.1          3.10.0-210.el7
        - RedHat EL7.2          3.10.0-327.el7
        - RedHat EL7.3          3.10.0-514.el7
        - SLES12                3.12.28-4
        - SLES12.1              3.12.49-11.1
        - SLES12.2              4.4.21-69-default
        - kernel.org            4.8    *

      * Minimal QA for these versions.

1.3 HCAs and RNICs Supported
This release supports IB HCAs by IBM, Intel and Mellanox Technologies, iWARP 
RNICs by Chelsio Communications and Intel and RoCE adapters by Emulex, IBM and 

InfiniBand Adapters
  o   IBM HCAs:
        - GX Dual-port SDR 4x IB HCA
        - GX Dual-port SDR 12x IB HCA
        - GX Dual-port DDR 4x IB HCA
        - GX Dual-port DDR 12x IB HCA

  o   Intel (formerly QLogic) HCAs:
        - Intel(R) True Scale DDR PCIe x8 and x16 HCAs
        - Intel(R) True Scale QDR PCIe x8 Gen2 HCAs

  o   Mellanox Technologies HCAs (SDR, DDR and QDR Modes are Supported):
        - ConnectX(R) and ConnectX EN (Rev 2.9.1000)
        - ConnectX-2 (Rev 2.9.1200 and above)

  o   Mellanox Technologies HCAs (FDR and FDR10 Modes are Supported):
        - ConnectX-3 (Rev 2.33.5100 and above)
        - ConnectX-3 Pro (Rev 2.33.5100 and above)

  o   Mellanox Technologies HCAs (FDR and FDR10 Modes are Supported):
        - Connect-IB (Rev 10.10.5054 and above)

        For official firmware versions please see:

iWARP Adapters
  o   Chelsio RNICs:
        - S310/S320 10GbE Storage Accelerators
        - R310/R320 10GbE iWARP Adapters
        - T4: T420-CR, T440-CR, T422-CR, T404-BT, T440-LP-CR, T420-LL-CR, 
        - T5: T502-BT, T580-CR, T580-LP-CR, T520-LL-CR, T520-CR, T522-CR, 
        - T6: T62100-CR, T62100-LP-CR, T6225-CR

  o   Intel RNICs:
        - NE020 10Gb iWARP Adapter

RoCE Adapters
 o   Emulex - using ocrdma
        - Emulex OCe14102 2-port 10 GbE RoCE
        - Emulex OCe14401 1-port 40 GbE RoCE

 o   IBM
        - IBM Flex System EN4132 2-port 10 GbE RoCE
        - IBM EL27 PCIe LP 2-Port 10GbE RoCE SFP+ adapter
        - IBM EC28 PCIe 2-Port 10GbE RoCE SFP+ adapter

  o   Mellanox
        - ConnectX-2 EN (Rev 2.9.1200 and above)
        - ConnectX-3 EN (Rev 2.31.5050 and above)

1.4 Switches Supported
This release was tested with switches and gateways provided by the following
InfiniBand Switches
  o   Flextronics
        - F-X430044
  o   Intel (formerly QLogic)
        - 12200
  o   Mellanox
        - MLNX-OS MSX6036/SX6025 w/w MLNX-OS version 3.3.4304
        - Grid Director 4036 w/w Grid Director version 3.9.2-992
        - FabricIT EFM IS5035 w/w FabricIT EFM version 1.1.3000
        - FabricIT BXM MBX5020 w/w FabricIT BXM version 2.1.2000

iWARP Switches
  o   Fujitsu
        - XG2000C 10Gb Ethernet Switch

RoCE Switches
  o   Arista
  o   BLADE Network Technologies (BNT)
  o   Mellanox
        - SX1036
        - SX1024
        - SX1016

1.5 Third Party Packages
The following third party packages have been tested with OFED 4.8:
- Open MPI - 1.8
- Intel MPI Library 2017

1.6 OFED Sources
All sources are located under git://git.openfabrics.org/

URL: git://git.openfabrics.org/compat-rdma/linux-4.8.git
Branch: master

- Linux kernel sub-tree that includes files relevant for the OFED project only.
  Based on v4.8. Used to shorten git clone time.
  Note: the regular Linux git tree can be used as well.

URL: git://git.openfabrics.org/compat-rdma/compat.git
Branch: ofed

- Based on compat project (https://github.com/mcgrof/compat). The compat module
  provides functionality introduced in newer kernels to older kernels through a
  set of header files and exported symbols.
  See https://github.com/mcgrof/compat/wiki for details.
- Used to replace kernel_addons in the previous OFED kernel tree.

URL: git://git.openfabrics.org/compat-rdma/compat-rdma.git
Branch: master

User level Sources are downloaded from http://www.openfabrics.org/downloads/
as written in the BUILD_ID

The kernel sources are based on Linux 4.8 mainline kernel. Its patches
are included in the OFED sources directory.
For details see HOWTO.build_ofed.

The list of maintainers is available under:

2. Change log
OFED-4.8 Main Changes from OFED 3.18-3 GA
1. Added support for RHEL7.3 and SLES12 SP2

2. compat-rdma:
- BACKPORT-ib_srp: synced with MLNX_OFED ib_srp backport
- IB/srp: force reconnect_delay module param to exceed fast_io_fail_tmo
- Final updates to ccl ABI between host and card to match OFED 4.8 and kernel 
- Fix rstream timeouts on 4k_lat test
- Support for Kernel 4.8 in QIB on OFED-4.8
- compat-rdma: Add i40iw to openibd
- Interop related fixes for iWarp drivers i40iw and nes
- configure: Fix typo
- compat: Cleanup auto-generated defines
- iw_cxgb4: Guard against null cm_id in dump_ep/qp
- Xeon Phi updates
- Fixes for i40iw which have been included in kernels > 4.8
- Updated 0003-add-the-ibp-client-and-server-drivers.patch

3. Updated packages:
- compat-rdma-4.8 - based on linux-4.8
- infiniband-diags-2.0.0
- Removed libibmad (deprecated)
- libfabric-1.4.2
- fabtests-1.4.2
- rdma-core v13
- libibscif-1.1.3
- rdma-core-12-1
- dapl-2.1.10
- fabtests-1.4.1rc1
- infiniband-diags-1.6.7
- libfabric-1.4.1rc1
- libibmad-1.3.13
- libibscif-1.1.2
- libvmw_pvrdma-1.0.0
- mstflint-4.5.0-1.17.g8a0c39d
- perftest-3.4-0.9.g98a9a17

4. install.pl
- Disable inband support for mstflint

3. Known Issues
The following is a list of general limitations and known issues of the various
components of the OFED 4.8 release.

01. When upgrading from an earlier OFED version, the installation script does
    not stop the earlier OFED version prior to uninstalling it.
    Workaround: Stop the old OFED stack (/etc/init.d/openibd stop) before
    upgrading to OFED 4.8 or reboot the server after OFED installation.
02. Memory registration by the user is limited according to administrator
    setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for
    system configuration.
03. Fork support from kernel 2.6.12 and above is available provided
    that applications do not use threads. fork() is supported as long as the
    parent process does not run before the child exits or calls exec().
    The former can be achieved by calling wait(childpid), and the latter can be
    achieved by application specific means.  The Posix system() call is
04. The qib driver is supported only on 64-bit platforms.
05. IPoIB: brctl utilities do not work on IPoIB interfaces. The reason for that
    is that these utilities support devices of type Ethernet only.
06. In case uninstall is failing, check the error log and remove
    the remaining RPMs manually using 'rpm -e <rpms list>'.
07. RDS is not supported.
08. Bug 2515: when an Intel HCA is attached directly to a Mellanox ConnectX3
    and the OpenSM is started on the Intel HCA, the link will not go to active.
    The workaround is to start the OpenSM on the Mellanox HCA.
09. Bug 2553: Unable to run UD traffic using qperf on a RoCE R-NIC
10. Bug 2561: Devices do not receive the echo reply of the first ping sequence 
    when the payload size is 65493 or higher. This is still an issues in the 
    4.3 kernel and should be resolved in a future OFED 4.x release
11. Bug 2574: Installing OFED (with --with_xeon_phi and qib=y in ofed.conf file)
    makes the access to the scif char device fail.
    psmd uses scif functions to allow psm to connect to the Phi devices. 
    It also makes sure it can connect to them when the service is started.
    Scif functions cannot be called unless the mpss service is running.
    Since openibd starts the psmd service, it should be loaded after mpss
12. Bug 2580:  NFSoRDMA client on PPC64LE will crash while trying to mount to 
    a NFSoRDMA server running on x86 box. This will be resolved in OFED 4.8
13. The ipath and eHCA drivers will be deprecated in the next OFED build that 
    uses a kernel >= 4.3 
14. Bug 2583: "iwpm_mapping_error_cb: Received msg seq = 381240 err code = 12
        client = 3" seen intermittently
15. Bug 2587: autoconf is needed by ibacm-1.1.0-1.x86_64
16. Bug 2596: Host crash related to mlx5 driver, ConnectIB HCAs, ib_srp, and
    performing IO
17. cmake and ninja packages are required to build rdma-core package.
        ninja binary RPMs can be downloaded from 
        On SLES12 cmake-3.5 or higher is required. The working cmake version 
can be
        taken from SLES12 SP2 and installed on SLES12 SP0 or SP1.
18. Bug 2622: ib_qib cannot be loaded after installing OFED 4.8 on RHEL 7.3 with
        stock kernel (3.10.0-514). Modeprobe fails with "Invalid argument". It 
        needed to pull a kernel update to 3.10.0-514.10.2 before installing 
19. Bug 2619:  IPoIB interface fails on RHEL 7.2 using FW release 12.18.2000 
with mlx5 MT_2180110032 adapter
        Issue: Trying to bring ib0 interface fails with 
        ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, 
status -22
        Workaround: revert MT_2180110032 firmware from 12.18.2000 to 12.12.1100.
20. NFSoRDMA is not supported
21. Bug 2628: rstream timeouts on 4k_lat test, qib driver
22. SRP issues on RHEL7.2. Bugs: 2631, 2632 and 2634
        Workaround with the SRP issue:
                It is possible to install the ib_srp-backport driver on top of 
OFED as follow:
                1. Install the OS (RHEL, SLES, ...).
                2. Install OFED.
                3. Install the ib_srp-backport driver, e.g. as follows 
(assuming $USER !=root):
                git clone https://github.com/bvanassche/ib_srp-backport
                cd ib_srp-backport
                make rpm
                sudo rpm -U $PWD/rpmbuilddir/RPMS/*/*.rpm

Note: See the release notes of each component for additional issues.
ewg mailing list

Reply via email to