Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI
This information is Copyright (c) 2010, Oracle and/or its affiliates. All
rights reserved.
1. Introduction
1.1. Project/Component Working Name:
Fibre Channel over IB Support
1.2. Name of Document Author/Supplier:
Author: Bill Taylor
1.3 Date of This Document:
30 June, 2010
4. Technical Description
Fibre Channel over IB Support
-----------------------------
4.1 Background
The Mellanox BridgeX product provides a gateway between InfiniBand
(IB) and 10 Gb Ethernet and/or 2/4/8 Gb Fibre Channel (FC). The
bridging protocol between IB and Fibre Channel is called "Fibre
Channel over IB" (FCoIB). Mellanox has added IB host adapter
extensions to the ConnectX IB adapter to support the BridgeX FCoIB
protocol. This case introduces interfaces into our Solaris InfiniBand
Transport Framework (IBTF, PSARC/2002/132 and follow-on cases) for the
Mellanox host adapter extensions used to support FCoIB initiator.
The expected consumer for FCoIB support is our "hermon" ConnectX IB
driver (PSARC/2008/497) and the FCoIB initiator driver.
4.2 Proposal
The proposal is to make additions to the IBTF Transport Interface (TI)
for IB Upper Level Protocol (ULP) clients and the Channel Interface
(CI) for HCA drivers.
All interface changes and additions in the proposal have a micro/patch
binding.
Transport Interface (ON Consolidation Private)
ibt_hca_flags2_t: add IBT_HCA2_FC flags
ibt_hca_attr_t: add hca_rfci_max_log2_qp, hca_fexch_max_log2_qp,
hca_fexch_log2_mem fields
ibt_chan_alloc_flags_t: add IBT_ACHAN_USES_RFCI, IBT_ACHAN_USES_FCMD,
IBT_ACHAN_USES_FEXCH flags
ibt_fc_attr_t: new struct for FC QP attributes
ibt_ud_chan_alloc_args_t: add ud_fc (ibt_fc_attr_t) field
ibt_fexch_query_attr_t: new struct for FEXCH query attributes
ibt_ud_chan_query_attr_t: add ud_fc (ibt_fc_attr_t), ud_query_fc
(ibt_fexch_query_attr_t) fields
ibt_cep_modify_flags_t: add IBT_CEP_SET_FEXCH_RANGE flag
ibt_ud_chan_modify_attr_t: add uc_fc (ibt_fc_attr_t) field
ibt_tran_srv_t: add IBT_RFCI_SRV, IBT_FCMD_SRV, IBT_FEXCH_SRV
ibt_wrc_opcode_t: add IBT_WRC_INIT_SEND_FCMD
ibt_wr_flags_t: add IBT_WR_SEND_FC_CRC flag
ibt_wr_rfci_send_t: new struct for RFCI version of Send WR
ibt_wr_init_send_t: new struct for FCMD WR to start I/O operation
ibt_wr_fc_t: new union of possible FC WRs
ibt_send_wr_t wr field: add fc (ibt_wr_fc_t) to union
ibt_wc_flags_t: add IBT_WC_FEXCH_FMT, IBT_WC_DIF_ERROR flags
ibt_wc_t wc_detail field: add IBT_WC_DETAIL_FEXCH_INIT_XFER,
IBT_WC_FEXCH_LAST, IBT_WC_DETAIL_RFCI_CRC_OK,
IBT_WC_DETAIL_FC_MATCH_MASK
ibt_wc_t: add wc_fexch_seq_cnt, wc_fexch_tx_bytes, wc_rx_bytes_xfer,
wc_fexch_seq_id fields
ibt_async_code_t: add IBT_FEXCH_ERROR
ibt_fc_syndrome_t: new enum for FEXCH async errors
ibt_async_event_t: add ev_fc (ibt_fc_syndrome_t)
Channel Interface (ON Consolidation Private)
ibt_hca_flags2_t: add IBT_HCA2_FC flags
ibt_hca_attr_t: add hca_rfci_max_log2_qp, hca_fexch_max_log2_qp,
hca_fexch_log2_mem fields
ibt_qp_alloc_flags_t: add IBT_QP_USES_RFCI, IBT_QP_USES_FCMD,
IBT_QP_USES_FEXCH flags
ibt_fc_attr_t: new struct for FC QP attributes
ibt_qp_alloc_attr_t: add qp_fc (ibt_fc_attr_t)
ibt_fexch_query_attr_t: new struct for FEXCH query attributes
ibt_qp_query_attr_t: add qp_query_fexch (ibt_fexch_query_attr_t)
ibt_cep_modify_flags_t: add IBT_CEP_SET_FEXCH_RANGE flag
ibt_qp_ud_attr_t: add ud_fc (ibt_fc_attr_t)
ibt_tran_srv_t: add IBT_RFCI_SRV, IBT_FCMD_SRV, IBT_FEXCH_SRV
ibt_wrc_opcode_t: add IBT_WRC_INIT_SEND_FCMD
ibt_wr_flags_t: add IBT_WR_SEND_FC_CRC
ibt_wr_rfci_send_t: new struct for RFCI version of Send WR
ibt_wr_init_send_t: new struct FCMD WR to start I/O operation
ibt_wr_fc_t: new union of possible FC WRs
ibt_send_wr_t wr field: add fc (ibt_wr_fc_t) to union
ibt_wc_flags_t: add IBT_WC_FEXCH_FMT, IBT_WC_DIF_ERROR flags
ibt_wc_t wc_detail field: add IBT_WC_DETAIL_FEXCH_INIT_XFER,
IBT_WC_FEXCH_LAST, IBT_WC_DETAIL_RFCI_CRC_OK,
IBT_WC_DETAIL_FC_MATCH_MASK
ibt_wc_t: add wc_fexch_seq_cnt, wc_fexch_tx_bytes, wc_rx_bytes_xfer,
wc_fexch_seq_id fields
ibt_async_code_t: add IBT_FEXCH_ERROR
ibt_fc_syndrome_t: new enum for FEXCH async errors
ibc_async_event_t: add ev_fc (ibt_fc_syndrome_t)
The changes in this case are part of IBTF ABI version 4.
Copies of all new and modified man pages are in the materials
directory (see section 4.3 below).
A. Detailed Description of FCoIB Support
While the FCoIB support could also support targets, the extensions in
this proposal are only sufficient to support initiators. Mellanox
support for FCoIB involves three variants of the IB Unreliable
Datagram (UD) transport service type.
The first is a Raw Fibre Channel Interface (RFCI) Queue Pair (QP),
which is used to establish the association between initiator and
targets. When it comes to I/O operations, typically, these QPs are
used to send confirmation (FCP_CONF) messages back to the target. In
IB terms, these QPs are like regular UD QPs, except that they have an
option for FC CRC support and End of Frame (EOF) values.
The second special QP type is the Fibre Channel Command (FCMD)
QP. This QP is used to initiate I/O operations through the associated
FEXCH QPs (see below) and perform memory registration through Work
Requests (WRs). Posting a special type of WR (Init & Send) causes a
data transfer operation to be assigned to an FEXCH QP and also causes
a command (FCP_CMND) message to be sent to the target. Registration is
accomplished on FCMDs by using the Registration WR previously defined
in PSARC/2009/060 only for Reliable Connected QPs. There is no receive
half to FCMD QPs.
The third special QP type is the Fibre Channel Exchange (FEXCH) QP. As
mentioned before, FCMD WRs assign data transfer operations to
associated FEXCH QPs. Allocation of FEXCH QPs is done via an
ibt_alloc_ud_channel_range() call to assign a group/range of QPs all
at once to an FCMD. The send half of these FEXCH QPs are not directly
used. The receive half of these QPs are used to signal the completion
of I/O operations. The work completions generated from the receive
queue have additional fields specific to Fibre Channel
operations. These FEXCH QPs also have associated memory regions (MRs)
which define the source/sink for a data transfer. Query QP operations
show which MRs are assigned to any given FEXCH QP. The MRs are
registered by using Registration WRs posted on the associated FCMD QP.
So pulling it all together, the typical I/O operation involves: (1)
registering memory belonging to an FEXCH QP through a Registration WR
on an FCMD QP, (2) initiating a data transfer with an Init & Send WR
on the FCMD, (3) waiting for the FEXCH completion and (4) a possible
confirmation message sent via an associated RFCI QP.
Add capability flag to ibt_hca_flags2_t to indicate support:
IBT_HCA2_FC = 1 << 12 /* FCoIB support */
Add other FC related HCA attributes to ibt_hca_attr_t:
uint8_t hca_rfci_max_log2_qp; /* log2 num RFCI QPs */
uint8_t hca_fexch_max_log2_qp; /* log2 num FEXCH QPs */
uint8_t hca_fexch_log2_mem; /* log2 size of FEXCH MRs (in bytes) */
Add flags and attributes (ibt_fc_attr_t) to for allocate QP ops:
ibt_chan_alloc_flags_t: add IBT_ACHAN_USES_RFCI, /* TI */
IBT_ACHAN_USES_FCMD, IBT_ACHAN_USES_FEXCH flags
add ibt_fc_attr_t struct to ibt_ud_chan_alloc_args_t
ibt_qp_alloc_flags_t: add IBT_QP_USES_RFCI, /* CI */
IBT_QP_USES_FCMD, IBT_QP_USES_FEXCH flags
add ibt_fc_attr_t to ibt_qp_alloc_attr_t
Add attributes (ibt_fc_attr_t & ibt_fexch_query_attr_t) for query QP ops:
add ibt_fc_attr_t & ibt_fexch_query_attr_t to /* TI */
ibt_ud_chan_query_attr_t
add ibt_fexch_query_attr_t to ibt_qp_query_attr_t /* CI */
add ibt_fc_attr_t to ibt_qp_ud_attr_t
Add flags and attributes (ibt_fc_attr_t) to modify QP ops:
add IBT_CEP_SET_FEXCH_RANGE flag to ibt_cep_modify_flags_t
add ibt_fc_attr_t to ibt_ud_chan_modify_attr_t /* TI */
add ibt_fc_attr_t to ibt_qp_ud_attr_t /* CI */
Add the new work requests structs:
define new transport service (QP) types for ibt_tran_srv_t:
IBT_RFCI_SRV, IBT_FCMD_SRV, IBT_FEXCH_SRV
for RFCI QPs, define a variant of UD Send (ibt_wr_rfci_send_t), this
operation can also use a new flag (IBT_WR_SEND_FC_CRC in
ibt_wr_flags_t)
for FCMD QPs, define a new Init & Send WR to assign initiate I/O
operations on FEXCH QPs (ibt_wr_init_send_t) and a new opcode
(IBT_WRC_INIT_SEND_FCMD in ibt_wrc_opcode_t)
define union of possible FC WRs (ibt_wr_fc_t) & add it to ibt_send_wr_t
Add new work completion fields:
add IBT_WC_DIF_ERROR flag to show Data Integrity Field errors
add IBT_WC_DETAIL_RFCI_CRC_OK flag for RFCI Send completions
add IBT_WC_FEXCH_FMT flag to show when using the special FEXCH
format; when set, the following are defined:
add wc_fexch_seq_cnt (sequence count), wc_fexch_tx_bytes (transmit
bytes), wc_rx_bytes_xfer (receive byes), wc_fexch_seq_id
(sequence id) fields
add IBT_WC_DETAIL_FEXCH_INIT_XFER (initiative transfer) and
IBT_WC_FEXCH_LAST (last sequence) flags
Add a new type of async error for FEXCH syndromes:
add IBT_FEXCH_ERROR to ibt_async_code_t
add ibt_fc_syndrome_t field to ibt_async_event_t
4.3 Man page summary
Man Page Disposition
(sorted by section)
-------------------------------------------
ibc_alloc_qp.9e changed
ibc_alloc_qp_range.9e changed
ibt_async_handler_t.9e changed
ibc_async_handler.9f changed
ibt_alloc_ud_channel.9f changed
ibt_alloc_ud_channel_range.9f changed
ibt_modify_ud_channel.9f changed
ibt_query_ud_channel.9f changed
ibc_qp_info_t.9s changed
ibc_qp_query_attr_9.9s changed
ibt_fc_attr_t.9s new
ibt_fexch_query_attr_t.9s new
ibt_hca_attr_t.9s changed
ibt_send_wr_t.9s changed
ibt_wc_t.9s changed
ibt_wr_fc_t.9s new
ibt_wr_rc_t.9s changed
ibt_wr_reg_pmr_t.9s new
Note: Since FCMD QPs can also do "Registration Work Request"
operations, we split the description of that WR out of ibt_wr_rc_t.9s
and put it into ibt_wr_reg_pmr_t.9s (where it can be referenced by
both ibt_wr_rc_t.9s and ibt_wr_fc_t.9s). Aside from reorganizing where
the information is, there is no change to the description of
Registration Work Request.
6. Resources and Schedule
6.4. Steering Committee requested information
6.4.1. Consolidation C-team Name:
ON
6.5. ARC review type: FastTrack
6.6. ARC Exposure: open
_______________________________________________
opensolaris-arc mailing list
opensolaris-arc@opensolaris.org
