Re: [PATCH RFC 2/9] IB/core: Introduce Signature Verbs API
On 10/22/2013 9:20 PM, Hefty, Sean wrote: Would we lose anything making this a new operation for the QP, versus trying to hook it into the existing ib_post_send call? If I understand correctly you are suggesting making it a verb? Well this operation is a fast-path operation - so I guess we will loose it in this case. Take SCSI for example, for each IO operation submitted by SCSI mid-layer, transport layer should perform any protection policy that SCSI asked for. From this point of view, signature operation resembles fast registration (since the transport does not own the IOP data buffers, so it uses fast registration methods). That is why we are hooking into ib_post_send. I'm suggesting multiple calls that can post to the send queue, rather than one call that does a giant switch statement at the beginning based on the opcode. Although I understand where you are coming from, We also lose in this case. If we go down this rode, we block the user from saving a HW doorbell by concatenating signature and RDMA WRs to a post list. I assume this is why fast_reg is also an extension of ib_post_send. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH RFC 2/9] IB/core: Introduce Signature Verbs API
I don't think so, Signature MR simply describes a signature associated memory region i.e. it is a memory region that also defines some signature operation offload aside from normal RDMA (for example validate strip). SGL are used to publish several rkeys for the server/target/peer to perform RDMA on each. In this case the user previously registered each MR which he wishes it's peer to RDMA over. Same story here, if user has several signature associated MRs, where he wish his peer to RDMA over (in a protected manner), he can use these rkeys to construct SGL. Why are the signature properties separate from the protection information? Well, Protection information is the actual protection block guards of the data (i.e. CRCs, XORs, DIFs etc..), while the signature properties structure is the descriptor telling the HCA how to treat/validate/generate the protection information. Note that signature support requires the HCA to be able to support INSERT operations. This means that there is no protection information and the HCA is asked to generate it and add it to the data stream (which may be incoming or outgoing...), Would we lose anything making this a new operation for the QP, versus trying to hook it into the existing ib_post_send call? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH RFC 2/9] IB/core: Introduce Signature Verbs API
Would we lose anything making this a new operation for the QP, versus trying to hook it into the existing ib_post_send call? If I understand correctly you are suggesting making it a verb? Well this operation is a fast-path operation - so I guess we will loose it in this case. Take SCSI for example, for each IO operation submitted by SCSI mid-layer, transport layer should perform any protection policy that SCSI asked for. From this point of view, signature operation resembles fast registration (since the transport does not own the IOP data buffers, so it uses fast registration methods). That is why we are hooking into ib_post_send. I'm suggesting multiple calls that can post to the send queue, rather than one call that does a giant switch statement at the beginning based on the opcode. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH RFC 2/9] IB/core: Introduce Signature Verbs API
The signature handover operation is binding all the necessary information for the HCA together: where is the data (data_mr), where is the protection information (prot_mr), what are the signature properties (sig_attrs). Once this step is taken (WR is posted), a single MR (sig_mr) describes the signature handover operation and can be used to perform RDMA under signature presence. Once the HCA will perform RDMA over this MR, it will take into account the signature context of the transaction and will follow the signature attributes configured. It seems like this changes loses the ability to use an SGL. Why are the signature properties separate from the protection information? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC 2/9] IB/core: Introduce Signature Verbs API
On 10/21/2013 5:34 PM, Hefty, Sean wrote: The signature handover operation is binding all the necessary information for the HCA together: where is the data (data_mr), where is the protection information (prot_mr), what are the signature properties (sig_attrs). Once this step is taken (WR is posted), a single MR (sig_mr) describes the signature handover operation and can be used to perform RDMA under signature presence. Once the HCA will perform RDMA over this MR, it will take into account the signature context of the transaction and will follow the signature attributes configured. It seems like this changes loses the ability to use an SGL. I don't think so, Signature MR simply describes a signature associated memory region i.e. it is a memory region that also defines some signature operation offload aside from normal RDMA (for example validate strip). SGL are used to publish several rkeys for the server/target/peer to perform RDMA on each. In this case the user previously registered each MR which he wishes it's peer to RDMA over. Same story here, if user has several signature associated MRs, where he wish his peer to RDMA over (in a protected manner), he can use these rkeys to construct SGL. Why are the signature properties separate from the protection information? Well, Protection information is the actual protection block guards of the data (i.e. CRCs, XORs, DIFs etc..), while the signature properties structure is the descriptor telling the HCA how to treat/validate/generate the protection information. Note that signature support requires the HCA to be able to support INSERT operations. This means that there is no protection information and the HCA is asked to generate it and add it to the data stream (which may be incoming or outgoing...), Hope this helps. Sagi. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC 2/9] IB/core: Introduce Signature Verbs API
On 10/18/2013 1:51 AM, Hefty, Sean wrote:
@@ -885,6 +901,19 @@ struct ib_send_wr {
u32 rkey;
struct ib_mw_bind_info bind_info;
} bind_mw;
+ struct {
+ struct ib_sig_attrs*sig_attrs;
+ struct ib_mr *sig_mr;
+ int access_flags;
+ /* Registered data mr */
+ struct ib_mr *data_mr;
+ u32 data_size;
+ u64 data_va;
+ /* Registered protection mr */
+ struct ib_mr *prot_mr;
+ u32 prot_size;
+ u64 prot_va;
+ } sig_handover;
At what point do we admit that this is a ridiculous structure?
If you are referring to ib_send_wr, I agree, shall we modify it to a
union of typedefs so it becomes more readable?
Help me understand what this WR is doing. Is this telling the HCA to copy data
between local MRs? What is a 'data MR' versus a 'protected MR'? (I'm not hip
on T10-DIF.)
No data copy, god forbids... :)
Let me start by giving a short intro on signature (and T10-DIF).
In signature world, data may exist with protection information which is
the guarding the data. In T10-DIF (Data Integrity Fields) for example,
these are 8-byte guards which includes CRC for each 512 bytes of data
(block).
An HCA which support signature offload, is expected to validate that the
data is intact (each block matches its guard) and send it correctly over
the wire (in T10-DIF case the data and protection should be interleaved
i.e. 512B of data followed by 8B of protection guard) or alternatively,
validate data (+ protection) coming from the wire and write it to the
associated memory areas.
In the general case, the data and the protection guards may lie in
different memory areas. SCSI mid-layer for instance, passes the
transport driver 2 buffers using 2 scatterlists.
The transport driver (or application in the general case), is expected
to register each buffer (as it normally would in order to use RDMA)
using 2 MRs.
The signature handover operation is binding all the necessary
information for the HCA together: where is the data (data_mr), where is
the protection information (prot_mr), what are the signature properties
(sig_attrs).
Once this step is taken (WR is posted), a single MR (sig_mr) describes
the signature handover operation and can be used to perform RDMA under
signature presence.
Once the HCA will perform RDMA over this MR, it will take into account
the signature context of the transaction and will follow the signature
attributes configured.
Hope this helps,
Sagi.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH RFC 2/9] IB/core: Introduce Signature Verbs API
@@ -885,6 +901,19 @@ struct ib_send_wr {
u32 rkey;
struct ib_mw_bind_info bind_info;
} bind_mw;
+ struct {
+ struct ib_sig_attrs*sig_attrs;
+ struct ib_mr *sig_mr;
+ int access_flags;
+ /* Registered data mr */
+ struct ib_mr *data_mr;
+ u32 data_size;
+ u64 data_va;
+ /* Registered protection mr */
+ struct ib_mr *prot_mr;
+ u32 prot_size;
+ u64 prot_va;
+ } sig_handover;
At what point do we admit that this is a ridiculous structure?
Help me understand what this WR is doing. Is this telling the HCA to copy data
between local MRs? What is a 'data MR' versus a 'protected MR'? (I'm not hip
on T10-DIF.)
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH RFC 2/9] IB/core: Introduce Signature Verbs API
This commit Introduces the Verbs Interface for signature related
operations. A signature handover operation shall configure the
layouts of data and protection attributes both in memory and wire
domains. Once the signature handover opration is done, the HCA will
offload data integrity generation/validation while performing
the actual data transfer.
Additions:
1. HCA signature capabilities in device attributes
Verbs provider supporting Signature handover operations shall
fill relevant fields in device attributes structure returned
by ib_query_device.
2. QP creation flag IB_QP_CREATE_SIGNATURE_EN
Creating QP that will carry signature handover operations
may require some special preperations from the verbs provider.
So we add QP creation flag IB_QP_CREATE_SIGNATURE_EN to declare
that the created QP may carry out signature handover operations.
Expose signature support to verbs layer (no support for now)
3. New send work request IB_WR_REG_SIG_MR
Signature handover work request. This WR will define
the signature handover properties of the memory/wire
domains as well as the domains layout.
* Currently expose just T10-DIF layout.
4. New Verb ib_check_sig_status
check_sig_status Verb shall check if any signature errors
are pending for a specific signature related ib_mr.
User should provide the ib_qp that executed the RDMA operation
involving the given ib_mr.
Signed-off-by: Sagi Grimberg sa...@mellanox.com
---
drivers/infiniband/core/verbs.c |8 ++
include/rdma/ib_verbs.h | 140 ++-
2 files changed, 147 insertions(+), 1 deletions(-)
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 1d94a5c..5636d65 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -1293,3 +1293,11 @@ int ib_dealloc_xrcd(struct ib_xrcd *xrcd)
return xrcd-device-dealloc_xrcd(xrcd);
}
EXPORT_SYMBOL(ib_dealloc_xrcd);
+
+int ib_check_sig_status(struct ib_mr *sig_mr,
+ struct ib_sig_err *sig_err)
+{
+ return sig_mr-device-check_sig_status ?
+ sig_mr-device-check_sig_status(sig_mr, sig_err) : -ENOSYS;
+}
+EXPORT_SYMBOL(ib_check_sig_status);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 65b7e79..cf46a83 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -116,7 +116,19 @@ enum ib_device_cap_flags {
IB_DEVICE_MEM_MGT_EXTENSIONS= (121),
IB_DEVICE_BLOCK_MULTICAST_LOOPBACK = (122),
IB_DEVICE_MEM_WINDOW_TYPE_2A= (123),
- IB_DEVICE_MEM_WINDOW_TYPE_2B= (124)
+ IB_DEVICE_MEM_WINDOW_TYPE_2B= (124),
+ IB_DEVICE_SIGNATURE_HANDOVER= (125),
+};
+
+enum ib_signature_prot_cap {
+ IB_PROT_T10DIF_TYPE_1 = 1,
+ IB_PROT_T10DIF_TYPE_2 = 1 1,
+ IB_PROT_T10DIF_TYPE_3 = 1 2,
+};
+
+enum ib_signature_guard_cap {
+ IB_GUARD_T10DIF_CRC = 1,
+ IB_GUARD_T10DIF_CSUM= 1 1,
};
enum ib_atomic_cap {
@@ -166,6 +178,8 @@ struct ib_device_attr {
unsigned intmax_fast_reg_page_list_len;
u16 max_pkeys;
u8 local_ca_ack_delay;
+ enum ib_signature_prot_cap sig_prot_cap;
+ enum ib_signature_guard_cap sig_guard_cap;
};
enum ib_mtu {
@@ -630,6 +644,7 @@ enum ib_qp_type {
enum ib_qp_create_flags {
IB_QP_CREATE_IPOIB_UD_LSO = 1 0,
IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK = 1 1,
+ IB_QP_CREATE_SIGNATURE_EN = 1 2,
/* reserve bits 26-31 for low level drivers' internal use */
IB_QP_CREATE_RESERVED_START = 1 26,
IB_QP_CREATE_RESERVED_END = 1 31,
@@ -780,6 +795,7 @@ enum ib_wr_opcode {
IB_WR_MASKED_ATOMIC_CMP_AND_SWP,
IB_WR_MASKED_ATOMIC_FETCH_AND_ADD,
IB_WR_BIND_MW,
+ IB_WR_REG_SIG_MR,
/* reserve values for low level drivers' internal use.
* These values will not be used at all in the ib core layer.
*/
@@ -885,6 +901,19 @@ struct ib_send_wr {
u32 rkey;
struct ib_mw_bind_info bind_info;
} bind_mw;
+ struct {
+ struct ib_sig_attrs*sig_attrs;
+ struct ib_mr *sig_mr;
+ int access_flags;
+ /* Registered data mr */
+ struct ib_mr *data_mr;
+ u32 data_size;
+ u64 data_va;
+ /* Registered protection mr */
+ struct ib_mr *prot_mr;
+ u32 prot_size;
+ u64 prot_va;
+ } sig_handover;
} wr;
