Re: [PATCH RFC v2 00/10] Introduce Signature feature
On 11/4/2013 8:41 PM, Nicholas A. Bellinger wrote: On Sat, 2013-11-02 at 14:57 -0700, Bart Van Assche wrote: On 1/11/2013 18:36, Nicholas A. Bellinger wrote: On Fri, 2013-11-01 at 08:03 -0700, Bart Van Assche wrote: On 31/10/2013 5:24, Sagi Grimberg wrote: In T10-DIF, when a series of 512-byte data blocks are transferred, each block is followed by an 8-byte guard. The guard consists of CRC that protects the integrity of the data in the block, and some other tags that protects against mis-directed IOs. Shouldn't that read logical block length divided by 2**(protection interval exponent) instead of 512 ? From the SPC-4 FORMAT UNIT section: Why should the protection interval in FORMAT_UNIT be mentioned when it's not supported by the hardware, nor by drivers/scsi/sd_dif.c itself..? Hello Nick, My understanding is that this patch series is not only intended for initiator drivers but also for target drivers like ib_srpt and ib_isert. As you know target drivers do not restrict the initiator operating system to Linux. Although I do not know whether there are already operating systems that support the protection interval exponent, It's my understanding that Linux is still the only stack that supports DIF, so AFAICT no one is actually supporting this. I think it is a good idea to stay as close as possible to the terminology of the SPC-4 standard. No, in this context it only adds pointless misdirection because 1) The hardware in question doesn't support it, and 2) Linux itself doesn't support it. I think that Bart is suggesting renaming block_size as pi_interval in ib_sig_domain. I tend to agree since even if support for that does not exist yet, it might be in the future. I think it is not a misdirection because it does represent the protection information interval. --nab -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC v2 00/10] Introduce Signature feature
On Tue, 2013-11-05 at 11:13 +0200, Sagi Grimberg wrote: On 11/4/2013 8:41 PM, Nicholas A. Bellinger wrote: On Sat, 2013-11-02 at 14:57 -0700, Bart Van Assche wrote: On 1/11/2013 18:36, Nicholas A. Bellinger wrote: On Fri, 2013-11-01 at 08:03 -0700, Bart Van Assche wrote: On 31/10/2013 5:24, Sagi Grimberg wrote: In T10-DIF, when a series of 512-byte data blocks are transferred, each block is followed by an 8-byte guard. The guard consists of CRC that protects the integrity of the data in the block, and some other tags that protects against mis-directed IOs. Shouldn't that read logical block length divided by 2**(protection interval exponent) instead of 512 ? From the SPC-4 FORMAT UNIT section: Why should the protection interval in FORMAT_UNIT be mentioned when it's not supported by the hardware, nor by drivers/scsi/sd_dif.c itself..? Hello Nick, My understanding is that this patch series is not only intended for initiator drivers but also for target drivers like ib_srpt and ib_isert. As you know target drivers do not restrict the initiator operating system to Linux. Although I do not know whether there are already operating systems that support the protection interval exponent, It's my understanding that Linux is still the only stack that supports DIF, so AFAICT no one is actually supporting this. I think it is a good idea to stay as close as possible to the terminology of the SPC-4 standard. No, in this context it only adds pointless misdirection because 1) The hardware in question doesn't support it, and 2) Linux itself doesn't support it. I think that Bart is suggesting renaming block_size as pi_interval in ib_sig_domain. I tend to agree since even if support for that does not exist yet, it might be in the future. I think it is not a misdirection because it does represent the protection information interval. The point is that changing the description from what the patch actually does, to something it does not do in order to 'stay as close as possible to the terminology of the SPC-4 standard' is pointlessly confusing. --nab -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC v2 00/10] Introduce Signature feature
On Sat, 2013-11-02 at 14:57 -0700, Bart Van Assche wrote: On 1/11/2013 18:36, Nicholas A. Bellinger wrote: On Fri, 2013-11-01 at 08:03 -0700, Bart Van Assche wrote: On 31/10/2013 5:24, Sagi Grimberg wrote: In T10-DIF, when a series of 512-byte data blocks are transferred, each block is followed by an 8-byte guard. The guard consists of CRC that protects the integrity of the data in the block, and some other tags that protects against mis-directed IOs. Shouldn't that read logical block length divided by 2**(protection interval exponent) instead of 512 ? From the SPC-4 FORMAT UNIT section: Why should the protection interval in FORMAT_UNIT be mentioned when it's not supported by the hardware, nor by drivers/scsi/sd_dif.c itself..? Hello Nick, My understanding is that this patch series is not only intended for initiator drivers but also for target drivers like ib_srpt and ib_isert. As you know target drivers do not restrict the initiator operating system to Linux. Although I do not know whether there are already operating systems that support the protection interval exponent, It's my understanding that Linux is still the only stack that supports DIF, so AFAICT no one is actually supporting this. I think it is a good idea to stay as close as possible to the terminology of the SPC-4 standard. No, in this context it only adds pointless misdirection because 1) The hardware in question doesn't support it, and 2) Linux itself doesn't support it. --nab -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC v2 00/10] Introduce Signature feature
On 11/2/2013 12:06 AM, Bart Van Assche wrote: On 31/10/2013 5:24, Sagi Grimberg wrote: While T10-DIF clearly defines that over the wire protection guards are interleaved into the data stream (each 512-Byte block followed by 8-byte guard), when in memory, the protection guards may reside in a buffer separated from the data. Depending on the application, it is usually easier to handle the data when it is contiguous. In this case the data buffer will be of size 512xN and the protection buffer will be of size 8xN (where N is the number of blocks in the transaction). It might be worth mentioning here that in the Linux block layer the approach has been chosen where actual data an protection information are in separate buffers. See also the bi_integrity field in struct bio. Bart. Hey Bart, I was expecting your input on this Thanks for the insightful comments! The explanation here is an attempt to Introduce T10-DIF to the mailing-list as simple as possible, so I tried not to dive into SBC-3/SPC-4. You are correct, the 8-byte protection guards will follow the protection interval which won't necessarily be 512 (only for DIF types 2,3). Sagi. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC v2 00/10] Introduce Signature feature
On 11/2/2013 12:06 AM, Bart Van Assche wrote: On 31/10/2013 5:24, Sagi Grimberg wrote: While T10-DIF clearly defines that over the wire protection guards are interleaved into the data stream (each 512-Byte block followed by 8-byte guard), when in memory, the protection guards may reside in a buffer separated from the data. Depending on the application, it is usually easier to handle the data when it is contiguous. In this case the data buffer will be of size 512xN and the protection buffer will be of size 8xN (where N is the number of blocks in the transaction). It might be worth mentioning here that in the Linux block layer the approach has been chosen where actual data an protection information are in separate buffers. See also the bi_integrity field in struct bio. Bart. This is true, but signature verbs interface supports also data and protection interleaving in memory space. A user wishes to do so will pass the same ib_sge both for data and protection. In fact this was a requirement we got from customers. Sagi. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC v2 00/10] Introduce Signature feature
On 31/10/2013 5:24, Sagi Grimberg wrote: In T10-DIF, when a series of 512-byte data blocks are transferred, each block is followed by an 8-byte guard. The guard consists of CRC that protects the integrity of the data in the block, and some other tags that protects against mis-directed IOs. Shouldn't that read logical block length divided by 2**(protection interval exponent) instead of 512 ? From the SPC-4 FORMAT UNIT section: quoteFor a type 2 protection or a type 3 protection format request, the protection interval exponent determines the length of user data to be transferred before protection information is transferred (i.e., the protection information interval). The protection information interval is calculated as follows: protection information interval = logical block length / 2**(protection interval exponent) where: logical block length is the number of bytes of user data in a logical block (see 4.5) and where protection interval exponent is zero if the short parameter list header (see table 36) is used or the contents of the PROTECTION INTERVAL EXPONENT field if the long parameter list header (see table 37) is used. If the protection information interval calculates to a value that is not an even number (e.g., 520 / 2**3 = 65) or not a whole number (e.g., 520 / 2**4 = 32.5 and 520 / 2**10 = 0.508), then the device server shall terminate the command with CHECK CONDITION status with the sense key set to ILLEGAL REQUEST and the additional sense code set to INVALID FIELD IN PARAMETER LIST./quote Bart. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC v2 00/10] Introduce Signature feature
On Fri, 2013-11-01 at 08:03 -0700, Bart Van Assche wrote: On 31/10/2013 5:24, Sagi Grimberg wrote: In T10-DIF, when a series of 512-byte data blocks are transferred, each block is followed by an 8-byte guard. The guard consists of CRC that protects the integrity of the data in the block, and some other tags that protects against mis-directed IOs. Shouldn't that read logical block length divided by 2**(protection interval exponent) instead of 512 ? From the SPC-4 FORMAT UNIT section: Why should the protection interval in FORMAT_UNIT be mentioned when it's not supported by the hardware, nor by drivers/scsi/sd_dif.c itself..? --nab -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC v2 00/10] Introduce Signature feature
Hi Sagi, I wander what's the performance overhead with this DIF support? And is there a roadmap for support SRP/ISER and target side for DIF? Regards, Jack On 10/31/2013 01:24 PM, Sagi Grimberg wrote: This patchset Introduces Verbs level support for signature handover feature. Siganture is intended to implement end-to-end data integrity on a transactional basis in a completely offloaded manner. There are several end-to-end data integrity methods used today in various applications and/or upper layer protocols such as T10-DIF defined by SCSI specifications (SBC), CRC32, XOR8 and more. This patchset adds verbs support only for T10-DIF. The proposed framework allows adding more signature methods in the future. In T10-DIF, when a series of 512-byte data blocks are transferred, each block is followed by an 8-byte guard. The guard consists of CRC that protects the integrity of the data in the block, and some other tags that protects against mis-directed IOs. Data can be protected when transferred over the wire, but can also be protected in the memory of the sender/receiver. This allows true end- to-end protection against bits flipping either over the wire, through gateways, in memory, over PCI, etc. While T10-DIF clearly defines that over the wire protection guards are interleaved into the data stream (each 512-Byte block followed by 8-byte guard), when in memory, the protection guards may reside in a buffer separated from the data. Depending on the application, it is usually easier to handle the data when it is contiguous. In this case the data buffer will be of size 512xN and the protection buffer will be of size 8xN (where N is the number of blocks in the transaction). There are 3 kinds of signature handover operation: 1. Take unprotected data (from wire or memory) and ADD protection guards. 2. Take protetected data (from wire or memory), validate the data integrity against the protection guards and STRIP the protection guards. 3. Take protected data (from wire or memory), validate the data integrity against the protection guards and PASS the data with the guards as-is. This translates to defining to the HCA how/if data protection exists in memory domain, and how/if data protection exists is wire domain. The way that data integrity is performed is by using a new kind of memory region: signature-enabled MR, and a new kind of work request: REG_SIG_MR. The REG_SIG_MR WR operates on the signature-enabled MR, and defines all the needed information for the signature handover (data buffer, protection buffer if needed and signature attributes). The result is an MR that can be used for data transfer as usual, that will also add/validate/strip/pass protection guards. When the data transfer is successfully completed, it does not mean that there are no integrity errors. The user must afterwards check the signature status of the handover operation using a new light-weight verb. This feature shall be used in storage upper layer protocols iSER/SRP implementing end-to-end data integrity T10-DIF. Following this patchset, we will soon submit krping patches which will demonstrate the usage of these signature verbs. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC v2 00/10] Introduce Signature feature
On 10/31/2013 2:55 PM, Jack Wang wrote: Hi Sagi, I wander what's the performance overhead with this DIF support? And is there a roadmap for support SRP/ISER and target side for DIF? Regards, Jack Well, all DIF operations are fully offloaded by the HCA so we don't expect any performance degradation other than the obvious 8-bytes integrity overhead. We have yet to take benchmarks on this and we definitely plan to do so. Regarding our roadmap, we plan to support iSER target (LIO) and initiator first. Some prior support for DIF needs to be added in target core level, then transport implementation is pretty straight-forward (iSER/SRP). So I aim for iSER DIF support (target+initiator) to make it into v3.14. Hope this helps, Sagi. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC v2 00/10] Introduce Signature feature
On 10/31/2013 02:20 PM, Sagi Grimberg wrote: On 10/31/2013 2:55 PM, Jack Wang wrote: Hi Sagi, I wander what's the performance overhead with this DIF support? And is there a roadmap for support SRP/ISER and target side for DIF? Regards, Jack Well, all DIF operations are fully offloaded by the HCA so we don't expect any performance degradation other than the obvious 8-bytes integrity overhead. We have yet to take benchmarks on this and we definitely plan to do so. Regarding our roadmap, we plan to support iSER target (LIO) and initiator first. Some prior support for DIF needs to be added in target core level, then transport implementation is pretty straight-forward (iSER/SRP). So I aim for iSER DIF support (target+initiator) to make it into v3.14. Hope this helps, Sagi. Good to know, thanks Jack -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
