On 01/30/14 09:19, Or Gerlitz wrote: > On 29/01/2014 19:56, Bart Van Assche wrote: >> On 01/29/14 16:06, Sagi Grimberg wrote: >>> Didn't understand why should it matter where the copy is done >>> (iser/block)? >> In the Linux kernel community it is considered important to avoid code >> duplication. Hence the proposal to keep code that copies data buffers >> in the block layer core and to avoid that such functionality has to be >> reimplemented in every block driver or SCSI LLD. > > Thanks for narrowing this down, I see your point, however the solution I > propose if to remove this copy altogether... for those rare cases where > fast-registration can't be done -- in SRP I think the code goes to > indirect mode under which there no copy (right?). As for iSER, we will > add to our TODO enhancing the code to avoid using bounce-buffer, few > designs might be possible, one way would be to create set of > FMRS/Fast-reg element which are capable to map chunks which are < > PAGE_SIZE, e.g in the order of single block, and hence can map what ever > arbitrary set of blocks provided by the upper layer.
I'm not sure how important this issue is on x86 systems since the page size on these systems is 4 KB and since many applications use an I/O size >= 4 KB. So if the I/O buffers are aligned on page boundaries the buffer memory can be registered as a single memory region without having to copy any data. However, on PPC systems, which have a page size of 64 KB, if e.g. a database issues a readv() or writev() call or if an I/O scheduler coalesces several small I/O requests into a single I/O request the data buffer of these I/O requests is discontiguous. I think it is important that data copying can be avoided for such I/O requests. By the way, is it documented somewhere what the alignment requirements are for FMR and FR requests ? As far as I can see the mlx4 driver requires page alignment for FMR requests but not for FR requests (see e.g. mlx4_check_fmr()). I haven't found a page aligment requirement in e.g. ehca_fmr_check_page_list(). Does this mean that alignment restrictions for FMR are HCA-dependent ? Would it be a good idea to add an attribute in e.g. ib_device_attr that allows ULP drivers to retrieve FMR alignment requirements in a generic way ? Thanks, Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
