Re: Integration of SCST in the mainstream Linux kernel
Bart Van Assche wrote: > On Feb 18, 2008 10:43 AM, Erez Zilber <[EMAIL PROTECTED]> wrote: > >> If you use a high value for FirstBurstLength, all (or most) of your data >> will be sent as unsolicited data-out PDUs. These PDUs don't use the RDMA >> engine, so you miss the advantage of IB. >> > > Hello Erez, > > Did you notice the e-mail Roland Dreier wrote on Februari 6, 2008 ? > This is what Roland wrote: > >> I think the confusion here is caused by a slight misuse of the term >> "RDMA". It is true that all data is always transported over an >> InfiniBand connection when iSER is used, but not all such transfers >> are one-sided RDMA operations; some data can be transferred using >> send/receive operations. >> > > Yes, I saw that. I tried to give an explanation with more details. > Or: data sent during the first burst is not transferred via one-sided > remote memory reads or writes but via two-sided send/receive > operations. At least on my setup, these operations are as fast as > one-sided remote memory reads or writes. As an example, I obtained the > following numbers on my setup (SDR 4x network); > ib_write_bw: 933 MB/s. > ib_read_bw: 905 MB/s. > ib_send_bw: 931 MB/s. > > According to these numbers one can think that you don't need RDMA at all, just send iSCSI PDUs over IB. The benchmarks that you use are synthetic IB benchmarks that are not equivalent to iSCSI over iSER. They just send IB packets. I'm not surprised that you got more or less the same performance because, AFAIK, ib_send_bw doesn't copy data (unlike iSCSI that has to copy data that is sent/received without RDMA). When you use RDMA with iSCSI (i.e. iSER), you don't need to create iSCSI PDUs and process them. The CPU is not busy as it is with iSCSI over TCP because no data copies are required. Another advantage is that you don't need header/data digest because the IB HW does that. Erez -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Integration of SCST in the mainstream Linux kernel
Bart Van Assche wrote: On Feb 18, 2008 10:43 AM, Erez Zilber [EMAIL PROTECTED] wrote: If you use a high value for FirstBurstLength, all (or most) of your data will be sent as unsolicited data-out PDUs. These PDUs don't use the RDMA engine, so you miss the advantage of IB. Hello Erez, Did you notice the e-mail Roland Dreier wrote on Februari 6, 2008 ? This is what Roland wrote: I think the confusion here is caused by a slight misuse of the term RDMA. It is true that all data is always transported over an InfiniBand connection when iSER is used, but not all such transfers are one-sided RDMA operations; some data can be transferred using send/receive operations. Yes, I saw that. I tried to give an explanation with more details. Or: data sent during the first burst is not transferred via one-sided remote memory reads or writes but via two-sided send/receive operations. At least on my setup, these operations are as fast as one-sided remote memory reads or writes. As an example, I obtained the following numbers on my setup (SDR 4x network); ib_write_bw: 933 MB/s. ib_read_bw: 905 MB/s. ib_send_bw: 931 MB/s. According to these numbers one can think that you don't need RDMA at all, just send iSCSI PDUs over IB. The benchmarks that you use are synthetic IB benchmarks that are not equivalent to iSCSI over iSER. They just send IB packets. I'm not surprised that you got more or less the same performance because, AFAIK, ib_send_bw doesn't copy data (unlike iSCSI that has to copy data that is sent/received without RDMA). When you use RDMA with iSCSI (i.e. iSER), you don't need to create iSCSI PDUs and process them. The CPU is not busy as it is with iSCSI over TCP because no data copies are required. Another advantage is that you don't need header/data digest because the IB HW does that. Erez -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Integration of SCST in the mainstream Linux kernel
Bart Van Assche wrote: > On Feb 5, 2008 6:01 PM, Erez Zilber <[EMAIL PROTECTED]> wrote: > >> Using such large values for FirstBurstLength will give you poor >> performance numbers for WRITE commands (with iSER). FirstBurstLength >> means how much data should you send as unsolicited data (i.e. without >> RDMA). It means that your WRITE commands were sent without RDMA. >> > > Sorry, but I'm afraid you got this wrong. When the iSER transport is > used instead of TCP, all data is sent via RDMA, including unsolicited > data. If you have look at the iSER implementation in the Linux kernel > (source files under drivers/infiniband/ulp/iser), you will see that > all data is transferred via RDMA and not via TCP/IP. > > When you execute WRITE commands with iSCSI, it works like this: EDTL (Expected data length) - the data length of your command FirstBurstLength - the length of data that will be sent as unsolicited data (i.e. as immediate data with the SCSI command and as unsolicited data-out PDUs) If you use a high value for FirstBurstLength, all (or most) of your data will be sent as unsolicited data-out PDUs. These PDUs don't use the RDMA engine, so you miss the advantage of IB. If you use a lower value for FirstBurstLength, EDTL - FirstBurstLength bytes will be sent as solicited data-out PDUs. With iSER, solicited data-out PDUs are RDMA operations. I hope that I'm more clear now. Erez -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Integration of SCST in the mainstream Linux kernel
Bart Van Assche wrote: On Feb 5, 2008 6:01 PM, Erez Zilber [EMAIL PROTECTED] wrote: Using such large values for FirstBurstLength will give you poor performance numbers for WRITE commands (with iSER). FirstBurstLength means how much data should you send as unsolicited data (i.e. without RDMA). It means that your WRITE commands were sent without RDMA. Sorry, but I'm afraid you got this wrong. When the iSER transport is used instead of TCP, all data is sent via RDMA, including unsolicited data. If you have look at the iSER implementation in the Linux kernel (source files under drivers/infiniband/ulp/iser), you will see that all data is transferred via RDMA and not via TCP/IP. When you execute WRITE commands with iSCSI, it works like this: EDTL (Expected data length) - the data length of your command FirstBurstLength - the length of data that will be sent as unsolicited data (i.e. as immediate data with the SCSI command and as unsolicited data-out PDUs) If you use a high value for FirstBurstLength, all (or most) of your data will be sent as unsolicited data-out PDUs. These PDUs don't use the RDMA engine, so you miss the advantage of IB. If you use a lower value for FirstBurstLength, EDTL - FirstBurstLength bytes will be sent as solicited data-out PDUs. With iSER, solicited data-out PDUs are RDMA operations. I hope that I'm more clear now. Erez -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Integration of SCST in the mainstream Linux kernel
Bart Van Assche wrote: > As you probably know there is a trend in enterprise computing towards > networked storage. This is illustrated by the emergence during the > past few years of standards like SRP (SCSI RDMA Protocol), iSCSI > (Internet SCSI) and iSER (iSCSI Extensions for RDMA). Two different > pieces of software are necessary to make networked storage possible: > initiator software and target software. As far as I know there exist > three different SCSI target implementations for Linux: > - The iSCSI Enterprise Target Daemon (IETD, > http://iscsitarget.sourceforge.net/); > - The Linux SCSI Target Framework (STGT, http://stgt.berlios.de/); > - The Generic SCSI Target Middle Level for Linux project (SCST, > http://scst.sourceforge.net/). > Since I was wondering which SCSI target software would be best suited > for an InfiniBand network, I started evaluating the STGT and SCST SCSI > target implementations. Apparently the performance difference between > STGT and SCST is small on 100 Mbit/s and 1 Gbit/s Ethernet networks, > but the SCST target software outperforms the STGT software on an > InfiniBand network. See also the following thread for the details: > http://sourceforge.net/mailarchive/forum.php?thread_name=e2e108260801170127w2937b2afg9bef324efa945e43%40mail.gmail.com_name=scst-devel. > > Sorry for the late response (but better late than never). One may claim that STGT should have lower performance than SCST because its data path is from userspace. However, your results show that for non-IB transports, they both show the same numbers. Furthermore, with IB there shouldn't be any additional difference between the 2 targets because data transfer from userspace is as efficient as data transfer from kernel space. The only explanation that I see is that fine tuning for iSCSI & iSER is required. As was already mentioned in this thread, with SDR you can get ~900 MB/sec with iSER (on STGT). Erez -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Integration of SCST in the mainstream Linux kernel
Bart Van Assche wrote: > On Jan 30, 2008 12:32 AM, FUJITA Tomonori <[EMAIL PROTECTED]> wrote: > >> iSER has parameters to limit the maximum size of RDMA (it needs to >> repeat RDMA with a poor configuration)? >> > > Please specify which parameters you are referring to. As you know I > had already repeated my tests with ridiculously high values for the > following iSER parameters: FirstBurstLength, MaxBurstLength and > MaxRecvDataSegmentLength (16 MB, which is more than the 1 MB block > size specified to dd). > > Using such large values for FirstBurstLength will give you poor performance numbers for WRITE commands (with iSER). FirstBurstLength means how much data should you send as unsolicited data (i.e. without RDMA). It means that your WRITE commands were sent without RDMA. Erez -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Integration of SCST in the mainstream Linux kernel
Bart Van Assche wrote: On Jan 30, 2008 12:32 AM, FUJITA Tomonori [EMAIL PROTECTED] wrote: iSER has parameters to limit the maximum size of RDMA (it needs to repeat RDMA with a poor configuration)? Please specify which parameters you are referring to. As you know I had already repeated my tests with ridiculously high values for the following iSER parameters: FirstBurstLength, MaxBurstLength and MaxRecvDataSegmentLength (16 MB, which is more than the 1 MB block size specified to dd). Using such large values for FirstBurstLength will give you poor performance numbers for WRITE commands (with iSER). FirstBurstLength means how much data should you send as unsolicited data (i.e. without RDMA). It means that your WRITE commands were sent without RDMA. Erez -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Integration of SCST in the mainstream Linux kernel
Bart Van Assche wrote: As you probably know there is a trend in enterprise computing towards networked storage. This is illustrated by the emergence during the past few years of standards like SRP (SCSI RDMA Protocol), iSCSI (Internet SCSI) and iSER (iSCSI Extensions for RDMA). Two different pieces of software are necessary to make networked storage possible: initiator software and target software. As far as I know there exist three different SCSI target implementations for Linux: - The iSCSI Enterprise Target Daemon (IETD, http://iscsitarget.sourceforge.net/); - The Linux SCSI Target Framework (STGT, http://stgt.berlios.de/); - The Generic SCSI Target Middle Level for Linux project (SCST, http://scst.sourceforge.net/). Since I was wondering which SCSI target software would be best suited for an InfiniBand network, I started evaluating the STGT and SCST SCSI target implementations. Apparently the performance difference between STGT and SCST is small on 100 Mbit/s and 1 Gbit/s Ethernet networks, but the SCST target software outperforms the STGT software on an InfiniBand network. See also the following thread for the details: http://sourceforge.net/mailarchive/forum.php?thread_name=e2e108260801170127w2937b2afg9bef324efa945e43%40mail.gmail.comforum_name=scst-devel. Sorry for the late response (but better late than never). One may claim that STGT should have lower performance than SCST because its data path is from userspace. However, your results show that for non-IB transports, they both show the same numbers. Furthermore, with IB there shouldn't be any additional difference between the 2 targets because data transfer from userspace is as efficient as data transfer from kernel space. The only explanation that I see is that fine tuning for iSCSI iSER is required. As was already mentioned in this thread, with SDR you can get ~900 MB/sec with iSER (on STGT). Erez -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
