Re: [LSF/MM TOPIC] iSCSI MQ adoption via MCS discussion
On 01/07/15 22:39, Mike Christie wrote: On 01/07/2015 10:57 AM, Hannes Reinecke wrote: On 01/07/2015 05:25 PM, Sagi Grimberg wrote: Hi everyone, Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI. My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions. In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue). iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal. As I see it, the todo's are: 1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution). 2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection. 3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible. 4. Use blk-mq pre-allocation and tagging facilities. I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this. Yes, that's a really good topic. I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good. Mike? Any comments? I have been working under the assumption that people would be ok with MCS upstream if we are only using it to handle the issue where we want to do something like have a tcp/iscsi connection per CPU then map the connection to a blk_mq_hw_ctx. In this more limited MCS implementation there would be no iscsi layer code to do something like load balance across ports or transport paths like how dm-multipath does, so there would be no feature/code duplication. For balancing across hctxs, then the iscsi layer would also leave that up to whatever we end up with in upper layers, so again no feature/code duplication with upper layers. So pretty non controversial I hope :) If people want to add something like round robin connection selection in the iscsi layer, then I think we want to leave that for after the initial merge, so people can argue about that separately. Hello Sagi and Mike, I agree with Sagi that adding scsi-mq support in the iSER initiator would help iSER users because that would allow these users to configure a single iSER target and use the multiqueue feature instead of having to configure multiple iSER targets to spread the workload over multiple cpus at the target side. And I agree with Mike that implementing scsi-mq support in the iSER initiator as multiple independent connections probably is a better choice than MC/S. RFC 3720 namely requires that iSCSI numbering is session-wide. This means maintaining a single counter for all MC/S sessions. Such a counter would be a contention point. I'm afraid that because of that counter performance on a multi-socket initiator system with a scsi-mq implementation based on MC/S could be worse than with the approach with multiple iSER targets. Hence my preference for an approach based on multiple independent iSER connections instead of MC/S. Bart. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [LSF/MM TOPIC] iSCSI MQ adoption via MCS discussion
On 01/07/2015 10:57 AM, Hannes Reinecke wrote: > On 01/07/2015 05:25 PM, Sagi Grimberg wrote: >> Hi everyone, >> >> Now that scsi-mq is fully included, we need an iSCSI initiator that >> would use it to achieve scalable performance. The need is even greater >> for iSCSI offload devices and transports that support multiple HW >> queues. As iSER maintainer I'd like to discuss the way we would choose >> to implement that in iSCSI. >> >> My measurements show that iSER initiator can scale up to ~2.1M IOPs >> with multiple sessions but only ~630K IOPs with a single session where >> the most significant bottleneck the (single) core processing >> completions. >> >> In the existing single connection per session model, given that command >> ordering must be preserved session-wide, we end up in a serial command >> execution over a single connection which is basically a single queue >> model. The best fit seems to be plugging iSCSI MCS as a multi-queued >> scsi LLDD. In this model, a hardware context will have a 1x1 mapping >> with an iSCSI connection (TCP socket or a HW queue). >> >> iSCSI MCS and it's role in the presence of dm-multipath layer was >> discussed several times in the past decade(s). The basic need for MCS is >> implementing a multi-queue data path, so perhaps we may want to avoid >> doing any type link aggregation or load balancing to not overlap >> dm-multipath. For example we can implement ERL=0 (which is basically the >> scsi-mq ERL) and/or restrict a session to a single portal. >> >> As I see it, the todo's are: >> 1. Getting MCS to work (kernel + user-space) with ERL=0 and a >>round-robin connection selection (per scsi command execution). >> 2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and >>using blk-mq based queue (conn) selection. >> 3. Rework iSCSI core locking scheme to avoid session-wide locking >>as much as possible. >> 4. Use blk-mq pre-allocation and tagging facilities. >> >> I've recently started looking into this. I would like the community to >> agree (or debate) on this scheme and also talk about implementation >> with anyone who is also interested in this. >> > Yes, that's a really good topic. > > I've pondered implementing MC/S for iscsi/TCP but then I've figured my > network implementation knowledge doesn't spread that far. > So yeah, a discussion here would be good. > > Mike? Any comments? I have been working under the assumption that people would be ok with MCS upstream if we are only using it to handle the issue where we want to do something like have a tcp/iscsi connection per CPU then map the connection to a blk_mq_hw_ctx. In this more limited MCS implementation there would be no iscsi layer code to do something like load balance across ports or transport paths like how dm-multipath does, so there would be no feature/code duplication. For balancing across hctxs, then the iscsi layer would also leave that up to whatever we end up with in upper layers, so again no feature/code duplication with upper layers. So pretty non controversial I hope :) If people want to add something like round robin connection selection in the iscsi layer, then I think we want to leave that for after the initial merge, so people can argue about that separately. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [LSF/MM TOPIC] iSCSI MQ adoption via MCS discussion
On 01/07/2015 08:25 AM, Sagi Grimberg wrote: > Hi everyone, > > Now that scsi-mq is fully included, we need an iSCSI initiator that > would use it to achieve scalable performance. The need is even greater > for iSCSI offload devices and transports that support multiple HW > queues. As iSER maintainer I'd like to discuss the way we would choose > to implement that in iSCSI. > > My measurements show that iSER initiator can scale up to ~2.1M IOPs > with multiple sessions but only ~630K IOPs with a single session where > the most significant bottleneck the (single) core processing > completions. > > In the existing single connection per session model, given that command > ordering must be preserved session-wide, we end up in a serial command > execution over a single connection which is basically a single queue > model. The best fit seems to be plugging iSCSI MCS as a multi-queued > scsi LLDD. In this model, a hardware context will have a 1x1 mapping > with an iSCSI connection (TCP socket or a HW queue). > > iSCSI MCS and it's role in the presence of dm-multipath layer was > discussed several times in the past decade(s). The basic need for MCS is > implementing a multi-queue data path, so perhaps we may want to avoid > doing any type link aggregation or load balancing to not overlap > dm-multipath. For example we can implement ERL=0 (which is basically the > scsi-mq ERL) and/or restrict a session to a single portal. > > As I see it, the todo's are: > 1. Getting MCS to work (kernel + user-space) with ERL=0 and a >round-robin connection selection (per scsi command execution). > 2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and >using blk-mq based queue (conn) selection. > 3. Rework iSCSI core locking scheme to avoid session-wide locking >as much as possible. > 4. Use blk-mq pre-allocation and tagging facilities. > > I've recently started looking into this. I would like the community to > agree (or debate) on this scheme and also talk about implementation > with anyone who is also interested in this. > > Cheers, > Sagi. I started looking at this last year (and Hannes' suggestion), and would love to join the discussion. Please add me to the list of those that wish to attend. -- Lee Duncan SUSE Labs -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [LSF/MM TOPIC] iSCSI MQ adoption via MCS discussion
On 01/07/2015 05:25 PM, Sagi Grimberg wrote: > Hi everyone, > > Now that scsi-mq is fully included, we need an iSCSI initiator that > would use it to achieve scalable performance. The need is even greater > for iSCSI offload devices and transports that support multiple HW > queues. As iSER maintainer I'd like to discuss the way we would choose > to implement that in iSCSI. > > My measurements show that iSER initiator can scale up to ~2.1M IOPs > with multiple sessions but only ~630K IOPs with a single session where > the most significant bottleneck the (single) core processing > completions. > > In the existing single connection per session model, given that command > ordering must be preserved session-wide, we end up in a serial command > execution over a single connection which is basically a single queue > model. The best fit seems to be plugging iSCSI MCS as a multi-queued > scsi LLDD. In this model, a hardware context will have a 1x1 mapping > with an iSCSI connection (TCP socket or a HW queue). > > iSCSI MCS and it's role in the presence of dm-multipath layer was > discussed several times in the past decade(s). The basic need for MCS is > implementing a multi-queue data path, so perhaps we may want to avoid > doing any type link aggregation or load balancing to not overlap > dm-multipath. For example we can implement ERL=0 (which is basically the > scsi-mq ERL) and/or restrict a session to a single portal. > > As I see it, the todo's are: > 1. Getting MCS to work (kernel + user-space) with ERL=0 and a >round-robin connection selection (per scsi command execution). > 2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and >using blk-mq based queue (conn) selection. > 3. Rework iSCSI core locking scheme to avoid session-wide locking >as much as possible. > 4. Use blk-mq pre-allocation and tagging facilities. > > I've recently started looking into this. I would like the community to > agree (or debate) on this scheme and also talk about implementation > with anyone who is also interested in this. > Yes, that's a really good topic. I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good. Mike? Any comments? Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage h...@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.