Re: [Lsf] It's time to put together the schedule
On 02/25/15 18:36, Sagi Grimberg wrote: - About shared tags. So for scsi commands and TMFs we don't have a problem since we are guaranteed ITTs are unique. I wander how will we allocate a unique ITT for iscsi specific tasks (LOGIN, TEXT, LOGOUT, NOOP_OUT). My implementation did it per-session, so I reserved a range of ITTs for iscsi specific commands (in a kfifo), I wander how we can do that for multiple sessions. We need some kind of tag allocator for iscsi specific commands. Perhaps Bart/Christoph can advise if a LLD can allocate a unique tag for an LLD specific command using block layer tags. It is possible to allocate a reserved tag from the blk-mq layer by passing true as the fourth argument when calling blk_mq_alloc_request(). However, using that mechanism from inside a SCSI LLD driver would require changes in the SCSI core. The code in scsi_mq_setup_tags() initializes the number of reserved tags to zero. Additionally, the flag use_blk_tags in the SCSI host template makes it easy to use the same code paths in a SCSI LLD in multiqueue and single queue mode. However, the single queue block layer tag allocator does not support reserved tags (see also blk_queue_init_tags()). So it is probably easier to use a custom tag allocator in the iSCSI initiator for iSCSI-specific tasks instead of trying to allocate tags for these tasks from the block layer. Bart. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [Lsf] It's time to put together the schedule
On 2/26/2015 5:30 PM, Bart Van Assche wrote: On 02/25/15 18:36, Sagi Grimberg wrote: - About shared tags. So for scsi commands and TMFs we don't have a problem since we are guaranteed ITTs are unique. I wander how will we allocate a unique ITT for iscsi specific tasks (LOGIN, TEXT, LOGOUT, NOOP_OUT). My implementation did it per-session, so I reserved a range of ITTs for iscsi specific commands (in a kfifo), I wander how we can do that for multiple sessions. We need some kind of tag allocator for iscsi specific commands. Perhaps Bart/Christoph can advise if a LLD can allocate a unique tag for an LLD specific command using block layer tags. It is possible to allocate a reserved tag from the blk-mq layer by passing true as the fourth argument when calling blk_mq_alloc_request(). However, using that mechanism from inside a SCSI LLD driver would require changes in the SCSI core. The code in scsi_mq_setup_tags() initializes the number of reserved tags to zero. Additionally, the flag use_blk_tags in the SCSI host template makes it easy to use the same code paths in a SCSI LLD in multiqueue and single queue mode. However, the single queue block layer tag allocator does not support reserved tags (see also blk_queue_init_tags()). Yes I looked at that, but as you said, it does not a really fit SCSI LLDs. So it is probably easier to use a custom tag allocator in the iSCSI initiator for iSCSI-specific tasks instead of trying to allocate tags for these tasks from the block layer. That's what I thought, a session would allocate an array of mgmt iscsi specific commands, each assigned with a unique ITT which is out of scsi_cmds_max range. I guess I just wanted to understand if there is a nicer way of doing this instead of each LLD with similar needs doing the same thing... Sagi. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [Lsf] It's time to put together the schedule
On 2/25/2015 12:36 PM, Mike Christie wrote: On 02/22/2015 09:25 PM, Mike Christie wrote: I just hit a bug in the userspace code. Will send that later. Hey Sagi, Attached is the userspace patch, user-mq6.patch. It is made over 0001-iscsid-make-sure-actor-is-delated-before-reschedulin.patch (the patch to fix that double schedule bug you guys found). I am also attaching a updated kernel patch. It has some fixes for logout and iscsi_tcp mq setup. To use the patches, just set the new iscsid.conf setting node.session.queue_ids. It is just a string of ints: node.session.queue_ids = 1 2 4 8 that get passed in to the kernel. For each id, iscsid will create a session and have the LLD map whatever they want to that id value. Login is the same: iscsiadm -m node -T yourtargget -p ip --login However, after you login you have to manually scan iscsiadm -m session --rescan For logout, you currently have to make sure you logout all the sessions, so use: iscsiadm -m node -T yourtargget -p ip --logout or iscsiadm -m session --logout If you just pass in a specific session id like here: iscsiadm -m session -r SID --logout then that will wait for all the other sessions in the group to be logged out before completion the task. I did this because I was not yet sure how to handle dynamic hctx updates in the kernel. For the LLD implementation, I hooked in iscsi_tcp to the session/group creation code. Like I said before, I was not sure what every driver/fw/hw was going to map to, so the queue id that is getting passed into the session/connection/ep creation functions is really generic and you can map it to whatever you like right now. For ib_iser, you should look at iscsi_tcp.c's create_session_grp and destroy_session_grp callouts to see how to allocate the host in a backward compatible way. I'll do that. Sofware iscsi/iser is doing a host per session still, then doing a session_grp per host and multiple sessions per group. HW iscsi offload will continue to do a host per some hw/fw resource, then it can have multiple groups and multiple sessions per group. I am passing in the queue_id to bind to in every object callout (ep_connect, conn_create, session_create), because I was not sure at what time all the drivers needed to bind/setup-mappings at. So pick which ever makes sense and let me know. I have not had time to break this into a proper patchset. Was not ready to send as a RFC set. There is debugging and // comments in places, but feel free to give me any feedback. That's not a problem, we will get it ready for submission together... If you did get my other mails/patches a while back then make sure you are using the new userspace patches/tools in this mail with the updated kernel patch in this mail. I have not yet added kernel/user compat code, so you will hit hangs/crashes if you mix and match. Thanks Mike, I'm working with upstream both in user and kernel. Couple of quick first comments: - Passing a list in node.session.queue_ids indeed allows the user a degree of freedom, but it might be an overshoot. We should allow giving a range type of queue_ids and the default should be a range [0-default_nr_queues]. On a side note, I suspect we will pretty soon find out that this linear assignment will be the only useful setting and leave only nr_queues setting. - In the single queue case, we need to pass the kernel a default WILDCARD_QUEUE_ID so drivers can spread the completion contexts of each session as they do today (and don't introduce a performance regression). - About the queue_id. sw_tcp will need it for the TX/RX threads CPU binding, so it is used as a conn attribute. iser (and I assume other offloads as well) will need it for MSIX vector assignments, so it is used as an endpoint attribute. The session is completely logical, thus it should not hold a queue_id assignment (IMO at least). - The session group should allocate session_map to only hold the number of sessions it was passed with (not nr_cpu_ids with possible holes). The session selection is based on the mq mapping, thus it should be a 1x1 mapping to hctx. So it is basically boils down to: idx = sc-request-q-mq_map[sc-request-mq_ctx-cpu]; session = grp-session_map[idx]; and when we will properly use block layer tagging: tag = blk_mq_unique_tag(sc-request); idx = blk_mq_unique_tag_to_hwq(tag); session = grp-session_map[idx]; So I guess my point is, we should not assign a queue_id to a session, the ep/conn queue_id was used at establishment for context assignment. - About shared tags. So for scsi commands and TMFs we don't have a problem since we are guaranteed ITTs are unique. I wander how will we allocate a unique ITT for iscsi specific tasks (LOGIN, TEXT, LOGOUT, NOOP_OUT). My implementation did it per-session, so I reserved a range of ITTs for iscsi specific commands (in a kfifo), I wander how we can do that for multiple sessions. We need some kind of tag allocator
Re: [Lsf] It's time to put together the schedule
On Mon, Feb 23, 2015 at 5:25 AM, Mike Christie micha...@cs.wisc.edu wrote: Dropping lsf for iscsi list. Sorry about that. I thought I sent this to you. It must have got lost. Attached is the kernel changes for session based mq support (patch made over linus's tree). It just has the core changes only. I did not yet modify iscsi_tcp/libiscsi to bind our threads and sockets to cpus or deal with the networking. I was waiting to hear back from you about how it could work for ib_iser and was waiting to hear back from some hw vendors about how they are going to support mq in their hw/fw. So, the queue_id that gets passed in is really generic at this point and can map to whatever the transport driver wants to map it to (actual hw/fw queue for offload, msix/cpu combo like srp for iser, cpu and something else for iscsi_tcp, etc). I just hit a bug in the userspace code. Will send that later. But yeah, just added a new struct between the host and sessions. Mike, such a big patch with 0-len change-log... could you come up with 3-4 sentences telling what we have here, would be very much appreciated (thanks), Or. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [Lsf] It's time to put together the schedule
On 2/23/2015 5:25 AM, Mike Christie wrote: Sorry about that. I thought I sent this to you. It must have got lost. Attached is the kernel changes for session based mq support (patch made over linus's tree). It just has the core changes only. I did not yet modify iscsi_tcp/libiscsi to bind our threads and sockets to cpus or deal with the networking. I was waiting to hear back from you about how it could work for ib_iser and was waiting to hear back from some hw vendors about how they are going to support mq in their hw/fw. So, the queue_id that gets passed in is really generic at this point and can map to whatever the transport driver wants to map it to (actual hw/fw queue for offload, msix/cpu combo like srp for iser, cpu and something else for iscsi_tcp, etc). I just hit a bug in the userspace code. Will send that later. But yeah, just added a new struct between the host and sessions. Hey Mike, I have been working on transforming iscsi to use block layer tags. I'll port these patches on top of this one. Thanks, Sagi. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
