On 2/25/2015 12:36 PM, Mike Christie wrote:
On 02/22/2015 09:25 PM, Mike Christie wrote:
I just hit a bug in the userspace code. Will send that later.

Hey Sagi,

Attached is the userspace patch, user-mq6.patch. It is made over
0001-iscsid-make-sure-actor-is-delated-before-reschedulin.patch (the
patch to fix that double schedule bug you guys found).

I am also attaching a updated kernel patch. It has some fixes for logout
and iscsi_tcp mq setup.

To use the patches, just set the new iscsid.conf setting
"node.session.queue_ids". It is just a string of ints:

node.session.queue_ids = 1 2 4 8

that get passed in to the kernel. For each id, iscsid will create a
session and have the LLD map whatever they want to that id value. Login
is the same:

iscsiadm -m node -T yourtargget -p ip --login

However, after you login you have to manually scan

iscsiadm -m session --rescan


For logout, you currently have to make sure you logout all the sessions,
so use:

iscsiadm -m node -T yourtargget -p ip --logout

or

iscsiadm -m session --logout

If you just pass in a specific session id like here:

iscsiadm -m session -r SID --logout

then that will wait for all the other sessions in the group to be logged
out before completion the task. I did this because I was not yet sure
how to handle dynamic hctx updates in the kernel.


For the LLD implementation, I hooked in iscsi_tcp to the session/group
creation code. Like I said before, I was not sure what every
driver/fw/hw was going to map to, so the queue id that is getting passed
into the session/connection/ep creation functions is really generic and
you can map it to whatever you like right now.

For ib_iser, you should look at iscsi_tcp.c's create_session_grp and
destroy_session_grp callouts to see how to allocate the host in a
backward compatible way.

I'll do that.

Sofware iscsi/iser is doing a host per session
still, then doing a session_grp per host and multiple sessions per
group. HW iscsi offload will continue to do a host per some hw/fw
resource, then it can have multiple groups and multiple sessions per group.

I am passing in the queue_id to bind to in every object callout
(ep_connect, conn_create, session_create), because I was not sure at
what time all the drivers needed to bind/setup-mappings at. So pick
which ever makes sense and let me know.

I have not had time to break this into a proper patchset. Was not ready
to send as a RFC set. There is debugging and // comments in places, but
feel free to give me any feedback.

That's not a problem, we will get it ready for submission together...


If you did get my other mails/patches a while back then make sure you
are using the new userspace patches/tools in this mail with the updated
kernel patch in this mail. I have not yet added kernel/user compat code,
so you will hit hangs/crashes if you mix and match.


Thanks Mike, I'm working with upstream both in user and kernel.

Couple of quick first comments:

- Passing a list in node.session.queue_ids indeed allows the user
  a degree of freedom, but it might be an overshoot. We should allow
  giving a range type of queue_ids and the default should be a range
  [0-default_nr_queues].
  On a side note, I suspect we will pretty soon find out that this
  linear assignment will be the only useful setting and leave only
  nr_queues setting.

- In the single queue case, we need to pass the kernel a default
  WILDCARD_QUEUE_ID so drivers can spread the completion contexts of
  each session as they do today (and don't introduce a performance
  regression).

- About the queue_id. sw_tcp will need it for the TX/RX threads
  CPU binding, so it is used as a conn attribute. iser (and I assume
  other offloads as well) will need it for MSIX vector assignments, so
  it is used as an endpoint attribute. The session is completely
  logical, thus it should not hold a queue_id assignment (IMO at least).

- The session group should allocate session_map to only hold the number
  of sessions it was passed with (not nr_cpu_ids with possible holes).
  The session selection is based on the mq mapping, thus it should be a
  1x1 mapping to hctx. So it is basically boils down to:

  idx = sc->request->q->mq_map[sc->request->mq_ctx->cpu];
  session = grp->session_map[idx];

  and when we will properly use block layer tagging:

  tag = blk_mq_unique_tag(sc->request);
  idx = blk_mq_unique_tag_to_hwq(tag);
  session = grp->session_map[idx];

  So I guess my point is, we should not assign a queue_id to a session,
  the ep/conn queue_id was used at establishment for context assignment.

- About shared tags. So for scsi commands and TMFs we don't have a
  problem since we are guaranteed ITTs are unique. I wander how will we
  allocate a unique ITT for iscsi specific tasks (LOGIN, TEXT, LOGOUT,
  NOOP_OUT). My implementation did it per-session, so I reserved a range
  of ITTs for iscsi specific commands (in a kfifo), I wander how we can
  do that for multiple sessions. We need some kind of tag allocator for
  iscsi specific commands. Perhaps Bart/Christoph can advise if a LLD
  can allocate a unique tag for an LLD specific command using block
  layer tags.

Let me work on some of the things mentioned here.

Cheers,
Sagi.

--
You received this message because you are subscribed to the Google Groups 
"open-iscsi" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to open-iscsi+unsubscr...@googlegroups.com.
To post to this group, send email to open-iscsi@googlegroups.com.
Visit this group at http://groups.google.com/group/open-iscsi.
For more options, visit https://groups.google.com/d/optout.

Reply via email to