Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Michael We could just let the user specify the Id Ext when adding Michael the device. How does this sound? Yes, that makes the most sense -- just add another optional option for use when adding a target. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Quoting r. Vu Pham [EMAIL PROTECTED]: Subject: Re: [PATCH] IB/SRP: Enable multichannel What is the advantage to have multiple connections/qps on the same physical port to the same target? The disavantages are wasting resources, instability, no fail-over on physical port error... The advantage is if the target in question is an IOC that connects to a FC SAN for example. In this case, the host is physically connected to the same IOC, but can maintain independent logical connections to specific storage devices on the SAN that are behind the IOC. Quoting r. Michael S. Tsirkin [EMAIL PROTECTED] Subject: Re: [PATCH] IB/SRP: Enable multichannel Maybe we should just use the port GUID instead of the node GUID to form the initiator ID? That would solve this pretty cleanly I think. Sounds good. I think we should also stick the pkey into the identifier extension - I think it's nice for each partition to be able to act as a separate virtual network, not affecting others. What do you think? -- MST Sticking the pkey into the identifier extension may once again restrict the ability of the host to have multiple logical connections to an SRP IOC target. The most flexible approach appears to be: Identifier ID = Port GUID Identifier Extension = User specified Ishai's IB-SRP patch of 09/27 appears to accomplish the above. Madhu ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Quoting r. Lakshmanan, Madhu [EMAIL PROTECTED]: Subject: RE: [openib-general] [PATCH] IB/SRP: Enable multichannel Quoting r. Vu Pham [EMAIL PROTECTED]: Subject: Re: [PATCH] IB/SRP: Enable multichannel What is the advantage to have multiple connections/qps on the same physical port to the same target? The disavantages are wasting resources, instability, no fail-over on physical port error... The advantage is if the target in question is an IOC that connects to a FC SAN for example. In this case, the host is physically connected to the same IOC, but can maintain independent logical connections to specific storage devices on the SAN that are behind the IOC. We could just let the user specify the Id Ext when adding the device. How does this sound? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Quoting r. Lakshmanan, Madhu [EMAIL PROTECTED]: gen2/trunk/src/linux-kernel/infiniband/ulp/srp. This is deprecated. You can get the exact code used for OFED 1.1 from ofed git tree. The instructions are here: https://openib.org/svn/gen2/branches/1.1/ofed/docs/HOWTO.build_ofed I'd like to collaborate with you on OFED SRP. Please note that OFED 1.1 is in freeze and only critical and documentation fixes are accepted. Note also that OFED is a distribution testing and packaging, not a development effort. OFED backports kernel.org code to older kernels, so there's no OFED SRP as such: to get your work into the next OFED release you should just work against the latest kernel.org tree, and get Roland to accept your patches. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Quoting r. Roland Dreier [EMAIL PROTECTED]: Subject: Re: [PATCH] IB/SRP: Enable multichannel Maybe we should just use the port GUID instead of the node GUID to form the initiator ID? That would solve this pretty cleanly I think. Sounds good. I think we should also stick the pkey into the identifier extension - I think it's nice for each partition to be able to act as a separate virtual network, not affecting others. What do you think? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Michael S. Tsirkin wrote: Quoting r. Vu Pham [EMAIL PROTECTED]: Most of srp targets that I tested don't support multiple channels. Which are these? Mellanox referenced srp target, Texas Memory System's SSD, Engenio. And what happens when you ask for multichannel support? For Texas' SSD the login req is rejected For Mellanox srp target the new multi channel/connection is established; however, if the host is in error recovery and do target reset, the host should terminate all outstanding channels/connections else the target have outstanding I/Os dangled on multi-channel/connection and try to complete the I/Os. This is violate scsi task management ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Quoting r. Vu Pham [EMAIL PROTECTED]: Either you can use multiple channels or derive different initiator_port_ID in the login req to have multiple paths on the same physical port So how about we just stick a pointer inside the indentifier extension instead of enabling multichannel? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Michael S. Tsirkin wrote: Quoting r. Vu Pham [EMAIL PROTECTED]: Either you can use multiple channels or derive different initiator_port_ID in the login req to have multiple paths on the same physical port So how about we just stick a pointer inside the indentifier extension instead of enabling multichannel? That's the simple change. Beside that you have to maintain a list of connections/channels connected to the same target, to manage/clean-up resource associated with these connections, how to handle error recovery especially target reset and host reset... What is the advantage to have multiple connections/qps on the same physical port to the same target? The disavantages are wasting resources, instability, no fail-over on physical port error... ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Vu Pham wrote:
Michael S. Tsirkin wrote:
Quoting r. Vu Pham [EMAIL PROTECTED]:
Either you can use multiple channels or derive different
initiator_port_ID in the login req to have multiple paths on
the same physical port
So how about we just stick a pointer inside the indentifier extension
instead of enabling multichannel?
That's the simple change. Beside that you have to maintain a
list of connections/channels connected to the same target,
to manage/clean-up resource associated with these
connections, how to handle error recovery especially target
reset and host reset...
What is the advantage to have multiple connections/qps on
the same physical port to the same target? The disavantages
are wasting resources, instability, no fail-over on physical
port error...
I see the limitation of current srp implementation. If we
have the following topoloty
host port 1 -- target port 1
host port 1 -- target port 2
the current srp implementation will use the same
initiator_port_id for both login requests and the target
will reject the second login if you don't turn on
SUPPORT_MULTI_CHANNEL
Another way to solve this is to use different
initiator_port_id for the logins ie.
path 1: initiator_port_ID{target_port1_GUID,
initiator_port1_GUID} and target_port_ID{id_ext, ioc_guid}
path 2:
initiator_port_ID{target_port2_GUID,initiator_port1_GUID}
and target_port_ID
This also will guarantee the uniqueness of initiator_port_id
in the fabric
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Maybe we should just use the port GUID instead of the node GUID to form the initiator ID? That would solve this pretty cleanly I think. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Roland Dreier wrote: Maybe we should just use the port GUID instead of the node GUID to form the initiator ID? That would solve this pretty cleanly I think. This is also Vu's idea. There are two issues: 1) My patch allows a sophisticated user to have two logical connections on the same physical solution. He can have different connection parameters (e.g., MAX_CMD_PER_LUN) according to the application needs. Do you think there is such need? 2) In the current implementation there is a problem when there are two connections on the same physical connection - when the second connection sends REQ to the target, the target sends a DREQ to the first connection, but when someone tries to access the first scsi_host, ib_srp tries to reconnect the first connection and then the second connection gets a DREQ - and so the ping pong goes. And if there is a multipath daemon that checks the status of the connections this ping pong can be for ever. We need to find a way to eliminate this behavior. Ishai ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH] IB/SRP: Enable multichannel
Hi Roland, SRP High Availability needs an initiator to connect to the same target several times, e.g., once from each IB port of the target (this way we can use device mapper multipath for failover). Note that both connections are actually active, e.g. multipath is issuing commands to to get the remote scsi id. Since multiple channel operation is currently disabled in connection request, each new connection request will cause the target to disconnect the existing connection which forces us to bounce a lot between the two channels. This patch enables multiple channel operation in connection requests, to avoid getting disconnects when multiple connections are active. There does not seem to be any harm in doing this even when multipath is not used. Signed-off-by: Ishai Rabinovitz [EMAIL PROTECTED] --- Index: last_stable/drivers/infiniband/ulp/srp/ib_srp.c === --- last_stable.orig/drivers/infiniband/ulp/srp/ib_srp.c2006-09-26 09:22:13.0 +0300 +++ last_stable/drivers/infiniband/ulp/srp/ib_srp.c 2006-09-26 14:54:35.0 +0300 @@ -329,6 +329,7 @@ static int srp_send_req(struct srp_targe req-priv.req_it_iu_len = cpu_to_be32(srp_max_iu_len); req-priv.req_buf_fmt = cpu_to_be16(SRP_BUF_FORMAT_DIRECT | SRP_BUF_FORMAT_INDIRECT); + req-priv.req_flags = SRP_MULTICHAN_MULTI; /* * In the published SRP specification (draft rev. 16a), the * port identifier format is 8 bytes of ID extension followed -- Ishai Rabinovitz ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Ishai Rabinovitz wrote: Hi Roland, SRP High Availability needs an initiator to connect to the same target several times, e.g., once from each IB port of the target (this way we can use device mapper multipath for failover). Note that both connections are actually active, e.g. multipath is issuing commands to to get the remote scsi id. It depends on how you define a path. An target-port-ID initiator-port-ID tuple will form a path. Current srp implementation uses port GID as initiator_port_ID, and target ioc_guid + id_ext as target_port_ID. With this implementation, a physical host port physical target port will form a path multipath driver will see the same scsi_id of a lun thru multiple paths Since multiple channel operation is currently disabled in connection request, each new connection request will cause the target to disconnect the existing connection which forces us to bounce a lot between the two channels. Either you can use multiple channels or derive different initiator_port_ID in the login req to have multiple paths on the same physical port Most of srp targets that I tested don't support multiple channels. -vu ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/SRP: Enable multichannel
Quoting r. Vu Pham [EMAIL PROTECTED]: Most of srp targets that I tested don't support multiple channels. Which are these? And what happens when you ask for multichannel support? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
