Re: Add initial DCB support
Anish, On Oct 18, 2011, at 1:30 PM, Anish Bhatt wrote: > Mark, > I was reading up on some previous correspondence regarding how > iSCSI traffic should be sent via a separate VLAN. I'm trying to figure > out the use case for said scenario. Is the idea that iscsi traffic is > run over a separate VLAN interface (created using the VLAN id from the > VLAN header for iSCSI configured on the switch) but with the priority > group pulled from DCBX ? The set_dcb_priority() calls upstream seem to > suggest this approach. Or were u suggesting that open-iscsi create some > sort of internal VLAN for explicit DCB use ? > -Anish > > One socket to bind them all. The reason to use a VLAN is to have a VLAN tag that can indicate the priority as packets pass through switches. iSCSI should then be using that VLAN interface for its traffic. You will see that the DCB code gets the desired priority from the app TLV provided for the underlying link interface. If the iSCSI traffic were not sent to the VLAN interface, the priority would only have meaning within the system. That is, the priority could still affect the queuing within the system, but without the VLAN tag, it would not be able to represent that priority over the wire to other devices in the network. If, for whatever reason, there is no need to indicate that priority to other network devices, then it would not be necessary to to have iSCSI use a VLAN. A VLAN that only existed internally would defeat the purpose of having the VLAN tag indicate the priority of the traffic to other devices. Again, if indicating the priority over the wire is not important, then this could be done, but an internal-only VLAN would seem to serve to purpose at all. The link device could just as well be used if there is no need to communicate that priority to other devices. I don't think that there is any need for open-iscsi to create any special VLAN device. >> -Original Message- >> From: open-iscsi@googlegroups.com [mailto:open-iscsi@googlegroups.com] >> On Behalf Of Rustad, Mark D >> Sent: Friday, October 07, 2011 2:35 PM >> To: open-iscsi@googlegroups.com; Anish Bhatt >> Cc: shyam_i...@dell.com >> Subject: Re: Add initial DCB support >> >> Anish, >> >> On Oct 7, 2011, at 1:47 PM, Anish Bhatt wrote: >> >>> Bringing up an old discussion for clarification here. If I > understand >>> correctly iscsi tlvs will be exchanged with protocol id set to >>> 3260(0xCBC) at all times, and this will get mapped to the correct >> iscsi >>> traffic flow at the endpoint irrespective of actual running port >> number. >>> Is that correct ? >>> -Anish >> >> Yes. The port number used in the tlv identifies the application by its >> well-known port number, but the application can still be configured to >> use whatever port it wants. >> >> -- >> Mark Rustad, LAN Access Division, Intel Corporation -- Mark Rustad, LAN Access Division, Intel Corporation -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
RE: Add initial DCB support
Mark, I was reading up on some previous correspondence regarding how iSCSI traffic should be sent via a separate VLAN. I'm trying to figure out the use case for said scenario. Is the idea that iscsi traffic is run over a separate VLAN interface (created using the VLAN id from the VLAN header for iSCSI configured on the switch) but with the priority group pulled from DCBX ? The set_dcb_priority() calls upstream seem to suggest this approach. Or were u suggesting that open-iscsi create some sort of internal VLAN for explicit DCB use ? -Anish One socket to bind them all. > -Original Message- > From: open-iscsi@googlegroups.com [mailto:open-iscsi@googlegroups.com] > On Behalf Of Rustad, Mark D > Sent: Friday, October 07, 2011 2:35 PM > To: open-iscsi@googlegroups.com; Anish Bhatt > Cc: shyam_i...@dell.com > Subject: Re: Add initial DCB support > > Anish, > > On Oct 7, 2011, at 1:47 PM, Anish Bhatt wrote: > > > Bringing up an old discussion for clarification here. If I understand > > correctly iscsi tlvs will be exchanged with protocol id set to > > 3260(0xCBC) at all times, and this will get mapped to the correct > iscsi > > traffic flow at the endpoint irrespective of actual running port > number. > > Is that correct ? > > -Anish > > Yes. The port number used in the tlv identifies the application by its > well-known port number, but the application can still be configured to > use whatever port it wants. > > -- > Mark Rustad, LAN Access Division, Intel Corporation > > > -- > You received this message because you are subscribed to the Google > Groups "open-iscsi" group. > To post to this group, send email to open-iscsi@googlegroups.com. > To unsubscribe from this group, send email to open- > iscsi+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/open-iscsi?hl=en. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Add initial DCB support
Anish, On Oct 7, 2011, at 1:47 PM, Anish Bhatt wrote: > Bringing up an old discussion for clarification here. If I understand > correctly iscsi tlvs will be exchanged with protocol id set to > 3260(0xCBC) at all times, and this will get mapped to the correct iscsi > traffic flow at the endpoint irrespective of actual running port number. > Is that correct ? > -Anish Yes. The port number used in the tlv identifies the application by its well-known port number, but the application can still be configured to use whatever port it wants. -- Mark Rustad, LAN Access Division, Intel Corporation -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
RE: Add initial DCB support
Bringing up an old discussion for clarification here. If I understand correctly iscsi tlvs will be exchanged with protocol id set to 3260(0xCBC) at all times, and this will get mapped to the correct iscsi traffic flow at the endpoint irrespective of actual running port number. Is that correct ? -Anish One socket to bind them all. > -Original Message- > From: open-iscsi@googlegroups.com [mailto:open-iscsi@googlegroups.com] > On Behalf Of Rustad, Mark D > Sent: Thursday, February 03, 2011 9:21 AM > To: open-iscsi@googlegroups.com; shyam_i...@dell.com > Subject: RE: Add initial DCB support > > Shyam, > > On Thursday, February 03, 2011 7:55 AM, shyam iyer wrote: > > > > And to change the priority in the kernel via userspace are you > posting > > a related patch to dcbtool ? > > > > Specifically adding the iSCSI subtype.. > > dcbtool has support for the CEE form for specifying iSCSI. The kernel > was not doing much with it, but patches that are in the works are > fixing that. lldptool will handle the DCBX forms and the kernel will > handle them as well. > > You will note that the netlink interface used here specifies the > protocol by well-known port number, adopting the DCBX means of > specifying protocol. So please don't change the use of the well-known > port in dcb_app.c to the port actually in use - the well-known port is > exactly what is wanted for this purpose. > > -- > Mark Rustad, LAN Access Division, Intel Corporation. > > -- > You received this message because you are subscribed to the Google > Groups "open-iscsi" group. > To post to this group, send email to open-iscsi@googlegroups.com. > To unsubscribe from this group, send email to open- > iscsi+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/open-iscsi?hl=en. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Add initial DCB support
On 2/2/2011 4:25 PM, Rustad, Mark D wrote:
> Shyam,
>
> On Wednesday, February 02, 2011 4:00 PM, shyam iyer wrote:
>>
>> On Feb 1, 6:10 pm, Mark Rustad wrote:
>
>
>
>>> +static void set_dcb_priority(struct iscsi_conn *conn, const char *devname)
>>> +{
>>> + int pri_mask = 0;
>>> +
>>> + pri_mask = get_dcb_app_pri_by_port(devname, ISCSI_DEFAULT_PORT);
>>> + if (pri_mask < 0)
>>> + log_debug(2, "Getting priority for %s returned %d",
>>> + devname, pri_mask);
>>> + else if (pri_mask == 0)
>>> + log_debug(2, "No priority for %s", devname);
>>> + else {
>>> + int pri = select_priority(conn, pri_mask);
>>> +
>>> + log_debug(1, "Setting socket priority to %d", pri);
>>> + setsockopt(conn->socket_fd, SOL_SOCKET,
>>> + SO_PRIORITY, &pri, sizeof(pri));
>>
>> Are we assuming 0-6 only as the priority ?
>
> No, it could be 0-7.
>
>> I thought it should be 0-7 and so requiring CAP_NET_ADMIN
>
> Hmmm. I think I always see it running as root. So what needs to be done
> to handle other security models?
>
>> Which brings me to the related question.
>>
>> How will the priority be handled when you have multipathed iSCSI
>> sessions through the same net device.
>>
>> Scenario -
>> 1st Session priority set to 3
>> Change priority to 4
>> Connect 2nd session. I am assuming the priority for the new connection
>> socket is 4.
>
> Yes. A deficiency in the current implementation is that the first session
> will stay at 3, but the second one would be at 4.
>
>> Are these two sessions going to be through different H/W queues if the
>> session is through the same ethX
>
> Yes, they would be.
>
>> So would it be correct to say that the mapping of the SO_PRIORITY is
>> to the tc configured HW queue?
>
> John Fastabend could answer this more specifically, but I do see the packets
> going
> to different hardware queues when different priorities are used. Right now I
> am not seeing the priority indicated in the VLAN header of transmitted
> packets and I think that is a bug. Note that I have been testing with kernels
> that include changes that may not be upstream yet. I guess I should check on
> where those changes are, but I have other things to do first.
>
Hi Shyam,
The mapping depends on the priority to TC mapping configured in the
net device. This can be configured through the mqprio qdisc or with
the base driver dflts which is 1:1 for ixgbe and 8 traffic classes.
Note the mqprio qdisc is in net-next-2.6 tree and should get merged into
Linus' tree during the next kernel merge window. Also there will need to
be some enabling of iproute2 that I haven't done yet. For now the
defaults are used.
John.
--
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com.
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/open-iscsi?hl=en.
RE: Add initial DCB support
Shyam, On Thursday, February 03, 2011 7:55 AM, shyam iyer wrote: > > And to change the priority in the kernel via userspace are you posting > a related patch to dcbtool ? > > Specifically adding the iSCSI subtype.. dcbtool has support for the CEE form for specifying iSCSI. The kernel was not doing much with it, but patches that are in the works are fixing that. lldptool will handle the DCBX forms and the kernel will handle them as well. You will note that the netlink interface used here specifies the protocol by well-known port number, adopting the DCBX means of specifying protocol. So please don't change the use of the well-known port in dcb_app.c to the port actually in use - the well-known port is exactly what is wanted for this purpose. -- Mark Rustad, LAN Access Division, Intel Corporation. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Add initial DCB support
On Feb 2, 7:25 pm, "Rustad, Mark D" wrote:
> Shyam,
>
> On Wednesday, February 02, 2011 4:00 PM, shyam iyer wrote:
>
> > On Feb 1, 6:10 pm, Mark Rustad wrote:
>
>
>
>
>
> > > +static void set_dcb_priority(struct iscsi_conn *conn, const char
> > > *devname)
> > > +{
> > > + int pri_mask = 0;
> > > +
> > > + pri_mask = get_dcb_app_pri_by_port(devname, ISCSI_DEFAULT_PORT);
> > > + if (pri_mask < 0)
> > > + log_debug(2, "Getting priority for %s returned %d",
> > > + devname, pri_mask);
> > > + else if (pri_mask == 0)
> > > + log_debug(2, "No priority for %s", devname);
> > > + else {
> > > + int pri = select_priority(conn, pri_mask);
> > > +
> > > + log_debug(1, "Setting socket priority to %d", pri);
> > > + setsockopt(conn->socket_fd, SOL_SOCKET,
> > > + SO_PRIORITY, &pri, sizeof(pri));
>
> > Are we assuming 0-6 only as the priority ?
>
> No, it could be 0-7.
>
> > I thought it should be 0-7 and so requiring CAP_NET_ADMIN
>
> Hmmm. I think I always see it running as root. So what needs to be done
> to handle other security models?
>
> > Which brings me to the related question.
>
> > How will the priority be handled when you have multipathed iSCSI
> > sessions through the same net device.
>
> > Scenario -
> > 1st Session priority set to 3
> > Change priority to 4
> > Connect 2nd session. I am assuming the priority for the new connection
> > socket is 4.
>
> Yes. A deficiency in the current implementation is that the first session
> will stay at 3, but the second one would be at 4.
>
And to change the priority in the kernel via userspace are you posting
a related patch to dcbtool ?
Specifically adding the iSCSI subtype..
> > Are these two sessions going to be through different H/W queues if the
> > session is through the same ethX
>
> Yes, they would be.
>
> > So would it be correct to say that the mapping of the SO_PRIORITY is
> > to the tc configured HW queue?
>
> John Fastabend could answer this more specifically, but I do see the packets
> going
> to different hardware queues when different priorities are used. Right now I
> am not seeing the priority indicated in the VLAN header of transmitted
> packets and I think that is a bug. Note that I have been testing with kernels
> that include changes that may not be upstream yet. I guess I should check on
> where those changes are, but I have other things to do first.
>
> --
> Mark Rustad, LAN Access Division, Intel Corporation.
--
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com.
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/open-iscsi?hl=en.
RE: Add initial DCB support
Shyam,
On Wednesday, February 02, 2011 4:00 PM, shyam iyer wrote:
>
> On Feb 1, 6:10 pm, Mark Rustad wrote:
> > +static void set_dcb_priority(struct iscsi_conn *conn, const char *devname)
> > +{
> > + int pri_mask = 0;
> > +
> > + pri_mask = get_dcb_app_pri_by_port(devname, ISCSI_DEFAULT_PORT);
> > + if (pri_mask < 0)
> > + log_debug(2, "Getting priority for %s returned %d",
> > + devname, pri_mask);
> > + else if (pri_mask == 0)
> > + log_debug(2, "No priority for %s", devname);
> > + else {
> > + int pri = select_priority(conn, pri_mask);
> > +
> > + log_debug(1, "Setting socket priority to %d", pri);
> > + setsockopt(conn->socket_fd, SOL_SOCKET,
> > + SO_PRIORITY, &pri, sizeof(pri));
>
> Are we assuming 0-6 only as the priority ?
No, it could be 0-7.
> I thought it should be 0-7 and so requiring CAP_NET_ADMIN
Hmmm. I think I always see it running as root. So what needs to be done
to handle other security models?
> Which brings me to the related question.
>
> How will the priority be handled when you have multipathed iSCSI
> sessions through the same net device.
>
> Scenario -
> 1st Session priority set to 3
> Change priority to 4
> Connect 2nd session. I am assuming the priority for the new connection
> socket is 4.
Yes. A deficiency in the current implementation is that the first session
will stay at 3, but the second one would be at 4.
> Are these two sessions going to be through different H/W queues if the
> session is through the same ethX
Yes, they would be.
> So would it be correct to say that the mapping of the SO_PRIORITY is
> to the tc configured HW queue?
John Fastabend could answer this more specifically, but I do see the packets
going
to different hardware queues when different priorities are used. Right now I am
not seeing the priority indicated in the VLAN header of transmitted packets and
I think that is a bug. Note that I have been testing with kernels that include
changes that may not be upstream yet. I guess I should check on where those
changes are, but I have other things to do first.
--
Mark Rustad, LAN Access Division, Intel Corporation.
--
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com.
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/open-iscsi?hl=en.
Re: Add initial DCB support
On Feb 1, 6:10 pm, Mark Rustad wrote:
> Add support for setting priorities provided by DCB in the kernel. This
> implementation does not track any priority changes which could occur
> over time.
>
> Signed-off-by: Mark Rustad
> ---
> usr/Makefile | 4 ++
> usr/io.c | 104
> +-
> 2 files changed, 105 insertions(+), 3 deletions(-)
>
> diff --git a/usr/Makefile b/usr/Makefile
> index b02a706..991efd9 100644
> --- a/usr/Makefile
> +++ b/usr/Makefile
> @@ -21,10 +21,12 @@ ifeq ($(OSNAME),Linux)
> endif
> endif
> IPC_OBJ=netlink.o
> +DCB_OBJ=dcb_app.o
> else
> ifeq ($(OSNAME),FreeBSD)
> IPC_CFLAGS=
> IPC_OBJ=ioctl.o
> +DCB_OBJ=
> endif
> endif
>
> @@ -39,7 +41,7 @@ SYSDEPS_SRCS = $(wildcard ../utils/sysdeps/*.o)
> # sources shared between iscsid, iscsiadm and iscsistart
> ISCSI_LIB_SRCS = iscsi_util.o io.o auth.o login.o log.o md5.o sha1.o iface.o
> \
> idbm.o sysfs.o host.o session_info.o iscsi_sysfs.o iscsi_net_util.o \
> - iscsid_req.o $(SYSDEPS_SRCS)
> + iscsid_req.o $(SYSDEPS_SRCS) $(DCB_OBJ)
> # core initiator files
> INITIATOR_SRCS = initiator.o scsi.o actor.o event_poll.o mgmt_ipc.o \
> transport.o cxgbi.o be2iscsi.o
> diff --git a/usr/io.c b/usr/io.c
> index 8fb806d..e9017a1 100644
> --- a/usr/io.c
> +++ b/usr/io.c
> @@ -26,11 +26,14 @@
> #include
> #include
> #include
> +#include
> +#include
> #include
> #include
>
> #include "types.h"
> #include "iscsi_proto.h"
> +#include "iscsi_settings.h"
> #include "initiator.h"
> #include "iscsi_ipc.h"
> #include "log.h"
> @@ -38,6 +41,7 @@
> #include "idbm.h"
> #include "iface.h"
> #include "sysdeps.h"
> +#include "dcb_app.h"
>
> #define LOG_CONN_CLOSED(conn) \
> do { \
> @@ -76,6 +80,80 @@ set_non_blocking(int fd)
>
> }
>
> +static int select_priority(struct iscsi_conn *conn, int pri_mask)
> +{
> + int msk;
> +
> + if (!pri_mask)
> + return 0;
> +
> + /*
> + * TODO: Configure priority selection from the mask
> + * For now, just always take the highest
> + */
> +
> + /* Find highest bit set */
> + while ((msk = pri_mask & (pri_mask - 1)))
> + pri_mask = msk;
> +
> + return ffs(pri_mask) - 1;
> +}
> +
> +static int inet_cmp_addr(struct sockaddr *s1, struct sockaddr *s2)
> +{
> + struct sockaddr_in *si1 = (struct sockaddr_in *)s1;
> + struct sockaddr_in *si2 = (struct sockaddr_in *)s2;
> +
> + return si1->sin_addr.s_addr != si2->sin_addr.s_addr;
> +}
> +
> +static int inet6_cmp_addr(struct sockaddr *s1, struct sockaddr *s2)
> +{
> + struct sockaddr_in6 *si1 = (struct sockaddr_in6 *)s1;
> + struct sockaddr_in6 *si2 = (struct sockaddr_in6 *)s2;
> +
> + return memcmp(&si1->sin6_addr, &si2->sin6_addr,
> sizeof(si1->sin6_addr));
> +}
> +
> +static char *find_ifname(struct ifaddrs *ifa, struct sockaddr *sa)
> +{
> + for (; ifa; ifa = ifa->ifa_next) {
> + if (sa->sa_family != ifa->ifa_addr->sa_family)
> + continue;
> + switch (sa->sa_family) {
> + case AF_INET:
> + if (inet_cmp_addr(sa, ifa->ifa_addr) == 0)
> + return ifa->ifa_name;
> + break;
> + case AF_INET6:
> + if (inet6_cmp_addr(sa, ifa->ifa_addr) == 0)
> + return ifa->ifa_name;
> + break;
> + }
> + }
> +
> + return NULL;
> +}
> +
> +static void set_dcb_priority(struct iscsi_conn *conn, const char *devname)
> +{
> + int pri_mask = 0;
> +
> + pri_mask = get_dcb_app_pri_by_port(devname, ISCSI_DEFAULT_PORT);
> + if (pri_mask < 0)
> + log_debug(2, "Getting priority for %s returned %d",
> + devname, pri_mask);
> + else if (pri_mask == 0)
> + log_debug(2, "No priority for %s", devname);
> + else {
> + int pri = select_priority(conn, pri_mask);
> +
> + log_debug(1, "Setting socket priority to %d", pri);
> + setsockopt(conn->socket_fd, SOL_SOCKET,
> + SO_PRIORITY, &pri, sizeof(pri));
Are we assuming 0-6 only as the priority ?
I thought it should be 0-7 and so requiring CAP_NET_ADMIN
Which brings me to the related question.
How will the priority be handled when you have multipathed iSCSI
sessions through the same net device.
Scenario -
1st Session priority set to 3
Change priority to 4
Connect 2nd session. I am assuming the priority for the new connection
socket is 4.
Are these two sessions going to be through different H/W queues if the
session is through the same ethX
So would it be correct to say that the mapping of the SO_PRIORITY is
to the tc configured HW queue?
> + }
> +}
> +
> #if 0
> /* not used by anyone */
> stati
