Re: [RFC] restore netdev_priv optimization
Hi, David Miller wrote: From: Stephen Hemminger [EMAIL PROTECTED] Date: Fri, 17 Aug 2007 15:40:22 -0700 Compile tested only!!! Obviously. The first loopback transmit is guarenteed to crash. [...] And this also breaks loopback again, which uses a static struct netdev in the kernel image, it doesn't use alloc_netdev(), so egress_subqueue of loopback will be NULL. Talking about loopback, don't you think it could be the right time to make it behave like any other kind of net devices, and allocate it dynamically. Having a dynamically allocated loopback could make maintenance easier (removing special cases). Also this is something we'll need to support multiple loopbacks for example for network namespaces. Eric Biederman has written a nice patch that does this. I'm using it on 2.6.23-rc2. Benjamin -- B e n j a m i n T h e r y - BULL/DT/Open Software RD http://www.bull.com - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] restore netdev_priv optimization
From: Stephen Hemminger [EMAIL PROTECTED] Date: Fri, 17 Aug 2007 15:40:22 -0700 Compile tested only!!! Obviously. The first loopback transmit is guarenteed to crash. Fix optimization of netdev_priv() lost by the addition of multiqueue. Move the variable size subqueues to after the constant size priv area. When putting back the old netdev_priv() code, I tried to make it clearer by using roundup() and ALIGN() macros. --- a/include/linux/netdevice.h 2007-08-17 12:08:51.0 -0400 +++ b/include/linux/netdevice.h 2007-08-17 12:48:03.0 -0400 @@ -575,16 +575,15 @@ struct net_device /* The TX queue control structures */ unsigned integress_subqueue_count; - struct net_device_subqueue egress_subqueue[1]; + struct net_device_subqueue *egress_subqueue; }; This just trades off the dev-priv dereference for a dev-egress_subqueue dereference. I bet they occur about equally in the data paths, at least on transmit. And this also breaks loopback again, which uses a static struct netdev in the kernel image, it doesn't use alloc_netdev(), so egress_subqueue of loopback will be NULL. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] restore netdev_priv optimization
On Fri, 17 Aug 2007 16:04:09 -0700 (PDT) David Miller [EMAIL PROTECTED] wrote: From: Stephen Hemminger [EMAIL PROTECTED] Date: Fri, 17 Aug 2007 15:40:22 -0700 Compile tested only!!! Obviously. The first loopback transmit is guarenteed to crash. That is fixable. Fix optimization of netdev_priv() lost by the addition of multiqueue. Move the variable size subqueues to after the constant size priv area. When putting back the old netdev_priv() code, I tried to make it clearer by using roundup() and ALIGN() macros. --- a/include/linux/netdevice.h 2007-08-17 12:08:51.0 -0400 +++ b/include/linux/netdevice.h 2007-08-17 12:48:03.0 -0400 @@ -575,16 +575,15 @@ struct net_device /* The TX queue control structures */ unsigned integress_subqueue_count; - struct net_device_subqueue egress_subqueue[1]; + struct net_device_subqueue *egress_subqueue; }; This just trades off the dev-priv dereference for a dev-egress_subqueue dereference. I bet they occur about equally in the data paths, at least on transmit. The subqueue is only referenced in start/stop queue and that only happens once per packet on normal tx, and only if multiqueue is used. The existing multiqueue penalizes all devices, not just multiqueue devices. And this also breaks loopback again, which uses a static struct netdev in the kernel image, it doesn't use alloc_netdev(), so egress_subqueue of loopback will be NULL. That can be overcome. -- Stephen Hemminger [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] restore netdev_priv optimization
From: Stephen Hemminger [EMAIL PROTECTED] Date: Fri, 17 Aug 2007 16:19:28 -0700 The subqueue is only referenced in start/stop queue and that only happens once per packet on normal tx, and only if multiqueue is used. If it only happens when multiqueue, then why does loopback need at least one queue entry there even though it's not multiqueue? :-) This thing is deref'd regardless of multi-queue. Once per TX packet is a lot, and it's the same amount of derefs as dev-priv will have on the TX path. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] restore netdev_priv optimization (planb)
From: Stephen Hemminger [EMAIL PROTECTED] Date: Fri, 17 Aug 2007 17:49:09 -0700 Fix optimization of netdev_priv() lost by the addition of multiqueue. Only configurations that define MULITQUEUE need the extra overhead in netdevice structure and the loss of the netdev_priv optimization. Signed-off-by: Stephen Hemminger [EMAIL PROTECTED] Every distribution vendor is going to turn MULTIQUEUE on. Therefore %99 of Linux users will not see any gain from your patch. You're walking around in a dark room and hitting walls every few seconds. Take a moment and think about how to really deal with this generically for a second before churning out another patch. For example, how about making the multiqueue limit fixed, say at 64 queues, and declare the egress_queues as an array of 64 entries. Then you can get constant pointer formation for both the netdev priv and the queues. This will also basically eliminate all of the alloc_mqueue() logic, since struct netdev will now always be a fixed size. And, amazingly, this deals with the loopback issue transparently as well. :-) It would also be nice to just get rid of netdev-priv, it will prevent more misuses like the cxgb3 case, for example. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] restore netdev_priv optimization (planb)
David Miller wrote: From: Stephen Hemminger [EMAIL PROTECTED] Date: Fri, 17 Aug 2007 17:49:09 -0700 Fix optimization of netdev_priv() lost by the addition of multiqueue. Only configurations that define MULITQUEUE need the extra overhead in netdevice structure and the loss of the netdev_priv optimization. Signed-off-by: Stephen Hemminger [EMAIL PROTECTED] Every distribution vendor is going to turn MULTIQUEUE on. Therefore %99 of Linux users will not see any gain from your patch. You're walking around in a dark room and hitting walls every few seconds. Take a moment and think about how to really deal with this generically for a second before churning out another patch. For example, how about making the multiqueue limit fixed, say at 64 queues, and declare the egress_queues as an array of 64 entries. Then you can get constant pointer formation for both the netdev priv and the queues. this sounds highly optimistic (64 queues is enough for everyone?) and probably will be quickly outdated by both hardware and demand... Auke - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] restore netdev_priv optimization (planb)
On Fri, 17 Aug 2007 18:21:25 -0700 Kok, Auke [EMAIL PROTECTED] wrote: David Miller wrote: From: Stephen Hemminger [EMAIL PROTECTED] Date: Fri, 17 Aug 2007 17:49:09 -0700 Fix optimization of netdev_priv() lost by the addition of multiqueue. Only configurations that define MULITQUEUE need the extra overhead in netdevice structure and the loss of the netdev_priv optimization. Signed-off-by: Stephen Hemminger [EMAIL PROTECTED] Every distribution vendor is going to turn MULTIQUEUE on. Therefore %99 of Linux users will not see any gain from your patch. You're walking around in a dark room and hitting walls every few seconds. Take a moment and think about how to really deal with this generically for a second before churning out another patch. For example, how about making the multiqueue limit fixed, say at 64 queues, and declare the egress_queues as an array of 64 entries. Then you can get constant pointer formation for both the netdev priv and the queues. this sounds highly optimistic (64 queues is enough for everyone?) and probably will be quickly outdated by both hardware and demand... Plan C was replacing MULTIQUEUE boolean with a int value 1 ... 256. All this was a one day what if exercise, not really a big churn.. -- Stephen Hemminger [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] restore netdev_priv optimization (planb)
From: Kok, Auke [EMAIL PROTECTED] Date: Fri, 17 Aug 2007 18:21:25 -0700 this sounds highly optimistic (64 queues is enough for everyone?) and probably will be quickly outdated by both hardware and demand... As such drivers appear in the tree we can adjust the value. Even the most aggressively multi-queued virtualization and 10GB ethernet chips I am aware of, both in production and in development, do not exceed this limit. Since you think this is worth complaining about, you must know of some exceptions? :-) - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] restore netdev_priv optimization (planb)
From: Stephen Hemminger [EMAIL PROTECTED] Date: Fri, 17 Aug 2007 18:28:07 -0700 Plan C was replacing MULTIQUEUE boolean with a int value 1 ... 256. All this was a one day what if exercise, not really a big churn.. Yes, that's another reasonable approach. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] restore netdev_priv optimization (planb)
David Miller wrote: From: Kok, Auke [EMAIL PROTECTED] Date: Fri, 17 Aug 2007 18:21:25 -0700 this sounds highly optimistic (64 queues is enough for everyone?) and probably will be quickly outdated by both hardware and demand... As such drivers appear in the tree we can adjust the value. Even the most aggressively multi-queued virtualization and 10GB ethernet chips I am aware of, both in production and in development, do not exceed this limit. Since you think this is worth complaining about, you must know of some exceptions? :-) I actually don't, but I assume that demand for queues will quickly increase once the feature becomes available. e.g. in e1000 hardware (pci-e) we support 2, and this is really old hardware already (laugh), 82575 has 4. the ixgbe driver me and Ayyappan posted already supports 64 rx queues and 32 tx... I can only expect the next generation to take a bigger jump and implement at least 128 queues... :) Auke - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
