Re: netif_stop_queue() and multiple hardware queues
David Miller wrote: From: Jeremy Jackson [EMAIL PROTECTED] Date: Sun, 02 Jul 2006 22:36:03 -0400 I propose that this is an architecture/framework/API issue that the Linux core networking code should be dealing with. Proposals are worthless if nobody wants to do the work to actually implement the proposal. Your tone seems to suggest that you feel if you propose something, someone will just jump out of their seat and implement your proposal. That's not how this works :) I apologize for the tone. I *am* interested working on this. I'm just surprised nobody has jumped out of their seat to do it :) I have changed a line or two of kernel code but am by no means an expert, so I guess I was hoping for some kind of feedback about the proposal. I guess submit a patch for discussion? If it's gross I guess that would generate some feedback. What's the best Git tree to base this on? (maybe it's name means make svn and and cvs users feel like a stupid git?) In the current 2.6 network driver interface, the TX path seems to be based on mydev-hard_start_xmit(). The device driver initiates flow control with netif_stop_queue(). netif_stop_queue() and other functions (macros in netdevice.h) touch mydev-state, the bit __LINK_STATE_XOFF: netif_schedule() netif_start_queue() netif_wake_queue() netif_stop_queue() netif_queue_stopped() There is only one queue now, implied by default. Adding a hwqueue parameter to all of the above would be a start. It would be ideal if mydev-xmit_lock could be separated out, one for each hwqueue. A quick hack to implement this would be for the device to set the mydev-features NETIF_F_LLTX, and implement per-queue locking. Teaching qdiscs to use the multiple queues looks tricky in comparison, a quick hack might be to have prio's bands map 1:1 with the hardware queues. Regards, Jeremy - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: netif_stop_queue() and multiple hardware queues
I'd like to raise the question again. How can the network stack's flow control function with hardware that has multiple physical queues? How can these queues be exposed to tc? After scanning the netdev mailing list activity for the last 6 months, I can see no furthur discussion since my original post. I did notice some churn in several 802.11 layer stacks trying to deal with multiple hardware queues, the IPW2200 in particular. I propose that this is an architecture/framework/API issue that the Linux core networking code should be dealing with. There is some fragmentation already; compare the QoS *policy* implemented by Madwifi vs IPW2200 vs devicescape's 80211 stack. It's worth noting that at the 80211 layer, the first 2 listed above have a priority argument to (80211)_hard_start_xmit(). It would seem ideal to expose the queues of Ethernet layer hardware (provide mechanism) and let userspace tc tools do the classifying (sysadmin can set the policy). Jeremy Jeremy Jackson wrote: Hi, I posted this briefly on linux-net, before being redirected here. Two pieces of hardware now have Linux driver support for multiple hardware queues: Intel's e1000 (two queues from what I can see in the code) and Atheros's 5212 and up, in support of 802.11e and WME (four hardware queues). In the GigE case, this just reduces latency due to hardware queueing. In the WiFi case, the queues actually have significance in access to the shared medium. (ACKs can be disabled as well) It is also worthy of note that ADSL2, VDSL, and ADSL2+ have 4 different latency queues. These last two are significant; real-time applications suffer the most from low speed, shared, and/or non-deterministic media. I wonder where DOCSIS 2 is in this regard. Anyone? Beuler? So my question is, what's it going to take to get dev-hard_start_xmit() to hook up tc queues directly to hardware/driver queues? Right now, it seems no matter how elaborate a tc setup you have, everything funnels through one queue, where the only thing that survives from the classifying/queueing is skb- priority (ie nothing). The hardware driver can then try to reclassify packets. I suppose you could hack up an iptables classifyer to set skb-priority... The Atheros driver tries to do it's own classifying by first wiping out skb-priority, then hard-coding a mapping (tsk - policy is for the sysadmin!) between VLAN tag priority, IP TOS/DSCP, and skb-priority, then to one of the 4 queues and ACK states, blithely ignoring any fine work done by tc. It's be sweet to head this nonsense off at the pass, before others discover the rabbit trail and make it into a trade route. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: netif_stop_queue() and multiple hardware queues
From: Jeremy Jackson [EMAIL PROTECTED] Date: Sun, 02 Jul 2006 22:36:03 -0400 I propose that this is an architecture/framework/API issue that the Linux core networking code should be dealing with. Proposals are worthless if nobody wants to do the work to actually implement the proposal. Your tone seems to suggest that you feel if you propose something, someone will just jump out of their seat and implement your proposal. That's not how this works :) - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: netif_stop_queue() and multiple hardware queues
Hi Jeremy, I implemented this functionality in Devicescape's 802.11 stack. The approach I took was for the driver to install a device specific qdisc as the root qdisc on the device. This root qdisc's purpose is to expose the hardware queues directly, so other qdiscs can be attached as leaf qdiscs. This hardware specific root qdisc cannot be deleted or changed. This makes it possible to use tc to inspect/set/modify per hardware queue statistics and parameters. In order for this to work my device driver never calls netif_stop. Instead the qdisc dequeue function for the root qdisc looks to see which hardware queues can accept a frame, and if none then it returns no data. The driver's frame completion function calls __netif_schedule appropriately too to ensure the queue runs when it should. This allows Devicescape's 802.11 stack to properly integrate with the Linux tc framework. I don't think any other 802.11 drivers achieve this. In the future I plan to extend Devicescape's 802.11 root qdisc to further expose the 802.11 MAC's internal queues, in cases where this is useful (e.g. the scheduled access implementation). The same principle could apply to Intel's e1000. Simon -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeremy Jackson Sent: Wednesday, December 14, 2005 2:31 PM To: netdev@vger.kernel.org Subject: netif_stop_queue() and multiple hardware queues Hi, I posted this briefly on linux-net, before being redirected here. Two pieces of hardware now have Linux driver support for multiple hardware queues: Intel's e1000 (two queues from what I can see in the code) and Atheros's 5212 and up, in support of 802.11e and WME (four hardware queues). In the GigE case, this just reduces latency due to hardware queueing. In the WiFi case, the queues actually have significance in access to the shared medium. (ACKs can be disabled as well) It is also worthy of note that ADSL2, VDSL, and ADSL2+ have 4 different latency queues. These last two are significant; real-time applications suffer the most from low speed, shared, and/or non-deterministic media. I wonder where DOCSIS 2 is in this regard. Anyone? Beuler? So my question is, what's it going to take to get dev-hard_start_xmit() to hook up tc queues directly to hardware/driver queues? Right now, it seems no matter how elaborate a tc setup you have, everything funnels through one queue, where the only thing that survives from the classifying/queueing is skb- priority (ie nothing). The hardware driver can then try to reclassify packets. I suppose you could hack up an iptables classifyer to set skb-priority... The Atheros driver tries to do it's own classifying by first wiping out skb-priority, then hard-coding a mapping (tsk - policy is for the sysadmin!) between VLAN tag priority, IP TOS/DSCP, and skb-priority, then to one of the 4 queues and ACK states, blithely ignoring any fine work done by tc. It's be sweet to head this nonsense off at the pass, before others discover the rabbit trail and make it into a trade route. -- Jeremy Jackson Coplanar Networks W:(519)489-4903 MSN: [EMAIL PROTECTED] ICQ: 43937409 http://www.coplanar.net - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: netif_stop_queue() and multiple hardware queues
Oh - and re: policy - my 802.11 qdisc first calls out to the tc classify function - allowing the sysadmin to do what he wants, then if no class is selected it has a default implementation that reflects the appropriate 802.11 and WiFi specs for classification. Of course another implementation would be to implement an 802.11 classifier, and install this by default on the 802.11 qdisc. Simon -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Simon Barber Sent: Wednesday, December 14, 2005 3:07 PM To: Jeremy Jackson; netdev@vger.kernel.org Subject: RE: netif_stop_queue() and multiple hardware queues Hi Jeremy, I implemented this functionality in Devicescape's 802.11 stack. The approach I took was for the driver to install a device specific qdisc as the root qdisc on the device. This root qdisc's purpose is to expose the hardware queues directly, so other qdiscs can be attached as leaf qdiscs. This hardware specific root qdisc cannot be deleted or changed. This makes it possible to use tc to inspect/set/modify per hardware queue statistics and parameters. In order for this to work my device driver never calls netif_stop. Instead the qdisc dequeue function for the root qdisc looks to see which hardware queues can accept a frame, and if none then it returns no data. The driver's frame completion function calls __netif_schedule appropriately too to ensure the queue runs when it should. This allows Devicescape's 802.11 stack to properly integrate with the Linux tc framework. I don't think any other 802.11 drivers achieve this. In the future I plan to extend Devicescape's 802.11 root qdisc to further expose the 802.11 MAC's internal queues, in cases where this is useful (e.g. the scheduled access implementation). The same principle could apply to Intel's e1000. Simon -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeremy Jackson Sent: Wednesday, December 14, 2005 2:31 PM To: netdev@vger.kernel.org Subject: netif_stop_queue() and multiple hardware queues Hi, I posted this briefly on linux-net, before being redirected here. Two pieces of hardware now have Linux driver support for multiple hardware queues: Intel's e1000 (two queues from what I can see in the code) and Atheros's 5212 and up, in support of 802.11e and WME (four hardware queues). In the GigE case, this just reduces latency due to hardware queueing. In the WiFi case, the queues actually have significance in access to the shared medium. (ACKs can be disabled as well) It is also worthy of note that ADSL2, VDSL, and ADSL2+ have 4 different latency queues. These last two are significant; real-time applications suffer the most from low speed, shared, and/or non-deterministic media. I wonder where DOCSIS 2 is in this regard. Anyone? Beuler? So my question is, what's it going to take to get dev-hard_start_xmit() to hook up tc queues directly to hardware/driver queues? Right now, it seems no matter how elaborate a tc setup you have, everything funnels through one queue, where the only thing that survives from the classifying/queueing is skb- priority (ie nothing). The hardware driver can then try to reclassify packets. I suppose you could hack up an iptables classifyer to set skb-priority... The Atheros driver tries to do it's own classifying by first wiping out skb-priority, then hard-coding a mapping (tsk - policy is for the sysadmin!) between VLAN tag priority, IP TOS/DSCP, and skb-priority, then to one of the 4 queues and ACK states, blithely ignoring any fine work done by tc. It's be sweet to head this nonsense off at the pass, before others discover the rabbit trail and make it into a trade route. -- Jeremy Jackson Coplanar Networks W:(519)489-4903 MSN: [EMAIL PROTECTED] ICQ: 43937409 http://www.coplanar.net - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
