David Burgess wrote: > On Fri, Jul 30, 2010 at 9:24 AM, Darryl Miles > <[email protected]> wrote: > >> I don't understand the relevance of your citation as an argument against >> anything I've said. What is your understanding of the mechanism I described >> and what is your understanding of section (9.2.2) in relation to that ? > > After reading the wiki page and re-reading section 9.2.2 I don't see > anything to suggest that the tbf is intentionally delaying packets. In > fact, my understanding (and experience) is that by using it to shorten > your egress queue you will actually see reduced latency and improved > interactivity on a congested interface.
Okay following the same steps: Wiki: "If fewer than n tokens are available, no tokens are removed from the bucket, and the packet is considered to be non-conformant." Wiki (discussing what to do with non-conformant packets): "They may be enqueued for subsequent transmission when sufficient tokens have accumulated in the bucket." Wiki (going back up top): "A token is added to the bucket every 1 / r seconds." So (staying with the wiki page) for arguments sake "r" maybe 102400 (meaning 102400 tokens per second). Tokens directly correlate to 1:1 for "tokens" to "bytes of network data" (again for argument sake, as per wiki info). So this means that if enqueuing takes place then after you have exhausted your burst volume, your network throughput drops to 102400 bytes/second. Note that tokens are only charged (subtracted from the pool in the bucket) when a packets leaves the outbound interface queue to be committed to the wire. Nothing is charged for being sat in an interface queue, the point of the queue is to hold the packets until they can be sent. This is the principal of what TBF is doing in the kernel. > While tbf is constantly filling a bucket of tokens, the tokens > themselves are not packets. A packet leaving the interface in question > is matched to a token, if available, then queued for transmission. At > this point the packet is out of the control of tbf. Although you can > create a long (and therefore laggy) queue through tbf, any latency > introduced as a result is not the express purpose of the qdisc. At > least that's my undertanding. A token is an accounting unit (re "tokens are not packets"). 1 token relates to some amount of data. In the case of the wiki article the example is 1 token means 1 byte of network data. Usually the unit of network data is raw data that includes IP header overhead and all OSI layers above in the packet. It may not include the whole ethernet frame header since that is not added until the last moment during committal to the wire. It doesn't quite work as you describe. The "packet leaving the interface" is the last thing to happen (I read this to mean, packets end up on the wire), but you talk about it first... then talk about token matching. You don't seem to talk in terms that inside the kernel (when TC is enabled) multiple queues exist. Sure there is a "network device driver outbound queue" (one per interface) and anything that makes it into this queue will be shoved onto the wire ASAP. It is the job of the network interface's device driver to do that all day long (shove packets from this queue on the wire ASAP). But you fail to talk in terms that other packet queues that exist inside TC that are logically separate to each device driver outbound packet queue (but are still associated with an interface). Everytime the LARTC documentation uses the term QDISC it is talking about this logical queues concept. The txqueue that you see in ifconfig, you can think of this as being a logical queue that is separate from the "network device driver outbound queue". The "network device driver outbound queue" is largely unseen by tooling and is more a device driver private queue to get work done. QDISC can be a simple single extra queue or in more complex scenarios can be a hierarchical queue structure (parent/children), the hierarchy exists to allow a single packet to be accounted for by multiple nodes in the queue structure and also allow for complex accounting situations to lend-out/borrow-from tokens with neighboring ancestry. These complex scenarios are not necessary for TBF support. In the case of TBF think of it as an extra queue inside the TBF itself, that only exists when TBF is enabled. This is also the same queue that the Wiki page references in my quote above (when it queues to hold it for later transmission). So as you can understand from this the packet(s) remain in control of the TBF and until the necessary tokens can be charged (subtracted from the bucket) it will not be allowed to pass. When it is allowed to pass, the TBF removes the packet from the QDISC queue and enqueues the packet into the "network device driver outbound queue". Until then the device drive can not see the packet. Another point I should make is that a device driver queue is actually quite small (relative to the maximum throughput) so for 100Mbit ethernet in my implementations of ethernet this queue is just big enough for 1 or 2 packets (one on wire being sent, and next one transfered to card to send next). But you must think of all this a device driver specific and as your comments indicate, once motions start it is largely outside of the control of the Linux router. But this packet queue is 1 or 2 packets deep and by the time they get into it they have passed though the TBF queue. The next aspect of TBF to think about, is what to do when the QDISC of delayed packets contains 2 or more packets. One major concept of QDISC is _WHICH_ packet do I pick next from the queue and _WHEN_ do I send it down to the device driver. Just because the device driver says it is empty and needs feeding with another packet, doesn't mean I have to give it the next packet right away. This is how TBF inserts a delay, since it knows how long to wait before it can send the next packet. Part of the concepts in LARTC is that you don't have to work on the First-In-First-Out basis. A network interface outbound queue by default works on FIFO basis. Well this is where such terms as SFQ and RED and such comes in. You can stick with FIFO but usually when there is some contention instead of hitting the hard wall of the number of packets/bytes in the queue limit, you usually want to do something pro-active to prevent that. SFQ might be better for access/edge routers. It is easier to be fair when at any one time there might only be a handful of connections actively sending packets. RED might be better for transit/core routers. It is hard to be fair when your packet-per-second rate and number of distinct connections passing through is so high. What you really need to do is try to get some of those connections to slowdown. There are many others implementations. I have somewhat simplified / skipped over some details here for the purpose of getting across the concept in a way that should have parity with practical experience. > I used tbf as my WAN parent qdisc for some time with great > satisfaction, until an apparent incompatibility arose between it and > libtorrent, whereby torrents would slow to a crawl when tbf was > active. Configuration problem. Something better for the LARTC list to look at. You did not state your goals nor the incompatability between your TC policy and what libtorrent is doing. High throughput, high connections active, high connection turnover, etc... There is no reason why a user should notice any performance change, unless you have configured the policy to enforce some rule. Darryl _______________________________________________ Soekris-tech mailing list [email protected] http://lists.soekris.com/mailman/listinfo/soekris-tech
