Good traffic discipline requires fine-grained control, which means getting
as close to the wire as possible. Otherwise, lots of buffering tends to
get in the way. For ALTQ, the external interface is the best place to set
up on, hence the recommendations for queueing on the ethernet interface
instead of tun0.
Queue tagging must happen on tun0, since IP is only visible there. Once
past tun0, it gets wrapped in PPPoE headers, so pf can't decode it on the
ethernet interface.
Unfortunately, the only option on OpenBSD appears to be userland ppp with
pppoe. As tun0 is an interface to userland, the kernel packet tags get
lost as soon as the packet gets pulled off by ppp. ppp feeds it to pppoe,
which wraps it and puts it back on the ethernet interface. Modifications
to get the tags through userland would probably break more things than
they would fix. So, as things stand now, queueing on the external
interface is impossible, since the queue tags can't get there.
Kernel PPPoE would be much better.
That leaves queueing on tun0, which seems to be troublesome. As Primož Gabrijelčič commented:
If I activate this with 4 download clients (totaling approx 40 KB/s) and 4
upload clients (totaling approx 18 KB/s) - IOW in unsaturated conditions -
the pppoe %CPU rockets skyhigh. Minimal value is 23% but I have observed
values as high as 75%. If pppoe CPU percentage rises too high (don't know
exact figures here), the download and upload speed starts to drop.
I did some playing around and discovered something. It seems that someone
forgot to fully ALTQify tun0. Specifically, select() always returns read-
ready if there's any data in the internal queue, whether ALTQ's discipline
is ready to release it or not.
The theory is that when the queues start filling, ppp's non-blocking I/O
loop starts spinning on tun0, trying to pull packets off when they aren't
ready. Once it starts spinning, pppoe will be adversely affected and
lose throughput.
As I don't have a PPPoE setup to work with, I did my own testing with just
tun0, and saw the spin effect. Below is a patch for if_tun.c, which fixed
the problem I observed. I'd like to know if it fixes pppoe queueing for
anyone brave enough to try patches from me. I've been known to make things
explode.
--- sys/net/if_tun.c Tue Jan 7 01:00:34 2003
+++ sys/net/if_tun.c Fri Jun 6 23:03:25 2003
@@ -433,6 +433,7 @@
int unit, s;
struct tun_softc *tp;
struct tuninfo *tunp;
+ struct mbuf *m; if ((unit = minor(dev)) >= ntun)
return (ENXIO);
@@ -494,8 +495,9 @@
tp->tun_flags &= ~TUN_ASYNC;
break;
case FIONREAD:
- if (tp->tun_if.if_snd.ifq_head)
- *(int *)data = tp->tun_if.if_snd.ifq_head->m_pkthdr.len;
+ IFQ_POLL(&tp->tun_if.if_snd, m);
+ if (m)
+ *(int *)data = m->m_pkthdr.len;
else
*(int *)data = 0;
break;
@@ -731,6 +733,7 @@
int unit, s;
struct tun_softc *tp;
struct ifnet *ifp;
+ struct mbuf *m; if ((unit = minor(dev)) >= ntun)
return (ENXIO);
@@ -742,7 +745,8 @@ switch (rw) {
case FREAD:
- if (ifp->if_snd.ifq_len > 0) {
+ IFQ_POLL(&ifp->if_snd, m);
+ if (m != NULL) {
splx(s);
TUNDEBUG(("%s: tunselect q=%d\n", ifp->if_xname,
ifp->if_snd.ifq_len));