Re: [Cake] Per-host fairness

[moeller0]

Hi Georgios,


> On Oct 18, 2016, at 03:33 , Georgios Amanakis  wrote:
> 
> I tried with "besteffort dual-dsthost nat" on ifb ingress and
> “besteffort dual-srchost nat" on WAN egress,

Thanks.

> and when doing simple
> things (i.e. downloading from a couple of websites) the bandwidth is
> divided fair per-host, i.e 1:1 between A and B.

Okay, does this stay fair if one of the hosts open a bunch of websites 
at the same time? Does the download test also give per-host fairness if the 
number of downloads is massively imbalanced between the hosts?


> However, when one of
> the hosts is using bittorrent, he also gets most of the bandwidth, i.e.
> if total ingress bandwidth is 3300kbit, A is using bittorrent and gets
> 2900kbit, B is downloading from a single website and gets 400kbit.

Bit-torrent is a hard problem it seems, especially the cobalts branch 
that you test is supposed to better deal with it though so this is abit of a 
puzzle. How long are your tests? Are you looking only at a few seconds or 
something like >30 seconds (maybe the shaper needs time to reach equilibrium)? 
What happens in the upload during theses tests? I also wonder how many new 
torrent connections are established during the test or are the torrent flows 
long lived?

I probably would try to instantiate cake not on the wan but on a lan 
interface of the router and re-do the test without the “nat” option just to 
reduce the number of tested parts. If you should do this, please remember that 
ingress and egress are per interface, so downloads from the internet use the 
LAN-egress while internet-uploads use LAN-ingress, so make sure to switch the 
bandwidth definitions for your two cake instances accordingly. 
Finally could you post the results of “tc -s qdisc” before, during and 
after your bit-torrent tests for the test you just reported and the LAN test in 
case you ago that route.


Best Regards
Sebastian

> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] diffserv based on firewall mark

[moeller0]

Hi Ching,

> On Oct 12, 2016, at 14:40 , ching lu <lschin...@gmail.com> wrote:
> 
> There is no need for cleansing dscp for wan ingress, I think it is 
> unnecessary, too
> 
> In https://www.bufferbloat.net/projects/codel/wiki/Cake/
> 
> There is a statement:
> 
> “The only way we know how to “fix” bittorrent is to classify it somewhat, 
> somehow, as “background”."

Which was true at the time. In the mean time cake grew new “isolation” 
modes that will attempt to provide on a first level per-(internal)-IP-address 
fairness and iside each internal IP-address “band” also per-flow fairness. This 
should allow to restrict the bad effect of bit-torrent traffic on the machine 
actually running the torrent client. Which seems like a goos compromise since 
most torrent clients allow configurations to alleviate the issue somewhat for 
that specific machine (like bandwidth limits). These additional modes do 
require a bit of testing and especially on ingress they will not be 100% robust 
(too many in-rushing bit-torrent connections might cause buffers upstream of 
the cake-managed link to fill and cause increased latency), but that just comes 
with instantiating a shaper on the wrong end of the real bottleneck. As a 
sidenote the more bandwidth difference exist between real bottleneck and the 
artifical cake-managed bottleneck the better ingress shaping will work…


> 
> But in fact, there is no simply way to classify bittorrent INGRESS traffic

Yes, and no…

> 
> DSCP -> unreliable, easily spoofed by attacker, and the value is most likely 
> 0x0\

Well, if BT clients would mark CS1/BK that would be a decent 1st step, 
except that will also tell ISPs which packets to drop first… (which might be 
actually in the users interest)

> firewall mark -> cake do not use firewall mark/connmark

If you can firewall mark you can also re-map dscp… But I believe the 
real issue is that bit-torrent was designed to have no clear unambiguous 
signature so figuring out which packets belong to bit-torrent flows is the 
tricky bit…

> 
> Finally, I guess most likely home users will use bit torrent.

But that is a guess? Numbers/real data would be better; that said with 
even windows update allowing peer-to-peer distribution of updates 
bit-torrent-like traffic probably is something most home-users will see 
occasionally.

Best Regards
Sebastian


> 
> 2016年10月12日 下午8:04，"moeller0" <moell...@gmx.de>寫道：
> Hi Ching?
> 
> > On Oct 12, 2016, at 12:17 , ching lu <lschin...@gmail.com> wrote:
> >
> >
> > 2016年10月12日 下午6:05，"moeller0" <moell...@gmx.de>寫道：
> > >
> > > Hi Ching,
> > >
> > > > On Oct 12, 2016, at 11:35 , ching lu <lschin...@gmail.com> wrote:
> > > >
> > > > How to archive "cake follows iptables"? is it “wan ingress -> iptables
> > >
> > > Yes.
> > >
> > > > -> wifi egress/LAN egress -> ifb egress -> cake”?
> > >
> > > Except that if you instantiate cake on the interface connecting 
> > > to the outers LAN/WLAN side (lets call this LAN for short), cake will 
> > > reside on that interfaces egress and hence you require no ifb for traffic 
> > > coming in from the internet (as a plus cake will even without the fancy 
> > > new deNAT options see the full intrnal IP addresses, useful for dual and 
> > > triple isolation options). In the direction facing the internet you can 
> > > instantiate cake on an ifb interface for LAN and then put the iptables 
> > > DSCP cleaner on the WAN egress side (and the WAN ingress side, unless you 
> > > trust your ISP to deliver reasonable DSCP values, which should be like 
> > > never*)
> >
> > The bandwidth shaper won’t work correctly if cake(s) are registered on 
> > multiple LAN interface, ifb is necessary
> >
> > e.g. if ingress bandwidth limit is 100M, then setting 50M on wifi, and 50M 
> > on LAN ?
> 
> Yes that seems true, if you instantiate cake on br-lan (which I 
> believe would be the relevant interface) you will shape both wireless and 
> wired traffic, but most likely also internal traffic… But that can be solved 
> by one more router/AP ;)
> 
> >
> > I think the diffserv support of cake model is not suitable for home network 
> > currently.
> 
> I have no real opinion on that, but could you explicitly state what 
> short coming you see that is a showstopper? DSCP cleaning on ingress is BTW 
> not really required to happen before cake, as long as cake is set to 
> besteffort it will ignore DSCP markings anyway, and if you want to 
> re-map/re-classify pa

Re: [Cake] diffserv based on firewall mark

[moeller0]

Hi Ching?

> On Oct 12, 2016, at 12:17 , ching lu <lschin...@gmail.com> wrote:
> 
> 
> 2016年10月12日 下午6:05，"moeller0" <moell...@gmx.de>寫道：
> >
> > Hi Ching,
> >
> > > On Oct 12, 2016, at 11:35 , ching lu <lschin...@gmail.com> wrote:
> > >
> > > How to archive "cake follows iptables"? is it “wan ingress -> iptables
> >
> > Yes.
> >
> > > -> wifi egress/LAN egress -> ifb egress -> cake”?
> >
> > Except that if you instantiate cake on the interface connecting to 
> > the outers LAN/WLAN side (lets call this LAN for short), cake will reside 
> > on that interfaces egress and hence you require no ifb for traffic coming 
> > in from the internet (as a plus cake will even without the fancy new deNAT 
> > options see the full intrnal IP addresses, useful for dual and triple 
> > isolation options). In the direction facing the internet you can 
> > instantiate cake on an ifb interface for LAN and then put the iptables DSCP 
> > cleaner on the WAN egress side (and the WAN ingress side, unless you trust 
> > your ISP to deliver reasonable DSCP values, which should be like never*)
> 
> The bandwidth shaper won’t work correctly if cake(s) are registered on 
> multiple LAN interface, ifb is necessary
> 
> e.g. if ingress bandwidth limit is 100M, then setting 50M on wifi, and 50M on 
> LAN ?

Yes that seems true, if you instantiate cake on br-lan (which I believe 
would be the relevant interface) you will shape both wireless and wired 
traffic, but most likely also internal traffic… But that can be solved by one 
more router/AP ;) 

> 
> I think the diffserv support of cake model is not suitable for home network 
> currently.

I have no real opinion on that, but could you explicitly state what 
short coming you see that is a showstopper? DSCP cleaning on ingress is BTW not 
really required to happen before cake, as long as cake is set to besteffort it 
will ignore DSCP markings anyway, and if you want to re-map/re-classify packets 
vie DSCP on ingress you are pretty much out of scope for a typical home 
network. Cleaning up on egress, as to not leak internal configuration to the 
upstream seems indeed sub-optimal, but cake is not alone in that regard…

> The setup is much more complex

Well, DSCP setup is complex no matter how you slice and dice it… but 
maybe you have an idea what a shaper (like cake) could/should do to make this 
simpler?

Best Regards
Sebastian

> 
> 
> 
> >
> > Best Regards
> > Sebastian
> >
> > 8) DSCP are only ever guranteed to be meaninful inside a dscp domain, and 
> > in reality your home net is a different domain from the ISP’s. It would 
> > have been nice if the DSCP field would have been separeted into 2 3bit 
> > fields, the first for the actual sender to request one of 8 differential 
> > classes and the other 3bits for the current domain to store its actually 
> > used DSCP bits. I claim the 3 bits should be enough for anybody  ;)
> >
> >
> > >
> > >
> > > On Wed, Oct 12, 2016 at 5:10 PM, moeller0 <moell...@gmx.de> wrote:
> > >> Hi,
> > >>
> > >>
> > >>> On Oct 12, 2016, at 10:11 , ching lu <lschin...@gmail.com> wrote:
> > >>>
> > >>> For egress, setting DSCP field should work.
> > >>>
> > >>> iptables -> wan egress -> cake
> > >>>
> > >>> But is it possible to set DSCP to 0x0 after cake's classification? i
> > >>> do not know how ISP handle non-zero DSCP, there seems to be no
> > >>> standard for this.
> > >>
> > >>Interestingly cake, at some point in the past offered exactly 
> > >> that functionality, but it got removed due to added complexity with very 
> > >> little practical applicability (and a potential layering violation, but 
> > >> one could equally argue that the current layering is partly 
> > >> sub-optimal/wrong and hence violating it to better reflect reality might 
> > >> be acceptable). But current cake does not offer this. If you are willing 
> > >> to daisy-chain two routers, you could run cake on the respective egress 
> > >> interfaces connecting both routers, and do the DSCP cleaning on the 
> > >> outer router’s egress interface toward the internet…
> > >>
> > >>>
> > >>>
> > >>> For ingress, DSCP field may not be set by network peer at all, and i
> > >>> have multiple LAN interfaces
> > >

Re: [Cake] Master branch updated

[moeller0]

Hi Jonathan,

> On Oct 4, 2016, at 13:18 , Jonathan Morton <chromati...@gmail.com> wrote:
> 
> 
>> On 4 Oct, 2016, at 11:46, moeller0 <moell...@gmx.de> wrote:
>> 
>> About that PTM accounting, could you explain why you want to perform the 
>> adjustment as a a “virtual” size increase per packet instead of a “virtual” 
>> rate reduction?
> 
> The shaper works by calculating the time occupied by each packet on the wire, 
> and advancing a virtual clock in step with a continuous stream of packets.
> 
> The time occupation, in turn, is calculated as the number of bytes which 
> appear on the wire divided by the number of bytes that wire can pass per 
> second.  As an optimisation, the division is turned into a multiplication by 
> the reciprocal.

Okay, but that seems not really relevant to the topic at hand as in PTM 
systems the effective payload-rate is 64/65 of the raw bit rate. The 65th byte 
is independent of the actual packet size sent so theoretically better modeled 
as a rate reduction than as a size increase, even though in essence for a 
shaper you can account for it either way.

> 
> I’m quite keen to keep the “bytes per second” purely derived from the raw 
> bitrate of the link, because that is the value widely advertised by ISPs and 
> network equipment manufacturers everywhere.  Hence, overhead compensation is 
> implemented purely by increasing the accounted size of the packets.

Sorry that does not make much sense, I realize that mathematically they 
are interchangeable, but that does not make them the same IMHO. Per packet 
overhead needs to be accounted on a per-packet basis you have no other real 
option (unless you work with a fixed packet length), but generic rate 
reductions do not need to be recomputed for each packet.


> 
> I have been careful here to calculate ceil(len * 65/64) here, so that the 
> overhead is never underestimated.

Which is Jonathanese for might be overestimated, so you at least agree 
with my point about the precision of the accounting being relevant. As I 
proposed in on of my comments “floor(shaper_rate * 64/64)” has the same 
property of being conservative, only with a lower possible error.

>  For example, a 1500-byte IP packet becomes 1519 with bridged PTM or 1527 
> with PPPoE over PTM, before the PTM calculation itself.  These both round up 
> to 1536 before division, so 24 more bytes will be added in both cases.

That is not one of the arguments I have made, but thanks for pointing 
that out.

> 
> This is less than 2 bits more than actually required (on average), so wastes 
> less than 1/6200 of the bandwidth when full-sized packets dominate the link 
> (as is the usual case).  Users are unlikely to notice this in practice.

Erm, VoIP packets are not close to full MTU so I am not sure whether 
“as is the usual case” is very convincing. Actually your tendency to always 
“wing it” instead of doing research as shown when you claimed 64/65 bit 
encoding for PTM instead of looking into the relevant standards (which I had to 
cite twice to make you at least fix that misconception) does not not fill me 
with confidence about those parts of cake where I do have zero expertise.

> 
> Next to all the other stuff Cake does for each packet, the overhead 
> compensation is extremely quick.  And, although the code looks very similar, 
> the PTM compensation is faster than the ATM compensation, because the factor 
> involved is a power of two (which GCC is very good at optimising into shifts 
> and masks).  This is fortunate, since PTM is typically used on 
> higher-bandwidth links than ATM.

I venture a guess that I have forgotten more about ATM/PTM ADSL/VDSL 
than you ever bothered to read up on, so why do you keep telling me these 
observations? If the goal is to annoy me, then mission accomplished.

> 
> Now, if you can show me that the above is in fact incorrect - that 
> significant bandwidth is wasted on some real traffic profile, or that 
> cake_overhead() figures highly in a CPU profile on real hardware - then I 
> will reconsider.

And that is great fun, the guy (you) that most often argues from first 
principle instead of from real world data, requests actual data in one of the 
cases where first principle seems quite applicable: when an operation can be 
(almost) completely avoided.

But I guess we just keep it as in the past; you keep not fully grasping 
the intricacies of different XDSL/DOCSIS encodings and I keep ridiculing you 
for the demonstrated lack of love to detail in these matters. 
In the past I sometimes wondered whether I did anything to offend you 
by voicing my concerns in too brash or impolite way, but now I simply assume 
that you (like most of us) simply do not react well to criticism (even if 
justified) and p

Re: [Cake] de-natting & host fairness

[moeller0]

Hi Jonathan,

> On Sep 26, 2016, at 16:30 , Jonathan Morton <chromati...@gmail.com> wrote:
> 
> 
>> On 26 Sep, 2016, at 16:28, moeller0 <moell...@gmx.de> wrote:
>> 
>> Does that mean an initial packet(s) for a flow will be “misclassified” (not 
>> really since there should be no record yet to snatch the translated IP from) 
>> do all those initially non-classified packets end up in the same bin?
> 
> The initial packet will normally be outgoing, so it’ll go through conntrack 
> before reaching the qdisc.  If it’s incoming, then it’ll be “related to” an 
> existing connection or else won’t be natted - though I’m not sure whether 
> “related” connections pre-emptively get conntrack entries before traffic has 
> been seen.  If not, that initial packet will be associated with the NAT box 
> by the qdisc, rather than the internal host, while subsequent packets will 
> correctly be associated with the internal host.

Okay, so the worst case is “regression” to simple flow fairness and 
only for initial packets. What about port redirects, will these be already 
included into the conntrack and what about UDP (I believe bit-torrent uses UDP 
so I believe this to be relevant).

> 
> That assumes we have qdiscs attached to the egress and ingress sides of a 
> WAN-facing interface, as normally desired.
> 
> The code looks sane at first glance, so I’ll give it a try at my end.  With 
> any luck, I’ll be able to improve triple-isolate’s performance enough to make 
> that the default, too.  I should probably use a different data structure than 
> a ring buffer, so that there is less in the way of linear searching for an 
> unblocked flow.
> 
> The current default is “flows”, which doesn’t need NAT information to 
> unambiguously distinguish flows from each other.  However, “hosts” mode does 
> need it when running in a NAT environment, otherwise internal hosts will 
> erroneously be lumped together with the NAT box.  Triple-isolate is 
> effectively a combination of “hosts” and “flows” - that is probably the 
> easiest way to understand it.

But the dual-[src\dst]host options are similar to triple-isolate in 
that regard except they also need directionality information which is hard to 
divine, so I fully agree that riple is the best candidate for a default. 
Assuming that it actually works…

> 
> I think it is reasonable to turn on conntrack lookups by default whenever 
> host information is relevant.  This is potentially true for all modes except 
> “flowblind” and “flows”.
> 
> Also long overdue are the more subtle overhead compensation factor for PTM, 
> and the two extra keywords for DOCSIS’ asymmetric overhead.

Teacher, teacher, ask me: the term you are looking for is “proper 
documentation”. 
About PTM what are you thinking about, the 64/64 encapsulation “tax”? 
About DOCSIS, while we have Greg White going on record for cable labs, would it 
not be wiser to survey more cable ISPs to check first whether everybody 
actually follows those recommendations before creating keywords? One thing I 
have learned about overhead compensation is, it never is as simple and nice as 
in real life as RFCs make you hope. I would love to be wrong on DOCSIS, but I 
believe the hypothesis should be someone set it up different. So in essence pan 
for a number of keywords for DOCSIS and name them in a way that allows future 
additions that do not sound awkward.

Best Regards
Sebastian

> 
> - Jonathan Morton
> 

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] de-natting & host fairness

[moeller0]

Hi Kevin,

this is like the missing puzzle piece, if you solved this, most home users 
might end up deep in your debt (without them realizing it of course).
Question, if I enable this on my link how will it deal with the typical 
differences between IPv4 and IPv6? I believe that the situation I have at home, 
NAT for IPv4 but no NAT for IPv6 (or if NAT, at least NAT with identifying last 
64 bits of the IPv6 addresses, no port remapping games) is quite common now a 
days. I assume it will do the right thing for IPv4 but will it still do the 
right thing for IPv6 flows as well? And what if for $DEITY’s sake someone would 
insist on using a port-remapping NAT on IPv6?
If, what I assume it will do the right thing by default, I would vote for 
enabling this by default and introduce keywords to disable this if required (in 
what I assume to be one of cake’s main ideas use reasonable defaults that in 
general do the right thing, but also allow crazy stuff if need be).
Do you have any idea how expensive this is computationally? I realize that this 
is a tad hard to measure as cake will not simply reduce the available bandwidth 
when running out of CPU cycles but first will allow the latency to increase.

Best Regards
Sebastian

> On Sep 26, 2016, at 05:20 , Kevin Darbyshire-Bryant 
>  wrote:
> 
> Greetings!
> 
> A while back I started on a quest to make cake 'nat' aware as the lack of 
> host fairness in a typical home router environment was the only thing that 
> prevented cake from being the ultimate qdisc in my opinion.  This involves 
> dealing with conntrack which on egress is easy (the kernel fills in a data 
> structure for us), ingress is less clear.  I hacked something together but 
> wasn't really happy with it.
> 
> Another github user 'tegularius' presented some beautifully crafted code that 
> did the lookups in a much neater way.  Originally it too had an 'ingress' 
> lookup problem.  This was worked on and I hacked some conditional 'denat' 
> options into cake & tc.
> 
> For your 'delight' a denat cake 
> https://github.com/kdarbyshirebryant/sch_cake/tree/natoptions along with a 
> matching tc https://github.com/kdarbyshirebryant/tc-adv/tree/denat
> 
> Typically I use 'dual-srchost srcnat' options on the egress interface, with 
> 'dual-dsthost dstnat' in the ingress ifb interface.  In *brief* testing, 
> bandwidth is shared fairly between hosts, and fairly by flow within each 
> host.  And it's not crashed yet.
> 
> Kevin
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Configuring cake for VDSL2 bridged connection

[moeller0]

Hi techicist,

> On Aug 26, 2016, at 13:15 , techic...@gmail.com wrote:
> 
> Is flowblind likely to give better performance?

That depends on your definition of better, I guess. Typically flow-fair 
queuing seems to be what most people prefer (unless an application either does 
not respond to AQM signals or open an excessive amount of individual flows 
flow-fair queueing effectively treats most traffic sources equal, pretty much 
what people seem to want, add to this a bit of classification to exempt e.g. 
VOIP traffic from only getting its flow-fair share of the bandwidth and the 
whole thing also works reasonably well with slow links). People suffering from 
unruly applications (like mis-configured? bit-torrent clients or recently 
windows update) often ask for per-application fairness, but that is not 
something a router will ever be able to deliver in my opinion; the closest we 
get to this would be fairnes by internal or external end-IP addresses. Luckily 
cake offers just these modes “dsthost”, “srchost” and even better offers a 
combination modes that will on a first level attempt per host-IP fairness and 
within each host IP also per-flow fairness (“dual-srchost” and “dual-dsthost”, 
and even “triple-isolate” which systematically might be better called 
“dual-srchost-dsthost” since it offers fist level fairness based on an 
under-documented mix of src and dst addresses, but I digress). Please note that 
on a typical homerouter, due to NAT, all the IP addressed based fairness modes 
will not work for IPv4 on the wan interface, IPv6 traffic should be fine, but 
IPv4 basically degrades into a computationally more intensive version of 
flow-fairness (as after NAT cake only sees the routers external IP for all 
internal hosts). This might have been more than you wanted to know…

Best Regards
Sebastian

> 
> netperfrunner looks very useful. Thank you for that.
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Configuring cake for VDSL2 bridged connection

[moeller0]

Hi techicist,


> On Aug 23, 2016, at 15:44 , techic...@gmail.com wrote:
> 
> I am using a TalkTalk (UK) VDSL2 connection via bridged PTM to my TP-LINK 
> Archer C7 V2. I am running LEDE.
> 
> TalkTalk uses DHCP to obtain an IP address and not DHCP as most other ISPs do.

I take it that one of the DHCPs should read PPPoE?

> 
> I am trying to configure cake and I see these options on bufferbloat.net:
> 
> There are eight new keywords which deal with the basic ADSL configurations. 
> These switch on ATM cell-framing compensation, and set the overhead based on 
> the raw IP packet as a baseline.
> 
> ipoa-vcmux (8)
> ipoa-llcsnap (16)
> bridged-vcmux (24)
> bridged-llcsnap (32)
> pppoa-vcmux (10)
> pppoa-llc (14)
> pppoe-vcmux (32)
> pppoe-llcsnap (40)bvn
> 
> How do I go about using these with OpenWRT?

(C) None of the above ;)

Really, all f the above keywords only deal with encapsulations used on ATM 
links, not PTM links. All of these will automatically enable ATM cell 
accounting and hence will not do the right thing on PTM links even if the 
per-packet overhead should be correct. cake offers two PTM specific keywords, 
pppoe-ptm and pppoe-bridged that translate into 27 and 19 bytes respectively. 
It looks like they are not well enough documented though*. I would recommend to 
simply ignoring these keywords wholesale and use the explicit “overhead 27” 
statement instead, but we are getting ahead of ourselves here:

The first question is what is the true bottleneck link and what bandwidth and 
encapsulation is in use on that link. Often the VDSL2-link is the true 
bottleneck, but some ISPS like DTAG in Germany actually implement a 
shaper/policer at the BRAS/BNG level with lower thresholds than the vdsl2 link 
itself. Be that is it may, the first issue is figuring out the relevant 
bottleneck link bandwidth (we just assume that your ISP has no shaper in use):

Look into the status page of your VDSL2-modem and write down the values of the 
actual synchronization bandwidth for uplink and downlink.

Multiply both with 64/65 = 0.984615384615 as VDSL2 uses a continuous 64/65 
encapsulation that will eat roughly 1.6% of the sync bandwidth; this now are 
your reference values for what can be sent over that link. Often 85 to 90%** of 
that reference bandwidth works well for downlinks, uplinks can work well up to 
100% of the reference. I initially recommend to set both uplink and downlink to 
50% of the reference values and run a speedtest (e.q. the dslreports of the 
sourceforge one that both also measure latency under load, or preferentially 
flent to a well connected netperf server) to get a feeling for the best case 
latency uunder load increase (at 50% of line rate both a potential ISPs shper 
as well as the real pe-packet overhead will not matter under real-world 
conditions).

Next you need to figure out the per-packet overhead, here is my handy cheat 
sheet:

###ATM: see RFC2684 http://www.faqs.org/rfcs/rfc2684.html
ATM (ADSL2+ PPPoE): 
2 Byte PPP + 6 Byte PPPoE + 6 destination MAC + 6 source MAC + 2 
ethertype + 3 ATM LLC + 5 ATM SNAP + 2 ATM pad + 8 ATM AAL5 SAR = 40 Byte
ATM (ADSL2+ PPPoE VLAN): 
2 Byte PPP + 6 Byte PPPoE + 4 Byte VLAN + + 6 destination MAC + 6 
source MAC + 2 ethertype + 3 ATM LLC + 5 ATM SNAP + 2 ATM pad + 8 ATM AAL5 SAR 
= 44 Byte

###VDSL2 see IEEE 802.3-2012 61.3 relevant for VDSL2: Note VDSL2 typically 
transports full ethernet frames including the FCS (shown as COMON below) 
VDSL2 (PPPoE VLAN)
2 Byte PPP + 6 Byte PPPoE + 4 Byte VLAN + 1 Byte Start of Frame (S), 1 
Byte End of Frame (Ck), 2 Byte TC-CRC (PTM-FCS), = 16 Byte
COMMON: 4 Byte Frame Check Sequence (FCS) + 6 (dest MAC) + 6 (src MAC) 
+ 2 (ethertype) = 18 byte
total: 34 Byte
VDSL2 (your case?)
1 Byte Start of Frame (S), 1 Byte End of Frame (Ck), 2 Byte TC-CRC 
(PTM-FCS), = 4 Byte
COMMON: 4 Byte Frame Check Sequence (FCS) + 6 (dest MAC) + 6 (src MAC) 
+ 2 (ethertype) = 18 byte
total: 22 Byte

### Ethernet
FAST-ETHERNET (should also be valid for GB ethernet): 
7 Byte Preamble + 1 Byte start of frame delimiter (SFD) + 12 Byte inter 
frame gap (IFG) = 20 Byte
COMMON: 4 Byte Frame Check Sequence (FCS) + 6 (dest MAC) + 6 (src MAC) 
+ 2 (ethertype) = 18 byte
total: 38 Bytes worth of transmission time


So a per-packet overhead of 22 seems correct for your case. But the linux 
kernel will already add 14 bytes (for the part of the ethernet header it 
actually send to the device the MACs and the ethertype) automatically for most 
interfaces, so in all likelihood (assuming you connect via ethernet from your 
router to the DSL modem) you should specify 22-14 = 8 Bytes per packet overhead 
for SQM.


Now redo the tests from before but first keep the uplink at 50% and iteratively 
increase the downlink until you encounter too much latency under load increase 
(for your taste); set the uplink to 50% and iteratively increase the 

Re: [Cake] fq_codel on 3g network in Mauritius

[moeller0]

Hi Loganaden,

> On Jul 24, 2016, at 19:13 , Loganaden Velvindron <logana...@gmail.com> wrote:
> 
> On Sun, Jul 24, 2016 at 6:40 PM, moeller0 <moell...@gmx.de> wrote:
>> Hi Jonathan,
>> 
>>> On Jul 24, 2016, at 13:28 , Jonathan Morton <chromati...@gmail.com> wrote:
>>> 
>>> 
>>>> On 24 Jul, 2016, at 13:53, moeller0 <moell...@gmx.de> wrote:
>>>> 
>>>> In theory interval can be different for ingress and egress (think 
>>>> old-school SAT-internet with modem upload) it probably is easiest to only 
>>>> configure one interval setting for the time being.
>>> 
>>> Since the interval parameter depends on the RTT, not the one-way delay, it 
>>> should always be the same both ways (except for inter-packet-time effects).
>> 
>>Thanks for setting this straight. I was confused; the thing that 
>> lingered at the back of my mind was that if we do the target extension for 
>> one direction and correct the interval in that direction, we should also 
>> correct the interval in the other direction, especially since as Jonathan 
>> points out the interval describes the full back-and-forth path…
>> 
>> Best Regards
>>Sebastian
>> 
> 
> I've updated the target to be 22ms based on my current interval of 450ms.
> 
> http://www.dslreports.com/speedtest/4523668
> 
> It's not pretty, but the quality is A or A+.
> 
> It's interesting to see the saw tooth pattern for the upload. This was
> not the case when the target was 5ms, which I believe was calculated
> based on a 100ms worse rtt.


I just noticed you use the dslreports pre-canned profiles for measuring. I 
would recommend against doing that. As these will use different number of 
upstream/downstream flows per profile making comparisons between different 
profiles harder. I would recommend to get a free dslreports account and use the 
speed test configuration to use a fixed number of flows per direction (I 
typically use 16/16, but on slower links often I do not get all 16 going, in 
that case I retry with a lower number of flows, also set manually). After 
registration you can also request high resolution buffer bloat measurements, 
which nicely illustrate a link’s behavior under load (but might not work well 
on very slow links, so maybe you are already running the optimum configuration).

Best Regards
Sebastian



___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] fq_codel on 3g network in Mauritius

[moeller0]

Hi Loganaden,

this is exactly the right idea; interval basically defines the “reaction time” 
window, or the time the endpoints of a connection minimally require to actually 
react to the drop/mark signal. So on a slow link with RTTs in the order of 
300ms set interval to 300ms.
Target should be set to around 5-10% of interval to optimze the bandwidth 
latency tradeoff, but in any case target should be larger than the time 
required to send an individual packet, so that no queue builds up for sparse 
flows.
It is not quite clear to me whether in your case you would account the long 
“pipeline” depth to the target or simply bandwidth/1540…

Best Regards
Sebastian

> On Jul 24, 2016, at 09:03 , Loganaden Velvindron  wrote:
> 
> I am getting A to A+  for quality when setting the interval to 350ms.
> 
> http://www.dslreports.com/speedtest/4520977
> 
> I am getting - to C  for quality when setting the interval to the
> default value of 100ms.
> 
> http://www.dslreports.com/speedtest/4520957

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] conntrack and ipv6

[moeller0]

Hi Dave,


> On Jul 2, 2016, at 14:47 , Dave Täht  wrote:
> 
> It is generally my hope that ipv6 nat will not be widely deployed.
> 
> Firewalls will be stateful instead, and thus there would be no need to
> access the conntrack information for ipv6 in cake.

I would hope that IPv6 NAT would not re-map ports but instead simply 
“hide” stuff behind the prefix, so internal hosts would still be differentiated 
by the remaining 64bits (since individual hosts are supposed to use multiple 
IPv6 addresses I would be amazed if IPv6NAT would hide behind a /128…). The 
bigger fairness issue is that individual host can use basically as many IPs as 
they want and per-IP fairness will not be the right thing anymore anf then all 
we can use is per-MAC, so no internal routers permitted anymore…

> 
> I'm not sure, however, to
> what extent ipv6 conntrack is in openwrt today, certainly udp and tcp,
> "in" is essentially blocked by default, and needs to be triggered by an
> outgoing message. Similarly I'm unfamiliar with the state of ipv6 upnp
> and pcp support in openwrt or client applications at present.
> 
> 
> On 6/30/16 10:33 AM, Kevin Darbyshire-Bryant wrote:
>> 
>> 
>> On 02/06/16 13:29, Jonathan Morton wrote:
 On 2 Jun, 2016, at 14:09, Kevin Darbyshire-Bryant
  wrote:
 
 Cake uses the flow dissector API to do flow hashing...including per
 host flows for dual/triple isolation.  The unfortunate bit is that
 the qdisc inevitably gets placed after packets have been NATed on
 egress and before they've been de-NATed on ingress.
 
 When mentioned before Johnathan said "flow dissector ideally needs to
 be tweaked to do this" or words to that effect.
 
 I'd like to progress that idea...the thought of me kernel programming
 should horrify everyone but really I'm asking for help in being
 pointed in the right direction to ask for help...and go from there :-)
>>> I believe Linux does NAT using a “connection tracker” subsystem.  That
>>> would contain the necessary data for resolving NAT equivalents.  I
>>> don’t know how easy it is to query in a qdisc context, though.
>> Imagine my joy of discovering http://fatooh.org/esfq-2.6/  - someone has
>> already bl**dy done itand I found it lurking in LEDE as part of a
>> patch.
>> 
>> So there relevant bits are something of the order:
>> 
>> 
>> +#ifdef CONFIG_NET_SCH_ESFQ_NFCT
>> +   enum ip_conntrack_info ctinfo;
>> +   struct nf_conn *ct = nf_ct_get(skb, );
>> +#endif
>> 
>> +#ifdef CONFIG_NET_SCH_ESFQ_NFCT
>> +   /* defaults if there is no conntrack info */
>> +   info.ctorigsrc = info.src;
>> +   info.ctorigdst = info.dst;
>> +   info.ctreplsrc = info.dst;
>> +   info.ctrepldst = info.src;
>> +   /* collect conntrack info */
>> +   if (ct && ct != _conntrack_untracked) {
>> +   if (skb->protocol == __constant_htons(ETH_P_IP)) {
>> +   info.ctorigsrc =
>> ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple.src.u3.ip;
>> +   info.ctorigdst =
>> ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple.dst.u3.ip;
>> +   info.ctreplsrc =
>> ct->tuplehash[IP_CT_DIR_REPLY].tuple.src.u3.ip;
>> +   info.ctrepldst =
>> ct->tuplehash[IP_CT_DIR_REPLY].tuple.dst.u3.ip;
>> +   }
>> +   else if (skb->protocol == __constant_htons(ETH_P_IPV6)) {
>> +   /* Again, hash ipv6 addresses into a single u32. */
>> +   info.ctorigsrc =
>> jhash2(ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple.src.u3.ip6, 4,
>> q->perturbation);
>> +   info.ctorigdst =
>> jhash2(ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple.dst.u3.ip6, 4,
>> q->perturbation);
>> +   info.ctreplsrc =
>> jhash2(ct->tuplehash[IP_CT_DIR_REPLY].tuple.src.u3.ip6, 4,
>> q->perturbation);
>> +   info.ctrepldst =
>> jhash2(ct->tuplehash[IP_CT_DIR_REPLY].tuple.dst.u3.ip6, 4,
>> q->perturbation);
>> +   }
>> +
>> +   }
>> +#endif
>> 
>> I'd rip out the IPv6 conntrack stuff as I'm much more concerned by
>> handling IPv4 NAT.  And I'm not sure how to get it into cake's host
>> handling yet but
>> 
>> I can feel an experiment and hackery coming on later today :-)
>> 
>> Am overjoyed!
>> ___
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] New to cake. Some questions

[moeller0]

Hi Dennis,

> On Jun 10, 2016, at 14:43 , Dennis Fedtke <dennisfed...@gmail.com> wrote:
> 
> Hi Sebastian,
> 
> i used the default setting of 1000.

Okay, that should work i assume unless you have a very fast link… What 
link at what ISP do you actually have?

> But it seems that my isp is dropping icmp packets if there are exceeding some 
> sending threshold.

I would be amazed if they did, a sympotom of that would be rsate 
reduction to all ICMP probe flows independent of target host. If however you 
only see this with specific hosts it is very likely that that host rate limits 
its ICMP responses. In either case try another host further upstream. II think 
I has reasonable decent results with targeting 8.8.8.8, googles dns servers.

> So there is a lot of none usable ping data.

Again, try another host…

> I increased the send delay to 50ms. 25 ms already shows dropped requests.

That might also help, as long as you stay below their throttling rate 
the chosen host might still work okay.

> This is the third run now. Waiting for completion.

Well, sorry that the method is not as slick and streamlined, but there 
are no guarded good ICMP reflectors available on the net.

> The ping target is my first hop.

Try the next hop then ;)

> 
> Actually my ping always varies around +-5ms even at idle and independently of 
> ping target.

This is via wifi/wlan? If so try from a wired connection instead.

> When i look through the ping file the increase in ping times are actually 
> appear to be random to me.

Well, we expect variability of the individual “trials” to exist, that 
is why we collect so many and try to select the best measure in the matlab code 
to remove the unwanted variance. Could you post a link to both of the generated 
plots please, the first one showing te different aggregation measures might be 
helpful in diagnosing the issues deeper.

> So how to test if my isp responses with fixed icmp packet size?

You could try manually. In the folloewing example I pinged gstatic.com 
(which belongs to googles CDN as far as I know):

bash-3.2$ ping -s 1 -c 1 gstatic.com
PING gstatic.com (216.58.213.195): 1 data bytes
9 bytes from 216.58.213.195: icmp_seq=0 ttl=55

--- gstatic.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss


bash-3.2$ ping -s 64 -c 1 gstatic.com
PING gstatic.com (216.58.213.195): 64 data bytes
72 bytes from 216.58.213.195: icmp_seq=0 ttl=55 time=19.446 ms

--- gstatic.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 19.446/19.446/19.446/0.000 ms


bash-3.2$ ping -s 65 -c 1 gstatic.com
PING gstatic.com (216.58.213.195): 65 data bytes
72 bytes from 216.58.213.195: icmp_seq=0 ttl=55 time=21.138 ms
wrong total length 92 instead of 93

--- gstatic.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 21.138/21.138/21.138/0.000 ms
bash-3.2$ 


bash-3.2$ ping -s 1400 -c 1 gstatic.com
PING gstatic.com (216.58.213.195): 1400 data bytes
72 bytes from 216.58.213.195: icmp_seq=0 ttl=55 time=6.878 ms
wrong total length 92 instead of 1428

--- gstatic.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 6.878/6.878/6.878/0.000 ms

Once I try to send 65 Bytes of ICMP payload the response is cut short to 92 
bytes, the same might happen with your isp. But also if all your ISP does is 
rate limiting the ICMP packests that still can lead to to much variance in the 
RTTs…


> 
> Im in central europe too :D

Ah, then you just have a different work/sleep cycle than I do ;). Where 
in central Europe, if I might as Ii am, as you might have guessed based in 
Germany…

Best Regards
Sebastian

> 
> Thanks :)
> 
> 
> Am 10.06.2016 um 07:20 schrieb moeller0:
>> Hi Dennis,
>> 
>> 
>>> On Jun 10, 2016, at 02:49 , Dennis Fedtke <dennisfed...@gmail.com> wrote:
>>> 
>>> Hi Sebastian,
>>> 
>>> Sorry this is positive or?
>>  I would say that is unclear…
>> 
>>> But i need more samples ?
>>  I would try with more samples, after checking that the ping times in 
>> the recorded data file actually are larger for larger probes than for 
>> smaller, some hosts will reply with a fixed maximum ICMP packet instead of 
>> returning the received packet, thereby reducing the signal range (as only 
>> the upload leg of the link is meaning fully contributing useful differential 
>> signal.
>>  BTW I am in central europe so at times of the day my responses can be 
>> very sporadic, as I either am at work or sleeping ;)
>> 
>> Best Regards
>>  Sebastian
>>

Re: [Cake] New to cake. Some questions

[moeller0]

Hi Dennis,

> On Jun 10, 2016, at 02:41 , Dennis Fedtke <dennisfed...@gmail.com> wrote:
> 
> Hi Sebastian,
> 
> thanks again :)
> 
> the first 2 pictures arent loading for me in the browser i had to save to 
> hard disk.
> here is my results: http://i67.tinypic.com/5cklcg.png
> 
> I think it is a negativ one?

Mmmh, his is quite noisy, more noisy than it should be; I would 
recommend to redo the test with more sampling points. How many did you use? And 
which target IP did you end up targeting? Also could you also post the first 
picture, which shows more of the data distribution?

> The script gave me following log:
> 
> Found 45400 ping packets in ping_sweep__20160610_001950.txt
> Elapsed time is 767.967 seconds.
> Minimum size of ping payload used: 16 bytes.
> warning: division by zero
> warning: legend: ignoring extra labels
> Unknown or ambiguous terminal name 'wxt'
> Unknown or ambiguous terminal name 'wxt'
> Saved figure (5) to: ping_sweep__20160610_001950_data.png
> lower bound estimate for one ATM cell RTT based of specified up and downlink 
> is 0.064431 ms.
> estimate for one ATM cell RTT based on linear fit of the ping sweep data is 
> 0.064431 ms.
> Starting brute-force search for optimal stair fit, might take a while...
> Unknown or ambiguous terminal name 'wxt'
> Unknown or ambiguous terminal name 'wxt'
> Best staircase fit cumulative difference is: 25.28
> Best linear fit cumulative difference is: 25.787
> Quantized ATM carrier LIKELY (cummulative residual: stair fit 25.28 linear 
> fit 25.787
> remaining ATM cell length after ICMP header is 11 bytes.
> ICMP RTT of a single ATM cell is 0.059921 ms.
> 
> Estimated overhead preceding the IP header: 42 bytes
> 
> Can the errors be ignored ?

I have never seen these before, so I need to see whether I can recreate 
them, which octave version are you using?

Best Regards
Sebastian

> 
> Best Regards
> Dennis
> 
> 
> Am 10.06.2016 um 01:11 schrieb moeller0:
>> Hi Dennis,
>> 
>>> On Jun 10, 2016, at 00:45 , Dennis Fedtke <dennisfed...@gmail.com> wrote:
>>> 
>>> Hi Sebastian,
>>> 
>>> thank you for your answers :)
>>> 
>>> The ATM overhead detector script is currently running.
>>> I read the wiki about it but im not quite sure how to interpret the plot.
>>> I mean what info should i read from it? maximum packet size?
>>  The relevant number is reported as “Estimated overhead preceding the IP 
>> header” in the top part of the second figure created by the script. But that 
>> is only relevant.useful if you see a nice step like plot in figure 2 as well 
>> ( the second figure in 
>> https://github.com/moeller0/ATM_overhead_detector/wiki as positive and the 
>> fourth figure as negative example.
>> 
>>> If yes do i set the overhead in cake? Or do i set iptables to clamp to new 
>>> mtu/mss?
>>  If you use plain cake and you know the numerical overhead (NN) the 
>> easiest is to add the following to your cake invocation: “atm overhead NN”
>> 
>> Please note that if you use cake on an ethernet interface the kernel will 
>> already account for 14 byte of ethernet overhead, so if the script told you 
>> 44 as actual overhead, you use ”overhead 30” to address that. If you use a 
>> pppoe interface the kernel will most likely not add the 14 bytes for you, so 
>> then you would use “overhead 44” (I excluded the atm option in the last 
>> examples for clarity…)
>> 
>>> Regarding UDP paket dropping problem:
>>> I just read some forums and users stated that under heavy load cake starts 
>>> to drop udp packets which causes lag ingame.
>>> My idea was to set ingress/egress to diffserv4 and apply the EF dscp mark 
>>> on those packets.
>>  Ell, not a bad idea, but often the problem are in the incoming traffic, 
>> and unfortunately with the ifb we use we can not use iptables, but only tc, 
>> and remarking with tc is unpleasant.
>> 
>>> Will this even work? if yes how to do this? iptables?
>>  No, you wuld need tp use tc.
>> 
>>> ipt -t mangle -A PREROUTING -p udp -m multiport --ports 5000:5500 -j DSCP 
>>> --set-dscp-class EF
>>> 
>>> Like thia? Is prerouting correct here? (Taken from layer cake script)
>>  This will affect outgoing packets and might be a good idea in your 
>> specific case.
>> 
>> BUT why don’t you try the default behaviour with specific rules and tricks 
>> and report success or failure back to us, after all the fastest/easiest 
>> classification is one one does not need to perform at all

Re: [Cake] CAKE upstream in LEDE

[moeller0]

Hi Dave,


> On Jun 7, 2016, at 18:07 , Dave Taht  wrote:
> 
> shiny! I could not resist and installed lede head on my 1200ac just now.
> 
> But:
> 
> root@linksys-1200ac:/etc/config# tc qdisc add dev eth0 root cake bandwidth 
> 9mbit
> Unknown qdisc "cake", hence option "bandwidth" is unparsable
> root@linksys-1200ac:/etc/config#
> 
> Also sqm-scripts fails silently when trying to configure this.
> 
> root@linksys-1200ac:/etc/config# /etc/init.d/sqm  start
> SQM: Stopping SQM on eth0
> SQM: Starting SQM script: piece_of_cake.qos on eth0, in: 10 Kbps,
> out: 9000 Kbps
> SQM: piece_of_cake.qos was started on eth0 successfully

That is bad, and should not have happened. Do you know which parts of 
the required packages were already fully installed when this situation occurred?

Best Regards
Sebastian

> 
> 
> 
> On Tue, Jun 7, 2016 at 2:53 AM, Kevin Darbyshire-Bryant
>  wrote:
>> Dear All,
>> 
>> For those interested, Jonathan Morton's CAKE qdisc has made it into LEDE.
>> See
>> https://git.lede-project.org/?p=source.git;a=commit;h=1a3c56f8322d27f4652d23eb7dc14a45e1631e1c
>> for proof :-)
>> 
>> The package to install for your target is 'kmod-sched-cake'
>> https://downloads.lede-project.org/snapshots/targets/
>> 
>> The base 'iproute2' package has also been taught how to speak 'CAKE' so
>> there's no longer any need to install a special version of 'tc' (previously
>> supplied by the 'tc-adv' package)
>> https://git.lede-project.org/?p=source.git;a=commit;h=23147dd43ac0835cc9077819c32a101dee42461d
>> 
>> Package 'sqm-scripts' has known how to speak 'CAKE' for a while, with
>> choices between 'layer_cake' and 'piece_of_cake'.  Also package
>> 'luci-app-sqm' provides a user friendly web based configuration page for
>> 'sqm-scripts'
>> 
>> Thanks must go to Hannu Nyman for distilling the relevant changes from the
>> sch_cake and tc-adv code repositories (https://github.com/dtaht/sch_cake &
>> https://github.com/dtaht/tc-adv) into nice, neat patches for LEDE.
>> 
>> It may take a few days for the LEDE build bots to get around to your target
>> platform of interest, but the import kernel module (kmod-sched-cake) is
>> there.
>> 
>> Enjoy!
>> 
>> Kevin
>> ___
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
> 
> 
> 
> -- 
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
> http://blog.cerowrt.org
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] [Codel] Proposing COBALT

[moeller0]

Hi Jonathan,


> On Jun 4, 2016, at 16:16 , Jonathan Morton <chromati...@gmail.com> wrote:
> 
> 
>> On 4 Jun, 2016, at 17:01, moeller0 <moell...@gmx.de> wrote:
>> 
>> Maybe cake should allow to switch from the default mark by ECN policy to 
>> mark by drop per command line argument? At least that would allow much 
>> easier in the field testing… As is there is only the option of disabling ECN 
>> at the endpoint(s)…
> 
> I consider ignoring ECN in the way I described to be a fault condition 
> inevitably resulting in unresponsive traffic.  As a fault condition, it 
> should be rare.

Operative word being “should” in my opinion; as long as we have no 
reliable statistics either way, assuming rarity seems overly optimistic to me. 
Not giving the user control over policy requires the default policy to be 
almost 100% applicable., here we have a demonstrated case where this 
requirement seems violated. Make out of that what you want, if cake were my 
project I would make ECN versus drop configurable at the qdisc, as the control 
via the endhosts seems comparatively tedious, especially for quick comparative 
testing. But cake is not my project, so all I can do is try to make a case for 
introducing a policy control toggle…

> 
> The main effect in practice is that the RTT for the affected flows grows well 
> beyond normal, but since they are bulk transfers,
> this has only a minor detrimental effect (much of which is incurred 
> sender-side in the form of retransmission buffers two orders of magnitude 
> larger than necessary).
> 
> Rather than further complicate Codel or Cake, I’d like to simply apply a 
> general solution for unresponsive traffic, ie. COBALT.

If adding a toggle for ECN versus drop is your only concern in the 
complexity of cake’s configuration you have not been reading my arguments 
regarding the labyrinthian overhead keywords… Really not exposing this control 
for this might actually be a reasonable thing to do, but trying to “blame” this 
on added complexity seems far fetched… but what do I know…

Best Regards
Sebastian

> 
> - Jonathan Mortob
> 

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] [Codel] Proposing COBALT

[moeller0]

Hi Jonathan,


> On Jun 4, 2016, at 15:55 , Jonathan Morton  wrote:
> 
> 
>> On 4 Jun, 2016, at 04:01, Andrew McGregor  wrote:
>> 
>> ...servers with ECN response turned off even though they negotiate ECN.
> 
> It appears that I’m looking at precisely that scenario.
> 
> A random selection of connections from a packet dump show very high marking 
> rates, which are apparently acknowledged using CWR, but a subsequent dropped 
> packet (probably due to queue overflow) takes many seconds to be 
> retransmitted (I’m using a rather high memory limit for observation purposes).
> 
> Overall the TCP behaviour is approximately normal for NewReno on a dumb FIFO, 
> and the ECN signalling is completely ignored.  This doesn’t rule out the 
> possibility that it’s a different Reno relative, such as Westwood+ or 
> Compound.
> 
> There’s often more than one CWR per RTT.  This isn’t a consistent 
> characteristic; some connections have normal-looking CWRs while others issue 
> them every three packets, as if they’re fishing for “more accurate” ECN 
> feedback.  It might vary by host; I didn’t keep track of that.  But this 
> can’t be DCTCP; even that should back off in the face of a 100% marking rate, 
> which is often achieved at my low bandwidth and with very persistent queues.
> 
> Other servers respond normally to ECN signals, ruling out interference by my 
> ISP. It’s possible the ECE flag is wiped and the CWRs are faked, but there’s 
> no legitimate reason to do that.  The CWRs ultimately make no difference, 
> since at 100% CE marks, every ack has ECE set anyway.
> 
> Turning off ECN negotiation at the client results in a much better managed 
> queue with similar throughput.  It’s not immediately obvious whether that’s 
> due to a functioning congestion response or simply the AQM clearing out the 
> queue the hard way.  It’ll be interesting to see what effect COBALT has here, 
> when I get it to actually work.
> 
> As for who these servers are: Valve Software’s Steam platform.  I did say 
> they were large and popular.

Maybe cake should allow to switch from the default mark by ECN policy 
to mark by drop per command line argument? At least that would allow much 
easier in the field testing… As is there is only the option of disabling ECN at 
the endpoint(s)…

Best Regards
Sebastian

> 
> - Jonathan Morton
> 
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake/tc - removal of atm/ptm/ethernet specific overhead keywords

[moeller0]

> On Jun 2, 2016, at 20:55 , Jonathan Morton <chromati...@gmail.com> wrote:
> 
> 
>> On 2 Jun, 2016, at 21:53, moeller0 <moell...@gmx.de> wrote:
>> 
>> “conservative”-keyword needs special care in documentation as it is the only 
>> keyword that compounds per-packet-overhead and specific framing
> 
> Not true.  All of the ATM-specific encapsulation keywords - of which there 
> are ten others - also force ATM compensation on.  This is obvious in the code.
> 
> - Jonathan Morton
> 

I might not have picked the best example, but the current keywords make it 
simple for me ;) 
Let me rephrase then , it is not self-evident which keywords are ATM-specific 
then… But humor me:
tc qdisc add cake help
Usage: ... cake [ bandwidth RATE | unlimited* | autorate_ingress ]
[ rtt TIME | datacentre | lan | metro | regional | internet* | 
oceanic | satellite | interplanetary ]
[ besteffort | precedence | diffserv8 | diffserv4* ]
[ flowblind | srchost | dsthost | hosts | flows* | dual-srchost 
| dual-dsthost | triple-isolate ]
[ atm | noatm* ] [ overhead N | conservative | raw* ]
[ wash | nowash* ]
[ memlimit LIMIT ]
(* marks defaults)

Where is it evident that “conservative” includes atm encapsulation? And what 
should a user expect that specifies “noatm conservative”?

So Jonathan, please, instead of trying to argue obvious inconsistencies 
in the currently assigned encapsulation keywords away, just go and make sure 
they are consistent and well documented. 
Your postings in the overhead-matter make me question whether you do 
fully understand the issue at hand in its full complexity; so by all means go 
and collect input from users (usability and self-evidence of the keywords) and 
experts (on the actual likelihood of encapsulations).

Best Regards
Sebastian
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Proposing COBALT

[moeller0]

> On May 20, 2016, at 15:41 , David Lang  wrote:
> 
> On Fri, 20 May 2016, Jonathan Morton wrote:
> 
>> Normal traffic does not include large numbers of fragmented packets (I would 
>> expect a mere handful from certain one-shot request-response protocols which 
>> can produce large responses), so it is better to shunt them to a single 
>> queue per host-pair.
> 
> I don't agree with this.
> 
> Normal traffic on a well setup network should not include large numbers of 
> fragmented packets. But I have seen too many networks that fragment almost 
> everything as a result of there being a hop that goes through one or more 
> tunneling layers that lower the effective MTU (and no, path mtu discovery 
> does not always work)

True, do you have a cheaper idea of getting the flow identity cheaply 
from fragmented packets, short of ressembly ;) ?

Best Regards
Sebastian

> 
> David Lang

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] new code point proposed

[moeller0]

> On Apr 5, 2016, at 22:06 , Jonathan Morton  wrote:
> 
> 
>> On 5 Apr, 2016, at 21:57, Dave Taht  wrote:
>> 
>> https://tools.ietf.org/html/draft-you-tsvwg-latency-loss-tradeoff-00
> 
> Interesting.  This is obviously written around the DualQ AQM, but it seems 
> feasible to implement within Cake’s framework.  I’m somewhat pleased that 
> this appears to mark a move away from using ECN alone to describe whether 
> DCTCP is in use or not.
> 
> Some of the requirements are written in a way that assumes a single queue of 
> fixed maximum size for each of “La" and "Lo", rather than a dynamic, 
> flow-isolated system as Cake is.  It might be reasonable to suggest 
> clarifying the language to take these cases into account.
> 
> We can already vary the Codel parameters between tins, which provides an 
> obvious mechanism to make the queue management much more aggressive for “La” 
> traffic, and more relaxed for “Lo” traffic.  I don’t think we need to 
> explicitly limit the queue allocation to “La” traffic as specified.
> 
> Another detail which differs from Cake’s view of the world is that neither La 
> nor Lo have priority over each other.  While Cake does not implement strict 
> priority, it does implement soft priority as part of its effort to minimise 
> latency for the upper tins; the most explicit part of this is that bandwidth 
> consumed by higher tins is removed from lower tins’ allocations.  However, 
> this effect could be hidden by making the two DRR weights equal for the lower 
> tin.
> 
> Obviously, traffic marked with the existing “low latency intent” DSCPs can be 
> treated as “Lo” traffic when there is no admission control in place, without 
> any requirement for re-marking.  This is consistent with what Cake does 
> anyway.
> 
> This seems like a good excuse to overhaul Cake’s Diffserv class allocations.  
> I could propose a 5-class system:
> 
> Tin 0 = LLT “Lo” traffic (inc. existing low-loss & high-throughput classes), 
> 256/256, 100%, increased target & interval.
> Tin 1 = Best Effort traffic, 256/256, 100%, standard target & interval.
> Tin 2 = LLT “La” traffic (inc. existing low-latency classes), 256/256, 100%, 
> standard target, reduced interval.

This might back fire, as far as I understand interval is the reaction 
time window for a flow, this needs to be roughly in the ballpark of the RTT, 
reducing it (significantly) will make the AQM quite trigger happy. This might 
be in line with the LA proposal, but what if LA traffic has to cross a 
satellite link?

Best Regards
Sebastian


> Tin 3 = Low Priority traffic, 2048/16, 6.25%, standard target & interval.
> Tin 4 = Network Control traffic, 4096/1, 6.25%, increased target & interval.
> 
> Note that Tin 4 is almost, but not quite, strictly admission-controlled, 
> discouraging the use of “network control” DSCPs for general traffic.  Tins 
> 0-2 implement LLT as specified.  Tin 3 takes the unusual and 
> counter-intuitive step of placing “low priority” traffic in a high tin, but 
> the effect should be very similar to existing behaviour, due to the 
> soft-priority system and the low bandwidth allocation.
> 
> - Jonathan Morton
> 
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake separate qos for lan

[moeller0]

Hi Allen,

> On Mar 28, 2016, at 14:25 , Allan Pinto <allan...@gmail.com> wrote:
> 
> Hi Sebastian, 
> I should have made this more clear please see below topology with added 
> comments. the customers connecting to the linux router can be in range from 
> 100 to 2000, so shaping on the switch is not really a option.

Oh, I did not advocate shaping on the switch, just on the Linux 
router’s interface connecting it to that switch, but it seems i misunderstood 
your issue, I assumed you want to shape both directions…. 

> I am right now testing on a i3 machine, but for actual live testing am 
> planning to test with i7 or a xeon.
> 
>   Cache-Server [ connected to internet gateway , traffic can 
> be sent to it via wccp or policy based routing ]
>|
>   internet>internet Gateway —> L2 switch [ MEN network on fiber ]   --> 
> LInux router with cake[ includes a pppoe server which authenticates with 
> radius ] - - [ pppoe connection over a fiber men network ]  --> customer [ 
> customers can be 100 to 2000 ]
> 
> basically the customer will create a broadband connection on his pc to 
> connect.
> 
> . > @Allan, what is the link technology you use?
> fiber 1g/10g/last hop cat5e

Nice, that means you can certainly skip using pppoe-vcmux, as that is 
ATM/AAl5 specific, I would assume that using “overhead NN” should be more 
effective. Since you run the show you will know exactly which overhead to 
account for (keep in mind that Linux will occasionally add 14 bytes for parts 
of the ethernet header). It might make sense to include preamble and 
inter-frame gap into the per packet overhead as effectively you are shaping on 
ethernet IIUC.

> 
> > As I just wrote, can’t we completely avoid the IMQ/IFB here and use dual 
> > egress shaping instead (once on the pppoe device and once on the interface 
> > connected to the switch, which effectively should shape both directions)?
> 
> i may be wrong here, but i think jonathan is advising the use of IMQ/IFB to 
> provide two different shaping scenarios on egress itself. not ingress. as i 
> need cache traffic to have higher bandwidth on the egress towards customer 
> but non- cache traffic [ pure internet ] to remain within the bandwidth 
> limits purchased by the customer.

Ah, you are right, I have not fully thought through your requirements 
then. I am quite curios to learn how this will work out ;) Especially since you 
will need to run (100 to 2000) * 2 cake instances on the router if you go for a 
“two shaper per customer” approach. 

Best Regards
Sebastian

> 
> 
> 
> On Mon, Mar 28, 2016 at 5:39 PM, moeller0 <moell...@gmx.de> wrote:
> Hi Jonathan,
> 
> > On Mar 28, 2016, at 12:31 , Jonathan Morton <chromati...@gmail.com> wrote:
> >
> >
> >> On 27 Mar, 2016, at 11:20, moeller0 <moell...@gmx.de> wrote:
> >>
> >> it might be more future-proof to just use IFBs from the get-go
> >
> > For this particular use-case, it seems to be more complicated to use IFB 
> > than IMQ, largely because there is no iptables rule to divert packets 
> > through an IFB device, and unlike iptables, the CBQ filter mechanism 
> > doesn’t directly support negative matches of any kind.
> 
> As I just wrote, can’t we completely avoid the IMQ/IFB here and use 
> dual egress shaping instead (once on the pppoe device and once on the 
> interface connected to the switch, which effectively should shape both 
> directions)?
> 
> >
> > However, I think this would work - though it’s completely untested:
> >
> > ip link set ifb0 up
> >
> > tc qdisc replace dev ppp0 root handle 1: cake pppoe-vcmux bandwidth 
> > $FULL_RATE triple-isolate
> 
> I wonder how you came up with pppoe-vcmux, I have not seen any 
> information about the link technology in Allan’s post. As far as I know a 
> number of (mislead) ISPs use PPPoE even on fiber links. @Allan, what is the 
> link technology you use?
> 
> >
> > tc qdisc replace dev imq0 root handle 2: cake raw bandwidth $NONCACHE_RATE 
> > flows
> 
> I believe this might work as egress on the interface facing the 
> L2-switch…
> 
> >
> > tc filter replace dev ppp0 protocol ip prio 1 handle 11 u32 match ip src 
> > $CACHE_IP/32
> >
> > tc filter replace dev ppp0 protocol ip prio 2 handle 12 u32 action mirred 
> > egress redirect dev ifb0
> >
> > The logic of the above is that a positive match is made on the cache 
> > traffic, but no action is taken.  This terminates filter processing for 
> > that traffic.  The remaining traffic is redirected unconditionally to th

Re: [Cake] cake separate qos for lan

[moeller0]

Hi Allan,

> On Mar 27, 2016, at 07:31 , Allan Pinto  wrote:
> 
> > Is the cache inside or outside the point where the router is fitted
>  outside..
> 
> Cache-Server
>|
> internet Gateway —> L2 switch   --> LInux router with cake - - [ pppoe 
> connection ]  --> customer

Looking at this I would assume you can skip ingress shaping completely, 
just shape on the pppoe devices egress (for customer ingress) as well as on 
egress to the L2-switch for customer egress. This should allow to completely 
avoid IMQ/IFB and the associated computation cost. On the other hand I am not 
sure how well this will scale with its two cake instances per customer…

> 
> 
> 
> >Also, the command shown will apply a limit to *egress* traffic on the given 
> >port.  If you need to do *ingress* shaping, there’s a different sequence of 
> >commands
> i have put the ingress commands, i did not mention them as i did not have any 
> need of changes there.
>  
> 
> 
> On Sun, Mar 27, 2016 at 3:44 AM, Jonathan Morton  
> wrote:
> 
> > On 26 Mar, 2016, at 17:14, Allan Pinto  wrote:
> >
> > I'm experimenting in replacing a mikrotik router with plain linux. by 
> > following the instructions on the cake page, i have setup the following 
> > line in the /etc/ppp/ip-ip script so that the user will be limited to 
> > bandwidth using cake.
> >
> > /usr/sbin/tc qdisc add dev $pppdev root cake bandwidth ${BURST_DOWN}bit
> >
> > but i have certain lan traffic available to the customer which should be 
> > available at higher speed, for eg. cache traffic and i want to set that 
> > speed to 20mbit default .
> >
> > if i understand correctly i will have to mark traffic coming in from that 
> > lan source using iptables, can someone guide me how to set bandwidth only 
> > for that source to be higher.
> 
> I’m not certain what your topology is here.  Is the cache inside or outside 
> the point where the router is fitted, or is it within the router, or off to 
> the side on a separate port?
> 
> Also, the command shown will apply a limit to *egress* traffic on the given 
> port.  If you need to do *ingress* shaping, there’s a different sequence of 
> commands.
> 
>  - Jonathan Morton
> 
> 
> 
> 
> -- 
> Thanx and regd's.
> 
> Allan.
> 
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] triple flow isolation

[moeller0]

Hi Kevin,

> On Jan 11, 2016, at 21:33 , Kevin Darbyshire-Bryant 
> <ke...@darbyshire-bryant.me.uk> wrote:
> 
> 
> 
> On 11/01/16 18:16, moeller0 wrote:
>> Hi Kevin,
>> 
>> I agree the triple mode seems under-documented ;)
> Yes that's true but it is experimental after all - and I'm experimenting
> with it :-)
>>> On Jan 11, 2016, at 18:40 , Kevin Darbyshire-Bryant 
>>> <ke...@darbyshire-bryant.me.uk> wrote:
>>> 
>>> Hello List,
>>> 
>>> I've been looking at latest 'triple flow isolation' features in latest
>>> cake git and find myself confused.  It's very likely to be a
>>> misunderstanding on my part, although if I'm confused I'm sure others
>>> will, sooner or later, fall into the same trap.
>>> 
>>> I thought that triple flow was a solution such that a host with many
>>> elephant flows couldn't dominate the bandwidth consumption thus starving
>>> another host with just one elephant flow.  I guess the typical example
>>> is a 'bittorrent' host pulling data from many places vs a host pulling
>>> data from a single place.  My recent 'test' example was a host doing 4
>>> simulatenous pulls of 9GByte+ files vs my wife downloading a movie via
>>> the TV box downstairs.  The bandwidth was evenly divided by the 5 flows,
>>> however my host got 4/5ths of the share.  I didn't think this was
>>> supposed to happen with triple flow isolation?  Ideally we'd both get
>>> 50% of the ISP bandwidth managed by my router (cake is running on the
>>> WAN facing interface to the ISP modem with appropriate bandwidth limits set)
>>  I believe you would want src_host on egress and (post-NAT) dat-host on 
>> ingress, assuming your goal is fairness by internal host, not external hosts 
>> ;). Now the challenges are to get the post-NAT IP addresses on ingress and 
>> how to counteract IPv6’s tendency to grow incredible amounts of addresses by 
>> host. In theory all of this could be solved with fairness by internal MAC 
>> addresses, except these will only work in a shallow networks (as far as I 
>> understand). I would argue that wifi aggregation has the same issue 
>> regarding IPv6, but I digress...
> Ah, yes, silly me - NAT.  As you say pre-NAT internal source address on
> egress and post-NAT internal destination address on ingress.  So the
> shaping to work, cake needs to be on the wan interface but that's too
> late to obtain the internal pre-NATed addresses.  Oh dear.  Help!

So a quick and dirty test would be to set up sqm with cake on the 
interface connecting the router SoC with its switch (aka LAN) then you have 
access to the internal addresses for both egress and ingress. Note that ingress 
and egress will be switched as compared to sqm on the wan interface so adjust 
the shaper settings accordingly (ingress egress are always from the view of the 
router, and on a LAN interface the egress direction of the interface is the 
ingress/download direction from the WAN). So, I would guess, destination 
address on egress and source address on ingress as seen by sqm should do the 
trick then.
This test will obviously only work if no additional traffic hits the 
router besided WAN ans LAN, so no WLAN on the router for this test… Or leave 
everything as is an run pure IPv6 tests, since I believe openwrt does not do 
NAT6 by default…
I am really curious how cake behaves in that setting...

Best Regards
Sebastian

> 
> 
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] triple flow isolation

[moeller0]

Hi Kevin,

I agree the triple mode seems under-documented ;)

> On Jan 11, 2016, at 18:40 , Kevin Darbyshire-Bryant 
>  wrote:
> 
> Hello List,
> 
> I've been looking at latest 'triple flow isolation' features in latest
> cake git and find myself confused.  It's very likely to be a
> misunderstanding on my part, although if I'm confused I'm sure others
> will, sooner or later, fall into the same trap.
> 
> I thought that triple flow was a solution such that a host with many
> elephant flows couldn't dominate the bandwidth consumption thus starving
> another host with just one elephant flow.  I guess the typical example
> is a 'bittorrent' host pulling data from many places vs a host pulling
> data from a single place.  My recent 'test' example was a host doing 4
> simulatenous pulls of 9GByte+ files vs my wife downloading a movie via
> the TV box downstairs.  The bandwidth was evenly divided by the 5 flows,
> however my host got 4/5ths of the share.  I didn't think this was
> supposed to happen with triple flow isolation?  Ideally we'd both get
> 50% of the ISP bandwidth managed by my router (cake is running on the
> WAN facing interface to the ISP modem with appropriate bandwidth limits set)

I believe you would want src_host on egress and (post-NAT) dat-host on 
ingress, assuming your goal is fairness by internal host, not external hosts 
;). Now the challenges are to get the post-NAT IP addresses on ingress and how 
to counteract IPv6’s tendency to grow incredible amounts of addresses by host. 
In theory all of this could be solved with fairness by internal MAC addresses, 
except these will only work in a shallow networks (as far as I understand). I 
would argue that wifi aggregation has the same issue regarding IPv6, but I 
digress...

> 
> The good news is that latency was well controlled and general web
> browsing/catching emails etc was oblivious to the elephant melee going on.

Well, I have not managed to get the new cake onto my router at all, so 
I am happy to read that it works.

Best Regards
Sebastian

> 
> Advice gratefully received.
> 
> 
> Kevin
> 
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake