Re: [Cake] total download rate with many flows

[Pete Heist]

> On Nov 20, 2017, at 9:06 AM, Jonathan Morton  wrote:
> It's a kind offer, and since I'm in Finland (also EU), there shouldn't be any 
> problem with VAT.
> 
> However, I just managed to find a copy of the failsafe code on one of my 
> other machines, and merged it to cobalt branch.  I honestly thought it had 
> been pushed already, and that the only reason for it being otherwise was that 
> it must be stranded.
> 

Great news, well, the offer stands if there’s something to deal with later…

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[Jonathan Morton]

It's a kind offer, and since I'm in Finland (also EU), there shouldn't be
any problem with VAT.

However, I just managed to find a copy of the failsafe code on one of my
other machines, and merged it to cobalt branch.  I honestly thought it had
been pushed already, and that the only reason for it being otherwise was
that it must be stranded.

- Jonathan Morton
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[Pete Heist]

> From: Jonathan Morton <chromati...@gmail.com>
> To: Georgios Amanakis <gamana...@gmail.com>,
>   cake@lists.bufferbloat.net
> Subject: Re: [Cake] total download rate with many flows
> Message-ID: <464c2c38-8ecc-b232-19b3-bf3eb0146...@gmail.com>
> Content-Type: text/plain; charset=utf-8; format=flowed
> 
> On 16/11/17 19:06, Georgios Amanakis wrote:
>> I am skimming through the code but cannot see where the failsafe 
>> shaper's rate of advance is set at one-quarter rate. Could you point 
>> this out?
> 
> Oddly enough, I can't find it either.  It could be that the code wasn't 
> pushed yet - or even that it only exists on my MBP hard drive, which I 
> still don't have a working MBP to put in again.  Due to HFS+ 
> compression, I can't actually read the file by attaching it to a Linux 
> machine.

I have a vintage 2007 MBP running El Capitan that isn’t doing much. If it’s a 
matter of getting to data or some other critical stuff, would this help?

Its battery is shot because I stupidly neglected to charge it for too long (for 
the second time, which multiplies the dumbness), but the machine otherwise 
works.

If it’s a matter of getting to data, would this help? If so, we can see if 
there’s a way to arrange getting it to you from Czech and back. I’d want to be 
careful not to pay VAT to re-import my own possession. The postal authorities 
here can be sticklers. Just let me know if this helps somehow...

Pete

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[Jonathan Morton]

On 16/11/17 19:06, Georgios Amanakis wrote:
I am skimming through the code but cannot see where the failsafe 
shaper's rate of advance is set at one-quarter rate. Could you point 
this out?


Oddly enough, I can't find it either.  It could be that the code wasn't 
pushed yet - or even that it only exists on my MBP hard drive, which I 
still don't have a working MBP to put in again.  Due to HFS+ 
compression, I can't actually read the file by attaching it to a Linux 
machine.


--
 - Jonathan Morton
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[Jonathan Morton]

I'm curious as to why you think Cobalt is more aggressive than Codel.  It
does use more accurate approximations to the mathematical ideal than the
"reference" codel does.

It is however very odd that the Diffserv mode has any effect on this at
all.  It could be explained if a lot of the traffic is marked CS1, since
the Bulk tin has looser AQM parameters.  That suggests that selecting
'satellite' might help similarly.

Something worth trying would be to alter the failsafe shaper's rate of
advance.  Currently it has a one-quarter rate, which might be too
restrictive.  Tests at one-half and three-quarters might therefore be
interesting.  Otherwise, I don't think trying to modify the way ingress
mode works will do the right things.

Ultimately, this arises because Cake is having to drop packets in order to
signal congestion, and when there's a lot of flows, a lot of signals must
be sent to get through to them all.  With ECN working, it doesn't need to
waste bandwidth merely to signal.

The only other reasonable approach is to somehow reduce the signalling rate
under heavy flow load.  That requires informing Cobalt of the number of
bulk flows, and using that to scale the signalling frequency somehow.

- Jonathan Morton
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[Dave Taht]

On Tue, Nov 14, 2017 at 8:04 PM, George Amanakis <g_amana...@yahoo.com> wrote:
> The problem I am describing in this thread appears only when using
> besteffort. If diffserv3 is used (without any further prioritization by
> manipulating DSCP bits) downstream is more stable and higher in throughput
> than with besteffort.
>
> Also, in my case disabling BLUE (by commenting out cake_queue_{empty,full}
> and all p_drop parts in sch_cake.c and cobalt.c) does not make any
> difference.

The only other thing I can think of is replacing the codel algorithm
in cake with the one in fq_codel.


>
>
> On 11/14/2017 5:23 PM, Dave Taht wrote:
>>
>> because cobalt is more aggressive than codel. I've never believed in it.
>>
>> I'm sitting here trying to figure out how to get good ole regular cake
>> to compile again.
>>
>> I just merged the new ack stuff into the codel branch.
>>
>>
>> On Tue, Nov 14, 2017 at 2:13 PM, G. Amanakis via Cake
>> <cake@lists.bufferbloat.net> wrote:
>>>
>>> ___
>>> Cake mailing list
>>> Cake@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cake
>>>
>>>
>>> -- Forwarded message --
>>> From: "G. Amanakis" <g_amana...@yahoo.com>
>>> To: Dave Taht <d...@taht.net>, George Amanakis via Cake
>>> <cake@lists.bufferbloat.net>
>>> Cc:
>>> Bcc:
>>> Date: Tue, 14 Nov 2017 17:13:16 -0500
>>> Subject: Re: [Cake] total download rate with many flows
>>> Yes, it is. I am building from the cobalt branch. What makes you believe
>>> otherwise? Would you expect a different behaviour?
>>>
>>> On November 14, 2017 3:11:04 PM EST, Dave Taht <d...@taht.net> wrote:
>>>>
>>>> George Amanakis via Cake <cake@lists.bufferbloat.net> writes:
>>>>
>>>>>   From: George Amanakis <g_amana...@yahoo.com>
>>>>>   Subject: Re: [Cake] total download rate with many flows
>>>>>   To: David Lang <da...@lang.hm>
>>>>>   Cc: cake@lists.bufferbloat.net
>>>>>   Date: Mon, 13 Nov 2017 21:49:53 -0500 (17 hours, 20 minutes, 42
>>>>> seconds ago)
>>>>>
>>>>>   Dear David,
>>>>>
>>>>>   I agree. My point is that currently ingress mode seems to be dropping
>>>>> more
>>>>>   packets than necessary to keep senders from bottlenecking the
>>>>> connection (when
>>>>>   there is a large number of concurrent flows, >8). And right now,
>>>>> ingress mode is
>>>>>   the only mode that achieves this in situations such as Windows
>>>>> updates.
>>>>
>>>>
>>>> Is cobalt enabled in your build?
>>>
>>>
>>> --
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>>
>>
>>
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[George Amanakis via Cake]

--- Begin Message ---
The problem I am describing in this thread appears only when using 
besteffort. If diffserv3 is used (without any further prioritization by 
manipulating DSCP bits) downstream is more stable and higher in 
throughput than with besteffort.


Also, in my case disabling BLUE (by commenting out 
cake_queue_{empty,full} and all p_drop parts in sch_cake.c and cobalt.c) 
does not make any difference.



On 11/14/2017 5:23 PM, Dave Taht wrote:

because cobalt is more aggressive than codel. I've never believed in it.

I'm sitting here trying to figure out how to get good ole regular cake
to compile again.

I just merged the new ack stuff into the codel branch.


On Tue, Nov 14, 2017 at 2:13 PM, G. Amanakis via Cake
<cake@lists.bufferbloat.net> wrote:

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake


-- Forwarded message --
From: "G. Amanakis" <g_amana...@yahoo.com>
To: Dave Taht <d...@taht.net>, George Amanakis via Cake 
<cake@lists.bufferbloat.net>
Cc:
Bcc:
Date: Tue, 14 Nov 2017 17:13:16 -0500
Subject: Re: [Cake] total download rate with many flows
Yes, it is. I am building from the cobalt branch. What makes you believe 
otherwise? Would you expect a different behaviour?

On November 14, 2017 3:11:04 PM EST, Dave Taht <d...@taht.net> wrote:

George Amanakis via Cake <cake@lists.bufferbloat.net> writes:


  From: George Amanakis <g_amana...@yahoo.com>
  Subject: Re: [Cake] total download rate with many flows
  To: David Lang <da...@lang.hm>
  Cc: cake@lists.bufferbloat.net
  Date: Mon, 13 Nov 2017 21:49:53 -0500 (17 hours, 20 minutes, 42 seconds ago)

  Dear David,

  I agree. My point is that currently ingress mode seems to be dropping more
  packets than necessary to keep senders from bottlenecking the connection (when
  there is a large number of concurrent flows, >8). And right now, ingress mode 
is
  the only mode that achieves this in situations such as Windows updates.


Is cobalt enabled in your build?


--
Sent from my Android device with K-9 Mail. Please excuse my brevity.






--- End Message ---
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[Dave Taht]

because cobalt is more aggressive than codel. I've never believed in it.

I'm sitting here trying to figure out how to get good ole regular cake
to compile again.

I just merged the new ack stuff into the codel branch.


On Tue, Nov 14, 2017 at 2:13 PM, G. Amanakis via Cake
<cake@lists.bufferbloat.net> wrote:
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>
> -- Forwarded message --
> From: "G. Amanakis" <g_amana...@yahoo.com>
> To: Dave Taht <d...@taht.net>, George Amanakis via Cake 
> <cake@lists.bufferbloat.net>
> Cc:
> Bcc:
> Date: Tue, 14 Nov 2017 17:13:16 -0500
> Subject: Re: [Cake] total download rate with many flows
> Yes, it is. I am building from the cobalt branch. What makes you believe 
> otherwise? Would you expect a different behaviour?
>
> On November 14, 2017 3:11:04 PM EST, Dave Taht <d...@taht.net> wrote:
>>
>> George Amanakis via Cake <cake@lists.bufferbloat.net> writes:
>>
>>>  From: George Amanakis <g_amana...@yahoo.com>
>>>  Subject: Re: [Cake] total download rate with many flows
>>>  To: David Lang <da...@lang.hm>
>>>  Cc: cake@lists.bufferbloat.net
>>>  Date: Mon, 13 Nov 2017 21:49:53 -0500 (17 hours, 20 minutes, 42 seconds 
>>> ago)
>>>
>>>  Dear David,
>>>
>>>  I agree. My point is that currently ingress mode seems to be dropping more
>>>  packets than necessary to keep senders from bottlenecking the connection 
>>> (when
>>>  there is a large number of concurrent flows, >8). And right now, ingress 
>>> mode is
>>>  the only mode that achieves this in situations such as Windows updates.
>>
>>
>> Is cobalt enabled in your build?
>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[Dave Taht]

George Amanakis via Cake <cake@lists.bufferbloat.net> writes:

> From: George Amanakis <g_amana...@yahoo.com>
> Subject: Re: [Cake] total download rate with many flows
> To: David Lang <da...@lang.hm>
> Cc: cake@lists.bufferbloat.net
> Date: Mon, 13 Nov 2017 21:49:53 -0500 (17 hours, 20 minutes, 42 seconds ago)
>
> Dear David,
>
> I agree. My point is that currently ingress mode seems to be dropping more
> packets than necessary to keep senders from bottlenecking the connection (when
> there is a large number of concurrent flows, >8). And right now, ingress mode 
> is
> the only mode that achieves this in situations such as Windows updates.

Is cobalt enabled in your build?
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[George Amanakis via Cake]

--- Begin Message ---

Dear David,

I agree. My point is that currently ingress mode seems to be dropping 
more packets than necessary to keep senders from bottlenecking the 
connection (when there is a large number of concurrent flows, >8). And 
right now, ingress mode is the only mode that achieves this in 
situations such as Windows updates.


George


On 11/13/2017 9:08 PM, David Lang wrote:
Ingres and Egress are fundamentally different due to the fact that in 
ingress mode you are having to throw away data that has successfully 
traversed the bottleneck (deliberatly wasting your limited resource 
now to avoid having senders bottleneck the queue on the far side of 
the link)


in Egress mode, you aren't doing that, and so you can get much closer 
to the actual bandwidth.


In addition, in Ingress mode, you are always working via second-order 
effects, you can't slow a transmission directly like you can in egress 
mode, all you can do is drop packets and wait until the sender 
notices, retransmits and slows down. Egress has full control of the 
queues and can send things in any order, and may be able to 
continually fill the pipe and avoid any timeouts and retransmissions 
(if the flow is short enough)


David Lang



--- End Message ---
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[David Lang]

Ingres and Egress are fundamentally different due to the fact that in ingress 
mode you are having to throw away data that has successfully traversed the 
bottleneck (deliberatly wasting your limited resource now to avoid having 
senders bottleneck the queue on the far side of the link)


in Egress mode, you aren't doing that, and so you can get much closer to the 
actual bandwidth.


In addition, in Ingress mode, you are always working via second-order effects, 
you can't slow a transmission directly like you can in egress mode, all you can 
do is drop packets and wait until the sender notices, retransmits and slows 
down. Egress has full control of the queues and can send things in any order, 
and may be able to continually fill the pipe and avoid any timeouts and 
retransmissions (if the flow is short enough)


David Lang

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[George Amanakis via Cake]

--- Begin Message ---

I meant proportionally to (1-1/sqrt(x)).


On 11/13/2017 8:51 PM, George Amanakis wrote:
I am exploring this idea further. If q->time_next_packet is 
incremented for dropped packets proportionally to (1-1/x), where x is 
the count of all flows in the tin that is being served, ingress mode 
works much more smoothly: latency is still <50ms and throughput is 
very near to the set limit.


I *tried* to make a patch from latest cobalt.

=8<=
diff --git a/sch_cake.c b/sch_cake.c
index 82f264f..752783a 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -145,6 +145,7 @@ struct cake_flow {
    struct list_head  flowchain;
    s32   deficit;
    struct cobalt_vars cvars;
+   struct cobalt_vars cvars2;
    u16   srchost; /* index into cake_host table */
    u16   dsthost;
    u8    set;
@@ -254,6 +255,7 @@ struct cake_sched_data {
    u32 avg_window_bytes;
    u32 avg_peak_bandwidth;
    u64 last_reconfig_time;
+   u32 drop_len;
 };

 enum {
@@ -820,7 +822,7 @@ static unsigned int cake_drop(struct Qdisc *sch, 
struct sk_buff **to_free)

    sch->qstats.drops++;

    if(q->rate_flags & CAKE_FLAG_INGRESS)
-   cake_advance_shaper(q, b, cake_overhead(q, len), now);
+   q->drop_len += len;

 #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 8, 0)
    kfree_skb(skb);
@@ -1274,7 +1276,9 @@ retry:
    /* drop this packet, get another one */
    if(q->rate_flags & CAKE_FLAG_INGRESS) {
    len = cake_overhead(q, qdisc_pkt_len(skb));
-   cake_advance_shaper(q, b, len, now);
+   flow->cvars2.count = 
b->bulk_flow_count+b->sparse_flow_count+b->decaying_flow_count+b->unresponsive_flow_count;

+   cobalt_invsqrt(&(flow->cvars2));
+   q->drop_len += (len - reciprocal_scale(len, 
flow->cvars2.rec_inv_sqrt));

    flow->deficit -= len;
    b->tin_deficit -= len;
    }
@@ -1286,8 +1290,6 @@ retry:
    qdisc_qstats_drop(sch);
    kfree_skb(skb);
 #endif
-   if(q->rate_flags & CAKE_FLAG_INGRESS)
-   goto retry;
    }

    b->tin_ecn_mark += !!flow->cvars.ecn_marked;
@@ -1340,7 +1342,7 @@ static void cake_advance_shaper(struct 
cake_sched_data *q, struct cake_tin_data

    if(q->rate_ns) {
    s64 tdiff1 = b->tin_time_next_packet - now;
    s64 tdiff2 = (len * (u64)b->tin_rate_ns) >> 
b->tin_rate_shft;

-   s64 tdiff3 = (len * (u64)q->rate_ns) >> q->rate_shft;
+   s64 tdiff3 = ((q->drop_len + len) * (u64)q->rate_ns) 
>> q->rate_shft;


    if(tdiff1 < 0)
    b->tin_time_next_packet += tdiff2;
@@ -1348,6 +1350,7 @@ static void cake_advance_shaper(struct 
cake_sched_data *q, struct cake_tin_data

    b->tin_time_next_packet = now + tdiff2;

    q->time_next_packet += tdiff3;
+   q->drop_len = 0;
    }
 }

@@ -1711,6 +1714,7 @@ static void cake_reconfigure(struct Qdisc *sch)
 {
    struct cake_sched_data *q = qdisc_priv(sch);
    int c, ft;
+   q->drop_len=0;

    switch (q->tin_mode) {
    case CAKE_MODE_BESTEFFORT:
@@ -1941,6 +1945,7 @@ static int cake_init(struct Qdisc *sch, struct 
nlattr *opt)


    INIT_LIST_HEAD(>flowchain);
    cobalt_vars_init(>cvars);
+   cobalt_vars_init(>cvars2);

    q->overflow_heap[k].t = i;
    q->overflow_heap[k].b = j;

=8<=



On 11/11/2017 10:48 PM, George Amanakis wrote:
I totally understand what you are saying. However, I believe cake's 
egress and ingress modes currently behave as two extremes. One could 
argue that neither of them is the golden mean. With a patch in 
ingress mode (see below) and a single host using 32 flows to download 
I managed to increase throughput from ~7Mbps to ~10Mbps (configured 
limit 12200kbps) while latency increased from ~10ms to ~50ms, which 
would still be acceptable. As a comparison egress mode in the same 
setup gives me throughput ~11.5Mbps and latency ~500ms.


I would like to hear your thoughts about this idea: the patch is 
incrementing q->time_next_packet for dropped packets differently than 
for passed-through ones. Please focus on the idea, not the actual 
implementation :) (also pasted in https://pastebin.com/SZ14WiYw)


=8<=

diff --git a/sch_cake.c b/sch_cake.c
index 82f264f..a3a4a88 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -769,6 +769,7 @@ static void cake_heapify_up(struct 
cake_sched_data *q, u16 i)

 }

 static void cake_advance_shaper(struct cake_sched_data *q, struct 
cake_tin_data *b, u32 len, u64 now);
+static 

Re: [Cake] total download rate with many flows

[George Amanakis via Cake]

--- Begin Message ---
I am exploring this idea further. If q->time_next_packet is incremented 
for dropped packets proportionally to (1-1/x), where x is the count of 
all flows in the tin that is being served, ingress mode works much more 
smoothly: latency is still <50ms and throughput is very near to the set 
limit.


I *tried* to make a patch from latest cobalt.

=8<=
diff --git a/sch_cake.c b/sch_cake.c
index 82f264f..752783a 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -145,6 +145,7 @@ struct cake_flow {
    struct list_head  flowchain;
    s32   deficit;
    struct cobalt_vars cvars;
+   struct cobalt_vars cvars2;
    u16   srchost; /* index into cake_host table */
    u16   dsthost;
    u8    set;
@@ -254,6 +255,7 @@ struct cake_sched_data {
    u32 avg_window_bytes;
    u32 avg_peak_bandwidth;
    u64 last_reconfig_time;
+   u32 drop_len;
 };

 enum {
@@ -820,7 +822,7 @@ static unsigned int cake_drop(struct Qdisc *sch, 
struct sk_buff **to_free)

    sch->qstats.drops++;

    if(q->rate_flags & CAKE_FLAG_INGRESS)
-   cake_advance_shaper(q, b, cake_overhead(q, len), now);
+   q->drop_len += len;

 #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 8, 0)
    kfree_skb(skb);
@@ -1274,7 +1276,9 @@ retry:
    /* drop this packet, get another one */
    if(q->rate_flags & CAKE_FLAG_INGRESS) {
    len = cake_overhead(q, qdisc_pkt_len(skb));
-   cake_advance_shaper(q, b, len, now);
+   flow->cvars2.count = 
b->bulk_flow_count+b->sparse_flow_count+b->decaying_flow_count+b->unresponsive_flow_count;

+   cobalt_invsqrt(&(flow->cvars2));
+   q->drop_len += (len - reciprocal_scale(len, 
flow->cvars2.rec_inv_sqrt));

    flow->deficit -= len;
    b->tin_deficit -= len;
    }
@@ -1286,8 +1290,6 @@ retry:
    qdisc_qstats_drop(sch);
    kfree_skb(skb);
 #endif
-   if(q->rate_flags & CAKE_FLAG_INGRESS)
-   goto retry;
    }

    b->tin_ecn_mark += !!flow->cvars.ecn_marked;
@@ -1340,7 +1342,7 @@ static void cake_advance_shaper(struct 
cake_sched_data *q, struct cake_tin_data

    if(q->rate_ns) {
    s64 tdiff1 = b->tin_time_next_packet - now;
    s64 tdiff2 = (len * (u64)b->tin_rate_ns) >> 
b->tin_rate_shft;

-   s64 tdiff3 = (len * (u64)q->rate_ns) >> q->rate_shft;
+   s64 tdiff3 = ((q->drop_len + len) * (u64)q->rate_ns) >> 
q->rate_shft;


    if(tdiff1 < 0)
    b->tin_time_next_packet += tdiff2;
@@ -1348,6 +1350,7 @@ static void cake_advance_shaper(struct 
cake_sched_data *q, struct cake_tin_data

    b->tin_time_next_packet = now + tdiff2;

    q->time_next_packet += tdiff3;
+   q->drop_len = 0;
    }
 }

@@ -1711,6 +1714,7 @@ static void cake_reconfigure(struct Qdisc *sch)
 {
    struct cake_sched_data *q = qdisc_priv(sch);
    int c, ft;
+   q->drop_len=0;

    switch (q->tin_mode) {
    case CAKE_MODE_BESTEFFORT:
@@ -1941,6 +1945,7 @@ static int cake_init(struct Qdisc *sch, struct 
nlattr *opt)


    INIT_LIST_HEAD(>flowchain);
    cobalt_vars_init(>cvars);
+   cobalt_vars_init(>cvars2);

    q->overflow_heap[k].t = i;
    q->overflow_heap[k].b = j;

=8<=



On 11/11/2017 10:48 PM, George Amanakis wrote:
I totally understand what you are saying. However, I believe cake's 
egress and ingress modes currently behave as two extremes. One could 
argue that neither of them is the golden mean. With a patch in ingress 
mode (see below) and a single host using 32 flows to download I 
managed to increase throughput from ~7Mbps to ~10Mbps (configured 
limit 12200kbps) while latency increased from ~10ms to ~50ms, which 
would still be acceptable. As a comparison egress mode in the same 
setup gives me throughput ~11.5Mbps and latency ~500ms.


I would like to hear your thoughts about this idea: the patch is 
incrementing q->time_next_packet for dropped packets differently than 
for passed-through ones. Please focus on the idea, not the actual 
implementation :) (also pasted in https://pastebin.com/SZ14WiYw)


=8<=

diff --git a/sch_cake.c b/sch_cake.c
index 82f264f..a3a4a88 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -769,6 +769,7 @@ static void cake_heapify_up(struct cake_sched_data 
*q, u16 i)

 }

 static void cake_advance_shaper(struct cake_sched_data *q, struct 
cake_tin_data *b, u32 len, u64 now);
+static void cake_advance_shaper2(struct cake_sched_data *q, struct 
cake_tin_data *b, u32 len, u64 

Re: [Cake] total download rate with many flows

[George Amanakis via Cake]

--- Begin Message ---
I totally understand what you are saying. However, I believe cake's 
egress and ingress modes currently behave as two extremes. One could 
argue that neither of them is the golden mean. With a patch in ingress 
mode (see below) and a single host using 32 flows to download I managed 
to increase throughput from ~7Mbps to ~10Mbps (configured limit 
12200kbps) while latency increased from ~10ms to ~50ms, which would 
still be acceptable. As a comparison egress mode in the same setup gives 
me throughput ~11.5Mbps and latency ~500ms.


I would like to hear your thoughts about this idea: the patch is 
incrementing q->time_next_packet for dropped packets differently than 
for passed-through ones. Please focus on the idea, not the actual 
implementation :) (also pasted in https://pastebin.com/SZ14WiYw)


=8<=

diff --git a/sch_cake.c b/sch_cake.c
index 82f264f..a3a4a88 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -769,6 +769,7 @@ static void cake_heapify_up(struct cake_sched_data 
*q, u16 i)

 }

 static void cake_advance_shaper(struct cake_sched_data *q, struct 
cake_tin_data *b, u32 len, u64 now);
+static void cake_advance_shaper2(struct cake_sched_data *q, struct 
cake_tin_data *b, u32 len, u64 now);


 #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 8, 0)
 static unsigned int cake_drop(struct Qdisc *sch)
@@ -1274,7 +1275,7 @@ retry:
    /* drop this packet, get another one */
    if(q->rate_flags & CAKE_FLAG_INGRESS) {
    len = cake_overhead(q, qdisc_pkt_len(skb));
-   cake_advance_shaper(q, b, len, now);
+   cake_advance_shaper2(q, b, len, now);
    flow->deficit -= len;
    b->tin_deficit -= len;
    }
@@ -1286,8 +1287,6 @@ retry:
    qdisc_qstats_drop(sch);
    kfree_skb(skb);
 #endif
-   if(q->rate_flags & CAKE_FLAG_INGRESS)
-   goto retry;
    }

    b->tin_ecn_mark += !!flow->cvars.ecn_marked;
@@ -1351,6 +1350,24 @@ static void cake_advance_shaper(struct 
cake_sched_data *q, struct cake_tin_data

    }
 }

+static void cake_advance_shaper2(struct cake_sched_data *q, struct 
cake_tin_data *b, u32 len, u64 now)

+{
+   /* charge packet bandwidth to this tin, lower tins,
+    * and to the global shaper.
+    */
+   if(q->rate_ns) {
+   s64 tdiff1 = b->tin_time_next_packet - now;
+   s64 tdiff2 = (len * (u64)b->tin_rate_ns) >> 
b->tin_rate_shft;

+   s64 tdiff3 = (len * (u64)q->rate_ns) >> q->rate_shft;
+
+   if(tdiff1 < 0)
+   b->tin_time_next_packet += tdiff2;
+   else if(tdiff1 < tdiff2)
+   b->tin_time_next_packet = now + tdiff2;
+
+   q->time_next_packet += (tdiff3*27)>>5;
+   }
+}
 static void cake_reset(struct Qdisc *sch)
 {
    u32 c;

=8<=

On 11/10/2017 4:50 PM, Jonathan Morton wrote:


In fact, that's why I put a failsafe into ingress mode, so that it 
would never stall completely.  It can happen, however, that throughput 
is significantly reduced when the drop rate is high.


If throughput is more important to you than induced latency, switch to 
egress mode.


Unfortunately it's not possible to guarantee both low latency and high 
throughput when operating downstream of the bottleneck link.  ECN 
gives you better results, though.


- Jonathan Morton



--- End Message ---
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[Jonathan Morton]

In fact, that's why I put a failsafe into ingress mode, so that it would
never stall completely.  It can happen, however, that throughput is
significantly reduced when the drop rate is high.

If throughput is more important to you than induced latency, switch to
egress mode.

Unfortunately it's not possible to guarantee both low latency and high
throughput when operating downstream of the bottleneck link.  ECN gives you
better results, though.

- Jonathan Morton
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[George Amanakis via Cake]

--- Begin Message ---
Indeed. Also interplanetary essentially omits cake_advance_shaper for 
dropped packets (since cobalt never drops that way), almost disabling 
ingress mode. Which leads to other hosts having pings to WAN > 500ms 
when one of them is using many flows.


What I think is happening in ingress mode is that codel starts dropping 
packets in many flows simultaneously. Each time this happens, 
q->time_next_packet is advanced, eventually dis-proportionally to the 
actual rate, and this leads to the actual download rate being lower than 
set.


I am a big fan of ingress mode, just if we could correct this behaviour...


On 11/10/2017 3:42 PM, Dave Taht wrote:

Interplanetary basically disables codel for most applications.


On 11/1/2017 6:01 PM, Dave Taht wrote:

Awesome. thx. I will try to make time to look at these over the weekend.

I still haven't had that weekend.



--- End Message ---
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[Dave Taht]

Interplanetary basically disables codel for most applications.

> On 11/1/2017 6:01 PM, Dave Taht wrote:
>> Awesome. thx. I will try to make time to look at these over the weekend.

I still haven't had that weekend.

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[George Amanakis via Cake]

--- Begin Message ---

I did more testing regarding this issue.

I found out that if I use "interplanetary", I do not observe the 
behaviour I mentioned before. I still need to test the responsiveness of 
other hosts while one of them is using too many flows with this setting.


Setup is the same, cake configuration was:

qdisc cake 8007: dev ens4 root refcnt 2 bandwidth 12200Kbit besteffort 
dual-dsthost wash ingress rtt 3600.0s noatm overhead 18 via-ethernet mpu 64
qdisc cake 8008: dev ens3 root refcnt 2 bandwidth 2500Kbit besteffort 
dual-srchost nat wash rtt 3600.0s noatm overhead 18 via-ethernet mpu 64



On 11/1/2017 6:01 PM, Dave Taht wrote:

Awesome. thx. I will try to make time to look at these over the weekend.


--- End Message ---
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[Dave Taht]

Awesome. thx. I will try to make time to look at these over the weekend.
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[George Amanakis via Cake]

--- Begin Message ---

Dear Dave,

I could get the following dumps:

*lan.cake.pcap
captured on router, lan iface, with cake

#lan.cake.ping.pcap
captured on router, lan iface, with cake, ping from host other than w10 
update


$lan.nocake.pcap
captured on router, lan iface, without cake

*wan.cake.pcap
captured on router, wan iface, with cake

#wan.cake.ping.pcap
captured on router, wan iface, with cake, ping from host other than w10 
update


$wan.nocake.pcap
captured on router, wan iface, without cake

*win10host.cake.pcapng
captured on w10 updating, with cake

#win10host.cake.ping.pcapng
captured on w10 updating, with cake, ping from another host

$win10host.nocake.pcapng
captured on w10 updating, without cake

===

ping to google.com, which resolved to an IPv6 address.
The signs(*,#,$) denote packet captures done (nearly) in parallel.

LAN:
qdisc cake 8001: dev ens4 root refcnt 2 bandwidth 12200Kbit besteffort 
dual-dsthost wash ingress rtt 100.0ms noatm overhead 18 via-ethernet mpu 64


WAN:
qdisc cake 8002: dev ens3 root refcnt 2 bandwidth 2500Kbit besteffort 
dual-srchost nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64


===

Links:
lan.cake.pcap
https://drive.google.com/open?id=0B2bwzgme1zsdbS1uZVdaWXRpVTg

lan.cake.ping.pcap
https://drive.google.com/open?id=0B2bwzgme1zsdd3lwamJFZ0hIZ1E

lan.nocake.pcap
https://drive.google.com/open?id=0B2bwzgme1zsdUzNFZXdOaHVmX3c

wan.cake.pcap
https://drive.google.com/open?id=0B2bwzgme1zsdaURjQ2RuZC1xcU0

wan.cake.ping.pcap
https://drive.google.com/open?id=0B2bwzgme1zsda3FpQmpfQ0M0Nm8

wan.nocake.pcap
https://drive.google.com/open?id=0B2bwzgme1zsdeDhlVWx5OTVhQ28

win10host.cake.pcapng
https://drive.google.com/open?id=0B2bwzgme1zsdMi1nTnlHcnBZZUU

win10host.cake.ping.pcapng
https://drive.google.com/open?id=0B2bwzgme1zsdQmJ1eTEwZ1RKcWc

win10host.nocake.pcapng
https://drive.google.com/open?id=0B2bwzgme1zsdUTJ5OWZsYnlLU1k


Best,
George
--- End Message ---
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[George Amanakis via Cake]

--- Begin Message ---
I think I have another install not updated to "Fall Creators Update". I 
will try and get a capture using Wireshark. Or should I rather capture 
on the router?


George

On 10/31/2017 9:06 PM, Antoine Deschênes wrote:
Fall Creators update, killed my whole internet connection for 1 or 2 
hours two times (two systems to update).


Even the Google homepage would timeout using cake (or fq_codel or no 
sqm at all), until I figured out the dual-src/dsthost flags in 
sqm-scripts.


It looks like it had a LOT of simultaneous flows... Maybe they got the 
P2P update distribution working


Le 31 oct. 2017 8:45 PM, "Dave Taht" > a écrit :



I would *dearly* love a packet capture of a windows 10 update in this
case.

___
Cake mailing list
Cake@lists.bufferbloat.net 
https://lists.bufferbloat.net/listinfo/cake




--- End Message ---
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] total download rate with many flows

[Dave Taht]

I would *dearly* love a packet capture of a windows 10 update in this
case.

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake