David,
Returning from a fortnight offlist...
I think your conception of how ECN works is incorrect. You describe
ECN as if the AQM marks one packet when it drops another packet. You
say that the ECN-mark speeds up the retransmission of the dropped
packet. On the contrary, the idea of classic ECN [RFC3168] is that
the ECN marks replace the drops. In all known testing (except
pathological cases), classic ECN effectively eliminates drops for all
ECN-capable packets.
Nonetheless, I do agree with your sentiment that the perfect is the
enemy of the good. We can remove most of the really bloat-induced
latency without ECN. So the message must be clear: Deploy AQM now. No
need to wait for ECN. But implementations SHOULD allow ECN packets to
be classifed into a separately configurable instance of the AQM algo. {Note 1}
Similarly, this WG made sure we did not deprecate RED in the AQM
recommendations. Because, in existing equipment, even a poorly tuned
RED is usually much better than a bloated buffer with no AQM.
Just as the WG mustn't confuse messages, you mustn't get confused by
a discussion about the potential for more ambitious reductions in
latency. Koen started the thread with reference to our presentation
in the ICCRG in the IRTF, where 'R' in both cases stands for Research.
And I believe it is valid for the ECN benefits draft (in the IETF AQM
WG) to point to the potential of ECN, by using insights from research
in progress.
There are a number of different contributions to unnecessary latency,
not just the one popularised as bufferbloat:
[In the following, I will use the term 'short message(s)' as
shorthand for either a short interactive flow or a long flow
consisting of short interactive messages (like in a game), or video
frames or voice datagrams or anything where the perceived latency
depends on the latency of each 'message', especially a string of
messages with serial dependency as is typical in Web.]
#1 The well-known bufferbloat problem, where a long-running flow
fills a bloated buffer, delays short messages.
- AQM and/or flow queuing can remove this delay, without needing ECN.
#2 A loss causes head-of-line blocking for short message(s) while
waiting for the retransmission.
- AQM without ECN cannot remove this delay.
- Flow queuing cannot remove this delay, if the losses are self-induced.
- FEC can remove this delay without needing ECN, but the increased
redundancy is equivalent to poorer utilisation (altho only the short
flows need the redundancy).
- ECN can remove this delay.
#3 A loss near the end of a short flow can lead to multi-RTT delay.
- AQM without ECN cannot remove this delay.
- Techniques like tail-loss probe and RTO-restart can mitigate
this delay, without ECN, but not remove it.
- ECN can remove this delay.
#4 The Reno/Cubic sawtooth causes variation in delay between 1 and 2
base RTTs. This can affect short messages.
- AQM without ECN cannot remove this delay unless configured to
sacrifice utilisation.
- Flow queuing removes this delay variation if caused by a
separate long-running flow.{Note 2}
- A change to the TCP algo (e.g. DCTCP) can remove this delay
variation. Smaller sawteeth imply a much higher signalling rate,
which in turn requires ECN, otherwise drop probability would be
excessive. (This was the main point Koen was making.)
- Therefore, ECN can remove this delay.
#5 Slow-starts{Note 3} cause spikes in delay.
- AQM without ECN cannot remove this delay, and typically AQM is
designed to allow such bursts of delay in the hope they will
disappear of their own accord.
- Flow queuing can remove the effect of these delay bursts on
other flows, but only if it gives all flows a separate queue from the
start.{Note 2}
- Delay-based softening of slow-start, such as Hybrid Slow-Start
in Linux, can mitigate these variations, but with increased risk of
coming out of SS early, causing significantly longer completion time.
- ECN with AQM based on the instantaneous queue limits this delay,
without the risk of longer completion time.
#6 Slow-starts{Note 3} can cause runs of losses, which in turn cause delays.
- AQM without ECN cannot remove these delays.
- Flow queuing cannot remove these losses, if self-induced.
- Delay-based SS like HSS can mitigate these losses, with
increased risk of longer completion time.
- ECN can remove these losses, and the consequent delays.
Summary:
* AQM alone solves the main problem
* Flow queuing solves or mitigates most of the remaining secondary problems.
* ECN has the potential to solve all the remaining secondary problems
(pending further research to prove some of them).
Whether flow queuing is applicable depends on the scale. The work I'm
doing with Koen is to reduce the cost of the queuing mechanisms on
our BNGs (broadband network gateways). We're trying to reduce the
cost of per-customer queuing at scale, so per-flow queuing is simply
out of the question. Whereas ECN requires no more processing than drop.
ECN has potential cheating problems, but we have per-customer queues anyway.
Using flow as the unit of allocation also has its own problems, with
no proposed solutions.
Bob
{Note 1}: Your general point is that the perfect can be the enemy of
the good. Here's the presentation I gave in TSVAREA straight after VJ
presented CoDel in 2012 entitled "DCTCP & CoDel; the Best is the
Friend of the Good."
<http://www.bobbriscoe.net/presents/1207ietf/1207-tsvarea-dctcp.pdf>
{Note 2}: A lone flow can cause this delay variation to itself, but
that's irrelevant because, if the delay were not in the network it
would be at the sender.
{Note 3}: Delay and loss spikes can equivalently be caused when
Cubic's window rises to seek out newly available capacity after
another flow finishes or the link rate varies.
At 05:16 30/03/2015, David Lang wrote:
On Sat, 28 Mar 2015, Scheffenegger, Richard wrote:
David,
Perhaps you would care to provide some text to address the
misconception that you pointed out? (To wait for a 100% fix as a
90% fix appears much less appealing, while the current state of art is at 0%)
Ok, you put me on the spot :-) Here goes.
If you think that aqm-recommendations is not strogly enough worded.
I think this particular discussion (to aqm or not) really belongs
there. The other document (ecn benefits) has a different target in
arguing for going those last 10%...
so here is my "elevator pitch" on the problem. Feel free to take
anything I say here for any purpose, and I'm sure I'll get corrected
for anything I am wong on
Problem statement: Transmit buffers are needed to keep the network
layer fully utilized, but excessive buffers result in poor latency
for all traffic. This latency is frequently bad enough to cause some
types of traffic to fail entirely.
<link to more background goes here, including how separate
benchmarks for throughput and latency have mislead people, "packet
loss considered evil", cheaper memory encouraging larger buffers,
etc. Include tests like netperf-wrapper and ping latency while
under load, etc. Include examples where buffers have resulted in
latencies so long that packets are retransmitted before the first
copy gets to the destination>
Traditionally, transmit buffers have been sized to handle a fixed
number of packets. Due to teh variation in packet sizes, it is
impossible to tune this value to both keep the link fully utilized
when small packets dominate the trafific without having the queue
size be large enough to cause latency problems when large packets
dominate the traffic.
Shifting to Byte Queue Lengths where queues are allowed to hold a
variable number of packets depending on how large they are makes it
possible to manually tune the transmit buffer size to get good
latency under all traffic conditions at a given speed. However, this
step forward revealed two additional problems.
1. whenever the data rate changes, this value needs to be manually
changed (multi-link paths loose a link, noise degrades max
throughput on a link, etc)
2. high volume flows (i.e. bulk downloads) can starve other flows
(DNS lookups, VoIP, Gaming, etc). this happens because space in tue
queue is on a first-com-first-served basis, so the high-volume
traffic fills the queue (at which point it starts to be dropped),
but all other traffic that tries to arrive is also dropped. It turns
out that these light flows tend to have a larger effect on the user
experience than heavier flows, because things tend to be serialized
behind the lighter flows (DNS lookup before doing a large download,
retrieving a small HTML page to find what additional resources need
to be fetched to display a page), or the user experience is directly
effected by light flows (gaming lag, VoIP drops, etc)
Active Queue Management addresses these problems by adapting the
amount of data that is buffered to match the data transmission
capacity, and prevents high volume flows from starving low-volume
flows without the need to implement QoS classifications.
<insert link about how you can't trust QoS tags that are made by
other organizations, ways that it can be abused, etc>
This is possible because AQM algoithms don't have to drop the new
packet that arrives, the algorithm can decide to drop the packet for
one of the heavy flows rather than for one of the lightweight flows.
<insert references to currently favored AQM options here, PIE,
fq_codel, cake, ???. Also links to failed approaches>
Turning on aqm on every bottleneck link makes the Internet usable
for everyone, no matter what sort of application they are using.
<insert link on how to deal with equipment you can't configure by
throttling bandwidth before the bottleneck oand/or doing ingress
shaping of traffic>
While AQM makes the network usable, there is still additional room
for improvement. While dropping packets does result in the TCP
senders slowing down,and eventually stabilizing at around the right
speed to keep the link fully utilized, the only way that senders
have been able to detect problems is to discover that they have not
received an ack for the traffic within the allowed time. This causes
a 'bubble' in the flow as teh dropped packet must be retransmitted
(and sometimes a significant amount of data after the dropped packet
that did make it to the destination, but could not be acked because
fo the missing packet).
This "bubble" in the data flow can be greatly compressed by
configuring the AQM algorithm to send an ECN packet to the sender
when it drops a packet in a flow. The sender can then adapt faster,
slowing down it's new data, and re-sending the dropped packet
without having to wait for the timeout. This has two major effects
by allowing the sender to retransmit the packet sooner the dealy on
the dropped data is not as long, and because the replacement data
can arrive before the timeout of the following packets, they may not
need to be re-sent. by configuring the AQM algorithm to send the ECN
notification to the sender only when the packet is being dropped,
the effect of failure of the ECN packet to get through to the sender
(the notification packet runs into congestion and gets dropped, some
network device blocks it, etc) is that the ECN enabled case devolves
to match the non-ECN case in that the sender will still detect the
dropped packet via the timeout waiting for the ack as if ENCN was not enabled.
<insert link to possible problems that can happen here, including
the potential for an app to 'game' things if packets are marked at a
different level than when they are dropped.>
So a very strong recommendation to enable Active Queue Management,
while the different algorithms have different advantages and levels
of testing, even the 'worst' of the set results in a night-and-day
improvement for usability compared to unmanaged buffers.
Enabling ECN at the same point as dropping packets as part of
enabling any AQM algorithm results in a noticable improvement over
the base algorithm without ECN. When compared to the baseline, the
improvement added by ECN is tiny compared to the improvement from enabling AQM.
Is it fair to say that plain aqm vs aqm+ecn variation is on the
same order of difference as the differences between the different
AQM algorithms?
Future research items (which others here may already have done, and
would not be part of my 'elevator pitch')
I believe that currently ECn triggers the exact same slowdown that a
missed packet does, and it may be appropriate to have the sender do
a less drastic slowdown.
It would be very interesing to provide soem way for the application
sending the traffic to detect dropped packets and ECN responses. For
example, a streaming media source (especially an interactive one
like video conferencing) could adjust the bitrate that it's sending.
David Lang
_______________________________________________
aqm mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/aqm
________________________________________________________________
Bob Briscoe, BT
_______________________________________________
aqm mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/aqm
