AQM folks,
We've shown that if you solely focus on AQM, TCP becomes the limit on
how much you can reduce delay - loss not queuing starts to become the
dominant cause of delay at high load. If you weren't in the IETF AQM
sesssion in Prague, see "questioning a fixed delay target
<https://www.ietf.org/proceedings/93/slides/slides-93-aqm-4.pdf>" or the
tech report <http://www.bobbriscoe.net/projects/latency/credi_tr.pdf>
that backs it up.
We also demonstrated developments (see URLs at end) that would make it
possible to deploy scalable low-latency low-loss protocols like Data
Center TCP alongside a mix of traffic, either in data centres and
private networks, or even on the public Internet. One approach was
demonstrated at the recent IETF in Prague, showing DCTCP giving
ultra-low latency (1ms at the 98th percentile) over a broadband Internet
access while competing with a mix of Internet traffic on roughly equal
terms.
As a result of an ad hoc meeting ("Bar BoF" = Birds of a Feather) at the
Prague IETF, we have formed a new mailing list.
I'd like to invite you to join the list via:
<https://www.ietf.org/mailman/listinfo/tcpprague>
The idea is to ensure that those working on DCTCP implementations across
platforms (Free BSD, Linux, Windows, ...) will converge on solutions
that will interwork with each other and with existing traffic. Although
it is under the IETF's umbrella, we hope and expect that discussion will
be as much about implementation as writing standards. However, we get
the benefit of the IETF's IPR disclosure rules
<https://www.ietf.org/about/note-well.html>, and of course it fits the
IETF's purpose of interoperability.
The draft notes of the meeting are below.
And below that, is the original announcement with some context and
background URLs.
You can catch up on any discussion you've missed using the list archives
via the link above.
If you want to respond about something most relevant to tcpprague, pls
avoid cross-posting to the bloat list as well.
Cheers
Bob Briscoe
-------- Forwarded Message --------
Subject: Notes: DCTCP evolution 'bar BoF': Tue 21 Jul 2015, 17:40, Prague
Date: Tue, 28 Jul 2015 14:00:46 +0100
From: Bob Briscoe <[email protected]>
To: TCP Prague IETF List <[email protected]>
Folks,
These notes have taken a week, because I've only just put my machine
back together after having to rebuild the hardware a little :|
_*Notes: DCTCP Evolution Bar BoF*_
6-7pm Tue 21 Jul 2015, Budapest room, The Hilton, Prague, CZ
**Summary of Actions:*
*Lars E: Set up tcpprague wiki page
Bob B: Request [email protected] mailing list, via IETF process
(requires Area Director approval)
Bob B: Document Rationale - initiate a para on wiki.
Lars E: fwd Dagstuhl invitee list to Bob
Bob B: Set up list on wiki to assign people to invite those not in the
room to join.
*
* 18:00 Introductions - name and interest **
***Present:
Marcelo Bagnulo Braun <[email protected]>
Praveen Balasubramanian <[email protected]>
Martin Bekker <[email protected]>
Bob Briscoe <[email protected]>
Anna Brunstrom <[email protected]>
Stuart Cheshire <[email protected]>
Koen De Schepper <[email protected]>
Fabien Duchen <[email protected]>
Phil Eardley <[email protected]>
Lars Eggert <[email protected]>
Michio Honda <[email protected]>
Per Hurtig <[email protected]>
Jana Iyengar <[email protected]>
Naeem Khademi <[email protected]>
Mirja Kuehlewind <[email protected]>
Matt Mathis <[email protected] <mailto:[email protected]>>
Andrew McGregor <[email protected]>
Karen Nielsen <[email protected]>
Tommy Pauly <[email protected]>
Andreas Petlund <[email protected] <mailto:[email protected]>>
Costin Raiciu <[email protected]>
Pasi Sarolahti <[email protected]>
Richard Scheffenegger <[email protected]>
David Schinazi <[email protected]>
Randall Stewart <[email protected]>
Dave Thaler <[email protected]>
Brian Trammell <[email protected] <mIlto:[email protected]>>
Michael Tuexen <[email protected]>
Felix Weinrank <[email protected]>
Michael Welzl <[email protected]>
Alex Zimmermann <[email protected]>
** Scope and Agenda Bashing**
***
[Non-italic text is from the materials pre-prepared by Koen De Schepper
and Bob Briscoe.
/Italic text summarises conversation in the room./]
Meeting is covered by the standard IETF "Note Well" concerning
intellectual property.
*Scope*:
* Evolving the e2e DCTCP protocol for use alongside existing traffic
(whether in DCs, private nets or public Internet).
* Primarily to get DCTCP /developers/ involved early (Windows, FreeBSD,
Linux), so that whatever we decide to standardise can be implemented in
parallel
(Doing implementation and standardisation in series is not desirable,
in whichever order).
* Primarily an organisational meeting about creating a forum / community
to do this work, using people's experience to know what will work best.
*Not in Scope:***
* Network changes are not in scope unless they impact the list of
changes needed to DCTCP
* The in-network side of the solution (two approaches exist [DCttH
<http://www.bobbriscoe.net/projects/latency/dctth_preprint.pdf>, Judd15
<https://www.usenix.org/system/files/conference/nsdi15/nsdi15-paper-judd.pdf>],
others may follow).*
** Identifier of DCTCP-like traffic (please discuss by email, not in
this meeting)
/Lars E: Informational draft recording Microsoft's DCTCP should not be
stalled by this, as it has value of its own.
Unanimous agreement.
/
/Praveen S: Microsoft has offered a royalty free license for DCTCP IPR
<https://datatracker.ietf.org/ipr/2319/>./
/Karen N: Is DCTCP over a non-TCP transport (e.g. SCTP) in scope//?
Unanimous "Yes"//
/
/Outcome of discussion on the features of this DCTCP-like congestion
control that define this work://
/
1. /Must use ECN, but unlike RFC3168 ECN, marking is not merely
equivalent to drop,//
//so ECN signals can be more plentiful and sooner than drop./
2. /P//acket rate is proportional to 1/p//, where p is the ECN marking
probability. //
/
/Matt M: 1/p makes congestion control scale with the bandwidth, //by
making//the intensity of congestion control signals per RTT invariant//./
//
/Stuart Ch: Apple is turning on ECN by default in clients. Currently in
developer seeds but probably in the next releases. Packet loss is also
not a mystery./
** 18:15 List of /must-have/ changes before deployment alongside
existing traffic.**
***
/Matt M: Rather than a "MUST-have" list, produce a prioritised list,
because where to draw the necessity line could depend on the use-case.//
/
The following list wasn't formally prioritised in the meeting, but items
where some people questioned necessity are shifted down.
1. Fall back to Reno or Cubic behaviour on loss;
/For how long?//quick consensus: 1 RTT, but needs further
discussion. ECN response continues to operate in parallel./
2. Negotiate altered feedback semantics, to convey the extent of ECN
marking, not just its existence, and this feedback needs to be
robust to loss [RFC-to-be 7560
<https://datatracker.ietf.org/doc/draft-ietf-tcpm-accecn-reqs/>];
/Mirja K, Richard S & Bob B plan to have spec of much simpler
solution out soon. /
3. Use of a standardised packet identifier (if ECN-capable is not enough)
/Identifier tbd.
/*/- - - 8< - - - - - - - - highest line between "must-have for
safety" and "would be nice for performance" - - - - - - 8< - - - -/*
4. Handle a window of less than 2 when the RTT is low, rather than
increase the queue [TCP-sub-mss-cwnd
<https://www.ietf.org/proceedings/93/slides/slides-93-iccrg-5.pdf>],
like TCP Nice
<http://www.cs.utexas.edu/users/dahlin/software/2002-nice.html>.
/Michael W: Is this "must-have"? Quite a complicated step. //
//Bob B: Yes, but, otherwise DCTCP will pollute ultra-low latency
queues from the start./
5. Average ECN feedback over its own RTT, not the hard-coded RTT
suitable only for data-centres, perhaps reduce cwnd by seg-size/2
per ECN Echo, like Relentless TCP [Mathis09
<staff.psc.edu/mathis/papers/PFLDNet.pdf>];
/???: How bad would long-RTT flows be? More generally, how can we
evaluate all this?/
/Bob B: With mixed RTTs, flows with RTT > a couple of ms will
respond too quickly to bursts.//Whatever, it's already been
implemented by Mohammad Alizadeh in Linux, and evaluated
<http://simula.stanford.edu/%7Ealizade/Site/DCTCP_files/dctcp_analysis-full.pdf>,
so this is easy./
6. Heuristic testing for classic ECN bottlenecks
//The idea would be to detect a 'classic' RFC316 bottleneck by
whether appreciable delay growth accompanies the marking (originally
suggested by Michael W).
/Bob B: Complex and slow to detect, so it would have to learn and
cache for new flows - suggest this should only be a must-have if
measurements prove it to be a problem - i.e. if a significant
proportion of classic ECN bottlenecks //exist//
//Matt M: No need for this - rate mismatch //no worse than TCP
already sees with RTT discrepancies./
/* - - - 8< - - - - - - - - lowest line between "must-have for
safety" and "would be nice for performance" - - - - - - 8< - - - -*/
7. /Costin R: //Faster-than-additive increase//(similar to Cubic)
//A performance improvement, not "must-have", but would be nice to
have while we're doing this./
8. /[Not discussed in the meeting, but added by Bob B for the record]:
Less drastic exit from slow-start, to match less drastic rate
reduction per mark.//
//Currently, because marking threshold is shallow, //slow start
exits earlier than with drop, unnecessarily increasing completion
time.//
/
/
//Costin R: Any other way to evolve towards DCTCP over mixed networks,
without separate queues in the network?//
////Bob B: To discuss on ML, and if we continue with the proposed
approach, we must record the rationale on the WIki.//
///
** 18:30 Brainstorm to identify people not present who will be important
to this.**
***
Mohammad Alizadeh <[email protected]>
Grenville Armitage <[email protected]>
Fred Baker <[email protected]>
Stephen Bensley <[email protected]>
Daniel Borkmann <[email protected]>
Yuchung Cheng <[email protected]>
Kenjiro Cho <[email protected]>
邓灵莉/Lingli Deng <[email protected]>
Eric Dumazet <[email protected]>
Gorry Fairhurst <[email protected]>
Jamal Hadi Salim <[email protected]>
Glenn Judd <[email protected]>
Midori Kato <[email protected]>
Kenneth Klette Jonassen <[email protected]
<mailto:[email protected]>> (already subscribed)
Sridharan, Murari <[email protected]>
Hiren Panchasara <[email protected]>
Hagen Pfeifer <[email protected]>
Balaji Prabhakar <[email protected]>
KK Ramakrishnan <[email protected]>
Lawrence Stewart <[email protected]>
Dave Taht <[email protected]>
Florian Westphal <[email protected]>
/Agreed to cc to the following for awareness, but no need to invite to
join the list://
/Stephen Hemminger <[email protected]>
David Miller <[email protected]>
/Missing types of organisations://
/
* /Network operators (not so relevant for e2e protocol, but need to be
motivated to deploy the network part)/
* /CDN//s/
/[Bob B adds: Subsequent to mtg, Erik Nygren tells me Xin Zhang leads
Akamai's congestion control team. Also I noticed Hiren used to work at
Limelight, so may have contacts]//
////
//Lars E: Co-organising a Dagstuhl retreat around DCTCP. Will forward
list of invitees to Bob to notify once the ML exists.//
//Also Lars's list of FreeBSD and Linux devs.//
///
** 18:40 What is the best way to ensure the outputs from a number of
separate developers all converge in parallel to standardisation?**
***/Common Test Suite//
//Interop//events
////Plugfests//
////Serving paths (e.g. Google's) capable of serving this/
** 18:50 Next steps: Actions to set up suitable MLs, tools, with
timesales etc.**
***
/Discussed pros and cons of hosting ML on ietf.org.//
//General agreement: use ietf.org for ML - because the IPR Note Well is
useful. //
//
/Name for ML? /
//Matt M: TCP Prague (for an evolving protocol, a meaningless tag is
best).//
//Karen N: ecn-prague, because it's not just TCP?//
//
//Agreed: //[email protected]////
//
//Actions://
//Bob B: ML - ask SpencerD/MartinS, following the documented process//
//Lars E: Set up wiki page - for assigning people to send out invitations//
///
** End 19:05**
***
Notes: Bob Briscoe, helped by Andrew McGregor
28 Jul 2015
-------- Forwarded Message --------
Subject: DCTCP evolution 'bar BoF': Tue 21 Jul 2015, 17:40, Prague
Date: Mon, 20 Jul 2015 22:46:14 +0100
From: Bob Briscoe <[email protected]>
To: Mirja Kuehlewind <[email protected]>, EGGERT, Lars
<[email protected]>, Dave Thaler <[email protected]>, Praveen
Balasubramanian <[email protected]>, Alex Zimmermann
<[email protected]>, Richard Scheffenegger
<[email protected]>, Fred Baker <[email protected]>, Matt Mathis
<[email protected]>, Andrew McGregor <[email protected]>, Dave
Taht <[email protected]>, Stuart Cheshire <[email protected]>,
Michael WELZL <[email protected]>, Andreas Petlund
<[email protected]>, Gorry Fairhurst <[email protected]>, Anna
Brunstrom <[email protected]>
CC: De Schepper, Koen (Koen) <[email protected]>
Folks,
DCTCP evolution 'bar BoF': Tue 21 Jul 2015, 17:40, Prague
Location: Unless I have emailed with a room location before then, pls
meet at the IETF reception.
Koen & I are trying to get together people in Prague who are involved in
development of DCTCP across platforms (Windows, Free BSD, Linux, etc),
and who are interested in evolving it for use on heterogeneous networks,
e.g.
* data centres with a mix of TCP flavours, not just DCTCP
* private networks
* the public Internet
Pls fwd this invite to anyone in Prague who ought to be involved that
I've missed (pls cc everyone else too).
Sorry for short notice.
One purpose of the session will be to build a community beyond the IETF,
so I'd like us to compose an email to a wider set of people after the
session, e.g.:
Stephen Bensley <[email protected]> <mailto:[email protected]>
Glenn Judd <[email protected]>
<mailto:[email protected]>
Daniel Borkmann <[email protected]>
<mailto:[email protected]>
Florian Westphal <[email protected]> <mailto:[email protected]>
邓 灵莉/Lingli Deng <[email protected]>
<mailto:[email protected]>
Mohammad Alizadeh <[email protected]> <mailto:[email protected]>
Stephen Hemminger <[email protected]>
<mailto:[email protected]>
David S. Miller <[email protected]> <mailto:[email protected]>
Sridharan, Murari <[email protected] <mailto:[email protected]>>
Yuchung Cheng <[email protected] <mailto:[email protected]>>
Koen & Bob
PS. Below is some background, and some agenda ideas. Pls discuss, bash
and add your own.
We've recently developed an AQM that allows DCTCP to co-exist with
Cubic/Reno etc. with zero config. Links below.
We have to add some necessary mechanisms to DCTCP (listed below) so it
will be safe alongside other traffic. Two questions:
Q1. We don't want to fork DCTCP. Does anyone think a fork optimised
for homogeneous DCTCP would be better?
Q2. Anyone interested in helping?
We have an idea how to do each one, but sharing the load would be
great - there's Linux, FreeBSD, Windows, etc. to cover.
List of the 4 essential 'safety' mods to DCTCP (copied from the IETF
Internet Draft linked below) and one that might need to be essential:
o fall back to Reno or Cubic behaviour on loss;
o negotiate its altered feedback semantics, which conveys the extent
of ECN marking, not just its existence, and this feedback needs to
be robust to loss [I-D.ietf-tcpm-accecn-reqs];
o handle a window of less than 2 when the RTT is low, rather than
increase the queue [TCP-sub-mss-w].
o average ECN feedback over its own RTT, not the hard-coded RTT
suitable only for data-centres, perhaps like Relentless
TCP [Mathis09];
o Use of a standardised packet identifier (if ECN-capable is not enough)
oHeuristic testing for classic ECN bottlenecks (optional?)
We're trying to move fast because if we can get on top of other
developments (e.g. Apple's recent release of ECN), it will mean less
messy classification code in the AQM.
So the links below are not on official sites yet.
‘Data Centre to the Home’: Ultra-Low Latency for All
<http://www.bobbriscoe.net/projects/latency/dctth_preprint.pdf>
Highlights:
* 1ms 99%-ile queuing delay for all DCTCP traffic in thousands of
expts incl. high load,
over an e2e test network with real broadband equipment.
* DCTCP co-existence with Reno & Cubic, with no transport ID inspection.
* less ops per packet than RED
* Zero config
IETF Draft to standardise those parts of the AQM relevant to
interop(not yet submitted to IETF):
<http://www.bobbriscoe.net/projects/latency/draft-briscoe-aqm-dualq-coupled-00.txt>
Koen & Bob
--
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/
_______________________________________________
aqm mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/aqm
