Hi Nicholas and other authors, Thank you for producing this document. I had some (long) comments where I have also suggested some solutions/text, hopefully you find them helpful. When quoting from the document, I have used quotes and I have listed my comments as 1-4.
1. The number of tests recommended per AQM scheme seems to be quite large as well as many tests that may be best done together are done independently. For example, there are different tests for fairness between flows (section 3.2.3), level of traffic (mild, medium, heavy) (section 3.2.4), different types of bursty traffic (3.2.2) and different network environments (section 3.3). In order to obtain a realistic estimate of AQM comparison, many of these effects may need to be tested together. For example to increase level of traffic (in section 3.2.4) only long term TCP flows of 50s are proposed. It would be better to test the level of congestion with realistic traffic (and with realistic proportions) as proposed in section 3.2.4. Another example would be to test fairness between flows of same type - e.g. long term TCP flows when multiple types of TCP flows are present at a AQM controlled buffer or test fairness between flows of different types - e.g. long term TCP flows and web flows. I noticed that in section 3.3 different network environments are mentioned. I suggest that tests with appropriate realistic traffic be done on each of these enviroments with various levels of congestion. One could measure fairness from these tests without the need for additional tests. This may have the benefit of reducing overall number of tests as well. 2. There are four network enviroments mentioned in the document (section 3.3): Wifi - in home network. Data-Center communications. BB wireless/satellite. Low and high buffers This is a good set, however, recommed to add: Access networks - and further split access networks into DSL/FTTH networks and Cable access networks. Cable access networks may have much larger number of flows than DSL/FTTH since they are shared by a large number of customers There is another type of network environment - narrowband wireless (mobile) networks where the AQM could be placed on the eNodeB, but perhaps this is a complicated scenario, and best left for later or in another document. 3. I noticed that the modeling of different types of realistic traffic could be made more accurate (section 3.2.2) "Bulk TCP transfer: (continuous file transmission, or repeating 5MB file transmission);" The largest type of traffic on the Internet that uses bulk TCP - or large files via TCP is progressive download. The average speed of a Youtube video is over 3Mbps and it is on average 4.5 minutes long - so the average file size is considerably larger than 5MB. We have used an average of 38MB assuming 480p video and 4.5 minutes for some of our experiments. The inter-arrival time of these files should have a realistic random distribution as well as size of the files. "Realistic HTTP web traffic (repeated download of 700kB);" According to statistics that Google collected from 2010 about web traffic, the average GET file size for web traffic was only 7KB. It is very important to have the size of these files correct since AQMs can help transfers of small files when large file downloads are also occurring. The effect of AQM on a 7KB file would be very different than on a 700KB file. Web traffic also has a complex inter-arrival time relationship between the GET requests as well as the pages that produce these GET requests. However to make the testing simpler, a repeated download of a 7KB file with a Pareto distributed size and Pareto distributed inter-arrival time may be an OK compromise. I noticed that adaptive streaming is not modelled as a type of realistic traffic. It is probably the most important and largest flow on the Internet today - if we look at a recent Sandvine report of Internet traffic, about 75% of Internet traffic is real-time entertainment of which the largest component is adaptive streaming based video traffic. In order to simulate adaptive streaming traffic, one would need a client that changes video quality rates. To keep testing simpler, an option may be to generate realistic chunk sized files at regular intervals. Average netflix speeds are around 2Mbps. Typical chunk sizes range from 2s to 10s, but 4s may be a good average. The chunk sizes should be varied with some random distribution as is typically observed from real codecs. 4. In section 2.2.5, QoE metrics are mentioned. It may be valuable to spell out some standard models or expressions to derive QoE from network parameters so that these QoE can be compared between AQMs. In the literature there is work that derives closed form expressions for QoE. The following are examples and suggestions: Video - the work by Johan de Vriendt et al , "Model for estimating QoE of Video delivered using HTTP Adaptive Streaming", IFIP/IEEE Workshop on QoE Centric Management, May 2013 may be useful here. The authors used over 500 samples to derive their a closed form expressions. The authors derive the MOS = alpha*(average bit rate) + beta*(standard deviation of bit rates) + sigma*(no of bit rate switches) + gamma and then produce good values for alpha, beta, sigma and gamma. Voice - The work in "Voice quality prediction models and their application in VoIP networks" by L. Sun et al has closed form expressions for MOS scores for most commercial codes. They derive a long closed form expression with parameters a-j for key codecs (and then provide a-j for most commercial codecs) based on end-to-end and packet loss. Web - The delay for one average web GET file could be used as the representative for the page load time Key metric for AQMs on access networks may be QoE since it provides and end-user view. We have seen that often the absolute goodput does not matter as much as the stability of the flow for Video (which tends to be the dominant traffic). Larger goodput may provide higher VQ levels for Video, but if higher volatility is also produced then that higher VQ level may not produce a higher QoE. On the other hand, best metrics inter data center traffic may be goodput and delay since most of that traffic is likely to be inter machine traffic. Best regards, Shahid Akhtar. -----Original Message----- From: aqm [mailto:[email protected]] On Behalf Of Nicolas KUHN Sent: Friday, February 07, 2014 5:58 AM To: Richard Scheffenegger Cc: [email protected] Subject: [aqm] [AQM Evaluation Guidelines] Dear all, On the behalf of the contributors to the AQM Evaluation Guidelines, I encourage active discussion on the draft that we have written. I just submitted the draft as an individual draft to the I-D Submission Tool [0]. Please let us know what you think of the current document. Kind regards, Nicolas KUHN [0] http://datatracker.ietf.org/doc/draft-kuhn-aqm-eval-guidelines/ On Feb 5, 2014, at 10:01 PM, "Scheffenegger, Richard" <[email protected]> wrote: > Hi, > > a new month, a new status report. > > First of all, Wes and I as chairs would like to thank the editors who have > stepped forward to work on the AQM Evaluation Guideline draft. We are really > thankful for their burst of efforts in the last couple weeks! > > We expect that that a document will be ready for submission into the I-D > repository well before cutoff, as an individual draft, so that the WG can see > what the state of thinking is currently. Also, once the document is published > on datatracker we'd like to encourage active discussion on it. > > If the submitted -00 draft is felt to have a fairly complete outline of what > the current thinking in the WG is, we may be able to ask the WG for formal > adoption during this IETF meeting, or shortly thereafter. > > > > > - WG Milestones: > - Submit AQM recommendations to IESG for publication, obsoleting RFC > 2309 (Goal: January 2014) > - draft-ietf-aqm-recommendation is accepted towards this milestone > - http://datatracker.ietf.org/doc/draft-ietf-aqm-recommendation/ > - the draft has been updated per comments received > - if the authors are comfortable, a WGLC might be made on the next > revision > - we would like to hear from other authors of RFC 2309 on this > document, if anyone has contacts to them. > > > - Submit AQM algorithm evaluation guidelines to IESG for publication > as Informational (Goal: July 2014) > - An editor team has come forth and is working on this > - A draft should be available for discussion in the London IETF > - We encourage discussion on the list and during the meeting, if this > draft should be adopted by the working group. > > - Submit first algorithm specification to IESG for publication as > Proposed Standard (Goal: December 2014) > - Since any Proposed Standard algorithm should be in line with the > recommendations and be passable versus the evaluation guidelines, this > milestone is dependend on the progress of the two work items above. > - Currently the only algorithm spec with a complete and active > individual-submission draft is PIE > > - Other items: > - draft-pan-aqm-pie is under active work as a proposed algorithm: > http://tools.ietf.org/id/draft-pan-aqm-pie-00.txt, however the draft > has expired and should be refreshed. > - draft-nichols-tsvwg-codel is expired; Dave Taht or others may > revive it and/or describe pairing with FQ/SFQ algorithms: > http://tools.ietf.org/id/draft-nichols-tsvwg-codel-01.txt > - Other algorithm specifications are welcome! > - Though, we are not planning on adopting algorithms until > recommendations and evaluation guidelines are mostly stable > > > Richard Scheffenegger > > _______________________________________________ > aqm mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/aqm _______________________________________________ aqm mailing list [email protected] https://www.ietf.org/mailman/listinfo/aqm _______________________________________________ aqm mailing list [email protected] https://www.ietf.org/mailman/listinfo/aqm
