Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines
Hi Wes, On 19.08.2015 at 04:52 Wesley Eddy wrote: There were only a couple of the major issues that I thought I should comment on as a co-chair of the WG: 3) the overall number of tests and parameter combinations is really high Are there particular permutations (or classes of permutations) that you can suggest to remove? There's a balancing act between including enough to satisfy people that want to find edge cases and thoroughly characterize an algorithm, and the desire for a more easily tractable suite of tests. Right now, I can't, it was merely an observation... 4) from the discussed end-to-end metrics only latency/goodput metrics are used in the scenarios and for some of the scenarios these metrics are not suitable to show the desired behavior It would be easier for the editors to improve this if you could suggest specific metrics to add to specific scenarios, I think. We included some considerations in the comments to the individual sections already. As a general notice, The document could benefit by cross-referencing the scenarios against the documents referenced in Major Issue 6. Regarding particular metrics, suggesting one would require exact understanding of particular test goals, proper argumentation, and thus more time than LC deadline allowed. Regards, Roland ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines
On 8/18/2015 6:07 PM, Dave Taht wrote: On Tue, Aug 18, 2015 at 3:03 PM, Roland Bless roland.bl...@kit.edu wrote: Hi, Am 10.08.2015 um 15:43 schrieb Wesley Eddy: As chairs, Richard and I would like to start a 2-week working group last call on the AQM characterization guidelines: https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/ Please make a review of this, and send comments to the list or chairs. Any comments that you might have will be useful to us, even if it's just to say that you've read it and have no other comments. Unfortunately, we (Polina and I) did a thorough review, which is attached. TL;DR: from our point-of-view the I-D needs a major revision. I am so tired of this document that I can hardly bear to read it again, but I agree with the majority of the comments. Sometimes I do wish we could do graphics and charts as the IEEE does. We can add any type of graphics that are necessary, they will just only show up in the PDF version of the RFC, with only references to the PDF version in the TXT copy. See, for instance: https://tools.ietf.org/pdf/rfc6687.pdf Are there particular figures that need to be added to this AQM document to strengthen it? -- Wes Eddy MTI Systems ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines
On 8/18/2015 6:03 PM, Roland Bless wrote: Hi, Am 10.08.2015 um 15:43 schrieb Wesley Eddy: As chairs, Richard and I would like to start a 2-week working group last call on the AQM characterization guidelines: https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/ Please make a review of this, and send comments to the list or chairs. Any comments that you might have will be useful to us, even if it's just to say that you've read it and have no other comments. Unfortunately, we (Polina and I) did a thorough review, which is attached. TL;DR: from our point-of-view the I-D needs a major revision. Many thanks for the detailed review. I think a majority of the comments could be addressed in an update, if the authors agree. There were only a couple of the major issues that I thought I should comment on as a co-chair of the WG: 3) the overall number of tests and parameter combinations is really high Are there particular permutations (or classes of permutations) that you can suggest to remove? There's a balancing act between including enough to satisfy people that want to find edge cases and thoroughly characterize an algorithm, and the desire for a more easily tractable suite of tests. 4) from the discussed end-to-end metrics only latency/goodput metrics are used in the scenarios and for some of the scenarios these metrics are not suitable to show the desired behavior It would be easier for the editors to improve this if you could suggest specific metrics to add to specific scenarios, I think. 5) some sections in this document (e.g., 7.3, 10, 13) specify requirements for an AQM standard(/draft) and not requirements for a performance evaluation, so these sections should be moved to [draft-ietf-aqm-recommendation] That one is now an RFC (7567), so hopefully they're already reflected if they were critical requirements. In any case, I agree with you that requirements themselves should not be conveyed in this document, but rather it should be just aimed at characterizing algorithm behavior with regard to the requirements (for ones that are applicable to verification by testing). -- Wes Eddy MTI Systems ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines
On Tue, Aug 18, 2015 at 3:03 PM, Roland Bless roland.bl...@kit.edu wrote: Hi, Am 10.08.2015 um 15:43 schrieb Wesley Eddy: As chairs, Richard and I would like to start a 2-week working group last call on the AQM characterization guidelines: https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/ Please make a review of this, and send comments to the list or chairs. Any comments that you might have will be useful to us, even if it's just to say that you've read it and have no other comments. Unfortunately, we (Polina and I) did a thorough review, which is attached. TL;DR: from our point-of-view the I-D needs a major revision. I am so tired of this document that I can hardly bear to read it again, but I agree with the majority of the comments. Sometimes I do wish we could do graphics and charts as the IEEE does. Regards, Roland ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines
Hi all, I read the latest version of the draft, and I found it useful. The draft addresses a comprehensive range of topics for AQM characterization. What I am not so happy with, is the description of the corresponding experiments. Some critical points of my first review https://mailarchive.ietf.org/arch/msg/aqm/OwPTGmXLpEmCChpgE7ZFqFnnT64 still persist. I would like to regard these experiments as initial proposals (which is good to have) that might undergo substantial revision in practice later on. In general I have the feeling that the combinatorial number of mandatory experiments is close to infinity. Not only that I doubt this will ever be done; but who is subsequently going to judge the huge amount of results? Here are some minor comments: Section 2.7 defines goodput/delay scatter plots in two different ways: On with reference to [HAYE2013], the other definition with reference to [WINS2014]. I would prefer to have only one definition, namely [WINS2014]. - [HAYE2013] depends on a parameter variation across certain range (e.g. traffic load, or buffer size) that is not defined in most of our experiments. - [WINS2014] depends only on randomized replication of otherwise identical experiments. This should be applicable to any of the evaluation experiments. (In fact, it is unavoidable anyway.) Section 4.3: The term long-lived non application-limited UDP is somewhat infinite bandwidth. What the authors probably mean is long-lived UDP flow from unresponsive application to make it clear that no application layer congestion control is present like in NFS. Section 2.1: Formula on flow completion time: mismatch of dimensions (Byte vs. Mbps) Wolfram Lautenschlaeger -Ursprüngliche Nachricht- Von: aqm [mailto:aqm-boun...@ietf.org] Im Auftrag von Wesley Eddy Gesendet: Montag, 10. August 2015 15:44 An: aqm@ietf.org Betreff: [aqm] WGLC on draft-ietf-aqm-eval-guidelines As chairs, Richard and I would like to start a 2-week working group last call on the AQM characterization guidelines: https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/ Please make a review of this, and send comments to the list or chairs. Any comments that you might have will be useful to us, even if it's just to say that you've read it and have no other comments. Thanks! -- Wes Eddy MTI Systems ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm smime.p7s Description: S/MIME cryptographic signature ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
[aqm] WGLC on draft-ietf-aqm-eval-guidelines
As chairs, Richard and I would like to start a 2-week working group last call on the AQM characterization guidelines: https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/ Please make a review of this, and send comments to the list or chairs. Any comments that you might have will be useful to us, even if it's just to say that you've read it and have no other comments. Thanks! -- Wes Eddy MTI Systems ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
