Dear Wesley,
I apologize for the long delay. Below are responses to your emails
regarding the draft-ietf-aqm-eval-guidelines.
------------------------------------------------------------------------
On 22.01.16 15:50, Wesley Eddy wrote:
On 12/7/2015 7:32 AM, Polina Goltsman wrote:
In the abstract, the document says that it describes characterization
guidelines for an AQM proposal, to decide whether it should be
adopted by the AQM WG. The WG currently has two AQMs
(dropping/marking policy) in last call. Did someone evaluate these
AQMs according to the specified guidelines?
In my opinion, for "standardization" it would be good to have crisp
guidelines. For simply publishing RFCs that are experimental (not
standardized), it is much less important.
I have a question regarding "standardization". As I understand, no
evaluation was performed based on these guidelines. Since the intended
status of draft-ietf-aqm-eval-guidelines is "informational" too, this is
also not an issue. Am I correct?
(see also an email from Dave Täht, quoted at the end of this email)
Moreover, it seems to me that the WG is about to conclude. What
exactly is the purpose of standardizing this document then ?
It's definitely true that the activity level has been low, and so
we're trying to wrap up the outstanding work items before it flattens
completely. This document is not proposed for standards track, only
"Informational". The current goal as I see it (with co-chair hat on)
is to see if we have rough consensus that this is a useful
contribution to the community going forward, and that any small issues
can be addressed.
As I understood your review (please correct if I'm wrong), a main
issue you see is that there isn't specific guidance about numeric
values or ranges to use in evaluations? Nicolas explained why this
might be a bit difficult to do in general (and specific values
published in 2016 may be moot by 2018), so as I understand, one of the
limitations of this document is that it is only going to be able to
provide general descriptive guidance in terms of what kinds of tests
to run, and what types of things to look for, but not prescriptive
values and conditions to be met. Do you think that's okay, even
though it's probably less directly useful than if there were specific
values
Yes, my concern was that the document did not contain enough detailed
description of a test setup (capacities, RTT, etc). However, as Nicolas
explained, the intention is that the test setup is selected by a tester
for some specific environment. The results of an evaluation will then
provide guidance for selecting an AQM for this specific environment.
From my point of view this is completely reasonable approach, however
IMO *the document should state this explicitly*.
From answers to our (mine and Roland's) reviews we found out that the
goals of the document are:
We believe that
"
> AQM schemes need to be compared against both performance and
> deployment categories.
"
(this one is present in the document in section 4.3)
The main idea was to present the available and useful metrics early
in the document, so that the tester can choose between them those that
are of interest for the considered scenario.
and finally
The scope of this document is to present how to assess the performance of
AQMs – without focusing on a specific context ...
...
The unspecified parameters do not require specific knowledge on AQMs – the unspecified
parameters are (1) the RTT or the capacities, which are related to the
network underneath; (2) and the characteristics of the traffic
generation, which is related to the >traffic that is run between clients
and servers.
(which I understand as stated above, correct me if I am wrong)
(this goal will be referred as [*] below)
Section 1.2 is intended to specify the goals of the document. I am not
sure that everybody will understand the above stated goals from the text
of Section 1.2. and the document would benefit if these goals were
stated more explicitly (Well, I could not understand them without
explanations from the authors).
------------------------------------------------------------------------
On 22.01.16 19:32, Klatsky, Carl wrote:
Wes and all,
My comment is in regard to Polina’s comment “The WG currently has two
AQMs (dropping/marking policy) in last call. Did someone evaluate
these AQMs according to the specified guidelines?”. As I read over
draft-ietf-aqm-eval-guidelines, I did not think the objective of this
memo was to arrive at consensus to select one specific AQM. I thought
the objective was to publish guidelines for implementers & service
providers on how they can select an AQM that is best for their
environment. And further that both AQMs in last call would complete
the process as valid AQMs for implementers & service providers as
available to use in their environment, with the guidelines helping
them to decide which is best for them. Some may chose PIE, some may
chose FQ_CODEL. And possibly other future AQMs would go through the
IETF process for publication, and that would be an additional option
for implementers & service providers to evaluate according to the
guidelines as best for their environment.
Is my understand of draft-ietf-aqm-eval-guidelines correct?
Regards,
Carl Klatsky
Comcast
Again, this is not what the document says. The document says (several
times) that it is intended to be used by the AQM WG to evaluate
proposals. If this (guidelines for the WG) is not the intended goal,
then it is probably a good idea to remove this goal from the document.
------------------------------------------------------------------------
Next topic. Assuming that my understanding of the goals of the document
(Section 1.2 + above) are correct, I find the following a little bit
confusing.
1) The document does contain specific numbers in some scenarios, which,
considering the goal [*], I find quite confusing ... . The RTTs in range
[5ms-560ms] in Section 6 are, according to some TCP evaluation
guidelines, currently seen RTTs in the Internet, where the latter is for
satellite links, while the capacities "between 10Mbps an 100Mbps" in
Section 8 are probably available Ethernet rates in range of available
ADSL speeds. None of this makes sense if I decide to evaluate an AQM for
a gigabit LAN in an office. So I think it might be a good idea to either
introduce these numbers as "for example" and mentioning the setups for
which these values are applicable (e.g., "for example the RTT range for
general Internet is ...").
2) The document says:
The guidelines also discuss methods to understand the
various aspects associated with safely deploying and operating the
AQM scheme.
I can't understand whether the document is intended to include
requirements for an AQM specification document or not. Sections 8.3, 11,
and 12 include requirements ("SHOULD") for an AQM proposal.
Since RFC7567 does not provide a recommendation in form of a list of
what should be present in an AQM specification document, it might be a
good idea to include this list in the eval-guidelines. However I suggest
that these requirements be present as a separate section in the
document, since AQM authors and testers are likely to be different people.
Moreover, since Sections 8.3, 11, and 12 include requirements for AQM
proposals, what part of the document evaluates aspects associated with
safe deployment of AQM? In particular:
Section 8.3 (stability analysis): an AQM proposal may or may not cover
the particular test setup (capacity, RTT, etc), so the tester may need
to specify whether their specific parameters are covered by an AQM
proposal, and if not, perform the required analysis themselves.
Section 11 (implementation): AQM proposal SHOULD provide pseudocode. If
the selected test setup includes specific device, then discussing
device-specific implementation could be a part of an evaluation result.
An AQM proposal cannot possibly discuss implementation on every
available hardware.
(On the other hand, Section 12 requires to specify AQM parameters if
they are not default, so it can be considered as an evaluation of safe
deployment.)
3) Section 9.2 includes comments about DNS requests and SYN packets. I
understand that this is a valid concern for an evaluation, but the
scenarios in Section 9.2 do not evaluate this. I think it might be a
good idea to provide a separate scenario with appropriate traffic mix
and metrics, or, as a simpler solution, to discuss DNS and SYN packets
in Section 2.2 - flow start-up metric.
4) The document currently suggests using TCP congestion control
according to RFC5681 with SACK. Is it even possible to use pure RFC5681
on a real end-system? I think it might be a good idea to instead require
that an implementation of a network stack on end hosts is disclosed in
the evaluation results. For example, implementation of loss recovery can
affect results and it may be not configurable: if I understand
correctly, both proportional rate reduction (rfc6937) and a replacement
for TCP ABC cannot be turned off on Linux.
(the issue of network stack implementation on end-system is raised in
*Yee-Ting Li, Douglas Leith, and Robert ******N. Shorten. 2007.
Experimental evaluation of TCP protocols for****high-speed networks.)*
P.S. Is RFC5681 really NewReno and not Reno?
------------------------------------------------------------------------
On 22.01.16 15:52, Wesley Eddy wrote:
With my co-chair hat on, I agree with this sentiment that it will
never be perfect. To my knowledge, there aren't other resources
that have better information and guidance than this document when it
comes to the topic of characterizing AQMs. The goal has been to
produce something useful to the community, compared to the current
guidance available (which is basically nothing), so I'm hopeful that
we can get a version of it with rough consensus to go forward.
I am not very familiar with the following topic, so if the following
concerns are rubbish, please ignore them.
The sections for evaluating AQMs' performance (as opposed to deployment
safety) can be divided into two groups: scenarios in sections 5 (except
for 5.3), 6, 8, and 9.2 evaluate long-lived TCP flows, while scenarios
in section 7 and 9.1 evaluate traffic mixes including multimedia and
web flows. There are several guidelines for evaluation of TCPs, some of
which suggest evaluating TCP with AQM deployed. For eval-guidelines,
they can be used as a source for additional information on how to choose
and calculate metrics for the first group of scenarios. For the second
group of scenarios, eval-guidelines includes suggestion on how to
generate "web-like" traffic and does not include suggestions on how to
generate multimedia traffic. For both scenarios the document suggests
using delay-throughput graph as well as "Metrics such as end-to-end
latency, jitter, flow completion time MAY be generated" without
suggesting particular metric for particular type of traffic (to my
understanding, FCT does not make sense for a video stream and jitter for
web flow).
Is there literature on how to generate and evaluate multimedia streams,
including what metrics to choose and how to calculate them. Although
there are some references in Section 2 (e.g., interval between
consecutive losses), I think that the readers could benefit from
additional references on the topic. Do I understand correctly that the
universally accepted metric for web-flows is flow completion time?
------------------------------------------------------------------------
On 22.01.16 19:57, Dave Täht wrote:
The evaluation guide needs to be executable, or rather, turned into
public code and a standardized benchmark suite. Eventually.
Iteratively, flent has many tests that have proven valuable and quite a
few that have not.
The tests in the aqm guide, need to be created, iterated on, and examined.
Totally agree. Implementing the tests for flent could be a perfect
analogue for an "interoperability" test for these guidelines. Without
any available result it is hard to say, whether the guidelines are
useful or not.
Regards,
Polina
_______________________________________________
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm
