Hi Nicolas, Preethi, David, and Naeem,
After discussing several performance-related topics
in AQM, BMWG, and IPPM with Richard Scheffenegger,
I agreed to review your draft from the point of view of existing
Benchmarking activities in IETF.
I'll just make a few suggestions in this first pass and see where
that takes us, didn't go the github route.
My comments are prefaced by ACM:
regards,
Al
ACM:
I read in Section 12:
12.1. Methodology
A sufficiently detailed description of the test setup SHOULD be
provided. Indeed, that would allow other to replicate the tests if
needed. This test setup MAY include software and hardware versions.
ACM:
The goal, IMHO, is to produce repeatable tests.
Thinking ahead to the results, at least one of the parties tested
will want to repeat the tests for various reasons, such as wanting
understand the details behind performance of their system. Accepting
this view would mean that requirement levels above are MUST, not
SHOULD or MAY.
The tester MAY make its data available.
The proposals SHOULD be experimented on real systems, or they MAY be
evaluated with event-driven simulations (such as NS-2, NS-3, OMNET,
etc.). The proposed scenarios are not bound to a particular
evaluation toolset.
12.2. Comments on metrics measurement
In this document, we present the end-to-end metrics that SHOULD be
evaluated to evaluate the trade-off between latency and goodput.
ACM:
You are better-off to agree on the mandatory set of metrics for
evaluation before hand, or you expose the analysis to variation
with the introduction of new metrics when the tests are repeated.
The
queue-related metrics enable a better understanding of the AQM
behavior under tests and the impact of its internal parameters.
Whenever it is possible, these guidelines advice to consider queue-
related metrics, such as link utilization, queuing delay, queue size
or packet loss.
ACM:
It's fine to collect internal metrics for information, but the
evaluation should be based on externally observed metrics (what
we call blackbox metrics in BMWG).
These guidelines could hardly detail the way the metrics can be
measured depends highly on the evaluation toolset.
ACM:
It's still important to say exactly how a given toolset
performs a measurement to collect each specific metric.
This way, at least what was measured can be understood,
and other tools might be modified to measure the same way.
ACM:
Alternatively, you could point to existing standards with
metric definitions and find a tool set that will measure them
(which is likely, because they *are* standardized).
More about this below.
---end of Section 12 comments ----
ACM:
Since we ended up on Metrics, let's go back to section
2. End-to-end metrics
...This section presents the metrics that COULD
be used to better quantify (1) the reduction of latency, (2)
maximization of goodput and (3) the trade-off between the two. These
metrics SHOULD be considered to better assess the performance of an
AQM scheme.
The metrics listed in this section are not necessarily suited to
every type of traffic detailed in the rest of this document. It is
therefore NOT REQUIRED to measure all of following metrics.
ACM:
It seems more clear to say what metrics will be certainly be measured,
and the applicability of those metrics. I see a key distinction under
UDP and TCP testing, where the "Goodput" metric only has to account for
retransmissions with TCP, for example. But this is a case where you normally
won't have retransmissions with UDP (unless there is some higher-
layer integrity on stream or messages imposed).
You can also have a set of OPTIONAL metrics, and with their
respective applicability.
2.1. Flow Completion time
The flow completion time is an important performance metric for the
end user. Considering the fact that an AQM scheme may drop packets,
the flow completion time is directly linked to the dropping policy of
the AQM scheme. This metric helps to better assess the performance
of an AQM depending on the flow size.
ACM:
It would be good to recognize a relationship for this metric:
Flow Completion Time, s = (flow size, bits) / (Goodput for the flow, bits/s)
because you later specify Goodput.
2.2. Packet loss
Packet losses, that may occur in a queue, impact on the end-to-end
performance at the receiver's side.
The tester MUST evaluate, at the receiver:
o the packet loss probability: this metric should be frequently
measured during the experiment as the long term loss probability
is of interests for steady state scenarios only;
o the interval between consecutive losses: the time between two
losses should be measured. From the set of interval times, the
tester should present the median value, the minimum and maximum
values and the 10th and 90th percentiles.
ACM:
In lab testing, there are two practical ways to assess loss:
- for all packets sent, check that a corresponding packet was received
within a reasonable time for transmission, Tmax (RFC2679)
- keep a count of all packets sent, and count the non-duplicate packets
received when done sending allowing time for queues to empty (RFC2544)
either can produce the Loss Ratio = Lost Packets / Total Sent
The interval between consecutive losses is called a Gap in RFC3611,
where the density of bursts can be variable but busts can be forced
to be consecutive losses by setting parameter Gmin = 0.
2.3. Packet loss synchronization
ACM:
This is sync between flows, it's not been covered in detail at IETF, AFAIK
but I simply add that many network events, such as failed link
restoration, cause correlated or synchronized loss across active flows.
2.4. Goodput
ACM:
BMWG has a definition of Goodput in RFC 2647:
http://tools.ietf.org/html/rfc2647#section-3.17
>
> Definition:
> The number of bits per unit of time forwarded to the correct
> destination interface of the DUT/SUT, minus any bits lost or
> retransmitted.
Here, DUT is Device Under Test, SUT is System Under Test, and
we recently clarified on the list that all bits/packets lost
were intended to be attributable to the DUT/SUT.
This means that the test setup needs to be qualified to assure
that it is not generating loss on its own.
2.5. Latency and jitter
The end-to-end latency differs from the queuing delay: it is linked
to the network topology and the path characteristics. Moreover, the
jitter strongly depends on the traffic and the topology as well. The
introduction of an AQM scheme would impact on these metrics and the
end-to-end evaluation of performance SHOULD consider them to better
assess the AQM schemes.
The guidelines advice that the tester SHOULD determine the minimum,
average and maximum measurements for these metrics and the
coefficient of variation for their average values as well.
ACM:
Latency is fairly easy to define, I suggest RFC2679.
"Jitter" has several definitions around, so we studied the trade-offs
years ago. The short answer is to measure delay variation according to
the difference between the packet in the stream/flow with minimum delay,
subtracted from all other packet delays, and then describe that shifted
distribution with a high percentile and other summary statistics.
This is Packet Delay Variation (PDV), in RFC 5481.
The comparison between two key delay variation metrics is tabulated here:
http://tools.ietf.org/html/rfc5481#section-7.3.
When there are few restrictions on measurement equipment, as we typically find
in
lab work, PDV serves all use cases quite nicely.
4. Various TCP variants
ACM:
I'll simply say that BMWG is concerned with the repeatability properties
of TCP testing, but we are investing some effort to understand the
issues and are working on a draft which enters into testing with stateful
flows: http://tools.ietf.org/html/draft-ietf-bmwg-traffic-management
Some folks from AQM have already commented on this, but it may be
worth a look.
_______________________________________________
aqm mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/aqm
