Re: [alto] Review for draft-ietf-alto-performance-metrics-12
Hi Jensen, Qin,
I thought a bit more on the suggestion by Jensen and agree with both of you
that it is OK. I have updated the document in -14.
Thanks a lot!
Richard
On Tue, Jan 12, 2021 at 4:37 AM Qin Wu wrote:
> Jensen:
>
> My impression, is an optional item in the ANBF, therefore the
> metric identifier can be used without as suffix.
>
> I think your proposed change makes sense to me.
>
>
>
> -Qin
>
> *发件人:* Jensen Zhang [mailto:jingxuan.n.zh...@gmail.com]
> *发送时间:* 2021年1月12日 15:36
> *收件人:* Qin Wu
> *抄送:* IETF ALTO ; Y. Richard Yang ; Kai
> Gao
> *主题:* Re: [alto] Review for draft-ietf-alto-performance-metrics-12
>
>
>
> Hi Qin,
>
>
>
> I agree with you that the constraints checking should rely on the
> implementation. I'm OK that it is not in the scope of this document.
>
>
>
> For other comments, I have checked v-13. I think most of them have been
> addressed, except for the following one.
>
>
>
> Section 2.2., paragraph 16:
>
>
>
> >If a metric has no (and hence no - as well), the metric MUST
>
>
>
> recommend adding " surrounding -, or using dash character instead;
>
> if possible, giving the precise BNF grammar will be better, as I
>
> see some metrics names also include the dash character ("-").
>
> >be considered as the 50 percentile (median). Since this scheme is
>
> >common for all metrics defined in this document, below we only
>
> >specify the base identifier.
>
>
>
> Although I can understand this sentence, I still think it should be better
> clarified.
>
> I would suggest giving the BNF grammar at the beginning of this section,
> e.g.,
>
>
>
>... Hence, each performance metric's identifier
>
>should indicate the statistic (i.e., an aggregation operation), to
>
>become
>
>
>
>::= [ '-' ]
>
>
>
>where MUST be one of the following:
>
>
>
> And changing the last paragraph of the section to:
>
>
>
>If '-' is not present in , the metric MUST
>
>be considered as the 50 percentile (median).
>
>
>
> Thanks,
>
> Jensen
>
>
>
>
>
> On Tue, Jan 12, 2021 at 2:22 PM Qin Wu wrote:
>
> Hi, Jensen:
>
> Speak as individual, My answer to your following question is false as
> well, even based on RFC7285, defining hopecount as float point value seem
> also weird.
>
> I think we can rely on implementation or some automation tools for
> constraints checking, but it is not scope of this document.
>
> For other comments, I think Richard have addressed in v-13. Please double
> check it. Thanks
>
>
>
> -Qin
>
> [alto] Review for draft-ietf-alto-performance-metrics-12
>
> Jensen Zhang Tue, 13 October 2020 04:17 UTCShow
> header
> <https://mailarchive.ietf.org/arch/msg/alto/qZrkPza-vEUcIqQMR3OJfk8G-uw/>
>
> Dear ALTOers and authors of draft-ietf-alto-performance-metrics-12,
>
>
>
> Below is my review for draft-ietf-alto-performance-metrics-12.
>
>
>
> Best regards,
>
> Jensen
>
>
>
> ==
>
> General issue:
>
>
>
> The document is well written. I only have one question about the design
>
> part:
>
>
>
> the base ALTO protocol only uses the cost-mode to infer the value format,
>
> e.g., "numerical" infers the cost value MUST be a floating-point value; but
>
> this document requires different value formats for different cost-metrics,
>
> e.g., "delay-ow" requires the non-negative floating-point value, and
>
> "hopcount" requires the non-negative integer value. But based on
>
> Sec 11.3.2.3 of RFC7285, in the "constraints" field, "ALTO servers SHOULD
>
> use at least IEEE 754 double-precision floating point [IEEE.754.2008] to
>
> store the cost value". I wonder if a test constraint expression like "eq
>
> 3.1" for the cost-metric "hopcount" is valid. Should the ALTO server reject
>
> such a request? According to RFC7285, it should be valid. But according to
>
> this document, it is clearly always false.
>
>
>
> ==
>
>
>
> Nits and writing suggestions:
>
>
>
> Section 1., paragraph 5:
>
>
>
> >The purpose of this document is to ensure proper usage of the
>
> >performance metrics defined in Table 1; it does not claim novelty of
>
> >the metrics. The Origin column of Table 1 gives the RFC which
>
> >defines each metric.
>
>
>
> Origin -> Origin Example (t
Re: [alto] Review for draft-ietf-alto-performance-metrics-12
Jensen:
My impression, is an optional item in the ANBF, therefore the metric
identifier can be used without as suffix.
I think your proposed change makes sense to me.
-Qin
发件人: Jensen Zhang [mailto:jingxuan.n.zh...@gmail.com]
发送时间: 2021年1月12日 15:36
收件人: Qin Wu
抄送: IETF ALTO ; Y. Richard Yang ; Kai Gao
主题: Re: [alto] Review for draft-ietf-alto-performance-metrics-12
Hi Qin,
I agree with you that the constraints checking should rely on the
implementation. I'm OK that it is not in the scope of this document.
For other comments, I have checked v-13. I think most of them have been
addressed, except for the following one.
Section 2.2., paragraph 16:
>If a metric has no (and hence no - as well), the metric MUST
recommend adding " surrounding -, or using dash character instead;
if possible, giving the precise BNF grammar will be better, as I
see some metrics names also include the dash character ("-").
>be considered as the 50 percentile (median). Since this scheme is
>common for all metrics defined in this document, below we only
>specify the base identifier.
Although I can understand this sentence, I still think it should be better
clarified.
I would suggest giving the BNF grammar at the beginning of this section, e.g.,
... Hence, each performance metric's identifier
should indicate the statistic (i.e., an aggregation operation), to
become
::= [ '-' ]
where MUST be one of the following:
And changing the last paragraph of the section to:
If '-' is not present in , the metric MUST
be considered as the 50 percentile (median).
Thanks,
Jensen
On Tue, Jan 12, 2021 at 2:22 PM Qin Wu
mailto:bill...@huawei.com>> wrote:
Hi, Jensen:
Speak as individual, My answer to your following question is false as well,
even based on RFC7285, defining hopecount as float point value seem also weird.
I think we can rely on implementation or some automation tools for constraints
checking, but it is not scope of this document.
For other comments, I think Richard have addressed in v-13. Please double check
it. Thanks
-Qin
[alto] Review for draft-ietf-alto-performance-metrics-12
Jensen Zhang mailto:jingxuan.n.zh...@gmail.com>>
Tue, 13 October 2020 04:17 UTCShow
header<https://mailarchive.ietf.org/arch/msg/alto/qZrkPza-vEUcIqQMR3OJfk8G-uw/>
Dear ALTOers and authors of draft-ietf-alto-performance-metrics-12,
Below is my review for draft-ietf-alto-performance-metrics-12.
Best regards,
Jensen
==
General issue:
The document is well written. I only have one question about the design
part:
the base ALTO protocol only uses the cost-mode to infer the value format,
e.g., "numerical" infers the cost value MUST be a floating-point value; but
this document requires different value formats for different cost-metrics,
e.g., "delay-ow" requires the non-negative floating-point value, and
"hopcount" requires the non-negative integer value. But based on
Sec 11.3.2.3 of RFC7285, in the "constraints" field, "ALTO servers SHOULD
use at least IEEE 754 double-precision floating point [IEEE.754.2008] to
store the cost value". I wonder if a test constraint expression like "eq
3.1" for the cost-metric "hopcount" is valid. Should the ALTO server reject
such a request? According to RFC7285, it should be valid. But according to
this document, it is clearly always false.
==
Nits and writing suggestions:
Section 1., paragraph 5:
>The purpose of this document is to ensure proper usage of the
>performance metrics defined in Table 1; it does not claim novelty of
>the metrics. The Origin column of Table 1 gives the RFC which
>defines each metric.
Origin -> Origin Example (to be consistent with the table)
>We can rough classify the performance metrics into two categories:
>those derived from the performance of individual packets (i.e., one-
>way delay, round-trip delay, delay variation, hop count, and loss
>rate), and those related with bandwidth (TCP throughput, residue
>bandwidth and max reservable bandwidth). These two categories are
>defined in Section 3 and Section 4 respectively. Note that all
>metrics except round trip delay are unidirectional. Hence, a client
>will need to query both directions if needed.
Section 2., paragraph 1:
>When defining the metrics in Table 1, this document considers the
>guidelines specified in [RFC6390], which requires fine-grained
>specification of (i) Metric Name, (ii) Metric Description, (iii)
>Method of Measurement or Calculation, (iv) Units of Measurement, (v)
>Measurement Points, and (vi) Measurement Timing. In particular, for
>each metric, this document defines (i) Metric Name, (ii) Metr
Re: [alto] Review for draft-ietf-alto-performance-metrics-12
Hi Qin,
I agree with you that the constraints checking should rely on the
implementation. I'm OK that it is not in the scope of this document.
For other comments, I have checked v-13. I think most of them have been
addressed, except for the following one.
Section 2.2., paragraph 16:
>If a metric has no (and hence no - as well), the metric MUST
recommend adding " surrounding -, or using dash character instead;
if possible, giving the precise BNF grammar will be better, as I
see some metrics names also include the dash character ("-").
>be considered as the 50 percentile (median). Since this scheme is
>common for all metrics defined in this document, below we only
>specify the base identifier.
Although I can understand this sentence, I still think it should be better
clarified.
I would suggest giving the BNF grammar at the beginning of this section,
e.g.,
... Hence, each performance metric's identifier
should indicate the statistic (i.e., an aggregation operation), to
become
::= [ '-' ]
where MUST be one of the following:
And changing the last paragraph of the section to:
If '-' is not present in , the metric MUST
be considered as the 50 percentile (median).
Thanks,
Jensen
On Tue, Jan 12, 2021 at 2:22 PM Qin Wu wrote:
> Hi, Jensen:
>
> Speak as individual, My answer to your following question is false as
> well, even based on RFC7285, defining hopecount as float point value seem
> also weird.
>
> I think we can rely on implementation or some automation tools for
> constraints checking, but it is not scope of this document.
>
> For other comments, I think Richard have addressed in v-13. Please double
> check it. Thanks
>
>
>
> -Qin
>
> [alto] Review for draft-ietf-alto-performance-metrics-12
>
> Jensen Zhang Tue, 13 October 2020 04:17 UTCShow
> header
> <https://mailarchive.ietf.org/arch/msg/alto/qZrkPza-vEUcIqQMR3OJfk8G-uw/>
>
> Dear ALTOers and authors of draft-ietf-alto-performance-metrics-12,
>
>
>
> Below is my review for draft-ietf-alto-performance-metrics-12.
>
>
>
> Best regards,
>
> Jensen
>
>
>
> ==
>
> General issue:
>
>
>
> The document is well written. I only have one question about the design
>
> part:
>
>
>
> the base ALTO protocol only uses the cost-mode to infer the value format,
>
> e.g., "numerical" infers the cost value MUST be a floating-point value; but
>
> this document requires different value formats for different cost-metrics,
>
> e.g., "delay-ow" requires the non-negative floating-point value, and
>
> "hopcount" requires the non-negative integer value. But based on
>
> Sec 11.3.2.3 of RFC7285, in the "constraints" field, "ALTO servers SHOULD
>
> use at least IEEE 754 double-precision floating point [IEEE.754.2008] to
>
> store the cost value". I wonder if a test constraint expression like "eq
>
> 3.1" for the cost-metric "hopcount" is valid. Should the ALTO server reject
>
> such a request? According to RFC7285, it should be valid. But according to
>
> this document, it is clearly always false.
>
>
>
> ==
>
>
>
> Nits and writing suggestions:
>
>
>
> Section 1., paragraph 5:
>
>
>
> >The purpose of this document is to ensure proper usage of the
>
> >performance metrics defined in Table 1; it does not claim novelty of
>
> >the metrics. The Origin column of Table 1 gives the RFC which
>
> >defines each metric.
>
>
>
> Origin -> Origin Example (to be consistent with the table)
>
> >We can rough classify the performance metrics into two categories:
>
> >those derived from the performance of individual packets (i.e., one-
>
> >way delay, round-trip delay, delay variation, hop count, and loss
>
> >rate), and those related with bandwidth (TCP throughput, residue
>
> >bandwidth and max reservable bandwidth). These two categories are
>
> >defined in Section 3 and Section 4 respectively. Note that all
>
> >metrics except round trip delay are unidirectional. Hence, a client
>
> >will need to query both directions if needed.
>
>
>
>
>
> Section 2., paragraph 1:
>
>
>
> >When defining the metrics in Table 1, this document considers the
>
> >guidelines specified in [RFC6390], which requires fine-grained
>
> >specification of (i) Metric Name, (ii) Metric Description, (iii)
>
> >Method of Measurement or Calculation, (iv) Units of M
[alto] Review for draft-ietf-alto-performance-metrics-12
Hi, Jensen: Speak as individual, My answer to your following question is false as well, even based on RFC7285, defining hopecount as float point value seem also weird. I think we can rely on implementation or some automation tools for constraints checking, but it is not scope of this document. For other comments, I think Richard have addressed in v-13. Please double check it. Thanks -Qin [alto] Review for draft-ietf-alto-performance-metrics-12 Jensen Zhang Tue, 13 October 2020 04:17 UTCShow header<https://mailarchive.ietf.org/arch/msg/alto/qZrkPza-vEUcIqQMR3OJfk8G-uw/> Dear ALTOers and authors of draft-ietf-alto-performance-metrics-12, Below is my review for draft-ietf-alto-performance-metrics-12. Best regards, Jensen == General issue: The document is well written. I only have one question about the design part: the base ALTO protocol only uses the cost-mode to infer the value format, e.g., "numerical" infers the cost value MUST be a floating-point value; but this document requires different value formats for different cost-metrics, e.g., "delay-ow" requires the non-negative floating-point value, and "hopcount" requires the non-negative integer value. But based on Sec 11.3.2.3 of RFC7285, in the "constraints" field, "ALTO servers SHOULD use at least IEEE 754 double-precision floating point [IEEE.754.2008] to store the cost value". I wonder if a test constraint expression like "eq 3.1" for the cost-metric "hopcount" is valid. Should the ALTO server reject such a request? According to RFC7285, it should be valid. But according to this document, it is clearly always false. == Nits and writing suggestions: Section 1., paragraph 5: >The purpose of this document is to ensure proper usage of the >performance metrics defined in Table 1; it does not claim novelty of >the metrics. The Origin column of Table 1 gives the RFC which >defines each metric. Origin -> Origin Example (to be consistent with the table) >We can rough classify the performance metrics into two categories: >those derived from the performance of individual packets (i.e., one- >way delay, round-trip delay, delay variation, hop count, and loss >rate), and those related with bandwidth (TCP throughput, residue >bandwidth and max reservable bandwidth). These two categories are >defined in Section 3 and Section 4 respectively. Note that all >metrics except round trip delay are unidirectional. Hence, a client >will need to query both directions if needed. Section 2., paragraph 1: >When defining the metrics in Table 1, this document considers the >guidelines specified in [RFC6390], which requires fine-grained >specification of (i) Metric Name, (ii) Metric Description, (iii) >Method of Measurement or Calculation, (iv) Units of Measurement, (v) >Measurement Points, and (vi) Measurement Timing. In particular, for >each metric, this document defines (i) Metric Name, (ii) Metric >Description, and (iv) Units of Measurement. The Measurement Points >are always specified by the specific ALTO services; for example, >endpoint cost service is between the two end points. end points -> endpoints Section 2.1., paragraph 11: >A particular type of "estimation is direct "import", which indicates >that the value of the metric is imported directly from a specific >existing protocol or system. Specifying "import" as source instead source -> the source >of the more generic "estimation" may allow better tracing of >information flow. For an "import" metric, it is RECOMMENDED that the >"parameters" field provides details to the system from which raw data >is imported. In particular, one may notice that the set of end-to- >end metrics defined in Table 1 has large overlap with the set defined >in [RFC8571], in the setting of IGP traffic engineering performance >metrics for each link (i.e., unidirectional link delay, min/max >unidirectional link delay, unidirectional delay variation, >unidirectional link loss, unidirectional residual bandwidth, >unidirectional available bandwidth, unidirectional utilized >bandwidth). Hence, an ALTO server may use "import" to indicate that >its end-to-end metrics are computed from link metrics imported from >[RFC8571]. Section 2.2., paragraph 2: >percentile, with letter p followed by a number p: a number p -> a number Section 2.2., paragraph 16: >If a metric has no (and hence no - as well), the metric MUST recommend adding " surrounding -, or using dash character instead; if pos
[alto] Review for draft-ietf-alto-performance-metrics-12
Dear ALTOers and authors of draft-ietf-alto-performance-metrics-12,
Below is my review for draft-ietf-alto-performance-metrics-12.
Best regards,
Jensen
==
General issue:
The document is well written. I only have one question about the design
part:
the base ALTO protocol only uses the cost-mode to infer the value format,
e.g., "numerical" infers the cost value MUST be a floating-point value; but
this document requires different value formats for different cost-metrics,
e.g., "delay-ow" requires the non-negative floating-point value, and
"hopcount" requires the non-negative integer value. But based on
Sec 11.3.2.3 of RFC7285, in the "constraints" field, "ALTO servers SHOULD
use at least IEEE 754 double-precision floating point [IEEE.754.2008] to
store the cost value". I wonder if a test constraint expression like "eq
3.1" for the cost-metric "hopcount" is valid. Should the ALTO server reject
such a request? According to RFC7285, it should be valid. But according to
this document, it is clearly always false.
==
Nits and writing suggestions:
Section 1., paragraph 5:
>The purpose of this document is to ensure proper usage of the
>performance metrics defined in Table 1; it does not claim novelty of
>the metrics. The Origin column of Table 1 gives the RFC which
>defines each metric.
Origin -> Origin Example (to be consistent with the table)
>We can rough classify the performance metrics into two categories:
>those derived from the performance of individual packets (i.e., one-
>way delay, round-trip delay, delay variation, hop count, and loss
>rate), and those related with bandwidth (TCP throughput, residue
>bandwidth and max reservable bandwidth). These two categories are
>defined in Section 3 and Section 4 respectively. Note that all
>metrics except round trip delay are unidirectional. Hence, a client
>will need to query both directions if needed.
Section 2., paragraph 1:
>When defining the metrics in Table 1, this document considers the
>guidelines specified in [RFC6390], which requires fine-grained
>specification of (i) Metric Name, (ii) Metric Description, (iii)
>Method of Measurement or Calculation, (iv) Units of Measurement, (v)
>Measurement Points, and (vi) Measurement Timing. In particular, for
>each metric, this document defines (i) Metric Name, (ii) Metric
>Description, and (iv) Units of Measurement. The Measurement Points
>are always specified by the specific ALTO services; for example,
>endpoint cost service is between the two end points.
end points -> endpoints
Section 2.1., paragraph 11:
>A particular type of "estimation is direct "import", which indicates
>that the value of the metric is imported directly from a specific
>existing protocol or system. Specifying "import" as source instead
source -> the source
>of the more generic "estimation" may allow better tracing of
>information flow. For an "import" metric, it is RECOMMENDED that the
>"parameters" field provides details to the system from which raw data
>is imported. In particular, one may notice that the set of end-to-
>end metrics defined in Table 1 has large overlap with the set defined
>in [RFC8571], in the setting of IGP traffic engineering performance
>metrics for each link (i.e., unidirectional link delay, min/max
>unidirectional link delay, unidirectional delay variation,
>unidirectional link loss, unidirectional residual bandwidth,
>unidirectional available bandwidth, unidirectional utilized
>bandwidth). Hence, an ALTO server may use "import" to indicate that
>its end-to-end metrics are computed from link metrics imported from
>[RFC8571].
Section 2.2., paragraph 2:
>percentile, with letter p followed by a number p:
a number p -> a number
Section 2.2., paragraph 16:
>If a metric has no (and hence no - as well), the metric MUST
recommend adding " surrounding -, or using dash character instead;
if possible, giving the precise BNF grammar will be better, as I
see some metrics names also include the dash character ("-").
>be considered as the 50 percentile (median). Since this scheme is
>common for all metrics defined in this document, below we only
>specify the base identifier.
Section 3., paragraph 1:
>This section introduces ALTO network performance metrics including
>one way delay, round trip delay, delay variation, hop count, and
>packet loss rate. They measure the "quality of experience" of the
>s
[alto] Review for draft-ietf-alto-performance-metrics-12
Dear ALTO WG I have done a review of the Performance Metrics draft (-12). No major problems regarding the technical part, Other comments related to consistency and clarity are in the attached file (marked with [DANNY]) Best regards, Danny Lachos ALTO Working Group Q. Wu Internet-DraftHuawei Intended status: Standards Track Y. Yang Expires: January 13, 2021Yale University Y. Lee Samsung D. Dhody Huawei S. Randriamasy Nokia Bell Labs L. Contreras Telefonica July 12, 2020 ALTO Performance Cost Metrics draft-ietf-alto-performance-metrics-12 Abstract [DANNY] Consider reducing the number of lines (https://www.ietf.org/standards/ids/guidelines/#6) Cost metric is a basic concept in Application-Layer Traffic Optimization (ALTO), and is used in basic ALTO services including both the cost map service and the endpoint cost service. Different applications may use different cost metrics, but the ALTO base protocol [RFC7285] defines only a single cost metric, i.e., the generic "routingcost" metric; see Sec. 14.2 of [RFC7285]. Hence, if the ALTO client of an application wants to issue a cost map or an endpoint cost request to determine the resource provider that offers better delay performance (i.e., low-delay) to a resource consumer, the base protocol does not define the cost metric to be used. This document addresses the issue by introducing network performance metrics, including network delay, jitter, packet loss rate, hop count, and bandwidth. The ALTO server may derive and aggregate such performance metrics from routing protocols such as BGP-LS, OSPF-TE and ISIS-TE, or from end-to-end traffic management tools, and then expose the information to allow applications to determine "where" to connect based on network performance criteria. There are multiple sources to derive the performance metrics. For example, whether the metric reported is an estimation based on measurements or it is a service-level agreement (SLA) can define the [DANNY] Consider adding a comma: "measurements, or it is a " meaning of the performance metric. Hence, an application may need additional contextual information beyond the metric value. This document introduces an additional "cost-context" field to the ALTO "cost-type" field to convey such information. Wu, et al. Expires January 13, 2021[Page 1] Internet-DraftALTO Performance Cost MetricsJuly 2020 Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on January 13, 2021. Copyright Notice Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4
[alto] Review for draft-ietf-alto-performance-metrics-12
Dear ALTO WG, Below is my review for the performance metrics draft. The document is well written and easy to follow. The specifications are also well-organized. I do not see major problems with the technical contents. However, there are a few typos and inconsistent usage of terms, as listed below. Best, Kai == Page 5: para 1: rough -> roughly, max -> maximum Page 6, sec 2.1 remove "and" before sla latency typically do -> does "SHOULD NOT" -> "MUST NOT", "can be placed" -> "must be placed" Page 7, para 1: a "sla" -> an "sla" after figure 1, para 1: "estimation -> "estimation" Page 8, with letter p -> with letter "p" Page 15, example: delay-var -> delay-variation Page 28: ALTO Server and Client -> ALTO server and client Page 29: "Since he ..." -> The paragraph is repeated and should be removed ==___ alto mailing list alto@ietf.org https://www.ietf.org/mailman/listinfo/alto
