Re: [alto] Benjamin Kaduk's Discuss on draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT)
Hi Richard, On Fri, Dec 17, 2021 at 03:52:04PM -0500, Y. Richard Yang wrote: > Hi Ben, > > Thank you so much for the wonderful review and we have taken a pass of the > document. One quick question: should we upload a newer version (v21) so > that you can check the detailed edits using diff or you prefer we post the > revised text inline in reply? The new version (v21) would be easier for me; thanks for asking! -Ben ___ alto mailing list alto@ietf.org https://www.ietf.org/mailman/listinfo/alto
Re: [alto] Benjamin Kaduk's Discuss on draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT)
Hi Ben, Thank you so much for the wonderful review and we have taken a pass of the document. One quick question: should we upload a newer version (v21) so that you can check the detailed edits using diff or you prefer we post the revised text inline in reply? Thank you so much! Richard On Wed, Dec 8, 2021 at 6:43 PM Benjamin Kaduk wrote: > Hi Qin, > > It looks like the only topic that's potentially unresolved is the BCP 18 > question. I think internationalization is a topic where we mostly look to > the ART ADs for guidance, and I'm reluctant to claim any kind of authority > on the "right thing to do". Mostly I wanted to raise the topic for > visibility in case anyone else had any thoughts; if no one else replies, I > think the authors should do what they feel best (which could include making > no change to the draft). > > Thanks, > > Ben > > On Mon, Dec 06, 2021 at 01:25:20PM +, Qin Wu wrote: > > Hi, Ben: > > -邮件原件- > > 发件人: alto [mailto:alto-boun...@ietf.org] 代表 Benjamin Kaduk > > 发送时间: 2021年12月4日 6:30 > > 收件人: Qin Wu > > 抄送: draft-ietf-alto-performance-metr...@ietf.org; alto@ietf.org; The > IESG ; Y. Richard Yang ; > alto-cha...@ietf.org > > 主题: Re: [alto] Benjamin Kaduk's Discuss on > draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT) > > > > Hi Qin, > > > > On Thu, Dec 02, 2021 at 09:04:18AM +, Qin Wu wrote: > > > Thanks Ben for detailed valuable review, see reply and clarification > below. > > > > > > -邮件原件- > > > >发件人: Benjamin Kaduk via Datatracker [mailto:nore...@ietf.org] > > > >发送时间: 2021年12月2日 13:05 > > > >收件人: The IESG > > > >抄送: draft-ietf-alto-performance-metr...@ietf.org; > > > >alto-cha...@ietf.org; alto@ietf.org; i...@j-f-s.de; i...@j-f-s.de > > > >主题: Benjamin Kaduk's Discuss on > > > >draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT) > > > > > > >Benjamin Kaduk has entered the following ballot position for > > > >draft-ietf-alto-performance-metrics-20: Discuss > > > > > > >When responding, please keep the subject line intact and reply to all > > > >email addresses included in the To and CC lines. (Feel free to cut > > > >this introductory paragraph, however.) > > > > > > > > > >Please refer to > > > >https://www.ietf.org/blog/handling-iesg-ballot-positions/ > > > >for more information about how to handle DISCUSS and COMMENT > positions. > > > > > > > > > >The document, along with other ballot positions, can be found here: > > > >https://datatracker.ietf.org/doc/draft-ietf-alto-performance-metrics/ > > > > > > > > > > > > >- > > > >- > > > >DISCUSS: > > > >- > > > >- > > > > > > >These should all be trivial to resolve -- just some minor internal > inconsistencies that need to be fixed before publication. > > > > > > >The discussion of percentile statistical operator in §2.2 is > internally inconsistent -- if the percentile number must be an integer, > then p99.9 is not valid. > > > [Qin Wu] Yes, the percentile is a number following the letter 'p', but > > > in some case when high precision is needed, this percentile number > will be further followed by an optional decimal part The decimal part > should start with the '.' separator. Maybe the separator cause your > confusion. See definition in section 2.2 for details: > > > " > > >percentile, with letter 'p' followed by a number: > > > gives the percentile specified by the number following the letter > > > 'p'. The number MUST be a non-negative JSON integer in the range > > > [0, 100] (i.e., greater than or equal to 0 and less than or equal > > > to 100), followed by an optional decimal part, if a higher > > > precision is needed. The decimal part should start with the '.' > > > separator (U+002E), and followed by a sequence of one or more > > > ASCII numbers between '0' and '9'. > > > " > > > Let us know if you think separator should be changed or you live with > the current form. > > > > Oops, that's my mistake and you are correct. Sorry about that; I agree > that no change is needed h
Re: [alto] Benjamin Kaduk's Discuss on draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT)
Hi, Ben: Since the current document clearly state the specification of SLA details is out of scope, we authors prefer to make no change to changes unless I hear objection for this. Thanks Ben's clarification on BCP 18 question. It is a very useful discussion. -Qin -邮件原件- 发件人: Benjamin Kaduk [mailto:ka...@mit.edu] 发送时间: 2021年12月9日 7:43 收件人: Qin Wu 抄送: draft-ietf-alto-performance-metr...@ietf.org; alto@ietf.org; The IESG ; Y. Richard Yang ; alto-cha...@ietf.org 主题: Re: [alto] Benjamin Kaduk's Discuss on draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT) Hi Qin, It looks like the only topic that's potentially unresolved is the BCP 18 question. I think internationalization is a topic where we mostly look to the ART ADs for guidance, and I'm reluctant to claim any kind of authority on the "right thing to do". Mostly I wanted to raise the topic for visibility in case anyone else had any thoughts; if no one else replies, I think the authors should do what they feel best (which could include making no change to the draft). Thanks, Ben On Mon, Dec 06, 2021 at 01:25:20PM +, Qin Wu wrote: > Hi, Ben: > -邮件原件- > 发件人: alto [mailto:alto-boun...@ietf.org] 代表 Benjamin Kaduk > 发送时间: 2021年12月4日 6:30 > 收件人: Qin Wu > 抄送: draft-ietf-alto-performance-metr...@ietf.org; alto@ietf.org; The > IESG ; Y. Richard Yang ; > alto-cha...@ietf.org > 主题: Re: [alto] Benjamin Kaduk's Discuss on > draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT) > > Hi Qin, > > On Thu, Dec 02, 2021 at 09:04:18AM +, Qin Wu wrote: > > Thanks Ben for detailed valuable review, see reply and clarification below. > > > > -邮件原件- > > >发件人: Benjamin Kaduk via Datatracker [mailto:nore...@ietf.org] > > >发送时间: 2021年12月2日 13:05 > > >收件人: The IESG > > >抄送: draft-ietf-alto-performance-metr...@ietf.org; > > >alto-cha...@ietf.org; alto@ietf.org; i...@j-f-s.de; i...@j-f-s.de > > >主题: Benjamin Kaduk's Discuss on > > >draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT) > > > > >Benjamin Kaduk has entered the following ballot position for > > >draft-ietf-alto-performance-metrics-20: Discuss > > > > >When responding, please keep the subject line intact and reply to > > >all email addresses included in the To and CC lines. (Feel free to > > >cut this introductory paragraph, however.) > > > > > > >Please refer to > > >https://www.ietf.org/blog/handling-iesg-ballot-positions/ > > >for more information about how to handle DISCUSS and COMMENT positions. > > > > > > >The document, along with other ballot positions, can be found here: > > >https://datatracker.ietf.org/doc/draft-ietf-alto-performance-metric > > >s/ > > > > > > > > >--- > > >-- > > >- > > >DISCUSS: > > >--- > > >-- > > >- > > > > >These should all be trivial to resolve -- just some minor internal > > >inconsistencies that need to be fixed before publication. > > > > >The discussion of percentile statistical operator in §2.2 is internally > > >inconsistent -- if the percentile number must be an integer, then p99.9 is > > >not valid. > > [Qin Wu] Yes, the percentile is a number following the letter 'p', > > but in some case when high precision is needed, this percentile number will > > be further followed by an optional decimal part The decimal part should > > start with the '.' separator. Maybe the separator cause your confusion. See > > definition in section 2.2 for details: > > " > >percentile, with letter 'p' followed by a number: > > gives the percentile specified by the number following the letter > > 'p'. The number MUST be a non-negative JSON integer in the range > > [0, 100] (i.e., greater than or equal to 0 and less than or equal > > to 100), followed by an optional decimal part, if a higher > > precision is needed. The decimal part should start with the '.' > > separator (U+002E), and followed by a sequence of one or more > > ASCII numbers between '0' and '9'. > > " > > Let us know if you think separator should be changed or you live with the > > current form. > > Oops, that's my mistake and you are correct. Sorry about that; I agree that > no change is needed here. > > [Qin Wu] G
Re: [alto] Benjamin Kaduk's Discuss on draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT)
Hi Qin, It looks like the only topic that's potentially unresolved is the BCP 18 question. I think internationalization is a topic where we mostly look to the ART ADs for guidance, and I'm reluctant to claim any kind of authority on the "right thing to do". Mostly I wanted to raise the topic for visibility in case anyone else had any thoughts; if no one else replies, I think the authors should do what they feel best (which could include making no change to the draft). Thanks, Ben On Mon, Dec 06, 2021 at 01:25:20PM +, Qin Wu wrote: > Hi, Ben: > -邮件原件- > 发件人: alto [mailto:alto-boun...@ietf.org] 代表 Benjamin Kaduk > 发送时间: 2021年12月4日 6:30 > 收件人: Qin Wu > 抄送: draft-ietf-alto-performance-metr...@ietf.org; alto@ietf.org; The IESG > ; Y. Richard Yang ; alto-cha...@ietf.org > 主题: Re: [alto] Benjamin Kaduk's Discuss on > draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT) > > Hi Qin, > > On Thu, Dec 02, 2021 at 09:04:18AM +, Qin Wu wrote: > > Thanks Ben for detailed valuable review, see reply and clarification below. > > > > -邮件原件- > > >发件人: Benjamin Kaduk via Datatracker [mailto:nore...@ietf.org] > > >发送时间: 2021年12月2日 13:05 > > >收件人: The IESG > > >抄送: draft-ietf-alto-performance-metr...@ietf.org; > > >alto-cha...@ietf.org; alto@ietf.org; i...@j-f-s.de; i...@j-f-s.de > > >主题: Benjamin Kaduk's Discuss on > > >draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT) > > > > >Benjamin Kaduk has entered the following ballot position for > > >draft-ietf-alto-performance-metrics-20: Discuss > > > > >When responding, please keep the subject line intact and reply to all > > >email addresses included in the To and CC lines. (Feel free to cut > > >this introductory paragraph, however.) > > > > > > >Please refer to > > >https://www.ietf.org/blog/handling-iesg-ballot-positions/ > > >for more information about how to handle DISCUSS and COMMENT positions. > > > > > > >The document, along with other ballot positions, can be found here: > > >https://datatracker.ietf.org/doc/draft-ietf-alto-performance-metrics/ > > > > > > > > >- > > >- > > >DISCUSS: > > >- > > >- > > > > >These should all be trivial to resolve -- just some minor internal > > >inconsistencies that need to be fixed before publication. > > > > >The discussion of percentile statistical operator in §2.2 is internally > > >inconsistent -- if the percentile number must be an integer, then p99.9 is > > >not valid. > > [Qin Wu] Yes, the percentile is a number following the letter 'p', but > > in some case when high precision is needed, this percentile number will be > > further followed by an optional decimal part The decimal part should start > > with the '.' separator. Maybe the separator cause your confusion. See > > definition in section 2.2 for details: > > " > >percentile, with letter 'p' followed by a number: > > gives the percentile specified by the number following the letter > > 'p'. The number MUST be a non-negative JSON integer in the range > > [0, 100] (i.e., greater than or equal to 0 and less than or equal > > to 100), followed by an optional decimal part, if a higher > > precision is needed. The decimal part should start with the '.' > > separator (U+002E), and followed by a sequence of one or more > > ASCII numbers between '0' and '9'. > > " > > Let us know if you think separator should be changed or you live with the > > current form. > > Oops, that's my mistake and you are correct. Sorry about that; I agree that > no change is needed here. > > [Qin Wu] Great, thanks. > > >Also, the listing of "cost-source" values introduced by this document (in > > >§5.1) does not include "nominal", but we do also introduce "nominal". > > [Qin Wu] I agree with this inconsistency issue, should be fixed in the next > > version. Thanks. > > >Similarly, in §3.1.3 we refer to the "-" component of a cost > > >metric string, that has been generalized to an arbitrary statistical > > >operator. > > [Qin Wu] No, it is not arbitrary statistics operator, We did add a > > statement to say " > >Since t
Re: [alto] Benjamin Kaduk's Discuss on draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT)
Hi, Ben: -邮件原件- 发件人: alto [mailto:alto-boun...@ietf.org] 代表 Benjamin Kaduk 发送时间: 2021年12月4日 6:30 收件人: Qin Wu 抄送: draft-ietf-alto-performance-metr...@ietf.org; alto@ietf.org; The IESG ; Y. Richard Yang ; alto-cha...@ietf.org 主题: Re: [alto] Benjamin Kaduk's Discuss on draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT) Hi Qin, On Thu, Dec 02, 2021 at 09:04:18AM +, Qin Wu wrote: > Thanks Ben for detailed valuable review, see reply and clarification below. > > -邮件原件- > >发件人: Benjamin Kaduk via Datatracker [mailto:nore...@ietf.org] > >发送时间: 2021年12月2日 13:05 > >收件人: The IESG > >抄送: draft-ietf-alto-performance-metr...@ietf.org; > >alto-cha...@ietf.org; alto@ietf.org; i...@j-f-s.de; i...@j-f-s.de > >主题: Benjamin Kaduk's Discuss on > >draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT) > > >Benjamin Kaduk has entered the following ballot position for > >draft-ietf-alto-performance-metrics-20: Discuss > > >When responding, please keep the subject line intact and reply to all > >email addresses included in the To and CC lines. (Feel free to cut > >this introductory paragraph, however.) > > > >Please refer to > >https://www.ietf.org/blog/handling-iesg-ballot-positions/ > >for more information about how to handle DISCUSS and COMMENT positions. > > > >The document, along with other ballot positions, can be found here: > >https://datatracker.ietf.org/doc/draft-ietf-alto-performance-metrics/ > > > > >- > >- > >DISCUSS: > >- > >- > > >These should all be trivial to resolve -- just some minor internal > >inconsistencies that need to be fixed before publication. > > >The discussion of percentile statistical operator in §2.2 is internally > >inconsistent -- if the percentile number must be an integer, then p99.9 is > >not valid. > [Qin Wu] Yes, the percentile is a number following the letter 'p', but > in some case when high precision is needed, this percentile number will be > further followed by an optional decimal part The decimal part should start > with the '.' separator. Maybe the separator cause your confusion. See > definition in section 2.2 for details: > " >percentile, with letter 'p' followed by a number: > gives the percentile specified by the number following the letter > 'p'. The number MUST be a non-negative JSON integer in the range > [0, 100] (i.e., greater than or equal to 0 and less than or equal > to 100), followed by an optional decimal part, if a higher > precision is needed. The decimal part should start with the '.' > separator (U+002E), and followed by a sequence of one or more > ASCII numbers between '0' and '9'. > " > Let us know if you think separator should be changed or you live with the > current form. Oops, that's my mistake and you are correct. Sorry about that; I agree that no change is needed here. [Qin Wu] Great, thanks. > >Also, the listing of "cost-source" values introduced by this document (in > >§5.1) does not include "nominal", but we do also introduce "nominal". > [Qin Wu] I agree with this inconsistency issue, should be fixed in the next > version. Thanks. > >Similarly, in §3.1.3 we refer to the "-" component of a cost > >metric string, that has been generalized to an arbitrary statistical > >operator. > [Qin Wu] No, it is not arbitrary statistics operator, We did add a > statement to say " >Since the identifier >does not include the - component, the values will >represent median values. > " > The median value has been defined in the section 2.1 as middle-point > of the observation, see median definition in section 2.2 " >median: > the mid-point (i.e., p50) of the observations. > " Hmm, I am not sure whether my point came through properly or not. Let me try again. In Section 3.1.3, we see the text: Comment: Since the "cost-type" does not include the "cost-source" field, the values are based on "estimation". Since the identifier does not include the - component, the values will represent median values. This is the only place in the document where the string "" appears, and in particular we do not define a "percentile component" anywhere that I can see. We do, however, define a "statistical operator" string (component) of a cost m
Re: [alto] Benjamin Kaduk's Discuss on draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT)
Hi Qin, On Thu, Dec 02, 2021 at 09:04:18AM +, Qin Wu wrote: > Thanks Ben for detailed valuable review, see reply and clarification below. > > -邮件原件- > >发件人: Benjamin Kaduk via Datatracker [mailto:nore...@ietf.org] > >发送时间: 2021年12月2日 13:05 > >收件人: The IESG > >抄送: draft-ietf-alto-performance-metr...@ietf.org; alto-cha...@ietf.org; > >alto@ietf.org; i...@j-f-s.de; i...@j-f-s.de > >主题: Benjamin Kaduk's Discuss on draft-ietf-alto-performance-metrics-20: > >(with DISCUSS and COMMENT) > > >Benjamin Kaduk has entered the following ballot position for > >draft-ietf-alto-performance-metrics-20: Discuss > > >When responding, please keep the subject line intact and reply to all email > >addresses included in the To and CC lines. (Feel free to cut this > >introductory paragraph, however.) > > > >Please refer to https://www.ietf.org/blog/handling-iesg-ballot-positions/ > >for more information about how to handle DISCUSS and COMMENT positions. > > > >The document, along with other ballot positions, can be found here: > >https://datatracker.ietf.org/doc/draft-ietf-alto-performance-metrics/ > > > > >-- > >DISCUSS: > >-- > > >These should all be trivial to resolve -- just some minor internal > >inconsistencies that need to be fixed before publication. > > >The discussion of percentile statistical operator in §2.2 is internally > >inconsistent -- if the percentile number must be an integer, then p99.9 is > >not valid. > [Qin Wu] Yes, the percentile is a number following the letter 'p', but in > some case when high precision is needed, this percentile number will be > further followed by an optional decimal part > The decimal part should start with the '.' separator. Maybe the separator > cause your confusion. See definition in section 2.2 for details: > " >percentile, with letter 'p' followed by a number: > gives the percentile specified by the number following the letter > 'p'. The number MUST be a non-negative JSON integer in the range > [0, 100] (i.e., greater than or equal to 0 and less than or equal > to 100), followed by an optional decimal part, if a higher > precision is needed. The decimal part should start with the '.' > separator (U+002E), and followed by a sequence of one or more > ASCII numbers between '0' and '9'. > " > Let us know if you think separator should be changed or you live with the > current form. Oops, that's my mistake and you are correct. Sorry about that; I agree that no change is needed here. > >Also, the listing of "cost-source" values introduced by this document (in > >§5.1) does not include "nominal", but we do also introduce "nominal". > [Qin Wu] I agree with this inconsistency issue, should be fixed in the next > version. Thanks. > >Similarly, in §3.1.3 we refer to the "-" component of a cost > >metric string, that has been generalized to an arbitrary statistical > >operator. > [Qin Wu] No, it is not arbitrary statistics operator, We did add a statement > to say > " >Since the identifier >does not include the - component, the values will >represent median values. > " > The median value has been defined in the section 2.1 as middle-point of the > observation, see median definition in section 2.2 > " >median: > the mid-point (i.e., p50) of the observations. > " Hmm, I am not sure whether my point came through properly or not. Let me try again. In Section 3.1.3, we see the text: Comment: Since the "cost-type" does not include the "cost-source" field, the values are based on "estimation". Since the identifier does not include the - component, the values will represent median values. This is the only place in the document where the string "" appears, and in particular we do not define a "percentile component" anywhere that I can see. We do, however, define a "statistical operator" string (component) of a cost metric string, in Section 2.2. In particular, we do have options for the statistical operator string that are *not* representable as percentile values, such as stddev and cur. So, I think it is inaccurate to write "-" component here. I propose to instead say "Since the identifier does not include a statistical operator component, the values will represent median values." > >-- > >COMMENT: > >-- > > >All things considered, this is a pretty well-written document that was easy > >to read. That helped a lot as I reviewed it, especially so on a week with a > >pretty full agenda for the IESG telechat. > > >Section 2.2 > > >Should we say anything about how to handle a situation where a base metric > >identifier is so long that the statistical operator string cannot be > >appended whil
Re: [alto] Benjamin Kaduk's Discuss on draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT)
Thanks Ben for detailed valuable review, see reply and clarification below. -邮件原件- >发件人: Benjamin Kaduk via Datatracker [mailto:nore...@ietf.org] >发送时间: 2021年12月2日 13:05 >收件人: The IESG >抄送: draft-ietf-alto-performance-metr...@ietf.org; alto-cha...@ietf.org; >alto@ietf.org; i...@j-f-s.de; i...@j-f-s.de >主题: Benjamin Kaduk's Discuss on draft-ietf-alto-performance-metrics-20: (with >DISCUSS and COMMENT) >Benjamin Kaduk has entered the following ballot position for >draft-ietf-alto-performance-metrics-20: Discuss >When responding, please keep the subject line intact and reply to all email >addresses included in the To and CC lines. (Feel free to cut this introductory >paragraph, however.) >Please refer to https://www.ietf.org/blog/handling-iesg-ballot-positions/ >for more information about how to handle DISCUSS and COMMENT positions. >The document, along with other ballot positions, can be found here: >https://datatracker.ietf.org/doc/draft-ietf-alto-performance-metrics/ >-- >DISCUSS: >-- >These should all be trivial to resolve -- just some minor internal >inconsistencies that need to be fixed before publication. >The discussion of percentile statistical operator in §2.2 is internally >inconsistent -- if the percentile number must be an integer, then p99.9 is not >valid. [Qin Wu] Yes, the percentile is a number following the letter 'p', but in some case when high precision is needed, this percentile number will be further followed by an optional decimal part The decimal part should start with the '.' separator. Maybe the separator cause your confusion. See definition in section 2.2 for details: " percentile, with letter 'p' followed by a number: gives the percentile specified by the number following the letter 'p'. The number MUST be a non-negative JSON integer in the range [0, 100] (i.e., greater than or equal to 0 and less than or equal to 100), followed by an optional decimal part, if a higher precision is needed. The decimal part should start with the '.' separator (U+002E), and followed by a sequence of one or more ASCII numbers between '0' and '9'. " Let us know if you think separator should be changed or you live with the current form. >Also, the listing of "cost-source" values introduced by this document (in >§5.1) does not include "nominal", but we do also introduce "nominal". [Qin Wu] I agree with this inconsistency issue, should be fixed in the next version. Thanks. >Similarly, in §3.1.3 we refer to the "-" component of a cost >metric string, that has been generalized to an arbitrary statistical operator. [Qin Wu] No, it is not arbitrary statistics operator, We did add a statement to say " Since the identifier does not include the - component, the values will represent median values. " The median value has been defined in the section 2.1 as middle-point of the observation, see median definition in section 2.2 " median: the mid-point (i.e., p50) of the observations. " >-- >COMMENT: >-- >All things considered, this is a pretty well-written document that was easy to >read. That helped a lot as I reviewed it, especially so on a week with a >pretty full agenda for the IESG telechat. >Section 2.2 >Should we say anything about how to handle a situation where a base metric >identifier is so long that the statistical operator string cannot be appended >while remaining under the 32-character limit? [Qin Wu] I think base metric identifier should not be randomly selected, full name of base metric is not recommended, probably short name or abbreviation should be used if cost metric string is too long. But I am not sure we should set rule for this. Maybe the rule "The total length of the cost metric string MUST NOT exceed 32 " defined in RFC7285 is sufficient? > min: > the minimal value of the observations. > max: > the maximal value of the observations. > [...] >Should we say anything about what sampling period of observations is in scope >for these operators? [Qin Wu] I think sampling period of observation is related to Method of Measurement or Calculation, based on earlier discussion and agreement in the group, we believe this more depends on measurement methodology or metric definition, which in some cases not necessary or feasible, we can look into metric definition RFC for more details. see clarification in section 2 for more details. >Section 3.x.4 >If we're going to be recommending that implementations link to external >human-readable resources (e.g., for the SLA details of estimation >methodology), does the guidance from BCP 18 in indicating the language >of text come into play? >It's also a bi
[alto] Benjamin Kaduk's Discuss on draft-ietf-alto-performance-metrics-20: (with DISCUSS and COMMENT)
Benjamin Kaduk has entered the following ballot position for draft-ietf-alto-performance-metrics-20: Discuss When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to https://www.ietf.org/blog/handling-iesg-ballot-positions/ for more information about how to handle DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: https://datatracker.ietf.org/doc/draft-ietf-alto-performance-metrics/ -- DISCUSS: -- These should all be trivial to resolve -- just some minor internal inconsistencies that need to be fixed before publication. The discussion of percentile statistical operator in §2.2 is internally inconsistent -- if the percentile number must be an integer, then p99.9 is not valid. Also, the listing of "cost-source" values introduced by this document (in §5.1) does not include "nominal", but we do also introduce "nominal". Similarly, in §3.1.3 we refer to the "-" component of a cost metric string, that has been generalized to an arbitrary statistical operator. -- COMMENT: -- All things considered, this is a pretty well-written document that was easy to read. That helped a lot as I reviewed it, especially so on a week with a pretty full agenda for the IESG telechat. Section 2.2 Should we say anything about how to handle a situation where a base metric identifier is so long that the statistical operator string cannot be appended while remaining under the 32-character limit? min: the minimal value of the observations. max: the maximal value of the observations. [...] Should we say anything about what sampling period of observations is in scope for these operators? Section 3.x.4 If we're going to be recommending that implementations link to external human-readable resources (e.g., for the SLA details of estimation methodology), does the guidance from BCP 18 in indicating the language of the text come into play? It's also a bit surprising that we specify the new fields in the "parameters" of a metric just in passing in the prose, without a more prominent indication that we're defining a new field. Section 3.1.4 "nominal": Typically network one-way delay does not have a nominal value. Does that mean that they MUST NOT be generated, or that they should be ignored if received, or something else? (Similarly for the other sections where we say the same thing.) This description can be either free text for possible presentation to the user, or a formal specification; see [IANA-IPPM] for the specification on fields which should be included. [...] Is the IANA registry really the best reference for what fields to include? Tpically we would only refer to the registry when we care about the current state of registered values, but the need here seems to effectively be the column headings of the registry, which could be obtained from the RFC defining the registry. Section 3.3.3 Intended Semantics: To specify spatial and temporal aggregated delay variation (also called delay jitter)) with respect to the minimum delay observed on the stream over the one-way delay from the specified source and destination. The spatial aggregation level is specified in the query context (e.g., PID to PID, or endpoint to endpoint). I do appreciate the note about how this is not the normal statistics variation that follows this paragraph, but I also don't think this is a particularly clear or precise specification for how to produce the number that is to be reported. It also doesn't seem to fully align with the prior art in the IETF, e.g., RFC 3393. It seems like it would be highly preferrable to pick an existing RFC and refer to its specification for computing a delay variation value. (To be clear, such a reference would then be a normative reference.) Section 3.4.3 Intended Semantics: To specify the number of hops in the path from the specified source to the specified destination. The hop count is a basic measurement of distance in a network and can be exposed as the number of router hops computed from the routing protocols originating this information. [...] It seems like this could get a little messy if there are multiple routing protocols in use (e.g., both normal IP routing and an overlay network, as for service function chaining or other overlay schemes). I don't have any suggestions for disambiguating things, though, and if the usage is consistent within a given ALTO Server it may not have much impact on the clients. Section 3.4.4 "sla": Typically hop count does not have an SLA value.
