Hi Roman, -----Original Message----- From: Roman Danyliw [mailto:[email protected]] Sent: Friday, November 15, 2019 10:01 AM To: Filippov Alexey <[email protected]>; Adam Roach <[email protected]>; The IESG <[email protected]> Cc: [email protected]; [email protected]; [email protected]; Mo Zanaty <[email protected]>; Elena Alshina <[email protected]> Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)
>Hi Alexey! >> -----Original Message----- >> From: Filippov Alexey <[email protected]> >> Sent: Tuesday, October 22, 2019 8:20 AM >> To: Roman Danyliw <[email protected]>; Adam Roach <[email protected]>; The >> IESG <[email protected]> >> Cc: [email protected]; [email protected]; draft-ietf-netvc- >> [email protected]; Mo Zanaty <[email protected]>; Elena Alshina >> <[email protected]> >> Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: >> (with DISCUSS and COMMENT) >> >> Dear Roman, >> >> I apologize for a very late reply. Unfortunately, I really had no time >> to provide my feedback earlier. Please, find my comments below. >> >> -- >> Best regards, >> Alexey Filippov >> >> >-----Original Message----- >> >From: Roman Danyliw [mailto:[email protected]] >> >Sent: Friday, June 21, 2019 5:09 PM >> >To: Filippov Alexey <[email protected]>; Adam Roach >> ><[email protected]>; The IESG <[email protected]> >> >Cc: [email protected]; [email protected]; >> >[email protected]; Mo Zanaty <[email protected]> >> >Subject: RE: Roman Danyliw's Discuss on >> >draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT) >> > >> >Hi! >> > >> >> -----Original Message----- >> >> From: iesg [mailto:[email protected]] On Behalf Of Filippov >> >> Alexey >> >> Sent: Thursday, June 13, 2019 1:15 PM >> >> To: Roman Danyliw <[email protected]>; Adam Roach <[email protected]>; >> The >> >> IESG <[email protected]> >> >> Cc: [email protected]; [email protected]; draft-ietf-netvc- >> >> [email protected]; Mo Zanaty <[email protected]> >> >> Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: >> >> (with DISCUSS and COMMENT) >> >> >> >> Dear Roman, >> >> >> >> Thank you a lot for your comments and raising the important questions. >> >> >> >> > How should a set of QPs be specified? >> >> As you probably know, a set of available QP values can be codec specific. >> >> Usually, a QP set is selected to cover medium bit-rate range that >> >> is considered to be the most complex for compressing. We proposed >> >> to set up the assessment process not only for this range but also >> >> for low and high QP ranges to cover a wider range of applications. >> >> According to my understanding, concrete QP values should be >> >> specified for candidate codecs in draft-ietf-netvc-testing >> >> >See below. Can you help with a specific pointer into >> >draft-ietf-netvc-testing >> which lists candidate codecs+QPs. >> AF: I guess the list of QPs for the candidate codec AV1 implemented in >> libaom is provided in Section 4.3 "Ranges" of draft-ietf-netvc-testing: >> "For the final evaluation described in [I-D.ietf-netvc-requirements], >> the quantizers used are 20, 24, 28, 32, 36, 39, 43, 47, 51, and 55. " > >>An explicit reference to netvc-testing would address my concern, and >>specifically make the following sentences clearer. > > >"Initially, for the codec selected as a reference one (e.g., HEVC or VP9), a >set of 10 QP (quantization parameter) values should be specified (in a >separate document on Internet video codec testing) and corresponding quality >values should be calculated." > >"A list of video sequences that should be used for testing as well as the 10 >QP values for the reference codec are defined in a separate document." > >Thanks. AF: Thank you for your comments. Explicit references to the -testing draft will be added to the next revision of the document. >> >> >> >> > How should the quality values be calculated? >> >> It is explained in Section 4.1, namely: “To assess the quality of >> >> output >> >> (decoded) sequences, two indexes, PSNR [3] and MS-SSIM [3,11] are >> >> separately computed. In the case of the YCbCr color format, PSNR >> >> should be calculated for each color plane whereas MS-SSIM is >> >> calculated for luma channel only. In the case of the RGB color >> >> format, both metrics are computed for R, G and B channels. Thus, >> >> for each sequence, 30 RD-points for PSNR (i.e. three RD-curves, one >> >> for each >> >> channel) and 10 RD-points for MS- SSIM (i.e. one RD-curve, for luma >> >> channel only) should be calculated in the case of YCbCr. If content >> >> is encoded as RGB, 60 RD-points (30 for PSNR and >> >> 30 for MS-SSIM) should be calculated, i.e. three RD-curves (one for >> >> each >> >> channel) are computed for PSNR as well as three RD-curves (one for >> >> each >> >> channel) for MS-SSIM.” In references [3, 11], these 2 quality >> >> assessment metrics and the ways of how to calculate them are >> >> described in >> detail. >> >> >Got it. Thanks. >> >> >> >-- What does the text “(in a separate document on Internet video >> >> >codec >> >> testing)” mean? >> >> > Per “A list of video sequences that should be used for testing as >> >> > well as the >> >> 10 QP values for the reference codec are defined in a separate >> >> document ", what document is that? Is it draft-ietf-netvc-testing? >> >> According to my understanding, it is draft-ietf-netvc-testing. >> > >> >Can you please help me with the specific section references in >> >draft-ietf-netvc- >> testing as I'm not seeing it -- where is the explicit reference codec >> being named and its associate 10 QP values? For what it's worth I see >> Section 4.1 of this draft saying that: >> >> > As >> > the reference for evaluation, state-of-the-art video codecs such as >> > HEVC/H.265 [4,5] or VP9 must be used. The reference source code of >> > the HEVC/H.265 codec can be found at [6]. The HEVC/H.265 codec must >> > be configured according to [13] and Table 9. >> > >> >I'm looking for something that either unambiguously points to the >> >reference >> codec+QPs; or clearer language that says that all of this is out of >> codec+scope in this >> and/or the -testing draft, but is clear on what input into the >> processes of this draft is required. >> AF: According to my understanding, VP9 implemented in libvpx is >> selected to be a reference codec and AV1 is a candidate (tested) one. >> So, the same QP range mentioned in Section 4.3 "Ranges" of >> draft-ietf-netvc-testing is applicable to both reference and candidate >> codecs. >> >> >> >What does “codec implementation (for both an encoder and a >> >> >decoder) >> >> should cover the worst case of computational complexity, memory >> >> bandwidth, and physical memory size” mean? >> >> I’d like to thank Adam for his comment. I also think that his >> >> formulation ("...should take into consideration the worst-case...") is >> >> clearer. >> >> A more detailed explanation on this Section can be found in my >> >> response to the secdir reviewer: >> >> https://mailarchive.ietf.org/arch/msg/secdir/QYx- >> >> hl07aec5XhgqkldqgWFG3Is >> > >> >I'd like to offer a further refinement of what Adam proposed: >> > >> >Original: >> >However, it is worth noting that a codec implementation (for both an >> >encoder >> and a decoder) should cover the worst case of computational >> complexity, memory bandwidth, and physical memory size (e.g., for >> decoded pictures used as references). >> > >> >Adam: >> >However, it is worth noting that a codec implementation (for both an >> >encoder >> and a decoder) should take into consideration the worst-case >> computational complexity, memory bandwidth, and physical memory size >> (e.g., for decoded pictures used as references). >> > >> >Roman+Adam: >> >However, it is worth noting that a codec implementation (for both an >> >encoder >> and a decoder) should take into consideration the worst-case >> computational complexity, memory bandwidth, and physical memory size >> needed to processes the input (e.g., the decoded pictures used as >> references). >> AF: The rephrasing labeled "Roman+Adam" sounds good to me. I guess it >> should replace the original sentence. > >The proposed text above works for me. Thanks. AF: Good to know. Thank you! -- Best regards, Alexey Filippov >> >> >Please add additional language that codec should be written in a >> >> >defensive >> >> style as they will be processing untrusted input. >> >> Good point. Thank you. I’ll add it. > >Thanks. > >Regards, >Roman > >> >Regards, >> >Roman >> > >> >> >> >> -- >> >> Best regards, >> >> Alexey Filippov >> >> >> >> -----Original Message----- >> >> From: Roman Danyliw via Datatracker [mailto:[email protected]] >> >> Sent: Wednesday, June 12, 2019 4:25 AM >> >> To: The IESG <[email protected]> >> >> Cc: [email protected]; Mo Zanaty >> >> <[email protected]>; [email protected]; [email protected]; >> >> video- [email protected] >> >> Subject: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: >> >> (with DISCUSS and COMMENT) >> >> >> >> Roman Danyliw has entered the following ballot position for >> >> draft-ietf-netvc-requirements-09: Discuss >> >> >> >> When responding, please keep the subject line intact and reply to >> >> all email addresses included in the To and CC lines. (Feel free to >> >> cut this introductory paragraph, however.) >> >> >> >> >> >> Please refer to >> >> https://www.ietf.org/iesg/statement/discuss-criteria.html >> >> for more information about IESG DISCUSS and COMMENT positions. >> >> >> >> >> >> The document, along with other ballot positions, can be found here: >> >> https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/ >> >> >> >> >> >> >> >> ------------------------------------------------------------------- >> >> -- >> >> - >> >> DISCUSS: >> >> ------------------------------------------------------------------- >> >> -- >> >> - >> >> >> >> (1) I worry that the level of detail in the in the Compression >> >> Performance Evaluation (Section 4.1) is insufficient for implementation. >> Specifically: >> >> >> >> (a) Per “Initially, for the codec selected as a reference one >> >> (e.g., HEVC or VP9), a set of 10 QP quantization parameter) values >> >> should be specified (in a separate document on Internet video codec >> >> testing) and corresponding quality values should be calculated.” >> >> >> >> -- How should a set of QPs be specified? >> >> >> >> --How should the quality values be calculated? >> >> >> >> -- What does the text “(in a separate document on Internet video >> >> codec testing)” mean? >> >> >> >> (b) Per “A list of video sequences that should be used for testing >> >> as well as the 10 QP values for the reference codec are defined in >> >> a separate document ", what document is that? Is it >> >> draft-ietf-netvc-testing? >> >> >> >> (2) Per the Security Considerations Section (Section 5) >> >> >> >> -- What does “codec implementation (for both an encoder and a >> >> decoder) should cover the worst case of computational complexity, >> >> memory bandwidth, and physical memory size” mean? >> >> >> >> -- Please add additional language that codec should be written in a >> >> defensive style as they will be processing untrusted input. >> >> >> >> >> >> ------------------------------------------------------------------- >> >> -- >> >> - >> >> COMMENT: >> >> ------------------------------------------------------------------- >> >> -- >> >> - >> >> >> >> (1) There is something odd about the document formatting – the >> >> title and the first author last name in the footer is wrapped in “< … >” >> >> >> >> (2) It would be helpful to forward reference that acronyms are >> >> explained in Appendix A. >> >> >> >> (3) This draft uses the words should and must to prescribe action. >> >> Why wasn’t >> >> RFC2119 cited to explain these words? >> >> >> >> (4) Section 2.0. A reference to explain “YCbCr 4:2:0” would be >> >> helpful >> >> >> >> (5) Section 2.1. Per “high encoder complexity” and “decoding >> >> complexity”, I initiate read that as a qualitative measure. >> >> However, the text says “up to 10x and more” so that implies some >> >> quantitative >> measure. What is that? >> >> >> >> (6) Section 2.1. Expand QP values on first use >> >> >> >> (7) Section 2.x. The language around content doesn’t appear to be >> >> consist. >> >> For example: >> >> >> >> -- Section 2.1, Internet Video Streaming says “movies, TV-series >> >> and shows, and animation.” >> >> >> >> -- Section 2.2, IPTV says “television content” >> >> >> >> -- Section 2.5, Screen casting says “business presentations …, >> >> animation (cartoons), gaming content, data visualization, …, >> >> virtual desktop infrastructure (VDI), screen/desktop sharing and >> >> collaboration, supervisory control and data acquisition (SCADA) >> >> display, automotive/navigation display, cloud gaming, factory >> >> automation display, wireless display, display wall, digital >>> >> operating room >> (DiOR), etc. >> >> >> >> What the difference between Section 2.1’s animation and Section >>>> >> 2.5’s cartoons? >> >> >> >> What’s the different between Section 2.1’s “movies, TV series …” >> >> and Section 2.2’s “television content”? >> >> >> >> (8) Section 2.5. The sentence “Currently, …” is very challenging >> >> to parse as it includes inline “i.e.,” and “etc”. >> >> >> >> (9) Section 2.5. Per “powerpoint, word documents”, these are >> >> specific Microsoft products. I recommend using more generic names. >> >> >> >> (10) Section 4. I found it confusing that an evaluation >> >> methodology was in a requirements document. I would have expected >>> >> it in the >> >> draft-ietf-netvc- testing > >>> >> >> (11) Section 4.1. VP9 needs a reference. > >>> > >>> (12) Section 6. I don’t think this entire section is necessary. > >>> > >>> (13) Editorial Nits: > >>> -- Section 3. Style nit. s/chapter/section/ > >>> > >>> -- Section 4.1. Typo. s/computged/computed > >>> _______________________________________________ video-codec mailing list [email protected] https://www.ietf.org/mailman/listinfo/video-codec
