Dear Roman, I apologize for a very late reply. Unfortunately, I really had no time to provide my feedback earlier. Please, find my comments below.
-- Best regards, Alexey Filippov >-----Original Message----- >From: Roman Danyliw [mailto:[email protected]] >Sent: Friday, June 21, 2019 5:09 PM >To: Filippov Alexey <[email protected]>; Adam Roach ><[email protected]>; The IESG <[email protected]> >Cc: [email protected]; [email protected]; >[email protected]; Mo Zanaty <[email protected]> >Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: >(with DISCUSS and COMMENT) > >Hi! > >> -----Original Message----- >> From: iesg [mailto:[email protected]] On Behalf Of Filippov Alexey >> Sent: Thursday, June 13, 2019 1:15 PM >> To: Roman Danyliw <[email protected]>; Adam Roach <[email protected]>; The >> IESG <[email protected]> >> Cc: [email protected]; [email protected]; draft-ietf-netvc- >> [email protected]; Mo Zanaty <[email protected]> >> Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: >> (with DISCUSS and COMMENT) >> >> Dear Roman, >> >> Thank you a lot for your comments and raising the important questions. >> >> > How should a set of QPs be specified? >> As you probably know, a set of available QP values can be codec specific. >> Usually, a QP set is selected to cover medium bit-rate range that is >> considered to be the most complex for compressing. We proposed to set >> up the assessment process not only for this range but also for low and >> high QP ranges to cover a wider range of applications. According to my >> understanding, concrete QP values should be specified for candidate >> codecs in draft-ietf-netvc-testing >See below. Can you help with a specific pointer into draft-ietf-netvc-testing >which lists candidate codecs+QPs. AF: I guess the list of QPs for the candidate codec AV1 implemented in libaom is provided in Section 4.3 "Ranges" of draft-ietf-netvc-testing: "For the final evaluation described in [I-D.ietf-netvc-requirements], the quantizers used are 20, 24, 28, 32, 36, 39, 43, 47, 51, and 55. " >> >> > How should the quality values be calculated? >> It is explained in Section 4.1, namely: “To assess the quality of >> output >> (decoded) sequences, two indexes, PSNR [3] and MS-SSIM [3,11] are >> separately computed. In the case of the YCbCr color format, PSNR >> should be calculated for each color plane whereas MS-SSIM is >> calculated for luma channel only. In the case of the RGB color format, >> both metrics are computed for R, G and B channels. Thus, for each >> sequence, 30 RD-points for PSNR (i.e. three RD-curves, one for each >> channel) and 10 RD-points for MS- SSIM (i.e. one RD-curve, for luma >> channel only) should be calculated in the case of YCbCr. If content is >> encoded as RGB, 60 RD-points (30 for PSNR and >> 30 for MS-SSIM) should be calculated, i.e. three RD-curves (one for >> each >> channel) are computed for PSNR as well as three RD-curves (one for >> each >> channel) for MS-SSIM.” In references [3, 11], these 2 quality >> assessment metrics and the ways of how to calculate them are described in >> detail. >Got it. Thanks. >> >-- What does the text “(in a separate document on Internet video >> >codec >> testing)” mean? >> > Per “A list of video sequences that should be used for testing as >> > well as the >> 10 QP values for the reference codec are defined in a separate >> document ", what document is that? Is it draft-ietf-netvc-testing? >> According to my understanding, it is draft-ietf-netvc-testing. > >Can you please help me with the specific section references in >draft-ietf-netvc-testing as I'm not seeing it -- where is the explicit >reference codec being named and its associate 10 QP values? For what it's >worth I see Section 4.1 of this draft saying that: > As > the reference for evaluation, state-of-the-art video codecs such as > HEVC/H.265 [4,5] or VP9 must be used. The reference source code of > the HEVC/H.265 codec can be found at [6]. The HEVC/H.265 codec must > be configured according to [13] and Table 9. > >I'm looking for something that either unambiguously points to the reference >codec+QPs; or clearer language that says that all of this is out of scope in >this and/or the -testing draft, but is clear on what input into the processes >of this draft is required. AF: According to my understanding, VP9 implemented in libvpx is selected to be a reference codec and AV1 is a candidate (tested) one. So, the same QP range mentioned in Section 4.3 "Ranges" of draft-ietf-netvc-testing is applicable to both reference and candidate codecs. >> >What does “codec implementation (for both an encoder and a decoder) >> should cover the worst case of computational complexity, memory >> bandwidth, and physical memory size” mean? >> I’d like to thank Adam for his comment. I also think that his >> formulation ("...should take into consideration the worst-case...") is >> clearer. >> A more detailed explanation on this Section can be found in my >> response to the secdir reviewer: >> https://mailarchive.ietf.org/arch/msg/secdir/QYx- >> hl07aec5XhgqkldqgWFG3Is > >I'd like to offer a further refinement of what Adam proposed: > >Original: >However, it is worth noting that a codec implementation (for both an encoder >and a decoder) should cover the worst case of computational complexity, memory >bandwidth, and physical memory size (e.g., for decoded pictures used as >references). > >Adam: >However, it is worth noting that a codec implementation (for both an encoder >and a decoder) should take into consideration the worst-case computational >complexity, memory bandwidth, and physical memory size (e.g., for decoded >pictures used as references). > >Roman+Adam: >However, it is worth noting that a codec implementation (for both an encoder >and a decoder) should take into consideration the worst-case computational >complexity, memory bandwidth, and physical memory size needed to processes the >input (e.g., the decoded pictures used as references). AF: The rephrasing labeled "Roman+Adam" sounds good to me. I guess it should replace the original sentence. >> >Please add additional language that codec should be written in a >> >defensive >> style as they will be processing untrusted input. >> Good point. Thank you. I’ll add it. > >Regards, >Roman > >> >> -- >> Best regards, >> Alexey Filippov >> >> -----Original Message----- >> From: Roman Danyliw via Datatracker [mailto:[email protected]] >> Sent: Wednesday, June 12, 2019 4:25 AM >> To: The IESG <[email protected]> >> Cc: [email protected]; Mo Zanaty >> <[email protected]>; [email protected]; [email protected]; video- >> [email protected] >> Subject: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: >> (with DISCUSS and COMMENT) >> >> Roman Danyliw has entered the following ballot position for >> draft-ietf-netvc-requirements-09: Discuss >> >> When responding, please keep the subject line intact and reply to all >> email addresses included in the To and CC lines. (Feel free to cut >> this introductory paragraph, however.) >> >> >> Please refer to >> https://www.ietf.org/iesg/statement/discuss-criteria.html >> for more information about IESG DISCUSS and COMMENT positions. >> >> >> The document, along with other ballot positions, can be found here: >> https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/ >> >> >> >> ---------------------------------------------------------------------- >> DISCUSS: >> ---------------------------------------------------------------------- >> >> (1) I worry that the level of detail in the in the Compression >> Performance Evaluation (Section 4.1) is insufficient for implementation. >> Specifically: >> >> (a) Per “Initially, for the codec selected as a reference one (e.g., >> HEVC or VP9), a set of 10 QP quantization parameter) values should be >> specified (in a separate document on Internet video codec testing) and >> corresponding quality values should be calculated.” >> >> -- How should a set of QPs be specified? >> >> --How should the quality values be calculated? >> >> -- What does the text “(in a separate document on Internet video codec >> testing)” mean? >> >> (b) Per “A list of video sequences that should be used for testing as >> well as the 10 QP values for the reference codec are defined in a >> separate document ", what document is that? Is it draft-ietf-netvc-testing? >> >> (2) Per the Security Considerations Section (Section 5) >> >> -- What does “codec implementation (for both an encoder and a decoder) >> should cover the worst case of computational complexity, memory >> bandwidth, and physical memory size” mean? >> >> -- Please add additional language that codec should be written in a >> defensive style as they will be processing untrusted input. >> >> >> ---------------------------------------------------------------------- >> COMMENT: >> ---------------------------------------------------------------------- >> >> (1) There is something odd about the document formatting – the title >> and the first author last name in the footer is wrapped in “< … >” >> >> (2) It would be helpful to forward reference that acronyms are >> explained in Appendix A. >> >> (3) This draft uses the words should and must to prescribe action. >> Why wasn’t >> RFC2119 cited to explain these words? >> >> (4) Section 2.0. A reference to explain “YCbCr 4:2:0” would be >> helpful >> >> (5) Section 2.1. Per “high encoder complexity” and “decoding >> complexity”, I initiate read that as a qualitative measure. However, >> the text says “up to 10x and more” so that implies some quantitative >> measure. What is that? >> >> (6) Section 2.1. Expand QP values on first use >> >> (7) Section 2.x. The language around content doesn’t appear to be consist. >> For example: >> >> -- Section 2.1, Internet Video Streaming says “movies, TV-series and >> shows, and animation.” >> >> -- Section 2.2, IPTV says “television content” >> >> -- Section 2.5, Screen casting says “business presentations …, >> animation (cartoons), gaming content, data visualization, …, virtual >> desktop infrastructure (VDI), screen/desktop sharing and >> collaboration, supervisory control and data acquisition (SCADA) >> display, automotive/navigation display, cloud gaming, factory >> automation display, wireless display, display wall, digital operating room >> (DiOR), etc. >> >> What the difference between Section 2.1’s animation and Section 2.5’s >> cartoons? >> >> What’s the different between Section 2.1’s “movies, TV series …” and >> Section 2.2’s “television content”? >> >> (8) Section 2.5. The sentence “Currently, …” is very challenging to >> parse as it includes inline “i.e.,” and “etc”. >> >> (9) Section 2.5. Per “powerpoint, word documents”, these are specific >> Microsoft products. I recommend using more generic names. >> >> (10) Section 4. I found it confusing that an evaluation methodology >> was in a requirements document. I would have expected it in the >> draft-ietf-netvc- testing >> >> (11) Section 4.1. VP9 needs a reference. >> >> (12) Section 6. I don’t think this entire section is necessary. >> >> (13) Editorial Nits: >> -- Section 3. Style nit. s/chapter/section/ >> >> -- Section 4.1. Typo. s/computged/computed >> _______________________________________________ video-codec mailing list [email protected] https://www.ietf.org/mailman/listinfo/video-codec
