Coming in very late on thread.... sequences need to be grouped in multiple dimensions. I'd suggest that one dimension be:
- Talking heads (still camera, little movement) - Cinematic material (slow camera moves, dramatic cuts, sudden moves) - Amateur video (camera shake, sudden moves) - Artificially generated (no camera shake, color gradients smoother than normal, no inter-frame movement blur) Another dimension should be available bandwidth - 0.1 Mbits, 1 Mbits, 10 Mbits, 100 Mbits I've come to the conclusion that varying resolution should be considered one tool for achieving lossy compression, and should not be a primary dimension - YMMV. (that said, existing source material has resolutions, and a codec should make sure to handle all resolutions well.) A third dimension should be available computing resources for encode and decode: - Infinite encode resources available (test style) - Encode 10x slower than real time (production environments) - Encode faster than real time (interactive + overhead) - Decode in real time, stalls acceptable - Decode in real time, stalls not acceptable A fourth dimension is about the use of lookahead: - Any lookahead, including multipass, is acceptable - Lookahead of a second or two is acceptable - Lookahead of 100 ms is acceptable - No lookahead at all is permissible (minimum delay) I'm running some codec comparisons on compare-codecs.appspot.com (open source project) - incorporating Derf's clip set is high on my TODO list. Den 27. jan. 2015 10:17, skrev Mo Zanaty (mzanaty): > Since a particularly interesting comparison will be HEVC, should we > consider some JCT-VC standard sequences and operating points? > > We can redefine the classes of sequences to be the most relevant for > initial experiments. For example: > > Class 1: Video Conferencing > a) 360p 30Hz 0.1-1Mbps > b) 720p 30Hz 0.2-2Mbps > c) 720p 60Hz 0.3-3Mbps > d) 1080p 30Hz 0.4-4Mbps > e) 1080p 60Hz 0.6-6Mbps > f) 2160p 30Hz 1.2-12Mbps > > Class 2: Video Streaming > a) 360p 30Hz 0.2-2Mbps > b) 720p 30Hz 0.3-3Mbps > c) 1080p 24Hz 0.6-6Mbps > d) 2160p 24Hz 2-20Mbps > > 1b and 2c may be the most interesting operating points for many. > > We can also consider other classes like professional applications, screen > content / wireless display, etc. with relatively higher rates. > > > Mo > > > On 1/26/15, 8:09 PM, Monty Montgomery <[email protected]> wrote: > > Right, and our use of AWCY and test clips being mostly regression > testing at the moment (and specific to our particular glass box), I > wasn't proposing the particulars what we're doing right now as the > starting point. Rather, it could be, but used a different way, not as > we're using it. > > Monty > > On Sun, Jan 25, 2015 at 1:14 PM, Eric Rescorla <[email protected]> wrote: >> Following up to myself: what i'm hoping here is to get a consensus list >> of >> the domains of interest so that we can then evaluate our progress against >> them. >> >> -Ekr >> >> >> On Sun, Jan 25, 2015 at 12:57 PM, Eric Rescorla <[email protected]> wrote: >>> >>> >>> On Fri, Jan 23, 2015 at 1:45 PM, Monty Montgomery <[email protected]> >>> wrote: >>>> >>>> Hi everyone, >>>> >>>> Testing discussions are already happening in private and elsewhere. >>>> The basic points are usually pretty similar and I expect they're going >>>> to keep coming up, so I'm going to try centralizing it here. >>>> >>>> The Daala team is using a small number of fairly short test sequences >>>> in an automated testing system ('Are We compressed Yet?) mostly as a >>>> way of doing regression testing. It's also used to sanity check small >>>> improvements that are unlikely to be apparent in visual testing. The >>>> various graphs we've shown in demos and the like are coming from AWCY. >>>> >>>> One point that's been raised several times is that we're not testing >>>> reasonable or practical bitrates. Or, rather, that our range includes >>>> the typical useful rates, but we also go way up to insanely high rates >>>> no one would ever use. That's mainly because AWCY is meant as a >>>> sanity/regression check, and many subtle bugs we've had in the past >>>> have only popped out at near-lossless quality levels. Automated >>>> testing should report these rates, but we don't intend to use them in >>>> evaluation of relative practical performance. >>> >>> >>> To sharpen this point a little bit, it's probably useful to distinguish >>> between testing for performance evaluation and testing for regression, >>> continuous integration etc. >>> >>> For the purpose of continuous integration, we should, as you >>> suggest, test all sorts of parameter sets, including insane ones, >>> as well as how the decoder handles bogus encodings, etc. For >>> the purpose of performance evaluation/comparison, I would suggest >>> that it would be useful to try to define a set of strawman scenarios >>> which we would use. I'm sure others have different opinions, but >>> for me the scenarios of most interest are basically videoconferencing, >>> so roughly speaking: >>> >>> - Some set of sizes like: >>> - QCIF >>> - VGA >>> - 1080p >>> >>> - 30fps. >>> - Bit rates on the order of 200kps - 5ish Mbps, depending on the size >>> (I could be a bit off here). >>> >>> >>> Presumably we want some streaming use cases, but I have a less good >>> handle on what typical streaming is... >>> >>> -Ekr >>> >>> >>> >>> >>> >>> >>>> Another point that keeps coming up is that we should come up with a >>>> standard set of sequences and rates for trading and evaluating first- >>>> and probably second-line evaluations. Full agreement there, though >>>> our own frontline tetsing right now is only using a handful of clips >>>> of a few seconds each (though our available library is much larger). >>>> >>>> Monty >>>> >>>> _______________________________________________ >>>> video-codec mailing list >>>> [email protected] >>>> https://www.ietf.org/mailman/listinfo/video-codec >>> >>> >> > > _______________________________________________ > video-codec mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/video-codec > > _______________________________________________ > video-codec mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/video-codec > _______________________________________________ video-codec mailing list [email protected] https://www.ietf.org/mailman/listinfo/video-codec
