>> My plan was to try to script some automated tests to run weekly on the
> build server. That way we would know if any avsync regressions occurred. But > it > didn't occur to me that we might have to fix up some producers/consumers. My > plan was to use +/-3ms as my pass/fail criteria (hence my previous comment). > But > maybe it would make more sense to use one frame duration - at least until > everything gets dialed in. >> > > I would like to develop a plan to test for regression against a > baseline so I can try to make a change or improvement and have more > automated testing to verify it. Starting with the avformat producer, > we need a set of reference inputs, but we do not have them, and it > does not make sense to produce them with the MLT avformat consumer. > But we can use blipflash producer to generate a raw DV and verify that > with dvthumbs.c. Then, we can use the ffmpeg command line tool to > generate a variety of encodings. We still do not know if those are > correct, but they do provide a best effort baseline against which we > can measure deviations to check for regressions, analyze trends, and > investigate the hotspots (popular or important formats that show a > large offset). I did some testing with ffmpeg and I'm pretty sure that much of the offset is comming from libav itself. For example, if you use libdv to round trip a blipflash through melt, you get an offset of 0. But if you use the dv output from melt and round trip it through ffmpeg as AVI, and then run it back through melt, you get an offset. So I tested a bunch of formats and I found that the A/V offset through melt is never worse than one video frame. So I set one video frame as the success threshold. The script is here: https://github.com/mltframework/mlt-scripts/blob/master/test/test_avsync.sh It will run every week along with the other autmoated tests. It tests 7 different output formats. We can add more or refine them. Let me know what formats you think are the most important. While the test won't tell us if the AV sync gets worse by a few milliseconds, it will tell us if there is a catostrophic AV sync error - which I think is probably the most important thing. Here are the current results, FYI: libdv: 0.0ms avformat-avi: -23.02ms avformat-dv: -33.38ms avformat-mkv: 0.0ms avformat-mov: 12.02ms avformat-mp4: 12.02ms avformat-mpg: -10.02ms Of course, the audio and video codecs probably have some effect on the sync in addition to the container format. I didn't dig too deep into that as most of the tests use default codecs. ~BM ------------------------------------------------------------------------------ Symantec Endpoint Protection 12 positioned as A LEADER in The Forrester Wave(TM): Endpoint Security, Q1 2013 and "remains a good choice" in the endpoint security space. For insight on selecting the right partner to tackle endpoint security challenges, access the full report. http://p.sf.net/sfu/symantec-dev2dev _______________________________________________ Mlt-devel mailing list Mlt-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mlt-devel
