This series adds decoding of H.264 MVC streams - Blu-ray 3D and the consumer camcorder recordings that use the same coding - to the existing h264 decoder, alongside the MV-HEVC support hevcdec already has, and using the same view_ids/view_ids_available/view_pos_available option interface so that the ffmpeg CLI view specifiers work for both.
Only the base view is decoded by default, so nothing changes for a caller that does not ask for more. This work was significantly helped by Claude Opus 5. I have tried to keep the commits as clean and self contained as possible. I will take ownership of the changes and help with resolving any future issues. The series is bisectable - every commit builds and passes fate-h264 - and it has had a good deal of testing beyond that, including a public LibreELEC test build: https://forum.libreelec.tv/thread/30548-3d-support-builds-for-raspberry-pi FATE SAMPLE =========== Patch 18 adds three tests that need a new sample, h264/mvc-2view.264 (18628 bytes, md5 99901e05413ff29aecc6f191c664d8b2): https://drive.google.com/uc?export=download&id=1y6xxbM8nwFCgtCvZizujChhcQoz8_S4w I have sent a separate mail to samples-request. I generated the file with the JMVC 8.5 reference encoder from synthetic input, so it is freely redistributable: nothing in the real world MVC material I tested against can be, being either commercial films or camcorder captures of unclear origin. Both views decode bit-identically to JMVC's own reconstruction, so the tests check conformance rather than only locking in current behaviour. APPROACH ======== Rather than address every view explicitly, as hevcdec does with HEVCLayerContext, the decoder keeps the per-view state that actually differs - POC and the reference lists - in a small H264ViewContext and swaps it in and out around a view change. That avoids touching the ~250 sites that would otherwise have needed a layer index threaded through them, and keeps the diff to the parts of the decoder that genuinely differ per view. Patch 8 converts the decoder to receive_frame(). An MVC access unit produces two frames from one packet, which the decode() callback cannot express. Patches 1 and 2 are not multiview at all. Both fix frame_num gap concealment on damaged input, and both reproduce on an unmodified tree without any of the rest of this series. They come first because the multiview paths make the second one about five times easier to reach: on one corpus file it deadlocks 1 run in 14 as things stand, and 5 in 14 once a view is selected with -map 0:v:0:view:1. WHAT WORKS ========== Both views, bit-identical to a single threaded decode, under frame threading, slice threading and no threading, at every thread count I tested, including on a Matroska file whose blocks do not start on an access unit boundary - the dependent view slices of one access unit sit at the head of the packet that goes on to carry the base view slices of the next. That case drove several of the fixes folded into patches 11 and 14. Field coded multiview works (patch 17). Hardware acceleration does not, and is refused with a warning: every h264 hwaccel builds its reference list from short_ref/long_ref and skips pictures with reference == 0, which is exactly what an inter-view reference looks like. Patch 15 has the detail. TESTING ======= - make fate: 5649/5649 passing. - Each of the 18 commits builds and passes fate-h264 standalone, so the series bisects cleanly. - A corpus of 43 real MVC and 3D files, 35 GB, decoded end to end with both views mapped: Blu-ray rips, camcorder .264/.MTS/.ssif, and Matroska. Every file that carries two views decodes both of them with no errors and with matching frame counts, the one exception being a Matroska file that is itself missing the dependent view slices of its last access unit (899 against 900, which is what the file contains). Frame threading and slice threading give byte-identical framecrc output over the whole corpus. Where I compared against an unpatched build - the files exercising the awkward paths, not all 43 - the base view output was bit-exact. - Corrupted input, as doc/developer.texi asks for: 12000 decodes of damaged streams across 16 files, under 10 threading configurations and 13 mapping/decoder-option combinations, with the noise bitstream filter, truncation and byte flipping. Clean under ASan and UBSan, and no hangs. Getting there found four deadlocks and a batch of undefined behaviour, all fixed in the patches that introduced them, except the concealment one described above. Three faults that turned up during that and are NOT this series' - each reproduces on an unmodified tree, and I will report them separately: - fftools/ffmpeg_sched.c:2168 asserts on some damaged input. - -err_detect +explode with frame threading deadlocks on plain single-view H.264. - Mapping both views in one ffmpeg command collapses the dependent view's timestamps in fftools when the two views share a PTS, which they do by definition. Mapping either view alone is correct, and the decoder emits correct timestamps in both cases. This looks like it would affect MV-HEVC equally. KNOWN GAPS ========== - mpegts carries the two views on separate PIDs (0x1011 and 0x1012) and presents them as separate streams, so a Blu-ray .ssif or .MTS still decodes only the view that was mapped. Associating the PIDs needs demuxer work that is not in this series. - cbs_h264 still refuses NAL 15 and NAL 20. Dom Cobley (18): avcodec/h264dec: always initialise a concealed frame_num gap picture avcodec/h264dec: do not await a field that was never decoded avcodec/h264_ps: fix swapped MVC profile comments avcodec/h264: parse subset SPS (NAL 15) and its MVC extension avcodec/h2645_parse: parse the H.264 MVC NAL unit header extension avcodec/h264dec: add multiview view_ids options avcodec/h264dec: add a per-view context avcodec/h264dec: switch to receive_frame() avcodec/h264: decode dependent views and inter-view references avcodec/h264_slice: do not reclaim the picture awaiting output avcodec/h264dec: output both views of an access unit together avcodec/h264: implement inter-view reference list modification avcodec/h264dec: pick up the view list wherever the subset SPS appears avcodec/h264dec: support frame and slice threading for multiview avcodec/h264dec: refuse multiview with hardware acceleration avformat/mov: read the mvcC box avcodec/h264dec: support field coded multiview doc/decoders, Changelog, fate: document and test H.264 MVC Changelog | 1 + doc/decoders.texi | 45 +++ libavcodec/bsf/extract_extradata.c | 2 +- libavcodec/h2645_parse.c | 38 ++ libavcodec/h2645_parse.h | 13 + libavcodec/h264_direct.c | 15 +- libavcodec/h264_parse.c | 4 + libavcodec/h264_parser.c | 3 + libavcodec/h264_picture.c | 2 + libavcodec/h264_ps.c | 161 +++++++- libavcodec/h264_ps.h | 57 +++ libavcodec/h264_refs.c | 285 +++++++++++++- libavcodec/h264_slice.c | 573 +++++++++++++++++++++++++++-- libavcodec/h264dec.c | 307 ++++++++++++++-- libavcodec/h264dec.h | 182 +++++++++ libavformat/mov.c | 122 ++++++ tests/fate/h264.mak | 9 + tests/ref/fate/flv-demux | 2 +- tests/ref/fate/h264-mvc-base | 21 ++ tests/ref/fate/h264-mvc-both | 42 +++ tests/ref/fate/h264-mvc-dependent | 21 ++ tests/ref/fate/mov-zombie | 2 +- tests/ref/fate/ts-small-demux | 2 +- 23 files changed, 1823 insertions(+), 86 deletions(-) create mode 100644 tests/ref/fate/h264-mvc-base create mode 100644 tests/ref/fate/h264-mvc-both create mode 100644 tests/ref/fate/h264-mvc-dependent -- 2.53.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
