PR #25012 opened by rogersnm URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25012 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25012.patch
# Summary of changes When a V4L2 stateful decoder reports a source change to a new capture size, `v4l2_handle_event()` reinitialized capture at once. `VIDIOC_STREAMOFF` on capture discards every buffer the decoder has filled that has not been dequeued yet, and the decoder interface signals the change before it has returned the frames decoded ahead of it (dev-decoder, "Dynamic Resolution Change"): > 1. After encountering a resolution change in the stream, the decoder sends a > `V4L2_EVENT_SOURCE_CHANGE` event [...] > 2. The decoder will then process and decode all remaining buffers from before > the resolution change point. The last buffer from before the change must be > marked with the `V4L2_BUF_FLAG_LAST` flag, similarly to the `Drain` sequence > above. `v4l2_dequeue_v4l2buf()` handles `POLLPRI` before `POLLIN`, so every frame still waiting when the event is read was lost: one or several, depending on timing. With this change, after a source change mid-stream the decoder goes on dequeuing until that LAST buffer (or `-EPIPE` past it), returning the frames before it, and then reinitializes capture. Two cases are handled at once, as before: - a change reported before any frame has been dequeued, at the start of a stream (no frames can be lost, and a driver need not send LAST there); - a change that leaves no capture buffer with the driver to return the frames in. ## Testing The only V4L2 stateful decoder I have is a virtio-media one: [lighter](https://github.com/fieldwork-ai/lighter), a Linux VM for macOS whose decoder runs on VideoToolbox, with a Debian bookworm guest. I would be glad of a run on hardware decoders (venus, mtk-vcodec, wave5, bcm2835-codec). Two H.264 streams that change size mid-stream: 90 frames at 640x360 then 30 at 320x180, and the reverse. Each run compares every frame's size and hash with the software decoder (`-noautoscale -f framemd5`): | | Runs with every frame identical | |---|---| | master | 38 of 40 (the other two lost 1 and 9 frames) | | this PR | 80 of 80 | - No change with this PR, against software: a 1080p H.264 stream with B-frames, an HEVC stream, end-of-stream draining, and `h264_v4l2m2m` encoding. - `make fate`: all 5,664 tests pass. To reproduce: ```sh ffmpeg -f lavfi -i testsrc2=size=640x360:rate=30 -frames:v 90 -c:v libx264 -threads 1 \ -profile:v baseline -bf 0 -g 90 -b:v 2M -f h264 large.h264 ffmpeg -f lavfi -i mandelbrot=size=320x180:rate=30 -frames:v 30 -c:v libx264 -threads 1 \ -profile:v baseline -pix_fmt yuv420p -bf 0 -g 30 -f h264 small.h264 cat large.h264 small.h264 > shrink.h264 ffmpeg -f h264 -r 30 -i shrink.h264 -c:v copy shrink.mkv ffmpeg -i shrink.mkv -noautoscale -pix_fmt yuv420p -f framemd5 sw.md5 ffmpeg -c:v h264_v4l2m2m -i shrink.mkv -noautoscale -pix_fmt yuv420p -f framemd5 hw.md5 diff <(grep -v '^#' sw.md5 | awk -F, '{print $5, $NF}') \ <(grep -v '^#' hw.md5 | awk -F, '{print $5, $NF}') ``` >From d879db8e5b65b8df75c1b0c3612aacdc1ff6b30d Mon Sep 17 00:00:00 2001 From: Nick Rogers <[email protected]> Date: Sun, 11 Oct 2026 16:57:53 +0100 Subject: [PATCH] avcodec/v4l2_context: return the frames decoded before a resolution change When the decoder reported a source change to a new capture size, capture was reinitialized at once, and VIDIOC_STREAMOFF on capture discards every buffer the decoder has filled that has not been dequeued yet. The stateful decoder interface (dev-decoder, "Dynamic Resolution Change") signals the change before the decoder has returned the frames decoded ahead of it, and ends them with a buffer flagged V4L2_BUF_FLAG_LAST, so those frames were lost: one or several, depending on how many were waiting when the event was read. After a source change mid-stream, keep dequeuing until that LAST buffer, or -EPIPE past it, then reinitialize. A change reported before any frame has been dequeued, at the start of a stream, is handled at once as before, as is one that leaves no capture buffer with the driver to return the frames in. Signed-off-by: Nick Rogers <[email protected]> --- libavcodec/v4l2_context.c | 110 +++++++++++++++++++++++++++++++------- libavcodec/v4l2_context.h | 13 +++++ 2 files changed, 105 insertions(+), 18 deletions(-) diff --git a/libavcodec/v4l2_context.c b/libavcodec/v4l2_context.c index be1df3785b..fcc5e2d4ca 100644 --- a/libavcodec/v4l2_context.c +++ b/libavcodec/v4l2_context.c @@ -169,6 +169,53 @@ static int v4l2_start_decode(V4L2Context *ctx) return 0; } +/** + * reinitialize capture for a new format + * returns 1 if reinit was successful, negative if it failed + */ +static int v4l2_change_capture_format(V4L2m2mContext *s, struct v4l2_format *cap_fmt) +{ + int ret = 0; + + s->capture.format_change = 0; + s->capture.height = v4l2_get_height(cap_fmt); + s->capture.width = v4l2_get_width(cap_fmt); + s->capture.sample_aspect_ratio = v4l2_get_sar(&s->capture); + + s->reinit = 1; + + if (s->avctx) + ret = ff_set_dimensions(s->avctx, s->capture.width, s->capture.height); + if (ret < 0) + av_log(logger(&s->capture), AV_LOG_WARNING, "update avcodec height and width\n"); + + ret = ff_v4l2_m2m_codec_reinit(s); + if (ret) { + av_log(logger(&s->capture), AV_LOG_ERROR, "v4l2_m2m_codec_reinit\n"); + return AVERROR(EINVAL); + } + + /* reinit executed */ + return 1; +} + +/** + * complete a source change that waited for the frames decoded before it + * returns 1 if reinit was successful, negative if it failed + */ +static int v4l2_finish_format_change(V4L2Context *ctx) +{ + V4L2m2mContext *s = ctx_to_m2mctx(ctx); + struct v4l2_format cap_fmt = s->capture.format; + + if (ioctl(s->fd, VIDIOC_G_FMT, &cap_fmt)) { + av_log(logger(ctx), AV_LOG_ERROR, "%s VIDIOC_G_FMT\n", s->capture.name); + return AVERROR(errno); + } + + return v4l2_change_capture_format(s, &cap_fmt); +} + /** * handle resolution change event and end of stream event * returns 1 if reinit was successful, negative if it failed @@ -201,30 +248,20 @@ static int v4l2_handle_event(V4L2Context *ctx) return 0; } - if (v4l2_resolution_changed(&s->capture, &cap_fmt)) { - s->capture.height = v4l2_get_height(&cap_fmt); - s->capture.width = v4l2_get_width(&cap_fmt); - s->capture.sample_aspect_ratio = v4l2_get_sar(&s->capture); - } else { + if (!v4l2_resolution_changed(&s->capture, &cap_fmt)) { v4l2_start_decode(ctx); return 0; } - s->reinit = 1; - - if (s->avctx) - ret = ff_set_dimensions(s->avctx, s->capture.width, s->capture.height); - if (ret < 0) - av_log(logger(ctx), AV_LOG_WARNING, "update avcodec height and width\n"); - - ret = ff_v4l2_m2m_codec_reinit(s); - if (ret) { - av_log(logger(ctx), AV_LOG_ERROR, "v4l2_m2m_codec_reinit\n"); - return AVERROR(EINVAL); + /* The decoder still returns the frames it decoded before the change, and + * ends them with a buffer flagged LAST (dev-decoder, "Dynamic Resolution + * Change"): reinitializing capture now would discard them. */ + if (s->capture.streamon && s->capture.dequeued) { + s->capture.format_change = 1; + return 0; } - /* reinit executed */ - return 1; + return v4l2_change_capture_format(s, &cap_fmt); } static int v4l2_stop_decode(V4L2Context *ctx) @@ -290,6 +327,20 @@ static V4L2Buffer* v4l2_dequeue_v4l2buf(V4L2Context *ctx, int timeout) "packets/frames.\n"); } + /* a pending source change is completed once the frames before it have + * been returned, or when no capture buffer is left to return them in */ + if (!V4L2_TYPE_IS_OUTPUT(ctx->type) && ctx->format_change && ctx->buffers) { + for (i = 0; i < ctx->num_buffers; i++) { + if (ctx->buffers[i].status == V4L2BUF_IN_DRIVER) + break; + } + if (ctx->format_change == 2 || i == ctx->num_buffers) { + if (v4l2_finish_format_change(ctx) < 0) + ctx->done = 1; + return NULL; + } + } + /* if we are draining and there are no more capture buffers queued in the driver we are done */ if (!V4L2_TYPE_IS_OUTPUT(ctx->type) && ctx_to_m2mctx(ctx)->draining) { for (i = 0; i < ctx->num_buffers; i++) { @@ -387,6 +438,11 @@ dequeue: ret = ioctl(ctx_to_m2mctx(ctx)->fd, VIDIOC_DQBUF, &buf); if (ret) { + if (errno == EPIPE && ctx->format_change) { + if (v4l2_finish_format_change(ctx) < 0) + ctx->done = 1; + return NULL; + } if (errno != EAGAIN) { ctx->done = 1; if (errno != EPIPE) @@ -396,6 +452,23 @@ dequeue: return NULL; } + if (!V4L2_TYPE_IS_OUTPUT(ctx->type)) + ctx->dequeued = 1; + +#ifdef V4L2_BUF_FLAG_LAST + if (ctx->format_change && (buf.flags & V4L2_BUF_FLAG_LAST)) { + int bytesused = V4L2_TYPE_IS_MULTIPLANAR(buf.type) ? + buf.m.planes[0].bytesused : buf.bytesused; + if (!bytesused) { + ctx->buffers[buf.index].status = V4L2BUF_AVAILABLE; + if (v4l2_finish_format_change(ctx) < 0) + ctx->done = 1; + return NULL; + } + /* return its frame first */ + ctx->format_change = 2; + } else +#endif if (ctx_to_m2mctx(ctx)->draining && !V4L2_TYPE_IS_OUTPUT(ctx->type)) { int bytesused = V4L2_TYPE_IS_MULTIPLANAR(buf.type) ? buf.m.planes[0].bytesused : buf.bytesused; @@ -570,6 +643,7 @@ int ff_v4l2_context_set_status(V4L2Context* ctx, uint32_t cmd) return AVERROR(errno); ctx->streamon = (cmd == VIDIOC_STREAMON); + ctx->dequeued = 0; return 0; } diff --git a/libavcodec/v4l2_context.h b/libavcodec/v4l2_context.h index fdd5cf5284..83475833b2 100644 --- a/libavcodec/v4l2_context.h +++ b/libavcodec/v4l2_context.h @@ -93,6 +93,19 @@ typedef struct V4L2Context { */ int done; + /** + * Whether a buffer has been dequeued since the stream was last started + * or stopped. + */ + int dequeued; + + /** + * A source change to a new capture format, put off until the frames + * decoded before it have been dequeued: 1 until the buffer flagged LAST + * that ends them, 2 once that buffer has been returned with a frame. + */ + int format_change; + } V4L2Context; /** -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
