PR #24973 opened by Kilian (lusoris)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24973
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24973.patch
This series fixes four bugs in the libvmaf and libvmaf_cuda error paths. After
a mid-stream failure the filter leaked a frame, could not pool the frames it
had already scored, and printed an uninitialized `VMAF score:` value.
## Changes (one commit each)
1. **Free the distorted frame on error.** `ff_framesync_dualinput_get()` hands
the caller ownership of the main frame. `do_vmaf()` and `do_vmaf_cuda()`
returned without freeing it when a picture copy or `vmaf_read_pictures()`
failed.
2. **Count a frame only after libvmaf accepted it.** `frame_cnt++` sat inside
the `vmaf_read_pictures()` call. A rejected frame was still counted, so
`uninit()` pooled over an index with no features, and pooling failed for every
model.
3. **Print no score when pooling fails.** `uninit()` logged the pooling error
and then printed `vmaf_score`, which the failed call never wrote.
4. **Pop the CUDA context when `cuMemcpy2D()` fails.**
`copy_picture_data_cuda()` returned with the context still pushed.
## How it was tested
Built master `7bc3576910` with `--toolchain=gcc-asan` against the
distribution's libvmaf 3.2.1, before and after the series. Uninitialized stack
variables were filled with a pattern (`-ftrivial-auto-var-init=pattern`).
A natural trigger needs no fault injection: a resolution change in the middle
of the stream with filter reinit disabled. libvmaf then rejects the first
larger frame.
```sh
cat 320x240.m2v 352x288.m2v > sizechange.m2v # two mpeg2video ES, 10 frames
each
ffmpeg -reinit_filter 0 -i sizechange.m2v -reinit_filter 0 -i sizechange.m2v \
-lavfi '[0:v][1:v]libvmaf' -f null -
```
| | before | after |
| --- | --- | --- |
| score line | `problem getting pooled vmaf score.` then `VMAF score:
-5314010372517807647...` (pattern bytes) | `VMAF score: 99.714265` over the 9
frames read |
| leaked AVFrame (LSan, `av_frame_clone` from `ff_framesync_get_frame`) | 1
direct + its pool buffers, 814028 bytes total | none |
A release n9.0.2 build without the pattern flag prints `VMAF score: 0.000000`
in the same spot.
The flush and pooling failures are hard to provoke naturally, so a small
LD_PRELOAD shim forced them. It makes `vmaf_read_pictures(NULL, NULL)` or
`vmaf_score_pooled()` return `-EIO`. Before the series, both cases printed the
uninitialized score after the error. After it, only the error is printed. A
clean run gives the same score before and after (99.742838).
The remaining LSan reports after the series are the `VmafPicture` buffers
handed to `vmaf_read_pictures()`. libvmaf documents that it owns them after the
call, but it does not free them on its own early-error path. That is a libvmaf
issue, not this filter's.
`make fate-source` passes. The series also applies cleanly to release/9.0,
which has the same code.
## Not tested
The CUDA path (commits 1, 2 and 4) is only compile-checked: no libvmaf with
CUDA support was available. I forced `CONFIG_LIBVMAF_CUDA_FILTER` on and ran a
syntax check against the libvmaf v3.2.1 headers and ffnvcodec 13.1. A planted
typo in the new `cuCtxPopCurrent()` call failed that check, so the check covers
the CUDA code. The CUDA path does not unref `pic_ref` when the distorted copy
fails. I left that alone because the ring-buffer semantics are libvmaf
internals.
## Relation to #24958
#24958 (`vmaf_picture_wrap()`) replaces the two `copy_picture_data()` calls in
`do_vmaf()` and does not touch the error branches. The two branches merge
cleanly in either order (`git merge-tree`). In the merged result the
`av_frame_free(&dist)` calls remain correct, because the wrapped picture holds
its own frame reference.
>From 0ddbc70e403d9141f4d598c3ce1bf37a588ff395 Mon Sep 17 00:00:00 2001
From: lusoris <[email protected]>
Date: Fri, 9 Oct 2026 18:15:48 +0200
Subject: [PATCH 1/4] avfilter/vf_libvmaf: free the distorted frame on error
The distorted frame returned by ff_framesync_dualinput_get() is owned
by the caller. do_vmaf() and do_vmaf_cuda() returned without freeing it
when copying a picture or vmaf_read_pictures() failed, leaking the
frame and keeping its buffer pool alive.
Signed-off-by: lusoris <[email protected]>
---
libavfilter/vf_libvmaf.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/libavfilter/vf_libvmaf.c b/libavfilter/vf_libvmaf.c
index 3ab67d3dba..58284d7f3c 100644
--- a/libavfilter/vf_libvmaf.c
+++ b/libavfilter/vf_libvmaf.c
@@ -150,6 +150,7 @@ static int do_vmaf(FFFrameSync *fs)
err = copy_picture_data(ref, &pic_ref, s->bpc);
if (err) {
av_log(ctx, AV_LOG_ERROR, "problem during vmaf_picture_alloc.\n");
+ av_frame_free(&dist);
return AVERROR(ENOMEM);
}
@@ -157,12 +158,14 @@ static int do_vmaf(FFFrameSync *fs)
if (err) {
av_log(ctx, AV_LOG_ERROR, "problem during vmaf_picture_alloc.\n");
vmaf_picture_unref(&pic_ref);
+ av_frame_free(&dist);
return AVERROR(ENOMEM);
}
err = vmaf_read_pictures(s->vmaf, &pic_ref, &pic_dist, s->frame_cnt++);
if (err) {
av_log(ctx, AV_LOG_ERROR, "problem during vmaf_read_pictures.\n");
+ av_frame_free(&dist);
return AVERROR(EINVAL);
}
@@ -777,6 +780,7 @@ static int do_vmaf_cuda(FFFrameSync* fs)
frames_ctx->sw_format);
if (err) {
av_log(ctx, AV_LOG_ERROR, "problem during copy_picture_data_cuda.\n");
+ av_frame_free(&dist);
return AVERROR(ENOMEM);
}
@@ -784,12 +788,14 @@ static int do_vmaf_cuda(FFFrameSync* fs)
frames_ctx->sw_format);
if (err) {
av_log(ctx, AV_LOG_ERROR, "problem during copy_picture_data_cuda.\n");
+ av_frame_free(&dist);
return AVERROR(ENOMEM);
}
err = vmaf_read_pictures(s->vmaf, &pic_ref, &pic_dist, s->frame_cnt++);
if (err) {
av_log(ctx, AV_LOG_ERROR, "problem during vmaf_read_pictures.\n");
+ av_frame_free(&dist);
return AVERROR(EINVAL);
}
--
2.52.0
>From 4239f5fd0b794a4d70c670850a1fd1444b15ab4a Mon Sep 17 00:00:00 2001
From: lusoris <[email protected]>
Date: Fri, 9 Oct 2026 18:15:57 +0200
Subject: [PATCH 2/4] avfilter/vf_libvmaf: count a frame only after libvmaf
accepted it
frame_cnt was incremented inside the vmaf_read_pictures() call, so a
frame libvmaf rejected was still counted. uninit() then asked for the
pooled score over [0, frame_cnt - 1], which includes the index that has
no features, and pooling failed for every model even though all
earlier frames were scored.
Increment the counter only when vmaf_read_pictures() succeeds.
Signed-off-by: lusoris <[email protected]>
---
libavfilter/vf_libvmaf.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/libavfilter/vf_libvmaf.c b/libavfilter/vf_libvmaf.c
index 58284d7f3c..3d90365dab 100644
--- a/libavfilter/vf_libvmaf.c
+++ b/libavfilter/vf_libvmaf.c
@@ -162,12 +162,13 @@ static int do_vmaf(FFFrameSync *fs)
return AVERROR(ENOMEM);
}
- err = vmaf_read_pictures(s->vmaf, &pic_ref, &pic_dist, s->frame_cnt++);
+ err = vmaf_read_pictures(s->vmaf, &pic_ref, &pic_dist, s->frame_cnt);
if (err) {
av_log(ctx, AV_LOG_ERROR, "problem during vmaf_read_pictures.\n");
av_frame_free(&dist);
return AVERROR(EINVAL);
}
+ s->frame_cnt++;
return ff_filter_frame(ctx->outputs[0], dist);
}
@@ -792,12 +793,13 @@ static int do_vmaf_cuda(FFFrameSync* fs)
return AVERROR(ENOMEM);
}
- err = vmaf_read_pictures(s->vmaf, &pic_ref, &pic_dist, s->frame_cnt++);
+ err = vmaf_read_pictures(s->vmaf, &pic_ref, &pic_dist, s->frame_cnt);
if (err) {
av_log(ctx, AV_LOG_ERROR, "problem during vmaf_read_pictures.\n");
av_frame_free(&dist);
return AVERROR(EINVAL);
}
+ s->frame_cnt++;
return ff_filter_frame(ctx->outputs[0], dist);
}
--
2.52.0
>From 13691b9dd0f9bfbe5ce71113b5e04e93b04c75c4 Mon Sep 17 00:00:00 2001
From: lusoris <[email protected]>
Date: Fri, 9 Oct 2026 18:16:06 +0200
Subject: [PATCH 3/4] avfilter/vf_libvmaf: do not print a score when pooling
fails
When vmaf_score_pooled() failed, uninit() logged the error and then
printed "VMAF score:" with vmaf_score, which the failed call never
wrote. The line reports an uninitialized stack value as a result.
Skip the score line for a model whose pooled score could not be
computed.
Signed-off-by: lusoris <[email protected]>
---
libavfilter/vf_libvmaf.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/libavfilter/vf_libvmaf.c b/libavfilter/vf_libvmaf.c
index 3d90365dab..c20f39706d 100644
--- a/libavfilter/vf_libvmaf.c
+++ b/libavfilter/vf_libvmaf.c
@@ -578,6 +578,7 @@ static av_cold void uninit(AVFilterContext *ctx)
if (err) {
av_log(ctx, AV_LOG_ERROR,
"problem getting pooled vmaf score.\n");
+ continue;
}
av_log(ctx, AV_LOG_INFO, "VMAF score: %f\n", vmaf_score);
--
2.52.0
>From e8803ed08e6b70be4ab9d626c853051885d53593 Mon Sep 17 00:00:00 2001
From: lusoris <[email protected]>
Date: Fri, 9 Oct 2026 18:16:15 +0200
Subject: [PATCH 4/4] avfilter/vf_libvmaf: pop the CUDA context when cuMemcpy2D
fails
copy_picture_data_cuda() pushes the device context before the copy and
returned without popping it when cuMemcpy2D() failed, leaving the
context current on the filter thread.
Signed-off-by: lusoris <[email protected]>
---
libavfilter/vf_libvmaf.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/libavfilter/vf_libvmaf.c b/libavfilter/vf_libvmaf.c
index c20f39706d..63f0dca6f5 100644
--- a/libavfilter/vf_libvmaf.c
+++ b/libavfilter/vf_libvmaf.c
@@ -747,8 +747,10 @@ static int copy_picture_data_cuda(VmafContext* vmaf,
m.Height = dst->h[i];
err = cu->cuMemcpy2D(&m);
- if (err)
+ if (err) {
+ cu->cuCtxPopCurrent(NULL);
return AVERROR_EXTERNAL;
+ }
break;
}
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
