PR #25117 opened by romainhedouin URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25117 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25117.patch
# Summary of changes Three small bug fixes: - Heap overflow: the prescreener buffer was only as wide as the plane and process_new() writes flags 4 at a time, so widths that weren't multiples of 4 wrote between 1 and 3 bytes past the end - Out of bounds reads: read_bytes() and read_words() read outside the line for planes shorter than 32 samples - Stale flags: after switching pscrn to none at runtime the old prescreener flags were kept and the predictor skipped those pixels (so the output was wrong) I found these 3 bugs when sweeping the entire ffmpeg codebase using an LLM <!-- If this PR requires new FATE test samples, attach them to the PR and list their target paths below (relative to the fate-suite root). Attached filenames must match the sample's filename: ```fate-samples # e.g. vorbis/new-sample.ogg ``` --> >From 6da469b66c5396cb11284f1be722a884d6aa73aa Mon Sep 17 00:00:00 2001 From: romainhedouin <[email protected]> Date: Fri, 16 Oct 2026 16:35:47 +0200 Subject: [PATCH 1/3] avfilter/vf_nnedi: fix a heap overflow in the prescreener buffer process_new() writes the prescreener flags four at a time, but the buffer was only as wide as the plane. If the width is not a multiple of four, up to three flags were written past the end of it. Found with ASan using the default settings, for example on gray 321x120. Found during a code review sweep of ffmpeg using an LLM. --- libavfilter/vf_nnedi.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavfilter/vf_nnedi.c b/libavfilter/vf_nnedi.c index 148791f804..8deba18b0c 100644 --- a/libavfilter/vf_nnedi.c +++ b/libavfilter/vf_nnedi.c @@ -1096,7 +1096,7 @@ static int config_input(AVFilterLink *inlink) return AVERROR(ENOMEM); for (int i = 0; i < s->nb_threads; i++) { - s->prescreen_buf[i] = av_calloc(s->planewidth[0], sizeof(**s->prescreen_buf)); + s->prescreen_buf[i] = av_calloc(FFALIGN(s->planewidth[0], 4), sizeof(**s->prescreen_buf)); if (!s->prescreen_buf[i]) return AVERROR(ENOMEM); } -- 2.54.0 >From f6e9857617e5a22c9d613c4b2f2b571a8b47c75d Mon Sep 17 00:00:00 2001 From: romainhedouin <[email protected]> Date: Fri, 16 Oct 2026 16:35:47 +0200 Subject: [PATCH 2/3] avfilter/vf_nnedi: fix out of bounds reads on narrow planes read_bytes() and read_words() pad each line with 32 mirrored samples on both sides. For a plane narrower than 32 samples they read before the start and past the end of the line. Clamp the indices. The output for wider planes is unchanged. Found with ASan on gray 16x16 and on the chroma of yuv420p 48x32. Found during a code review sweep of ffmpeg using an LLM. --- libavfilter/vf_nnedi.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/libavfilter/vf_nnedi.c b/libavfilter/vf_nnedi.c index 8deba18b0c..5c4b47e0a2 100644 --- a/libavfilter/vf_nnedi.c +++ b/libavfilter/vf_nnedi.c @@ -445,13 +445,13 @@ static void read_bytes(const uint8_t *src, float *dst, { for (int y = 0; y < height; y++) { for (int x = 0; x < 32; x++) - dst[-x - 1] = src[x]; + dst[-x - 1] = src[FFMIN(x, width - 1)]; for (int x = 0; x < width; x++) dst[x] = src[x]; for (int x = 0; x < 32; x++) - dst[width + x] = src[width - x - 1]; + dst[width + x] = src[FFMAX(width - x - 1, 0)]; dst += dst_stride; src += src_stride; @@ -468,13 +468,13 @@ static void read_words(const uint8_t *srcp, float *dst, for (int y = 0; y < height; y++) { for (int x = 0; x < 32; x++) - dst[-x - 1] = src[x] * scale; + dst[-x - 1] = src[FFMIN(x, width - 1)] * scale; for (int x = 0; x < width; x++) dst[x] = src[x] * scale; for (int x = 0; x < 32; x++) - dst[width + x] = src[width - x - 1] * scale; + dst[width + x] = src[FFMAX(width - x - 1, 0)] * scale; dst += dst_stride; src += src_stride; -- 2.54.0 >From bde805d4d243b53886839f167700b06a4a5b1d3c Mon Sep 17 00:00:00 2001 From: romainhedouin <[email protected]> Date: Fri, 16 Oct 2026 16:35:47 +0200 Subject: [PATCH 3/3] avfilter/vf_nnedi: fix stale prescreener flags after pscrn=none The prescreener flags are only written when a prescreener is used. If pscrn is changed to none at runtime, the flags of the previous frames stayed in the buffer, so the predictor skipped those pixels and the output was wrong. Clear the flags when pscrn is none. Checked by switching pscrn with sendcmd: the frames after the switch now match a run that started with pscrn=none. Found during a code review sweep of ffmpeg using an LLM. --- libavfilter/vf_nnedi.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/libavfilter/vf_nnedi.c b/libavfilter/vf_nnedi.c index 5c4b47e0a2..dff5915e21 100644 --- a/libavfilter/vf_nnedi.c +++ b/libavfilter/vf_nnedi.c @@ -631,6 +631,9 @@ static int filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) src_linesize * 2, srcbuf_stride, width, 1, in_scale); + if (!s->pscrn) + memset(prescreen_buf, 0, width); + for (int y = 0; y < slice_end - slice_start; y += 2) { if (s->pscrn > 0) s->prescreen[s->pscrn > 1](ctx, srcbuf + (y / 2) * srcbuf_stride + 32, -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
