PR #25893 opened by Kilian (lusoris)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25893
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25893.patch

avfilter/vf_hqdn3d: clamp the lowpass result to 16 bits

`lowpass()` returns `cur + coef[d]`, where `coef[d]` is a signed value from the
denoising coefficient table. The table entries are built from the midpoint of
each bin, so a step can round the result slightly outside the
`[min(prev, cur), max(prev, cur)]` interval of its two inputs.

`denoise_spatial()` chains that result across a row through `pixel_ant`, a
`uint32_t` that is never clamped, and also stores it into the `frame_ant` /
`line_ant` accumulators. Once a value leaves the `[0, 65535]` range that
`LOAD()` guarantees, a later `d = (prev - cur) >> (8 - LUT_BITS)` indexes the
coefficient table outside its valid range, which is an out-of-bounds read, and
the `uint16_t` accumulator stores wrap.

In a normal x86 build that per-row C loop is reached only for 12-bit and 14-bit
input; 8/9/10/16-bit use the SIMD row functions
(`ff_hqdn3d_row_{8,9,10,16}_x86`) after the first line, so the overshoot does
not accumulate there.

Clamp the lowpass result to 16 bits. This keeps `pixel_ant` and the stored
accumulators within `[0, 65535]`, so `d` stays in range at every depth, and it
removes the wrap. It is a no-op for any value that was already in range, so the
output is unchanged where it was already correct.

hqdn3d is a GPL filter, so this needs `--enable-gpl`.

Reproduce on a GPL build:

    ffmpeg -f lavfi -i 
"nullsrc=s=32x32:r=5:d=1,format=yuv420p12le,geq=lum='4095*gt(mod(X*X*7+Y*13+N*29,17),8)':cb=2048:cr=2048"
 -vf hqdn3d=10:10:10:10 -f null -

Under AddressSanitizer this reports a heap-buffer-overflow read in `lowpass`;
without a sanitizer it reads adjacent heap and the denoised output depends on
whatever was there.

A regression test is added: `fate-filter-hqdn3d-highdepth` runs the filter on a
12-bit high-frequency source built from `geq`, so it exercises the overshooting
C path without an external sample.



>From f1c8ac8b2ac67e1ebab1b304b0f4995fde6b445c Mon Sep 17 00:00:00 2001
From: lusoris <[email protected]>
Date: Mon, 9 Nov 2026 20:38:20 +0100
Subject: [PATCH] avfilter/vf_hqdn3d: clamp the lowpass result to 16 bits

lowpass() returns cur + coef[d]. coef[d] comes from the denoising
coefficient table, whose entries are built from the midpoint of each
bin, so a step can round the result slightly outside the
[min(prev, cur), max(prev, cur)] interval of its two inputs.

denoise_spatial() feeds that result back as prev on the next sample
through pixel_ant (an unclamped uint32_t) and the frame_ant / line_ant
accumulators. Once a value leaves the [0, 65535] range that LOAD()
guarantees, the next

    d = (prev - cur) >> (8 - LUT_BITS)

indexes the coefficient table out of bounds (an out-of-bounds read), and
the uint16_t accumulator store wraps.

On x86 that per-row C loop runs for every row only at 12- and 14-bit,
which have no SIMD row function; 8/9/10/16-bit use ff_hqdn3d_row_*_x86
after the first line, so the overshoot does not accumulate there.

Clamp the result to 16 bits so pixel_ant and the stored accumulators
stay in [0, 65535], d is a valid table index at every depth, and the
wrap is removed. The clamp is a no-op for any value already in range, so
output is unchanged wherever it was already correct.

Add fate-filter-hqdn3d-highdepth, a self-contained 12-bit framecrc
regression test that exercises the overshooting C path without an
external sample.

Signed-off-by: lusoris <[email protected]>
---
 libavfilter/vf_hqdn3d.c                |  2 +-
 tests/fate/filter-video.mak            |  3 +++
 tests/ref/fate/filter-hqdn3d-highdepth | 10 ++++++++++
 3 files changed, 14 insertions(+), 1 deletion(-)
 create mode 100644 tests/ref/fate/filter-hqdn3d-highdepth

diff --git a/libavfilter/vf_hqdn3d.c b/libavfilter/vf_hqdn3d.c
index 3b46b6d4f80..30270b3f4aa 100644
--- a/libavfilter/vf_hqdn3d.c
+++ b/libavfilter/vf_hqdn3d.c
@@ -51,7 +51,7 @@ av_always_inline
 static uint32_t lowpass(int prev, int cur, int16_t *coef, int depth)
 {
     int d = (prev - cur) >> (8 - LUT_BITS);
-    return cur + coef[d];
+    return av_clip_uint16(cur + coef[d]);
 }
 
 av_always_inline
diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak
index 82ab48dc68d..8f7d5a0cb73 100644
--- a/tests/fate/filter-video.mak
+++ b/tests/fate/filter-video.mak
@@ -238,6 +238,9 @@ fate-filter-gradfun: CMD = framecrc -c:v pgmyuv -i $(SRC) 
-vf gradfun
 FATE_FILTER_VSYNTH_PGMYUV-$(CONFIG_HQDN3D_FILTER) += fate-filter-hqdn3d
 fate-filter-hqdn3d: CMD = framecrc -c:v pgmyuv -i $(SRC) -vf hqdn3d
 
+FATE_FILTER-$(call FILTERFRAMECRC, NULLSRC GEQ FORMAT HQDN3D) += 
fate-filter-hqdn3d-highdepth
+fate-filter-hqdn3d-highdepth: CMD = framecrc -flags bitexact -lavfi 
"nullsrc=s=32x32:r=5:d=1,format=yuv420p12le,geq=lum=4095*gt(mod(X*X*7+Y*13+N*29\,17)\,8):cb=2048:cr=2048,hqdn3d=10:10:10:10"
 -frames:v 5
+
 FATE_FILTER_VSYNTH_PGMYUV-$(CONFIG_INTERLACE_FILTER) += fate-filter-interlace
 fate-filter-interlace: CMD = framecrc -c:v pgmyuv -i $(SRC) -vf interlace
 
diff --git a/tests/ref/fate/filter-hqdn3d-highdepth 
b/tests/ref/fate/filter-hqdn3d-highdepth
new file mode 100644
index 00000000000..fb1c72444b7
--- /dev/null
+++ b/tests/ref/fate/filter-hqdn3d-highdepth
@@ -0,0 +1,10 @@
+#tb 0: 1/5
+#media_type 0: video
+#codec_id 0: rawvideo
+#dimensions 0: 32x32
+#sar 0: 1/1
+0,          0,          0,        1,     3072, 0x5ec70518
+0,          1,          1,        1,     3072, 0x8a970758
+0,          2,          2,        1,     3072, 0xe00f1439
+0,          3,          3,        1,     3072, 0x1d9a09ad
+0,          4,          4,        1,     3072, 0xb2a509b6
-- 
2.54.0

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]

Reply via email to