PR #25893 opened by Kilian (lusoris)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25893
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25893.patch
avfilter/vf_hqdn3d: clamp the lowpass result to 16 bits
`lowpass()` returns `cur + coef[d]`, where `coef[d]` is a signed value from the
denoising coefficient table. The table entries are built from the midpoint of
each bin, so a step can round the result slightly outside the
`[min(prev, cur), max(prev, cur)]` interval of its two inputs.
`denoise_spatial()` chains that result across a row through `pixel_ant`, a
`uint32_t` that is never clamped, and also stores it into the `frame_ant` /
`line_ant` accumulators. Once a value leaves the `[0, 65535]` range that
`LOAD()` guarantees, a later `d = (prev - cur) >> (8 - LUT_BITS)` indexes the
coefficient table outside its valid range, which is an out-of-bounds read, and
the `uint16_t` accumulator stores wrap.
In a normal x86 build that per-row C loop is reached only for 12-bit and 14-bit
input; 8/9/10/16-bit use the SIMD row functions
(`ff_hqdn3d_row_{8,9,10,16}_x86`) after the first line, so the overshoot does
not accumulate there.
Clamp the lowpass result to 16 bits. This keeps `pixel_ant` and the stored
accumulators within `[0, 65535]`, so `d` stays in range at every depth, and it
removes the wrap. It is a no-op for any value that was already in range, so the
output is unchanged where it was already correct.
hqdn3d is a GPL filter, so this needs `--enable-gpl`.
Reproduce on a GPL build:
ffmpeg -f lavfi -i
"nullsrc=s=32x32:r=5:d=1,format=yuv420p12le,geq=lum='4095*gt(mod(X*X*7+Y*13+N*29,17),8)':cb=2048:cr=2048"
-vf hqdn3d=10:10:10:10 -f null -
Under AddressSanitizer this reports a heap-buffer-overflow read in `lowpass`;
without a sanitizer it reads adjacent heap and the denoised output depends on
whatever was there.
A regression test is added: `fate-filter-hqdn3d-highdepth` runs the filter on a
12-bit high-frequency source built from `geq`, so it exercises the overshooting
C path without an external sample.
>From f1c8ac8b2ac67e1ebab1b304b0f4995fde6b445c Mon Sep 17 00:00:00 2001
From: lusoris <[email protected]>
Date: Mon, 9 Nov 2026 20:38:20 +0100
Subject: [PATCH] avfilter/vf_hqdn3d: clamp the lowpass result to 16 bits
lowpass() returns cur + coef[d]. coef[d] comes from the denoising
coefficient table, whose entries are built from the midpoint of each
bin, so a step can round the result slightly outside the
[min(prev, cur), max(prev, cur)] interval of its two inputs.
denoise_spatial() feeds that result back as prev on the next sample
through pixel_ant (an unclamped uint32_t) and the frame_ant / line_ant
accumulators. Once a value leaves the [0, 65535] range that LOAD()
guarantees, the next
d = (prev - cur) >> (8 - LUT_BITS)
indexes the coefficient table out of bounds (an out-of-bounds read), and
the uint16_t accumulator store wraps.
On x86 that per-row C loop runs for every row only at 12- and 14-bit,
which have no SIMD row function; 8/9/10/16-bit use ff_hqdn3d_row_*_x86
after the first line, so the overshoot does not accumulate there.
Clamp the result to 16 bits so pixel_ant and the stored accumulators
stay in [0, 65535], d is a valid table index at every depth, and the
wrap is removed. The clamp is a no-op for any value already in range, so
output is unchanged wherever it was already correct.
Add fate-filter-hqdn3d-highdepth, a self-contained 12-bit framecrc
regression test that exercises the overshooting C path without an
external sample.
Signed-off-by: lusoris <[email protected]>
---
libavfilter/vf_hqdn3d.c | 2 +-
tests/fate/filter-video.mak | 3 +++
tests/ref/fate/filter-hqdn3d-highdepth | 10 ++++++++++
3 files changed, 14 insertions(+), 1 deletion(-)
create mode 100644 tests/ref/fate/filter-hqdn3d-highdepth
diff --git a/libavfilter/vf_hqdn3d.c b/libavfilter/vf_hqdn3d.c
index 3b46b6d4f80..30270b3f4aa 100644
--- a/libavfilter/vf_hqdn3d.c
+++ b/libavfilter/vf_hqdn3d.c
@@ -51,7 +51,7 @@ av_always_inline
static uint32_t lowpass(int prev, int cur, int16_t *coef, int depth)
{
int d = (prev - cur) >> (8 - LUT_BITS);
- return cur + coef[d];
+ return av_clip_uint16(cur + coef[d]);
}
av_always_inline
diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak
index 82ab48dc68d..8f7d5a0cb73 100644
--- a/tests/fate/filter-video.mak
+++ b/tests/fate/filter-video.mak
@@ -238,6 +238,9 @@ fate-filter-gradfun: CMD = framecrc -c:v pgmyuv -i $(SRC)
-vf gradfun
FATE_FILTER_VSYNTH_PGMYUV-$(CONFIG_HQDN3D_FILTER) += fate-filter-hqdn3d
fate-filter-hqdn3d: CMD = framecrc -c:v pgmyuv -i $(SRC) -vf hqdn3d
+FATE_FILTER-$(call FILTERFRAMECRC, NULLSRC GEQ FORMAT HQDN3D) +=
fate-filter-hqdn3d-highdepth
+fate-filter-hqdn3d-highdepth: CMD = framecrc -flags bitexact -lavfi
"nullsrc=s=32x32:r=5:d=1,format=yuv420p12le,geq=lum=4095*gt(mod(X*X*7+Y*13+N*29\,17)\,8):cb=2048:cr=2048,hqdn3d=10:10:10:10"
-frames:v 5
+
FATE_FILTER_VSYNTH_PGMYUV-$(CONFIG_INTERLACE_FILTER) += fate-filter-interlace
fate-filter-interlace: CMD = framecrc -c:v pgmyuv -i $(SRC) -vf interlace
diff --git a/tests/ref/fate/filter-hqdn3d-highdepth
b/tests/ref/fate/filter-hqdn3d-highdepth
new file mode 100644
index 00000000000..fb1c72444b7
--- /dev/null
+++ b/tests/ref/fate/filter-hqdn3d-highdepth
@@ -0,0 +1,10 @@
+#tb 0: 1/5
+#media_type 0: video
+#codec_id 0: rawvideo
+#dimensions 0: 32x32
+#sar 0: 1/1
+0, 0, 0, 1, 3072, 0x5ec70518
+0, 1, 1, 1, 3072, 0x8a970758
+0, 2, 2, 1, 3072, 0xe00f1439
+0, 3, 3, 1, 3072, 0x1d9a09ad
+0, 4, 4, 1, 3072, 0xb2a509b6
--
2.54.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]