PR #25179 opened by Marti2203 URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25179 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25179.patch
`interpolate_bilinear()` and `interpolate_biquadratic()` in `libavfilter/transform.c` accept sample positions down to -1 (`if (x < -1 || ...) return def;`), so that a position between -1 and 0 blends the edge pixel with the fill value that `PIXEL()` returns for index -1. But `x_f = (int)x` truncates towards zero, so for -1 < x < 0 the neighbours are 0 and 1 instead of -1 and 0: - bilinear extrapolates past the edge, and the result can fall outside 0..255 before the conversion to `uint8_t`, which is undefined. With an edge pixel of 200 next to 50, sampling at x = -0.5 computes 275, written as 19. The expected value is 100 with `edge=blank` (half the zero fill) and 200 with `edge=original`. - biquadratic takes `sqrt()` of a negative number and returns NaN. Nothing in the tree uses it at the moment. This was checked by calling `ff_affine_transform()` from a small harness built with `-fsanitize=float-cast-overflow`: master reports "275 is outside the range of representable values of type 'unsigned char'" for bilinear and "-nan is outside the range ..." for biquadratic; with the patch there are no reports, and the results are 100 / 200 as above. Only deshake uses this code, and only with bilinear. It is affected with `edge=blank` and `edge=original`; the default `edge=mirror` and `edge=clamp` remap the position first. On jittery input (`testsrc2` cropped with a moving offset), `deshake=edge=blank` output changes in 5111 luma samples with the patch, 1385 of them by more than 64 levels; `edge=original` changes in 1624. `mirror` and `clamp` output is byte-identical. There is no FATE coverage for deshake. This issue was found by an LLM-Enhanced Static Analysis Engine and verified manually. 🤖 Generated with [Claude Code](https://claude.com/claude-code) From ffa25238ca95baef460250fc47144954164123d7 Mon Sep 17 00:00:00 2001 From: Martin Mirchev <[email protected]> Date: Sun, 18 Oct 2026 17:38:33 +0800 Subject: [PATCH] avfilter/transform: round sample positions down for interpolation The bilinear and biquadratic interpolators accept positions down to -1 and rely on PIXEL() returning the fill value for the neighbour at -1, but (int) truncates towards zero, so for -1 < x < 0 both neighbours were 0 and 1. Bilinear then extrapolated past the edge and converted a value outside 0..255 to uint8_t (e.g. 275 next to a 200 edge pixel, written as 19); biquadratic took the square root of a negative number. This affects deshake with edge=blank and edge=original; mirror and clamp remap the position first. Use floorf() so that the lower neighbour is -1. Signed-off-by: Martin Mirchev <[email protected]> --- libavfilter/transform.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/libavfilter/transform.c b/libavfilter/transform.c index 1f91436f73..c7da565bbc 100644 --- a/libavfilter/transform.c +++ b/libavfilter/transform.c @@ -57,10 +57,10 @@ INTERPOLATE_METHOD(interpolate_bilinear) if (x < -1 || x > width || y < -1 || y > height) { return def; } else { - x_f = (int)x; + x_f = floorf(x); x_c = x_f + 1; - y_f = (int)y; + y_f = floorf(y); y_c = y_f + 1; v1 = PIXEL(src, x_c, y_c, width, height, stride, def); @@ -85,9 +85,9 @@ INTERPOLATE_METHOD(interpolate_biquadratic) if (x < - 1 || x > width || y < -1 || y > height) return def; else { - x_f = (int)x; + x_f = floorf(x); x_c = x_f + 1; - y_f = (int)y; + y_f = floorf(y); y_c = y_f + 1; v1 = PIXEL(src, x_c, y_c, width, height, stride, def); -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
