PR #20387 opened by Niklas Haas (haasn) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20387 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20387.patch
This wrapping logic still considered any nonzero return from the ASM function to be the overall result, but this is not true since the addition of FF_ALPHA_TRANSPARENT. Fix it by only early returning if FF_ALPHA_STRAIGHT is detected. Fixes: 9b8b78a8153f9ca94b7eb9cba87442954787912a See-Also: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20301#issuecomment-4802 >From 9931f416a09941ca5f415f5a5626847760ca7850 Mon Sep 17 00:00:00 2001 From: Niklas Haas <g...@haasn.dev> Date: Mon, 1 Sep 2025 16:39:00 +0200 Subject: [PATCH] avfilter/x86/vf_colordetect: fix alpha detect tail handling This wrapping logic still considered any nonzero return from the ASM function to be the overall result, but this is not true since the addition of FF_ALPHA_TRANSPARENT. Fix it by only early returning if FF_ALPHA_STRAIGHT is detected. Fixes: 9b8b78a8153f9ca94b7eb9cba87442954787912a See-Also: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20301#issuecomment-4802 --- libavfilter/x86/vf_colordetect_init.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/libavfilter/x86/vf_colordetect_init.c b/libavfilter/x86/vf_colordetect_init.c index 828f01937d..72fa021bf2 100644 --- a/libavfilter/x86/vf_colordetect_init.c +++ b/libavfilter/x86/vf_colordetect_init.c @@ -31,11 +31,11 @@ static int FUNC_NAME(const uint8_t *src, ptrdiff_t stride, { \ ptrdiff_t bytes = (width << SHIFT) & ~(MMSIZE - 1); \ int ret = ASM_FUNC_NAME(src, stride, bytes, height, min, max); \ - if (ret) \ + if (ret == FF_ALPHA_STRAIGHT) \ return ret; \ \ - return C_FUNC_NAME(src + bytes, stride, width - (bytes >> SHIFT), \ - height, min, max); \ + return ret | C_FUNC_NAME(src + bytes, stride, width - (bytes >> SHIFT), \ + height, min, max); \ } #define DETECT_ALPHA_FUNC(FUNC_NAME, ASM_FUNC_NAME, C_FUNC_NAME, SHIFT, MMSIZE) \ @@ -50,11 +50,12 @@ static int FUNC_NAME(const uint8_t *color, ptrdiff_t color_stride, ptrdiff_t bytes = (width << SHIFT) & ~(MMSIZE - 1); \ int ret = ASM_FUNC_NAME(color, color_stride, alpha, alpha_stride, \ bytes, height, p, q, k); \ - if (ret) \ + if (ret == FF_ALPHA_STRAIGHT) \ return ret; \ \ - return C_FUNC_NAME(color + bytes, color_stride, alpha + bytes, alpha_stride,\ - width - (bytes >> SHIFT), height, p, q, k); \ + return ret | C_FUNC_NAME(color + bytes, color_stride, alpha + bytes, \ + alpha_stride, width - (bytes >> SHIFT), height, \ + p, q, k); \ } #if HAVE_X86ASM -- 2.49.1 _______________________________________________ ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org To unsubscribe send an email to ffmpeg-devel-le...@ffmpeg.org
