PR #22305 opened by Alex Teaca (alex-teaca) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/22305 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/22305.patch
Large frame sizes produce scaling increments that overflow when stored in the c->lumXInc and c->lumYInc int variables. The calculation is performed in int64_t but truncated to int. Later, the incorrect values are passed to initFilter which applies arithmetic operations that produce undefined-behavior. This patch rejects frame sizes where the scaling increments don't fit into int (c->lumXInc and c->lumYInc). Fixes: https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/21588 Signed-off-by: Alex Teaca <[email protected]> >From 72ca1460b24eed0b043115d89a1ae24cda1fbcfa Mon Sep 17 00:00:00 2001 From: Alex Teaca <[email protected]> Date: Fri, 27 Feb 2026 11:01:33 +0200 Subject: [PATCH] swscale/utils: sanity check scaling increments Large frame sizes produce scaling increments that overflow when stored in the c->lumXInc and c->lumYInc int variables. The calculation is performed in int64_t but truncated to int. Later, the incorrect values are passed to initFilter which applies arithmetic operations that produce undefined-behavior. This patch rejects frame sizes where the scaling increments don't fit into int (c->lumXInc and c->lumYInc). Fixes: https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/21588 Signed-off-by: Alex Teaca <[email protected]> --- libswscale/utils.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/libswscale/utils.c b/libswscale/utils.c index 52095ab2c7..ced51c6147 100644 --- a/libswscale/utils.c +++ b/libswscale/utils.c @@ -1128,6 +1128,7 @@ av_cold int ff_sws_init_single_context(SwsContext *sws, SwsFilter *srcFilter, int dstH = sws->dst_h; int dst_stride = FFALIGN(dstW * sizeof(int16_t) + 66, 16); int flags, cpu_flags; + int64_t tmpLumXInc, tmpLumYInc; enum AVPixelFormat srcFormat, dstFormat; const AVPixFmtDescriptor *desc_src; const AVPixFmtDescriptor *desc_dst; @@ -1219,8 +1220,18 @@ av_cold int ff_sws_init_single_context(SwsContext *sws, SwsFilter *srcFilter, if (!srcFilter) srcFilter = &dummyFilter; - c->lumXInc = (((int64_t)srcW << 16) + (dstW >> 1)) / dstW; - c->lumYInc = (((int64_t)srcH << 16) + (dstH >> 1)) / dstH; + tmpLumXInc = (((int64_t)srcW << 16) + (dstW >> 1)) / dstW; + tmpLumYInc = (((int64_t)srcH << 16) + (dstH >> 1)) / dstH; + + if (tmpLumXInc > INT_MAX || tmpLumYInc > INT_MAX) { + av_log(c, AV_LOG_ERROR, "%dx%d -> %dx%d makes invalid scaling increments\n", + srcW, srcH, dstW, dstH); + return AVERROR(EINVAL); + } + + c->lumXInc = tmpLumXInc; + c->lumYInc = tmpLumYInc; + c->dstFormatBpp = av_get_bits_per_pixel(desc_dst); c->srcFormatBpp = av_get_bits_per_pixel(desc_src); c->vRounder = 4 * 0x0001000100010001ULL; -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
