#10852: sws_scale overflows buffer for some resolutions using ssse3 instructions
------------------------------------+-----------------------------------
Reporter: Jerome | Owner: (none)
Type: defect | Status: new
Priority: important | Component: swscale
Version: git-master | Resolution:
Keywords: memory | Blocked By:
Blocking: | Reproduced by developer: 0
Analyzed by developer: 0 |
------------------------------------+-----------------------------------
Description changed by Jerome:
Old description:
> Summary of the bug:
> This bug occurs when using the sws_scale function to convert the color
> space of an image from yuv420 to bgr24 (it might occur for other color
> spaces, I haven't verified).
> It started happening with version 5.0 and is still happening with the
> latest master. It does not systematically happen, only for some image
> resolution. When writing the output image, sws_scale goes beyond the
> image buffer by 8 bytes. This memory violation often triggers a segfault.
>
> Valgrind output:
> ==1050090== Memcheck, a memory error detector
> ==1050090== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et
> al.
> ==1050090== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright
> info
> ==1050090== Command: ./test_memcheck
> ==1050090==
> [swscaler @ 0x62cb680] deprecated pixel format used, make sure you did
> set range correctly
> Convert unscaled 3240
> ==1050090== Invalid write of size 8
> ==1050090== at 0x48EEAF6: ??? (libavutil/x86/x86inc.asm:1274)
> ==1050090== by 0x48ED6F2: yuv420_bgr24_ssse3 (yuv2rgb_template.c:193)
> ==1050090== by 0x48B5B5F: scale_internal (swscale.c:1042)
> ==1050090== by 0x48B6668: sws_scale (swscale.c:1214)
> ==1050090== by 0x109393: main (main_memcheck.cpp:40)
> ==1050090== Address 0x69aac40 is 0 bytes after a block of size 6,220,800
> alloc'd
> ==1050090== at 0x4847203: operator new[](unsigned long) (in
> /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==1050090== by 0x10931A: main (main_memcheck.cpp:23)
> ==1050090==
> ==1050090== Invalid write of size 8
> ==1050090== at 0x48EEAFB: ??? (libavutil/x86/x86inc.asm:1274)
> ==1050090== by 0x48ED6F2: yuv420_bgr24_ssse3 (yuv2rgb_template.c:193)
> ==1050090== by 0x48B5B5F: scale_internal (swscale.c:1042)
> ==1050090== by 0x48B6668: sws_scale (swscale.c:1214)
> ==1050090== by 0x109393: main (main_memcheck.cpp:40)
> ==1050090== Address 0x69aac48 is 8 bytes after a block of size 6,220,800
> alloc'd
> ==1050090== at 0x4847203: operator new[](unsigned long) (in
> /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==1050090== by 0x10931A: main (main_memcheck.cpp:23)
>
> How to reproduce:
> I'm attaching a small C++ program that illustrates how to reproduce the
> bug.
New description:
**Summary of the bug:**
This bug occurs when using the sws_scale function to convert the color
space of an image from yuv420 to bgr24 (it might occur for other color
spaces, I haven't verified).
It started happening with version 5.0 and is still happening with the
latest master. It does not systematically happen, only for some image
resolution. When writing the output image, sws_scale goes beyond the image
buffer by 8 bytes. This memory violation often triggers a segfault.
**Valgrind output:**
{{{
==1050090== Memcheck, a memory error detector
==1050090== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et
al.
==1050090== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright
info
==1050090== Command: ./test_memcheck
==1050090==
[swscaler @ 0x62cb680] deprecated pixel format used, make sure you did set
range correctly
Convert unscaled 3240
==1050090== Invalid write of size 8
==1050090== at 0x48EEAF6: ??? (libavutil/x86/x86inc.asm:1274)
==1050090== by 0x48ED6F2: yuv420_bgr24_ssse3 (yuv2rgb_template.c:193)
==1050090== by 0x48B5B5F: scale_internal (swscale.c:1042)
==1050090== by 0x48B6668: sws_scale (swscale.c:1214)
==1050090== by 0x109393: main (main_memcheck.cpp:40)
==1050090== Address 0x69aac40 is 0 bytes after a block of size 6,220,800
alloc'd
==1050090== at 0x4847203: operator new[](unsigned long) (in
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1050090== by 0x10931A: main (main_memcheck.cpp:23)
==1050090==
==1050090== Invalid write of size 8
==1050090== at 0x48EEAFB: ??? (libavutil/x86/x86inc.asm:1274)
==1050090== by 0x48ED6F2: yuv420_bgr24_ssse3 (yuv2rgb_template.c:193)
==1050090== by 0x48B5B5F: scale_internal (swscale.c:1042)
==1050090== by 0x48B6668: sws_scale (swscale.c:1214)
==1050090== by 0x109393: main (main_memcheck.cpp:40)
==1050090== Address 0x69aac48 is 8 bytes after a block of size 6,220,800
alloc'd
==1050090== at 0x4847203: operator new[](unsigned long) (in
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1050090== by 0x10931A: main (main_memcheck.cpp:23)
}}}
**How to reproduce:**
I'm attaching a small C++ program that illustrates how to reproduce the
bug.
**Environment:**
Ubuntu 22.04, but was also reproduced on Rocky9.
**Build:**
The ffmpeg library was built with the following flags:
{{{
--enable-libx265 \
--enable-libx264 \
--enable-shared \
--enable-gnutls \
--enable-libfreetype \
--enable-libfontconfig \
--enable-gpl \
--enable-nonfree \
--enable-nvdec \
--disable-static \
--enable-debug=3 --disable-stripping \
--nvccflags="-gencode arch=compute_86,code=sm_86 -O2"
}}}
--
--
Ticket URL: <https://trac.ffmpeg.org/ticket/10852#comment:3>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker_______________________________________________
FFmpeg-trac mailing list
FFmpeg-trac@avcodec.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-trac
To unsubscribe, visit link above, or email
ffmpeg-trac-requ...@ffmpeg.org with subject "unsubscribe".
