PR #22457 opened by mkver URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/22457 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/22457.patch
ff_h[yc]scale_fast_mmxext() call other functions from inline assembly; these functions look like leaf functions to GCC, so it may use the red zone to avoid modifying the stack. But this makes the call instructions in the inline asm corrupt the stack. In order to fix this 424bcc46b5fb0d662e0fb9ad6319c5b9ef3d770f made libswscale/x86/swscale_mmx.o be compiled with -mno-red-zone. Later Libav fixed it in their version in commit b14fa5572c2a3bb1d8cd6327c4687a2eee363bbb by saving and restoring the memory clobbered by the call (as is still done now). This was merged into FFmpeg in 0e7fc3cafe838b32c3af73954acf9de2f9240ffd, without touching the -mno-red-zone hack. Libav later renamed swscale_mmx.c to just swscale.c in 16d2a1a51c1dbdd69ee47b19c8ab66b905b7c5ce which was merged into FFmpeg in commit 2cb4d516549526b5e17e941f6d2375a2c501ade6, without removing the -mno-red-zone hack, although the file it applies to no longer existed. This commit removes the special red-zone handling given that it is inactive anyway. Btw: IMO a better fix would be to just subtract 128 from the stack before the calls and add it back afterwards. >From 47c5290499b4e8792ca8a3a394f5d79c1fc71504 Mon Sep 17 00:00:00 2001 From: Andreas Rheinhardt <[email protected]> Date: Mon, 9 Mar 2026 15:35:07 +0100 Subject: [PATCH] configure,swscale/x86/Makefile: Remove special red-zone handling ff_h[yc]scale_fast_mmxext() call other functions from inline assembly; these functions look like leaf functions to GCC, so it may use the red zone to avoid modifying the stack. But this makes the call instructions in the inline asm corrupt the stack. In order to fix this 424bcc46b5fb0d662e0fb9ad6319c5b9ef3d770f made libswscale/x86/swscale_mmx.o be compiled with -mno-red-zone. Later Libav fixed it in their version in commit b14fa5572c2a3bb1d8cd6327c4687a2eee363bbb by saving and restoring the memory clobbered by the call (as is still done now). This was merged into FFmpeg in 0e7fc3cafe838b32c3af73954acf9de2f9240ffd, without touching the -mno-red-zone hack. Libav later renamed swscale_mmx.c to just swscale.c in 16d2a1a51c1dbdd69ee47b19c8ab66b905b7c5ce which was merged into FFmpeg in commit 2cb4d516549526b5e17e941f6d2375a2c501ade6, without removing the -mno-red-zone hack, although the file it applies to no longer existed. This commit removes the special red-zone handling given that it is inactive anyway. Signed-off-by: Andreas Rheinhardt <[email protected]> --- configure | 6 ------ libswscale/x86/Makefile | 2 -- 2 files changed, 8 deletions(-) diff --git a/configure b/configure index 1759694274..e246dea0fc 100755 --- a/configure +++ b/configure @@ -8004,11 +8004,6 @@ fi enabled ftrapv && check_cflags -ftrapv -test_cc -mno-red-zone <<EOF && noredzone_flags="-mno-red-zone" -int x; -EOF - - if enabled icc; then # Just warnings, no remarks check_allcflags -w1 @@ -8666,7 +8661,6 @@ SLIB_INSTALL_EXTRA_LIB=${SLIB_INSTALL_EXTRA_LIB} SLIB_INSTALL_EXTRA_SHLIB=${SLIB_INSTALL_EXTRA_SHLIB} VERSION_SCRIPT_POSTPROCESS_CMD=${VERSION_SCRIPT_POSTPROCESS_CMD} SAMPLES:=${samples:-\$(FATE_SAMPLES)} -NOREDZONE_FLAGS=$noredzone_flags LIBFUZZER_PATH=$libfuzzer_path IGNORE_TESTS=$ignore_tests VERSION_TRACKING=$version_tracking diff --git a/libswscale/x86/Makefile b/libswscale/x86/Makefile index 9c9d286600..a6aa89e21a 100644 --- a/libswscale/x86/Makefile +++ b/libswscale/x86/Makefile @@ -1,5 +1,3 @@ -$(SUBDIR)x86/swscale_mmx.o: CFLAGS += $(NOREDZONE_FLAGS) - OBJS += x86/rgb2rgb.o \ x86/swscale.o \ x86/yuv2rgb.o \ -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
