PR #25892 opened by Ramiro Polla (ramiro)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25892
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25892.patch
The rounding constant for alpha was the one for 8-bit output regardless
of depth, which biased alpha by 2, 8 and 32 codes for 10, 12 and 14 bits.
Tested with:
./libswscale/tests/swscale -scaler none -src yuva444p -dst gbrap12le
```
Before: SSIM={Y=1.000000 U=0.999999 V=0.999999 A=0.999949} loss=5.149841e-05
After: SSIM={Y=1.000000 U=0.999999 V=0.999999 A=1.000000} loss=5.364418e-07
```
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Ramiro Polla <[email protected]>
>From 3b66e38039534bddb90417ecd0662b464f01967e Mon Sep 17 00:00:00 2001
From: Ramiro Polla <[email protected]>
Date: Mon, 9 Nov 2026 20:05:49 +0100
Subject: [PATCH] swscale/output: fix alpha rounding in yuv2gbrp_full_X_c()
above 8 bits
The rounding constant for alpha was the one for 8-bit output regardless
of depth, which biased alpha by 2, 8 and 32 codes for 10, 12 and 14 bits.
Tested with:
./libswscale/tests/swscale -scaler none -src yuva444p -dst gbrap12le
Before: SSIM={Y=1.000000 U=0.999999 V=0.999999 A=0.999949} loss=5.149841e-05
After: SSIM={Y=1.000000 U=0.999999 V=0.999999 A=1.000000} loss=5.364418e-07
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Ramiro Polla <[email protected]>
---
libswscale/output.c | 2 +-
libswscale/x86/output.asm | 6 ++++--
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/libswscale/output.c b/libswscale/output.c
index 55cba1ea5f1..bd4fe04b4ca 100644
--- a/libswscale/output.c
+++ b/libswscale/output.c
@@ -2434,7 +2434,7 @@ yuv2gbrp_full_X_c(SwsInternal *c, const int16_t
*lumFilter,
V >>= 10;
if (hasAlpha) {
- A = 1 << 18;
+ A = 1 << (SH - 4);
for (j = 0; j < lumFilterSize; j++)
A += alpSrc[j][i] * (unsigned)lumFilter[j];
diff --git a/libswscale/x86/output.asm b/libswscale/x86/output.asm
index e1b369c551d..8f749b713af 100644
--- a/libswscale/x86/output.asm
+++ b/libswscale/x86/output.asm
@@ -42,7 +42,9 @@ pd_65535_invf: times 8 dd 0x37800080 ;1.0/65535.0
pd_yuv2gbrp16_start: times 8 dd -0x40000000
pd_yuv2gbrp_y_start: times 8 dd (1 << 9)
pd_yuv2gbrp_uv_start: times 8 dd ((1 << 9) - (128 << 19))
-pd_yuv2gbrp_a_start: times 8 dd (1 << 18)
+pd_yuv2gbrp_a_start_8: times 8 dd (1 << 18)
+pd_yuv2gbrp_a_start_10: times 8 dd (1 << 16)
+pd_yuv2gbrp_a_start_12: times 8 dd (1 << 14)
pd_yuv2gbrp16_offset: times 8 dd 0x10000 ;(1 << 16)
pd_yuv2gbrp16_round13: times 8 dd 0xE0002000 ;(1 << 13) - (1 << 29)
pd_yuv2gbrp16_a_offset: times 8 dd 0x20002000
@@ -758,7 +760,7 @@ endstruc
%define Y_START [pd_yuv2gbrp_y_start]
%define Y_ROUND m9
%define UV_START [pd_yuv2gbrp_uv_start]
- %define A_START [pd_yuv2gbrp_a_start]
+ %define A_START [pd_yuv2gbrp_a_start_ %+ DEPTH]
%define A_CLIP2P 27
%endif
--
2.54.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]