Re: [FFmpeg-devel] [PATCH] avcodec/vp8dsp: vp7_luma_dc_wht_c: Fix multiple runtime error: signed integer overflow: -1366381240 + -1262413604 cannot be represented in type 'int'

[Michael Niedermayer]

On Wed, May 10, 2017 at 02:50:40PM +0200, Michael Niedermayer wrote:
> Fixes: 1440/clusterfuzz-testcase-minimized-5785716111966208
> 
> Found-by: continuous fuzzing process 
> https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
> Signed-off-by: Michael Niedermayer 
> ---
>  libavcodec/vp8dsp.c | 19 ++-
>  1 file changed, 10 insertions(+), 9 deletions(-)

applied

[...]
-- 
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

it is not once nor twice but times without number that the same ideas make
their appearance in the world. -- Aristotle


signature.asc
Description: Digital signature
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

[FFmpeg-devel] [PATCH] avcodec/vp8dsp: vp7_luma_dc_wht_c: Fix multiple runtime error: signed integer overflow: -1366381240 + -1262413604 cannot be represented in type 'int'

[Michael Niedermayer]

Fixes: 1440/clusterfuzz-testcase-minimized-5785716111966208

Found-by: continuous fuzzing process 
https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Signed-off-by: Michael Niedermayer 
---
 libavcodec/vp8dsp.c | 19 ++-
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/libavcodec/vp8dsp.c b/libavcodec/vp8dsp.c
index 7d9cfa8278..fed5c67a90 100644
--- a/libavcodec/vp8dsp.c
+++ b/libavcodec/vp8dsp.c
@@ -53,7 +53,8 @@ static void name ## _idct_dc_add4y_c(uint8_t *dst, int16_t 
block[4][16],  \
 #if CONFIG_VP7_DECODER
 static void vp7_luma_dc_wht_c(int16_t block[4][4][16], int16_t dc[16])
 {
-int i, a1, b1, c1, d1;
+int i;
+unsigned a1, b1, c1, d1;
 int16_t tmp[16];
 
 for (i = 0; i < 4; i++) {
@@ -61,10 +62,10 @@ static void vp7_luma_dc_wht_c(int16_t block[4][4][16], 
int16_t dc[16])
 b1 = (dc[i * 4 + 0] - dc[i * 4 + 2]) * 23170;
 c1 = dc[i * 4 + 1] * 12540 - dc[i * 4 + 3] * 30274;
 d1 = dc[i * 4 + 1] * 30274 + dc[i * 4 + 3] * 12540;
-tmp[i * 4 + 0] = (a1 + d1) >> 14;
-tmp[i * 4 + 3] = (a1 - d1) >> 14;
-tmp[i * 4 + 1] = (b1 + c1) >> 14;
-tmp[i * 4 + 2] = (b1 - c1) >> 14;
+tmp[i * 4 + 0] = (int)(a1 + d1) >> 14;
+tmp[i * 4 + 3] = (int)(a1 - d1) >> 14;
+tmp[i * 4 + 1] = (int)(b1 + c1) >> 14;
+tmp[i * 4 + 2] = (int)(b1 - c1) >> 14;
 }
 
 for (i = 0; i < 4; i++) {
@@ -73,10 +74,10 @@ static void vp7_luma_dc_wht_c(int16_t block[4][4][16], 
int16_t dc[16])
 c1 = tmp[i + 4] * 12540 - tmp[i + 12] * 30274;
 d1 = tmp[i + 4] * 30274 + tmp[i + 12] * 12540;
 AV_ZERO64(dc + i * 4);
-block[0][i][0] = (a1 + d1 + 0x2) >> 18;
-block[3][i][0] = (a1 - d1 + 0x2) >> 18;
-block[1][i][0] = (b1 + c1 + 0x2) >> 18;
-block[2][i][0] = (b1 - c1 + 0x2) >> 18;
+block[0][i][0] = (int)(a1 + d1 + 0x2) >> 18;
+block[3][i][0] = (int)(a1 - d1 + 0x2) >> 18;
+block[1][i][0] = (int)(b1 + c1 + 0x2) >> 18;
+block[2][i][0] = (int)(b1 - c1 + 0x2) >> 18;
 }
 }
 
-- 
2.11.0

___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel