Re: [FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-14 Thread Michael Niedermayer
On Thu, Jan 14, 2016 at 11:42:47AM -0500, Ronald S. Bultje wrote: > Hi, > > On Thu, Jan 14, 2016 at 11:34 AM, Michael Niedermayer < > mich...@niedermayer.cc> wrote: > > > On Thu, Jan 14, 2016 at 10:09:13AM -0500, Ronald S. Bultje wrote: > > > Hi, > > > > > > On Wed, Jan 13, 2016 at 9:50 PM,

Re: [FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-14 Thread Ronald S. Bultje
Hi, On Thu, Jan 14, 2016 at 12:06 PM, Michael Niedermayer < mich...@niedermayer.cc> wrote: > On Thu, Jan 14, 2016 at 11:42:47AM -0500, Ronald S. Bultje wrote: > > Hi, > > > > On Thu, Jan 14, 2016 at 11:34 AM, Michael Niedermayer < > > mich...@niedermayer.cc> wrote: > > > > > On Thu, Jan 14, 2016

Re: [FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-14 Thread Michael Niedermayer
On Thu, Jan 14, 2016 at 08:58:20AM +0100, Christophe Gisquet wrote: > > -#define YUVRGB_TABLE_HEADROOM 256 > > +#define YUVRGB_TABLE_HEADROOM 512 > [...] > > -const int yoffs = fullRange ? 384 : 326; > > +const int yoffs = fullRange ? 896 : 838; > > I think it's time to use that macro

Re: [FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-14 Thread Christophe Gisquet
Hi, 2016-01-14 9:41 GMT+01:00 Michael Niedermayer : > there are 2 seperate tables, the headroom in them also differs > curently, also the exact relation between input values and headroom > needed in the 2nd table depends on the yuv-rgb coefficients > > i can replace the

Re: [FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-14 Thread Paul B Mahol
On 1/14/16, Christophe Gisquet wrote: > Hi, > > 2016-01-14 9:41 GMT+01:00 Michael Niedermayer : >> there are 2 seperate tables, the headroom in them also differs >> curently, also the exact relation between input values and headroom >> needed

Re: [FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-14 Thread Michael Niedermayer
On Thu, Jan 14, 2016 at 10:09:02AM +, Paul B Mahol wrote: > On 1/14/16, Christophe Gisquet wrote: > > Hi, > > > > 2016-01-14 9:41 GMT+01:00 Michael Niedermayer : > >> there are 2 seperate tables, the headroom in them also differs > >>

Re: [FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-14 Thread Michael Niedermayer
On Thu, Jan 14, 2016 at 05:34:25PM +0100, Michael Niedermayer wrote: > On Thu, Jan 14, 2016 at 10:09:13AM -0500, Ronald S. Bultje wrote: > > Hi, > > > > On Wed, Jan 13, 2016 at 9:50 PM, Michael Niedermayer > > wrote: > > > > > From: Michael Niedermayer

Re: [FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-14 Thread Michael Niedermayer
On Thu, Jan 14, 2016 at 10:09:13AM -0500, Ronald S. Bultje wrote: > Hi, > > On Wed, Jan 13, 2016 at 9:50 PM, Michael Niedermayer > wrote: > > > From: Michael Niedermayer > > > > This makes SWS more robust > > Fixes: > >

Re: [FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-14 Thread Ronald S. Bultje
Hi, On Thu, Jan 14, 2016 at 11:34 AM, Michael Niedermayer < mich...@niedermayer.cc> wrote: > On Thu, Jan 14, 2016 at 10:09:13AM -0500, Ronald S. Bultje wrote: > > Hi, > > > > On Wed, Jan 13, 2016 at 9:50 PM, Michael Niedermayer > > wrote: > > > > > From: Michael Niedermayer

Re: [FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-14 Thread Michael Niedermayer
On Thu, Jan 14, 2016 at 12:30:32PM -0500, Ronald S. Bultje wrote: > Hi, > > On Thu, Jan 14, 2016 at 12:06 PM, Michael Niedermayer < > mich...@niedermayer.cc> wrote: > > > On Thu, Jan 14, 2016 at 11:42:47AM -0500, Ronald S. Bultje wrote: > > > Hi, > > > > > > On Thu, Jan 14, 2016 at 11:34 AM,

Re: [FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-14 Thread Ronald S. Bultje
Hi, On Wed, Jan 13, 2016 at 9:50 PM, Michael Niedermayer wrote: > From: Michael Niedermayer > > This makes SWS more robust > Fixes: > 07650a772d98aa63b0fed6370dc89037/asan_heap-oob_27ddeaf_2657_2c81ff264dee5d9712cb3251fb9c3bbb.264 > Fixes: out of array

Re: [FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-13 Thread Christophe Gisquet
> -#define YUVRGB_TABLE_HEADROOM 256 > +#define YUVRGB_TABLE_HEADROOM 512 [...] > -const int yoffs = fullRange ? 384 : 326; > +const int yoffs = fullRange ? 896 : 838; I think it's time to use that macro everywhere it is actually used without showing up. Best regards, Christophe

[FFmpeg-devel] [PATCH] swscale/yuv2rgb: Increase YUV2RGB table headroom

2016-01-13 Thread Michael Niedermayer
From: Michael Niedermayer This makes SWS more robust Fixes: 07650a772d98aa63b0fed6370dc89037/asan_heap-oob_27ddeaf_2657_2c81ff264dee5d9712cb3251fb9c3bbb.264 Fixes: out of array read Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind Signed-off-by: Michael