"Ronald S. Bultje" <[email protected]> writes: > From: "Ronald S. Bultje" <[email protected]> > > This fixes the same overflow as in the RGB48/16-bit YUV scaling; > some filters can overflow both negatively and positively (e.g. > spline/lanczos), so we bias a signed integer so it's "half signed" > and "half unsigned", and can cover overflows in both directions > while maintaining full 31-bit depth. > --- > libswscale/swscale.c | 12 ++++++------ > 1 files changed, 6 insertions(+), 6 deletions(-) > > diff --git a/libswscale/swscale.c b/libswscale/swscale.c > index 592c9d6..60652d9 100644 > --- a/libswscale/swscale.c > +++ b/libswscale/swscale.c > @@ -386,8 +386,8 @@ yuv2gray16_X_c_template(SwsContext *c, const int16_t > *lumFilter, > > for (i = 0; i < (dstW >> 1); i++) { > int j; > - int Y1 = 1 << 14; > - int Y2 = 1 << 14; > + int Y1 = (1 << 14) - 0x40000000; > + int Y2 = (1 << 14) - 0x40000000; > > for (j = 0; j < lumFilterSize; j++) { > Y1 += lumSrc[j][i * 2] * lumFilter[j]; > @@ -396,11 +396,11 @@ yuv2gray16_X_c_template(SwsContext *c, const int16_t > *lumFilter, > Y1 >>= 15; > Y2 >>= 15; > if ((Y1 | Y2) & 0x10000) {
This overflow check will now trigger for anything with a negative value, even if in range. The clipping is of course a no-op in those cases, so it does no damage as such, but it might not be what you intended. > - Y1 = av_clip_uint16(Y1); > - Y2 = av_clip_uint16(Y2); > + Y1 = av_clip_int16(Y1); > + Y2 = av_clip_int16(Y2); > } > - output_pixel(&dest[i * 2 + 0], Y1); > - output_pixel(&dest[i * 2 + 1], Y2); > + output_pixel(&dest[i * 2 + 0], 0x8000 + Y1); > + output_pixel(&dest[i * 2 + 1], 0x8000 + Y2); > } > } > > -- > 1.7.6 > -- Måns Rullgård [email protected] _______________________________________________ libav-devel mailing list [email protected] https://lists.libav.org/mailman/listinfo/libav-devel
