On 06/23/2012 03:45 PM, John Stebbins wrote: > On 12/18/2011 05:28 PM, Ronald S. Bultje wrote: >> Module: libav >> Branch: master >> Commit: d49352c7cc22fd8928a761a373c3508be17c9f19 >> >> Author: Ronald S. Bultje<[email protected]> >> Committer: Ronald S. Bultje<[email protected]> >> Date: Sun Dec 18 08:27:43 2011 -0800 >> >> swscale: fix overflows in vertical scaling at top/bottom edges. >> >> This fixes integer multiplication overflows in RGB48 output >> (vertical) scaling as detected by IOC. What happens is that for >> certain types of filters (lanczos, spline, bicubic), the >> intermediate sum of coefficients in the middle of a filter can >> be larger than the fixed-point equivalent of 1.0, even if the >> final sum is 1.0. This is fine and we support that. >> >> However, at frame edges, initFilter() will merge the coefficients >> for the off-screen pixels into the top or bottom pixel, such as >> to emulate edge extension. This means that suddenly, a single >> coefficient can be larger than the fixed-point equivalent of >> 1.0, which the vertical scaling routines do not support. >> >> Therefore, remove the merging of coefficients for edges for >> the vertical scaling filter, and instead add edge detection >> to the scaler itself so that it copies the pointers (not data) >> for the edges (i.e. it uses line[0] for line[-1] as well), so >> that a single coefficient is never larger than the fixed-point >> equivalent of 1.0. >> > Hi, > > I've noticed that this commit is triggering the assert in swscale.c:632. > assert(chrUSrcPtr + vChrFilterSize - 1 < chrUPixBuf + vChrBufSize * 2); > > This happens when converting with lanczos from YUV420P to RGB32 and it > is upscaling the chroma. The comment indicates that there is edge > detection in the scaler. So I'm wondering if the edge detection is > failing for this case or is the assert just vestigial and in need of > removal?
Which is your testcase? Trying probably is fast. lu -- Luca Barbato Gentoo/linux http://dev.gentoo.org/~lu_zero _______________________________________________ libav-devel mailing list [email protected] https://lists.libav.org/mailman/listinfo/libav-devel
