On 06/24/2012 09:53 AM, John Stebbins wrote: > On 06/23/2012 10:07 PM, John Stebbins wrote: >> On 06/23/2012 07:41 PM, Luca Barbato wrote: >>> On 06/23/2012 03:45 PM, John Stebbins wrote: >>>> On 12/18/2011 05:28 PM, Ronald S. Bultje wrote: >>>>> Module: libav >>>>> Branch: master >>>>> Commit: d49352c7cc22fd8928a761a373c3508be17c9f19 >>>>> >>>>> Author: Ronald S. Bultje<[email protected]> >>>>> Committer: Ronald S. Bultje<[email protected]> >>>>> Date: Sun Dec 18 08:27:43 2011 -0800 >>>>> >>>>> swscale: fix overflows in vertical scaling at top/bottom edges. >>>>> >>>>> This fixes integer multiplication overflows in RGB48 output >>>>> (vertical) scaling as detected by IOC. What happens is that for >>>>> certain types of filters (lanczos, spline, bicubic), the >>>>> intermediate sum of coefficients in the middle of a filter can >>>>> be larger than the fixed-point equivalent of 1.0, even if the >>>>> final sum is 1.0. This is fine and we support that. >>>>> >>>>> However, at frame edges, initFilter() will merge the coefficients >>>>> for the off-screen pixels into the top or bottom pixel, such as >>>>> to emulate edge extension. This means that suddenly, a single >>>>> coefficient can be larger than the fixed-point equivalent of >>>>> 1.0, which the vertical scaling routines do not support. >>>>> >>>>> Therefore, remove the merging of coefficients for edges for >>>>> the vertical scaling filter, and instead add edge detection >>>>> to the scaler itself so that it copies the pointers (not data) >>>>> for the edges (i.e. it uses line[0] for line[-1] as well), so >>>>> that a single coefficient is never larger than the fixed-point >>>>> equivalent of 1.0. >>>>> >>>> Hi, >>>> >>>> I've noticed that this commit is triggering the assert in >>>> swscale.c:632. >>>> assert(chrUSrcPtr + vChrFilterSize - 1< chrUPixBuf + vChrBufSize * 2); >>>> >>>> This happens when converting with lanczos from YUV420P to RGB32 and it >>>> is upscaling the chroma. The comment indicates that there is edge >>>> detection in the scaler. So I'm wondering if the edge detection is >>>> failing for this case or is the assert just vestigial and in need of >>>> removal? >>> Which is your testcase? Trying probably is fast. >>> >>> lu >>> >> I'm afraid I don't understand your question. Are you asking me to try >> something? Or are you asking for more details about how you can >> reproduce this yourself? I haven't tried to reproduce this yet with >> avconv or such. I'm using libswscale directly. I've already >> described the circumstances. Of coarse, the assert is silent on a >> standard debug build of libav. You would need to add "-DDEBUG" to >> your CFLAGS to enable it. I only bumped into it because the assert >> doesn't seem to be disabled when building on mingw64 and I was testing >> all the various platforms HandBrake supports.
I was sure I answered you but looks like the email never reached the ml... >> > Here's a sample app and image that can be used to reproduce the assert. > Thank you! I'll have a look later. lu -- Luca Barbato Gentoo/linux http://dev.gentoo.org/~lu_zero _______________________________________________ libav-devel mailing list [email protected] https://lists.libav.org/mailman/listinfo/libav-devel
