On 06/24/2012 11:52 PM, Luca Barbato wrote:
On 06/24/2012 09:53 AM, John Stebbins wrote:
On 06/23/2012 10:07 PM, John Stebbins wrote:
On 06/23/2012 07:41 PM, Luca Barbato wrote:
On 06/23/2012 03:45 PM, John Stebbins wrote:
On 12/18/2011 05:28 PM, Ronald S. Bultje wrote:
Module: libav
Branch: master
Commit: d49352c7cc22fd8928a761a373c3508be17c9f19
Author: Ronald S. Bultje<[email protected]>
Committer: Ronald S. Bultje<[email protected]>
Date: Sun Dec 18 08:27:43 2011 -0800
swscale: fix overflows in vertical scaling at top/bottom edges.
This fixes integer multiplication overflows in RGB48 output
(vertical) scaling as detected by IOC. What happens is that for
certain types of filters (lanczos, spline, bicubic), the
intermediate sum of coefficients in the middle of a filter can
be larger than the fixed-point equivalent of 1.0, even if the
final sum is 1.0. This is fine and we support that.
However, at frame edges, initFilter() will merge the coefficients
for the off-screen pixels into the top or bottom pixel, such as
to emulate edge extension. This means that suddenly, a single
coefficient can be larger than the fixed-point equivalent of
1.0, which the vertical scaling routines do not support.
Therefore, remove the merging of coefficients for edges for
the vertical scaling filter, and instead add edge detection
to the scaler itself so that it copies the pointers (not data)
for the edges (i.e. it uses line[0] for line[-1] as well), so
that a single coefficient is never larger than the fixed-point
equivalent of 1.0.
Hi,
I've noticed that this commit is triggering the assert in
swscale.c:632.
assert(chrUSrcPtr + vChrFilterSize - 1< chrUPixBuf + vChrBufSize * 2);
This happens when converting with lanczos from YUV420P to RGB32 and it
is upscaling the chroma. The comment indicates that there is edge
detection in the scaler. So I'm wondering if the edge detection is
failing for this case or is the assert just vestigial and in need of
removal?
Which is your testcase? Trying probably is fast.
lu
I'm afraid I don't understand your question. Are you asking me to try
something? Or are you asking for more details about how you can
reproduce this yourself? I haven't tried to reproduce this yet with
avconv or such. I'm using libswscale directly. I've already
described the circumstances. Of coarse, the assert is silent on a
standard debug build of libav. You would need to add "-DDEBUG" to
your CFLAGS to enable it. I only bumped into it because the assert
doesn't seem to be disabled when building on mingw64 and I was testing
all the various platforms HandBrake supports.
I was sure I answered you but looks like the email never reached the ml...
Here's a sample app and image that can be used to reproduce the assert.
Thank you! I'll have a look later.
You may already know this. But just FYI, ffmpeg merged this change then
immediately reverted the part that skipped zeroing vertical filterPos
values that are negative (along with some other changes). I don't yet
understand swscale workings well enough to know which project is headed
in the "right" direction, but ffmpeg does not trigger this assert.
John
_______________________________________________
libav-devel mailing list
[email protected]
https://lists.libav.org/mailman/listinfo/libav-devel
