Hi guys, attached patches fix integer overflows in swscale as detected by IOC.
As for #3, the one that fixes overflows in RGB48 by changing initFilter: for certain types of filters, sums of coefficients at any intermediate stage can be >1 (as long as individual coefficients are <=1), because subsequent coefficients can be negative (bicubic, spline, lanczos). We support that and it works fine, basically up to a maximum of 2 in any intermediate (which is enough to cover anything). What initFilter did is to merge coefficients on edges, e.g. if there are 2 coefficients for position -1 and 0 (assuming edge extension), then -1 and 0 have the same pixel value, so we can also use coeffA+coeffB for pixel 0 and add a zero as last coefficient to the end. This prevents overreads at the line edges. The problem is that with spline/lanczos etc filters, the intermediate sum of coefficients can be >1 at any point, so the coeffA+coeffB can (and is) in some cases >1, which makes a single coefficient >1. We don't support that for the vertical filter (the horizontal one does it fine), and I don't think it's possible to support it unless I decrease the number of bits precision within the filters. Therefore, this patch basically removes the merging at edges for the vertical filter and instead duplicates lines in the 2D array by using the same pointer (line[0]) for "line[-1]" as well as line[0]. This fixes the integer overflows in the vertical filter. We don't copy data, just pointers, and only at a per-line basis, so it has very little effect on performance. The rest of the patches are straightforward. Please comment. Ronald
0001-Fix-add-overflow-in-swscale.patch
Description: Binary data
0001-Fix-overflow-in-swscale.patch
Description: Binary data
0001-swscale-fix-overflow-in-gray16-vertical-scaling.patch
Description: Binary data
0001-swscale-fix-overflows-in-RGB-rounding-constants.patch
Description: Binary data
_______________________________________________ libav-devel mailing list [email protected] https://lists.libav.org/mailman/listinfo/libav-devel
