Hi,
On Tue, Oct 18, 2011 at 9:05 AM, Ronald S. Bultje <[email protected]> wrote:
> Hi,
>
> On Tue, Oct 18, 2011 at 8:52 AM, Uoti Urpala <[email protected]> wrote:
>> On Tue, 2011-10-18 at 16:35 +0100, Måns Rullgård wrote:
>>> Uoti Urpala <[email protected]> writes:
>>>
>>> > On Tue, 2011-10-18 at 12:50 +0100, Måns Rullgård wrote:
>>> >> "Ronald S. Bultje" <[email protected]> writes:
>>> >>
>>> >> > + int64_t dd = d > SQRT_INT64_MAX ? ((d >> 1) *
>>> >> > d) >> 29 : (d * d) >> 30;
>>> >> > + int64_t ddd = d > SQRT_INT64_MAX || dd >
>>> >> > SQRT_INT64_MAX ?
>>> >> > + ((dd >> 2) *
>>> >> > d) >> 28 : (dd * d) >> 30;
>>> >>
>>> >> It is still overflowing. If d uses 32 bits, d*d >> 30 uses 34, and
>>> >> (dd >> 2) * d will still overflow. It needs >> 3 or some other
>>> >> equivalent change.
>>> >
>>> > Where did you get "32" from? I see no obvious reason why exactly that
>>> > would be the upper limit - the types are signed so 32 bits is not a
>>> > natural limit, and the values used look like it could be larger than 32
>>> > bits unless there's something special which prevents that.
>>>
>>> I saw it use exactly 32 bits in testing.
>>
>> Well I just tested it and saw d = 134093824000 (37 bits). Sinc scaler
>> 1280x720->104x104.
>
> That code only executes for the bicubic case, but you're right that
> there's no strict max, it can go all the way to 48bits for 192000 to
> 16 width scaling (idiotic example, I know).
Duh. Looking further down (4 lines):
if (d < 1LL<<30)
coeff = (12*(1<<24)-9*B-6*C)*ddd +
(-18*(1<<24)+12*B+6*C)*dd + (6*(1<<24)-2*B)*(1<<30);
else if (d < 1LL<<31)
coeff = (-B-6*C)*ddd + (6*B+30*C)*dd +
(-12*B-48*C)*d + (8*B+24*C)*(1<<30);
else
coeff=0.0;
in other words we never use dd/ddd unless they fit in 31 bits / the
overflow result is never used. I just royally wasted my time. :-).
(I did manage to fix the overflow, but that's kind of pointless now.)
Ronald
_______________________________________________
libav-devel mailing list
[email protected]
https://lists.libav.org/mailman/listinfo/libav-devel
