On Tue, 2011-10-18 at 16:35 +0100, Måns Rullgård wrote: > Uoti Urpala <[email protected]> writes: > > > On Tue, 2011-10-18 at 12:50 +0100, Måns Rullgård wrote: > >> "Ronald S. Bultje" <[email protected]> writes: > >> > >> > + int64_t dd = d > SQRT_INT64_MAX ? ((d >> 1) * d) > >> > >> 29 : (d * d) >> 30; > >> > + int64_t ddd = d > SQRT_INT64_MAX || dd > > >> > SQRT_INT64_MAX ? > >> > + ((dd >> 2) * d) > >> > >> 28 : (dd * d) >> 30; > >> > >> It is still overflowing. If d uses 32 bits, d*d >> 30 uses 34, and > >> (dd >> 2) * d will still overflow. It needs >> 3 or some other > >> equivalent change. > > > > Where did you get "32" from? I see no obvious reason why exactly that > > would be the upper limit - the types are signed so 32 bits is not a > > natural limit, and the values used look like it could be larger than 32 > > bits unless there's something special which prevents that. > > I saw it use exactly 32 bits in testing.
Well I just tested it and saw d = 134093824000 (37 bits). Sinc scaler 1280x720->104x104. _______________________________________________ libav-devel mailing list [email protected] https://lists.libav.org/mailman/listinfo/libav-devel
