Anton Khirnov <[email protected]> writes: >> >> - s->coefs_finished[c] |= (1LL << (se + 1)) - (1LL << ss); >> >> + s->coefs_finished[c] |= (2LL << se) - (1LL << ss); >> >> last_scan = !~s->coefs_finished[c]; >> >> } >> >> >> > >> > how big is se ? >> >> se can be up to 63, because all other values are caught by a if before >> (se < ss || se >= 64) >> however (se + 1) makes it up to 64 again. This is bigger than the size >> of the variable, leading to undefined behaviour. > > How does the patch fix anything? 2 << 63 is just as undefined as 1 << 64
On the other hand, 2ULL << 63 would be well defined (assuming unsigned long is 64 bits, and then the value is zero). But maybe there's a good reason for using signed types here? Regards, /Niels -- Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26. Internet email is subject to wholesale government surveillance. _______________________________________________ libav-devel mailing list [email protected] https://lists.libav.org/mailman/listinfo/libav-devel
