1. i * i + (1 << (NMSEDC_FRACBITS - 1)) is always positive (and it doesn't overflow), so one can remove the FFMAX in the computation of lut_nmsedec_sig0. 2. The discriminant of the polynomial i * i - 2^(F + 1) * i + 2^(2 * F) + 2^(F - 1) is negative; hence this polynomial has no real solutions, i.e. its sign doesn't change and is always positive. This allows to remove the FFMAX in the computation of lut_nmsedec_ref0. 3. After that, one sees that the computation of lut_nmsedec_ref0 actually contains lut_nmsedec_sig0. This is obscured by masking the last NMSEDEC_FRACBITS away, but the other summands are multiples of 2^NMSEDEC_FRACBITS and so masking doesn't affect them.
Signed-off-by: Andreas Rheinhardt <andreas.rheinha...@gmail.com> --- Supersedes https://ffmpeg.org/pipermail/ffmpeg-devel/2019-September/250687.html libavcodec/j2kenc.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libavcodec/j2kenc.c b/libavcodec/j2kenc.c index 38643c9a28..c5c04bc4bf 100644 --- a/libavcodec/j2kenc.c +++ b/libavcodec/j2kenc.c @@ -522,13 +522,13 @@ static void init_luts(void) for (i = 0; i < (1 << NMSEDEC_BITS); i++){ lut_nmsedec_sig[i] = FFMAX((3 * i << (13 - NMSEDEC_FRACBITS)) - (9 << 11), 0); - lut_nmsedec_sig0[i] = FFMAX((i*i + (1<<NMSEDEC_FRACBITS-1) & mask) << 1, 0); + lut_nmsedec_sig0[i] = ((i * i + (1 << (NMSEDEC_FRACBITS - 1))) & mask) << 1; a = (i >> (NMSEDEC_BITS-2)&2) + 1; lut_nmsedec_ref[i] = FFMAX((a - 2) * (i << (13 - NMSEDEC_FRACBITS)) + (1 << 13) - (a * a << 11), 0); - lut_nmsedec_ref0[i] = FFMAX(((i * i - (i << NMSEDEC_BITS) + (1 << 2 * NMSEDEC_FRACBITS) + (1 << (NMSEDEC_FRACBITS - 1))) & mask) - << 1, 0); + lut_nmsedec_ref0[i] = lut_nmsedec_sig0[i] - (i << (NMSEDEC_BITS + 1)) + + (1 << (2 * NMSEDEC_FRACBITS + 1)); } } -- 2.20.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
