How useful are very large values of "a"?
With a range of +/- 2**23 in Q16, it seems you can scale luma values by
as much as 128. As a result, when subsampling the prediction for 4:2:0,
you require independent quantization and clipping steps for all four
input pixels, and then a second quantization and rounding step for the
average. If you restricted the range of "a" to +/- 2**21, you could
still expand luma by a factor of 32, but introduce less rounding noise
into the averaging process in most cases, and clip less often. Since
that also removes the non-linear operations applied to each individual
value of yr(i, j), it lets you save 3 of the 4 multiplies, too, i.e.,
c(i, j) = clip(
a*(yr(2*i, 2*j) + yr(2*i+1, 2*j) + yr(2*i, 2*j+1) + yr(2*i+1, 2*j+1)
+ b >> (16 + 2)).
In a similar vein, it seems possible to use a scaled y'(i, j) (i.e.,
without the final right shift by two) without overflowing an unsigned
32-bit integer for any of the sums in Figure 3, assuming 8-bit input and
at most 64x64 blocks. All you'd have to do is remove the "+ 2" in the
shift at the end of the above equation.
For 10-bit input you would still have to downscale in your average to
keep these intermediate calculations in 32-bit, and 12-bit input already
would not fit.
It also seems like the prediction is biased, since the calculation of
c(i, j) from yr(i, j) does not include a rounding offset. Simply adding
(1 << 15) to "b" before clipping should remove this bias. Correctly
rounding the division used to compute "a" would also slightly reduce the
error, but that effect should be much smaller (and rounding in the
calculation of SSyy, SScc, and SSyc is not helpful).
Just as an editorial note, I think there are a couple of mistakes in the
formulas in Section 4.
I think you meant SSyc = YCsum - ((Ysum * Csum) >> 2*log2(N))
(instead of SSyc = YCsum - ((YCsum * YCsum) >> 2*log2(N))).
And then also b = ((Csum << 16) - a*Ysum) >> 2*log2(N)
(instead of b = ((Csum << 16) - a*YCsum) >> 2*log2(N)).
_______________________________________________
video-codec mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/video-codec
