> also, have you tried adding a small constant to tmp ? > i would expect that this or a similar operation would allow moving > away from all "unstable" points without really changing the output in a > relevant way. > > Can't find an op, in mult mode, in order to not raise the assert But if i use add calc instead of mult, I can have a bigger tolerance without raising the assert
In 8 bits mode Tolerance 0.0000000000001 : pass 0.000000000001 : fail for i = 1 0.00000000001 : fail for i in [1, 2, 4, 8] 0.0000000001 : fail for i in [1, 2, 4, 8, 16, 32, 64] In 16 bits mode 0.0000000000001 : pass 0.000000000001 : fail for i in [1, 257, 259, 261, 263, 265, 267, 269, 271, 273, 275, 277, 279, 281, 65407] using this code : #define assert_stable_float(x) av_assert0((float)(x+0.0000000000001) == (float)(x-0.0000000000001)) static void inline fill_uint_to_float_lut(SwsContext *c, int bitdepth) { static const double float_mult8 = 1.0 / 255.0; static const double float_mult16 = 1.0 / 65535.0; int i; double tmp = 0.; if (bitdepth == 8) { /*! fill uint8 to float lut */ for (i = 0; i < 256; ++i){ c->uint2float_lut[i] = (float)tmp; tmp += float_mult8; assert_stable_float(tmp); } } else if (bitdepth == 16) { /*! fill uint16 to float lut */ for (i = 0; i < 65536; ++i){ c->uint2float_lut[i] = (float)tmp; tmp += float_mult16; assert_stable_float(tmp); } } else { /*! unsupported bitdepth */ av_assert0(0); } } Do you think 0.0000000000001, is enough for stable result in each supported platform ? Martin _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel