On Mon, Oct 08, 2018 at 05:04:43PM +0200, Martin Vignali wrote:
> >
> > also there are 2 divisions in this that you can trivially eliminate
> > /255 and /65535 (extra precission beyond IEEE float/double could change
> > these)
> >
> > also the whole could be done with fewer floats and no extra
>
> also there are 2 divisions in this that you can trivially eliminate
> /255 and /65535 (extra precission beyond IEEE float/double could change
> these)
>
> also the whole could be done with fewer floats and no extra complexity
> for example:
> int64_t tmp2 = 16843009LL * i;
>
On Mon, Sep 17, 2018 at 09:04:06AM +0200, Martin Vignali wrote:
> > also, have you tried adding a small constant to tmp ?
> > i would expect that this or a similar operation would allow moving
> > away from all "unstable" points without really changing the output in a
> > relevant way.
> >
> >
>
> also, have you tried adding a small constant to tmp ?
> i would expect that this or a similar operation would allow moving
> away from all "unstable" points without really changing the output in a
> relevant way.
>
>
Can't find an op, in mult mode, in order to not raise the assert
But if i use
On Sun, Sep 16, 2018 at 09:25:14PM +0200, Martin Vignali wrote:
> > you can just use something like (possibly with more or less 0 or a
> > magnitude
> > related value)
> > #define assert_stable_int(x) av_assert(llrintf(x+0.0001) ==
> > llrintf(x-0.0001))
> > #define assert_stable_float(x)
> you can just use something like (possibly with more or less 0 or a
> magnitude
> related value)
> #define assert_stable_int(x) av_assert(llrintf(x+0.0001) ==
> llrintf(x-0.0001))
> #define assert_stable_float(x) av_assert((float)(x+0.0001) ==
> (float)(x-0.0001))
>
> and then
On Mon, Sep 10, 2018 at 07:57:42PM +0200, Martin Vignali wrote:
> >
> > does the LUT generation code produce different results on platforms ?
> > if so i would suggest to try to use double and to add a small offset if
> > needed
> >
> > a 8bit table has 256 entries, a 16bit table 65536
> > a
>
> does the LUT generation code produce different results on platforms ?
> if so i would suggest to try to use double and to add a small offset if
> needed
>
> a 8bit table has 256 entries, a 16bit table 65536
> a difference would occur if a source value from 64bit floats gets rounded
>
On Thu, Aug 23, 2018 at 08:13:13PM +0200, Martin Vignali wrote:
[...]
> swscale_internal.h |2 -
> utils.c| 70
> ++---
> 2 files changed, 68 insertions(+), 4 deletions(-)
> 911c6d681b09ab719e2c8abe337887732c28b65e
>
Updated patch in attach
001 and 002 : unchanged
003 : update from prev 003 : change in lut generation : move coeff calc
outside the loop, and avoid "i - min_loop" calc.
004 : update from prev 004 : change in lut generation : avoid "i -
min_loop" calc.
Martin
Hello,
Patch in attach add some bitexact conversion for grayf32
and unscaled conv for gray16 to grayf32.
Pass fate test for me (x86_64, macos 10.12)
001 : Add bit exact conv for float to uint16
002 : Add bit exact conv for float to uint8
003 : Add bit exact lut generation for 8b to float
11 matches
Mail list logo
