On 30/09/16 10:35, Szabolcs Nagy wrote: > On 29/09/16 14:37, Andre Vieira (lists) wrote: >> >> On arm-none-eabi I'm seeing a failure for the long double type and inputs: >> { 1e-2l, 1e-4l, 1e-4l, 0.010000999950004999375l } >> >> The abs(frac) is higher than the toler: 1.73455e-16 vs 1e-16. Is that a >> reasonable difference? Should we raise toler3 to 1e-15? >> >> The last line is also too high: >> { 2147483647.l, 2147483647.l, 2147483647.l, 3719550785.027307813987l } >> Yields a frac of: 1.28198e-16 >> >> Those are the only ones that pass the 1e-16 threshold. >> > > i think the tolerance should be defined in > terms of LDBL_EPSILON (or numeric_limits). > > n*LDBL_EPSILON tolerance would accept hypot > with about n ulp error. >
now i see that there are huge ulp errors.. toler = 10*eps; should work for all formats, but currently there is >1000 ulp error on one of the double test cases.. so tolerance is carefully set to avoid triggering the failure there i'd set toler to 10000*eps if this test case is not for testing hypot quality.