Hi everyone, > On 3. Dec 2021, at 12:58, Berthold Stoeger via subsurface > <[email protected]> wrote: > > Indeed, by rounding one down, I get the "alternative" value. And of course, > ceil() and floor() would exhibit the same instability depending on whether you > are eps above or below the integer. So I reckon you should use round(...) or > lrint(…) > .
this is an excellent observation. I completely missed the fact that for no good reason, the fraction fn2 and fhe are rounded to integers. As intermediate values, they should be kept double. While I was at it, I made MOD, END, EAD, EAAD integers since those are all depths and thus they should be mm in ints. My hope would now be that rounded to mm, the difference in floating point precision should not really show up (at least in most of the cases). Alan, would you please check if my PR passes the tests on your architectures? https://github.com/subsurface/subsurface/pull/3357 <https://github.com/subsurface/subsurface/pull/3357> Best Robert
signature.asc
Description: Message signed with OpenPGP
_______________________________________________ subsurface mailing list [email protected] http://lists.subsurface-divelog.org/cgi-bin/mailman/listinfo/subsurface
