On Fri, Aug 22, 2008 at 7:07 PM, Waldemar Horwat <[EMAIL PROTECTED]> wrote: > Sam Ruby wrote: >> I find it easier to talk about real examples than abstractions. I've >> done the following quickly, so forgive me if I get some detail wrong. >> >> A binary floating point number has 52 bits of fraction, and by >> assuming an implicit leading one, they get an additional bit. This >> means that 1.1 is stored as (for brevity, I'll use hex) >> >> [1].1999999999999 >> >> A conversion of that to decimal128 would be equivalent to computing >> >> 4953959590107545m / 4503599627370496m >> >> Which would produce >> >> 1.099999999999999866773237044981215 > > That's incorrect. The correct answer is 1.100000000000000088817841970012523, > which is closer mathematically to 1.1.
Apparently, I rounded incorrectly. Rounding the first number correctly produces: 1.199999999999A 4953959590107546m / 4503599627370496m 1.100000000000000088817841970012523 >> Repeating that for 1.2 produces >> >> 0x13333333333333L >> 5404319552844595m / 4503599627370496m >> 1.199999999999999955591079014993738 > > This one is correct. One ramification of requiring up-conversion to decimal128 of mixed mode operations would mean that the following would be true (1.1 < 1.1m) && (1.2m < 1.2) > Waldemar > _______________________________________________ > Es-discuss mailing list > [email protected] > https://mail.mozilla.org/listinfo/es-discuss > _______________________________________________ Es-discuss mailing list [email protected] https://mail.mozilla.org/listinfo/es-discuss
