Sam Ruby wrote:
> On Fri, Aug 22, 2008 at 7:07 PM, Waldemar Horwat <[EMAIL PROTECTED]> wrote:
>> Sam Ruby wrote:
>>> I find it easier to talk about real examples than abstractions. I've
>>> done the following quickly, so forgive me if I get some detail wrong.
>>>
>>> A binary floating point number has 52 bits of fraction, and by
>>> assuming an implicit leading one, they get an additional bit. This
>>> means that 1.1 is stored as (for brevity, I'll use hex)
>>>
>>> [1].1999999999999
>>>
>>> A conversion of that to decimal128 would be equivalent to computing
>>>
>>> 4953959590107545m / 4503599627370496m
>>>
>>> Which would produce
>>>
>>> 1.099999999999999866773237044981215
>> That's incorrect. The correct answer is
>> 1.100000000000000088817841970012523, which is closer mathematically to 1.1.
>
> Apparently, I rounded incorrectly. Rounding the first number
> correctly produces:
>
> 1.199999999999A
> 4953959590107546m / 4503599627370496m
> 1.100000000000000088817841970012523
>
>>> Repeating that for 1.2 produces
>>>
>>> 0x13333333333333L
>>> 5404319552844595m / 4503599627370496m
>>> 1.199999999999999955591079014993738
>> This one is correct.
>
> One ramification of requiring up-conversion to decimal128 of mixed
> mode operations would mean that the following would be true
>
> (1.1 < 1.1m) && (1.2m < 1.2)
That's if we don't do the conversions the way you advocate in the other thread.
It's an interesting dilemma.
Double and float arithmetic already has the same property. If you take 1.1f
(i.e. the float) and compare it with 1.1 (the double), you'll find that one is
less than the other. At least the operators are consistent.
Waldemar
_______________________________________________
Es-discuss mailing list
[email protected]
https://mail.mozilla.org/listinfo/es-discuss
