Perhaps the BASIC code was only displaying the number to a few decimal
places. In C, printf defaults to 6 decimal places, which isn't enough to
show the actual result as any different from the ideal result.
It's also possible that the spreadsheet was doing something more
complicated internally than just 28*(1/7). Playing around with an
example in C, I can see the error in representing (1/7), but multiplying
that by 28 gives exactly 4:
Decimal In-memory
(1/7) = 0.142857149243354797363281250000 0x3e124925
28*(1/7) = 4.000000000000000000000000000000 0x40800000
On the other hand, storing 1/490 in a float, then storing the result of
multiplying that by 70 (should be the same as 1/7), then storing the
result of multiplying that by 28, gives 3.99999976...:
Decimal In-memory
(1/490) = 0.002040816238150000572204589844 0x3b05bf37
70*(1/490) = 0.142857134342193603515625000000 0x3e124924
28*70*(1/490) = 3.999999761581420898437500000000 0x407fffff
I'm not suggesting that's exactly what the spreadsheet was doing, but it
shows that slight changes to the detail of the calculation (which
wouldn't affect the result with unlimited precision) can have an effect
on whether the result is equal to or slightly more or less than the
expected result.
Mark.
Virgil Arrington wrote:
Thanks for the explanation. Oddly enough, it makes some sense, even to
my non-techy brain. What I found interesting was that, in the
spreadsheet, I got the "wrong" 3.9999, but I wrote a simple BASIC
program on my beloved Tandy Model 100 and got the "correct" answer of 4.
I was excited to see that my little first generation laptop gave better
results than my state of the art (for the time) PC.
Virgil
On 02/10/2016 06:01 PM, [email protected] wrote:
Virgil Arrington wrote:
Ken Springer wrote
I remember years ago when Intel turned out a chip that had an error in
it's math calculations. It was a rare happening, but when they
finally admitted it publicly, trying to say it wasn't important do to
the rare occurrence, it did not go over well at all! <G>
About 25 years ago, I was the treasurer of my children's preschool. I
created a spreadsheet to calculate paychecks, and I found that the
paycheck was consistently off by .01 (a penny). It drove me nuts. As it
turned out, one part of the calculation required the division of 28 by
7, which every third grader knows is 4. Well, my spreadsheet gave an
answer of 3.9999999999_. By itself, it wasn't a big problem, but later
in the chain of operations, the 3.99999_ produced a result that rounded
*down* to the nearest penny instead of *up*, which it would have done if
the 28/7 had resulted in 4 instead of 3.9999. I complained to a computer
friend of mine who tried to explain that the computer's answer was more
"precise" than my mental math of 28/7=4. I didn't buy it.
I wouldn't say 3.9999999... is more precise, but it sounds like your
problem is related to the precision of floating-point numbers. 1/7,
when represented in binary, is a recurring fraction 0.001001001...
(like how 1/3 is 0.3333... in decimal), so cannot be represented
precisely in binary with a finite number of bits (just as you can add
as many '3's as you like to 0.3333, but it still doesn't exactly
represent 1/3).
I don't think there should be a problem with calculating 28/7 = 4 as a
single floating point operation, but if the calculation was done
(either in the way your formula was expressed or the way the
application processed it) as (1/7)*28, that may well give a slightly
inaccurate result due to rounding of the (1/7). A bit like calculating
1/3 to 0.333, then multiplying that by 12 to get 3.996 instead of 4.
If using a round-down function, and the result was slightly less than
4, it would round down to 3. I guess the solution was to round to the
nearest integer.
I learned a valuable lesson in blindly accepting a computer's
calculation simply because it was made by a computer.
Indeed. They do exactly what they're told by a combination of
hardware, software and user input - but for various reasons that might
not amount to what you thought you were telling it to do.
Mark.
--
To unsubscribe e-mail to: [email protected]
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/global/users/
All messages sent to this list will be publicly archived and cannot be deleted
