On 5/09/2007 6:18 AM, John Stanton wrote:
These are regular floating point numbers, and behave accordingly.
Utter nonsense. round(98926650.5, 1) -> 98926650.5000001 is a BUG.
98926650.5 is representable EXACTLY in "regular" floating point.
The round function should calculate a scale factor (10.0) in this case,
and then:
shifted = 98926650.5 * 10.0 // i.e. 989266505.0, with no loss of precision
floored = floor(shifted) // result: 989266505.0 (no change)
answer = floored / 10.0 // result: 98926650.5, exactly
And here, using Python as a calculator and exemplar, is what should
happen with Selena's second case:
| >>> original = 85227887.01
| >>> original
| 85227887.010000005
| >>> shifted = original * 10.0
| >>> shifted
| 852278870.10000002
| >>> floored = float(int(shifted))
| >>> floored
| 852278870.0
| >>> import math
| >>> math.floor(shifted)
| 852278870.0
| >>> answer = floored / 10.0
| >>> answer
| 85227887.0
| >>>
Serena Lien wrote:
I have read some of the postings/faq about the difficulties with the
round
function, when certain numbers do not have a finite representation in
binary, which SQLite uses. eg 9.95 is actually 9.9499...etc so that
round(
9.95, 1) rounds down.
But, I have found several numbers which don't get rounded at all, and in
fact return more decimal places!
round(98926650.5, 1) -> 98926650.5000001
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
