It does explain it, but still results are not obvious :)
For example, this code:
=======
set bn [set rn 0]
for {set i 0} {$i < 1000} {incr i} {
set f "${i}.005"
set r [format %.2f $f]
set d [expr $r - $i]
if {$d > 0.0} {
incr bn
} else {
incr rn
}
}
puts "Rounded to 0.01 in $bn cases, to 0.0 in $rn cases"
============
produces results:
Rounded to 0.01 in 41 cases, to 0.00 in 959 cases
Thanks,
~ Alex.
On Sat, May 3, 2008 at 7:08 PM, Dossy Shiobara <[EMAIL PROTECTED]> wrote:
> On 2008.05.03, William Scott Jordan <[EMAIL PROTECTED]> wrote:
> > This is really more of a tcl question, but I'm hoping that someone on
> > the list might have an explanation. Why does [format %.2f 18.005] round
> > down to "18.00" and [format %.2f 1.415] round up to "1.42"? Any
> > guesses? Am I missing something obvious here?
>
> Classic floating-point precision and rounding issue. See:
>
>
> http://en.wikipedia.org/wiki/Floating_point#Representable_numbers.2C_conversion_and_rounding
>
> Read the third paragraph in that section.
>
> --
> Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/
> Panoptic Computer Network | http://panoptic.com/
> "He realized the fastest way to change is to laugh at your own
> folly -- then you can let go and quickly move on." (p. 70)
>
>
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to <[EMAIL
> PROTECTED]> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
> field of your email blank.
>
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
