The only way to get what you expect is to do the rounding using integer
arithmetic.
The number 18.005 must be multiplied by 1000 to keep the implied precision.
Then do the rounding and reformat as a decimal.
TCL 8.5 has libtomath which promises arbitary precision integer arithetic
but 8.4 has
Here is a helpful link, that explains the theory:
http://www2.hursley.ibm.com/decimal/
tom jackson
On Sunday 04 May 2008 10:20, Bernhard van Woerden wrote:
The only way to get what you expect is to do the rounding using integer
arithmetic.
The number 18.005 must be multiplied by 1000 to keep
Hey all!
This is really more of a tcl question, but I'm hoping that someone on
the list might have an explanation. Why does [format %.2f 18.005] round
down to 18.00 and [format %.2f 1.415] round up to 1.42? Any
guesses? Am I missing something obvious here?
Tcl version 8.4, if it matters.
On 2008.05.03, William Scott Jordan [EMAIL PROTECTED] wrote:
This is really more of a tcl question, but I'm hoping that someone on
the list might have an explanation. Why does [format %.2f 18.005] round
down to 18.00 and [format %.2f 1.415] round up to 1.42? Any
guesses? Am I missing
On Sat, 2008-05-03 at 15:31 -0700, William Scott Jordan wrote:
Hey all!
This is really more of a tcl question, but I'm hoping that someone on
the list might have an explanation. Why does [format %.2f 18.005] round
down to 18.00 and [format %.2f 1.415] round up to 1.42? Any
guesses? Am
William Scott Jordan wrote:
Hey all!
This is really more of a tcl question, but I'm hoping that someone on
the list might have an explanation. Why does [format %.2f 18.005] round
down to 18.00 and [format %.2f 1.415] round up to 1.42? Any
guesses? Am I missing something obvious here?
The plot thickens:
% format %.2f 18.0051
18.01
No ideas, though.
Bas.
On 04/05/2008, at 8:01 AM, William Scott Jordan wrote:
Hey all!
This is really more of a tcl question, but I'm hoping that someone
on the list might have an explanation. Why does [format %.2f
18.005] round down to
It does explain it, but still results are not obvious :)
For example, this code:
===
set bn [set rn 0]
for {set i 0} {$i 1000} {incr i} {
set f ${i}.005
set r [format %.2f $f]
set d [expr $r - $i]
if {$d 0.0} {
incr bn
} else {
incr rn
}
}
puts
Yuck. Okay, so is there any practical work-around for getting X.XX5 to
consistently round up? I suppose I could do something like add
0.001 to any number that I'm rounding, but that seems pretty sloppy.
Is there a best practice for dealing with this?
-William
Dossy Shiobara wrote:
On 2008.05.03, William Scott Jordan [EMAIL PROTECTED] wrote:
Yuck. Okay, so is there any practical work-around for getting X.XX5 to
consistently round up? I suppose I could do something like add
0.001 to any number that I'm rounding, but that seems pretty sloppy.
Is there a best
You have to remember that floating point math is done in base 2, but your are
inputing your numbers in base 10.
Another weired thing to keep in mind is that the default precision in Tcl has
changed in Tcl 8.5, so some answers are now different than they were in 8.4.
Another thing to keep in
11 matches
Mail list logo
