Re: [AOLSERVER] Tcl rounding question
The only way to get what you expect is to do the rounding using integer arithmetic. The number 18.005 must be multiplied by 1000 to keep the implied precision. Then do the rounding and reformat as a decimal. TCL 8.5 has libtomath which promises arbitary precision integer arithetic but 8.4 has wide which gives a fair bit of precision before it blows up. Watch out expr 46341*46341 -2147479015 I've attached some procs that I use with 8.4 to round as I would expect without having to keep everything as an integer. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl rounding question
Here is a helpful link, that explains the theory: http://www2.hursley.ibm.com/decimal/ tom jackson On Sunday 04 May 2008 10:20, Bernhard van Woerden wrote: The only way to get what you expect is to do the rounding using integer arithmetic. The number 18.005 must be multiplied by 1000 to keep the implied precision. Then do the rounding and reformat as a decimal. TCL 8.5 has libtomath which promises arbitary precision integer arithetic but 8.4 has wide which gives a fair bit of precision before it blows up. Watch out expr 46341*46341 -2147479015 I've attached some procs that I use with 8.4 to round as I would expect without having to keep everything as an integer. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl rounding question
On 2008.05.03, William Scott Jordan [EMAIL PROTECTED] wrote: This is really more of a tcl question, but I'm hoping that someone on the list might have an explanation. Why does [format %.2f 18.005] round down to 18.00 and [format %.2f 1.415] round up to 1.42? Any guesses? Am I missing something obvious here? Classic floating-point precision and rounding issue. See: http://en.wikipedia.org/wiki/Floating_point#Representable_numbers.2C_conversion_and_rounding Read the third paragraph in that section. -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl rounding question
On Sat, 2008-05-03 at 15:31 -0700, William Scott Jordan wrote: Hey all! This is really more of a tcl question, but I'm hoping that someone on the list might have an explanation. Why does [format %.2f 18.005] round down to 18.00 and [format %.2f 1.415] round up to 1.42? Any guesses? Am I missing something obvious here? Hello William, As far as I can see, it's just doing the correct math. By convention, when you round: 1.00 = 1.0 1.01 = 1.0 1.02 = 1.0 1.03 = 1.0 1.04 = 1.0 1.05 = 1.1 1.06 = 1.1 1.07 = 1.1 1.08 = 1.1 1.09 = 1.1 and so on. This way you get .0 for 5 numbers, .1 for the other numbers, and then the distribution is uniform. And that's not tcl-only , but they way they teached me to round numbers in school ;-P If you simply want to discard the decimal numbers... you'd try a method different than format's :-) Regards, Juan José - Juan José del Río| Comercio online / e-commerce (+34) 616 512 340| [EMAIL PROTECTED] Simple Option S.L. Tel: (+34) 951 930 122 Fax: (+34) 951 930 122 http://www.simpleoption.com -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl rounding question
William Scott Jordan wrote: Hey all! This is really more of a tcl question, but I'm hoping that someone on the list might have an explanation. Why does [format %.2f 18.005] round down to 18.00 and [format %.2f 1.415] round up to 1.42? Any guesses? Am I missing something obvious here? Tcl version 8.4, if it matters. Thanks! -William Set tcl_precision to 17 to see the fullest expansion of the value that tcl will work with. % set tcl_precision 17 17 % expr 18.005 18.004 % expr 1.415 1.415 % The rounding I think is obvious at that point. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl rounding question
The plot thickens: % format %.2f 18.0051 18.01 No ideas, though. Bas. On 04/05/2008, at 8:01 AM, William Scott Jordan wrote: Hey all! This is really more of a tcl question, but I'm hoping that someone on the list might have an explanation. Why does [format %.2f 18.005] round down to 18.00 and [format %.2f 1.415] round up to 1.42? Any guesses? Am I missing something obvious here? Tcl version 8.4, if it matters. Thanks! -William -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl rounding question
It does explain it, but still results are not obvious :)
For example, this code:
===
set bn [set rn 0]
for {set i 0} {$i 1000} {incr i} {
set f ${i}.005
set r [format %.2f $f]
set d [expr $r - $i]
if {$d 0.0} {
incr bn
} else {
incr rn
}
}
puts Rounded to 0.01 in $bn cases, to 0.0 in $rn cases
produces results:
Rounded to 0.01 in 41 cases, to 0.00 in 959 cases
Thanks,
~ Alex.
On Sat, May 3, 2008 at 7:08 PM, Dossy Shiobara [EMAIL PROTECTED] wrote:
On 2008.05.03, William Scott Jordan [EMAIL PROTECTED] wrote:
This is really more of a tcl question, but I'm hoping that someone on
the list might have an explanation. Why does [format %.2f 18.005] round
down to 18.00 and [format %.2f 1.415] round up to 1.42? Any
guesses? Am I missing something obvious here?
Classic floating-point precision and rounding issue. See:
http://en.wikipedia.org/wiki/Floating_point#Representable_numbers.2C_conversion_and_rounding
Read the third paragraph in that section.
--
Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on. (p. 70)
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to [EMAIL
PROTECTED] with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to [EMAIL PROTECTED]
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Tcl rounding question
Yuck. Okay, so is there any practical work-around for getting X.XX5 to consistently round up? I suppose I could do something like add 0.001 to any number that I'm rounding, but that seems pretty sloppy. Is there a best practice for dealing with this? -William Dossy Shiobara wrote: On 2008.05.03, William Scott Jordan [EMAIL PROTECTED] wrote: This is really more of a tcl question, but I'm hoping that someone on the list might have an explanation. Why does [format %.2f 18.005] round down to 18.00 and [format %.2f 1.415] round up to 1.42? Any guesses? Am I missing something obvious here? Classic floating-point precision and rounding issue. See: http://en.wikipedia.org/wiki/Floating_point#Representable_numbers.2C_conversion_and_rounding Read the third paragraph in that section. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl rounding question
On 2008.05.03, William Scott Jordan [EMAIL PROTECTED] wrote: Yuck. Okay, so is there any practical work-around for getting X.XX5 to consistently round up? I suppose I could do something like add 0.001 to any number that I'm rounding, but that seems pretty sloppy. Is there a best practice for dealing with this? Best practice? Use integers, format as decimal value as needed. This is often why you'll see financial applications store money values in cents or hundredths of a cent, i.e., a dollar is stored as either 100 (100 pennies) or 1 ... if they deal in quantities that include fractional cents. Only at the point where the data leaves the system, either to a UI or to another system, do they format it with the decimal point in place. This approach generally eliminates all floating-point precision and rounding issues and can even result in a performance increase in compute-heavy applications where the machine's CPU handles integer math faster than floating-point math. -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl rounding question
You have to remember that floating point math is done in base 2, but your are
inputing your numbers in base 10.
Another weired thing to keep in mind is that the default precision in Tcl has
changed in Tcl 8.5, so some answers are now different than they were in 8.4.
Another thing to keep in mind is that there is a difference between rounding
and actual significant digits. You should carry out your calculations with as
much precision as possible, then round to something less than that.
But, back to the question you have below. Tcl [format] uses a rounding
function, it isn't a truncation of the numbers.
If you use [expr {round(18.01)}] you get 18 (an integer).
The question is why doesn't Tcl offer a math function to round to some other
decimal precision?
The answer is simple. You should never 'round' an intermediate result. That
means you don't need this in Tcl. Final results can be formatted, and
[format] offers many more options than just the number of decimal points.
(Example of a problem with intermediate rounding:
Two step round:
round(18.449, 2) = 18.45
round(18.45, 1) = 18.5
One step round:
round(18.449, 1) = 18.4
)
tom jackson
On Saturday 03 May 2008 16:10, Bas Scheffers wrote:
The plot thickens:
% format %.2f 18.0051
18.01
No ideas, though.
Bas.
On 04/05/2008, at 8:01 AM, William Scott Jordan wrote:
Hey all!
This is really more of a tcl question, but I'm hoping that someone
on the list might have an explanation. Why does [format %.2f
18.005] round down to 18.00 and [format %.2f 1.415] round up to
1.42? Any guesses? Am I missing something obvious here?
Tcl version 8.4, if it matters.
Thanks!
-William
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
[EMAIL PROTECTED]
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the
Subject: field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
[EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the
email message. You can leave the Subject: field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to [EMAIL PROTECTED]
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
