Just one remark in addition to what Thomas Steinmaurer has already shown
you. Do not believe blindly in what stored procedure debuggers show you.
What you see in the debugger window, as being the value of a SP
variable, is not neccesarily the same as the actual value in the server
process. Try to check (inside the procedure itself) whether the result
fits in a range defined by the desired precision, e.g.:
if (A < 8.6999 or A > 8.7001) then
[raise an exception or signal the error in other way]
and run the modified procedure without the debugger. Inserting the
result into a table may be a good idea as well, provided that you check
the inserted value by other means than by looking at it ("seeing"
requires conversion to string, which may or may not be accurate).
regards
Tomasz
On 2012-03-07 21:16, Rick Debay wrote:
> No one? My (probably unjustified) fear is that since the variables are
> stored in a four byte primitive instead of a packed-decimal like format,
> I may randomly run in to this.
>
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Rick Debay
> Sent: Tuesday, March 06, 2012 12:08 PM
> To: [email protected]
> Subject: [firebird-support] Precision in stored procedures
>
> If a stored procedure has three variables
>
> DECLARE VARIABLE A NUMERIC(18,3);
> DECLARE VARIABLE B NUMERIC(18,3);
> DECLARE VARIABLE C DOUBLE PRECISION;
>
> And a function is evaluated with these values
>
> A = B / C;
> A = 7.3 / 0.839080459770115
>
> Then the value stored in A will be 8.7 not 8.70000000000119 (a firebird
> stored procedure debugger) or 8.6999999999999994041095890410959 (windows
> 7 calculator), correct?
>
>
> Disclaimer: This message (including attachments) is confidential and may be
> privileged. If you have received it by mistake please notify the sender by
> return e-mail and delete this message from your system. Any unauthorized use
> or dissemination of this message in whole or in part is strictly prohibited.
> Please note that e-mails are susceptible to change. RxStrategies, Inc. shall
> not be liable for the improper or incomplete transmission of the information
> contained in this communication or for any delay in its receipt or damage to
> your system. RxStrategies, Inc. does not guarantee that the integrity of this
> communication has been maintained nor that this communication is free from
> viruses, interceptions or interference.
>
>
>
--
__--==============================--__
__--== Tomasz Tyrakowski ==--__
__--== SOL-SYSTEM ==--__
__--== http://www.sol-system.pl ==--__
__--==============================--__
