On 5/30/07, Jonas Maebe <[EMAIL PROTECTED]> wrote:
0.005, as extended, is actually 0.004999999999999999999898356046329484 As double, it is actually 0.0050000000000000001
For now I have managed to implement a work-around in my code which passes all unit tests I could throw at it. I first look at what precision is required from the user, by looking at the format string (eg: '. I then use a tiSetPrecision function that rounds the Extended (or Double or Single) value passed in to the correct amount of decimals using a default rounding method (my default is Common Rounding aka HalfUp). It then passes the new extended which contains the already rounded value to the FormatFloat() so no round needs to be done, only formatting. This works perfectly and gives consistent results under both FPC and Delphi. The rounding routine is based on the code posted to the Borland Developer Networks forums for all to use. Here is an extract which highlights the exact problem we are talking about: --------------------------------------- These routines round input values to fit as closely as possible to an output number with desired number of decimal fraction digits. Because, in general, numbers with decimal fractions cannot be exactly represented in IEEE-754 floating binary point variables, error limits are used to determine if the input numbers are intended to represent an exact decimal fraction rather than a nearby value. Thus an error limit will be taken into account when deciding that a number input such as 1.295, which internally is represented 1.29499 99999 …, really should be considered exactly 1.295 and that 0.29999 99999 ... really should be interpreted as 0.3 when applying rounding rules. Nine rounding rules are implemented. -------------------------------- The implementation as used in the tiOPF project can be viewed here (around line 3200): http://tinyurl.com/2jawq9 The function comment also includes the original link to the BDN page. I think the Free Pascal developers will find this code very useful. Hopefully it can find it's way into the FPC compiler. -- Graeme Geldenhuys General error, hit any user to continue. _______________________________________________ fpc-devel maillist - [email protected] http://lists.freepascal.org/mailman/listinfo/fpc-devel
