On 8/21/2017 3:02 PM, James Richters wrote: > I am having an issue with a simple floating point application. I am setting > a variable to a specific value and immediately after I set it, it is not > exactly what I set it to. Here's an example > > Draw_GX_Min:=999.999; > Writeln(Draw_GX_Min:3:30); > > The writeln results in 999.999000000000020000000000000000 > > Why is it not 999.999000000000000000000000000000 where did 0.00000000000002 > come from? > Out of thin air... Well, kind of. Double floating point means 16 digits of precision, so when you force a 30 digit precision output, anything past 16 digits is random garbage, at best...
Ralf --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus _______________________________________________ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
