A Monday 03 March 2008, Francesc Altet escrigué: > This has been sent to the PyTables users' list, but bounced. Please, > remember to always subscribe to the list before sending messages. > > Thanks, > > ---------- Missatge transmès ---------- > > Subject: Float32 accuracy > Date: Sunday 02 March 2008 > From: "Christiaan Putter" <[EMAIL PROTECTED]> > To: pytables-users@lists.sourceforge.net > > Hi guys and girls, > > I'm having some strange inaccuracies when storing data into float32 > columns in a table. > > I'm storing historical financial data so I'm rounding most things to > 2 decimal places. Which means the values I write into tables tend to > look like > > 107.43000000000001 or 110.31999999999999 > > Which is the 'normal' inaccuracy I'd expect from floating point > values in python ( double precision ?) > > So after writing 37.14 to my table I'd ge 37.1399993896 back. So > this is the normal single precision inaccuracy I'd expect. > > The problem: > > Storing 12266.39 gives me 12266.3896484 > > Normally I'd expect 7 digits to be accurate when working with single > precision floats, though here only 6 digits are. > > Am I wrong to assume 7 digits to be accurate or is something else > behind this?
You should have in mind that accuracy is not always exact in terms of (decimal) digits. To be exact, IEEE single precision numbers have a mantissa of 23 bits, that in general it is said that corresponds to 7 digits, but if we do the calculations the figure is rather 6.92 digits (i.e. math.log(2**23, 10)). So, you should always expect a precision in the *order* than 7 decimal digits, which is not exactly 7. In fact, if you compute the difference in your case: 12266.39-12266.3896484 = 0.00035159999970346689 you can see that 12266.39 has been represented by a float32 number up to 8 digits of precision (5 to the left of the decimal point and 3 to the rigth). So, the representation chosen is not bad at all. If what you really want is to achieve better accuracy for the first two decimal digits, you can always do this: math.ceil(12266.3896484*100)/100. = 12266.389999999999 In any case, in order to understand better the intricacies of floating point (in-) accuracies, it is always good to have a look at: http://en.wikipedia.org/wiki/Floating_point Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-" ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
