On 09/03/10 23:50 +0100, Cédric Krier wrote: > On 09/03/10 18:41 +0100, Cédric Krier wrote: > > On 09/03/10 13:35 +0100, Cédric Krier wrote: > > > Here is a patch to fix digits issue in product uom due to new validation > > > of > > > digits. > > > > > > Please review this at http://codereview.appspot.com/325042/show > > > > > > > I'm facing an issue. > > The float8 of postgresql can not store correctly the value: > > 1.093613298338 > > It is rounded to: > > 1.09361329834 > > > > This value comes from the Yard UOM which is: > > > > 1 yard = 0.9144 meters > > > > So: > > 1 meter = round(1.0 / 0.9144, 12) meters = 1.093613298338 > > > > The issue is that when trying to compute back the 0.9144 with the rounded > > value we get: > > > > round(1.0 / 1.09361329834, 12) = 0.914399999998 > > > > But with the 12 digits value: > > > > round(1.0 / 1.093613298338, 12) = 0.9144 > > > > So float8 is not large enough to store the Yard conversion correctly. I > > propose to store rate and factor with numeric/decimal field instead of > > float. > > The precision has no limit but it will require to convert into float when > > calling compute_qty(). This function is called in the stock module to > > compute > > stock quantity, it is already well optimized to reduce the number of call > > but > > it will be a little slower due to the convertion of Decimal into float but I > > don't think we should care about. > > > > In fact, float8 are large enough so no need to switch to Decimal for this > reason. But I think it is still a good progress. > > In fact the issue comes from psycopg2, it seems that it round float and > Decimal to 11 decimal digits. > Is there anybody aware of this? I will check the code source of psycopg2 to > understand. >
I understood. pyscopg2 convert float and Decimal by using str function but according to [1], str gives only 12 significant digits. That is why the float value are rounded. I will see how I could make psycopg2 to use repr instead of str. And perhaps report issue to psycopg2. [1] http://docs.python.org/tutorial/floatingpoint.html -- Cédric Krier B2CK SPRL Rue de Rotterdam, 4 4000 Liège Belgium Tel: +32 472 54 46 59 Email/Jabber: [email protected] Website: http://www.b2ck.com/
pgpd9eJ7W7dB2.pgp
Description: PGP signature
