On 09/03/10 18:41 +0100, Cédric Krier wrote: > On 09/03/10 13:35 +0100, Cédric Krier wrote: > > Here is a patch to fix digits issue in product uom due to new validation of > > digits. > > > > Please review this at http://codereview.appspot.com/325042/show > > > > I'm facing an issue. > The float8 of postgresql can not store correctly the value: > 1.093613298338 > It is rounded to: > 1.09361329834 > > This value comes from the Yard UOM which is: > > 1 yard = 0.9144 meters > > So: > 1 meter = round(1.0 / 0.9144, 12) meters = 1.093613298338 > > The issue is that when trying to compute back the 0.9144 with the rounded > value we get: > > round(1.0 / 1.09361329834, 12) = 0.914399999998 > > But with the 12 digits value: > > round(1.0 / 1.093613298338, 12) = 0.9144 > > So float8 is not large enough to store the Yard conversion correctly. I > propose to store rate and factor with numeric/decimal field instead of float. > The precision has no limit but it will require to convert into float when > calling compute_qty(). This function is called in the stock module to compute > stock quantity, it is already well optimized to reduce the number of call but > it will be a little slower due to the convertion of Decimal into float but I > don't think we should care about. >
In fact, float8 are large enough so no need to switch to Decimal for this reason. But I think it is still a good progress. In fact the issue comes from psycopg2, it seems that it round float and Decimal to 11 decimal digits. Is there anybody aware of this? I will check the code source of psycopg2 to understand. -- Cédric Krier B2CK SPRL Rue de Rotterdam, 4 4000 Liège Belgium Tel: +32 472 54 46 59 Email/Jabber: [email protected] Website: http://www.b2ck.com/
pgpKu0qrEKuYk.pgp
Description: PGP signature
