Hi Glynn Thanks for the explanation. Good to know that the problem lies with the exceeding the integer range.. something I have to be careful about in the future.
On Thu, Sep 25, 2014 at 1:04 AM, Glynn Clements <gl...@gclements.plus.com> wrote: > > Paulo van Breugel wrote: > > > I have the following function (to calculate cell area) > > > > PI=3.1415926536 > > RES=0.0008333 > > R=6371007 > > r.mapcalc "test3 = (sin(y()+$RES/2) - sin(y()-$RES/2)) * ($RES * > $PI/180) * > > $R^2" > > > > This will give me a wrong results, apparently because R is an integer > > value. If I adapt the formula: > > > > r.mapcalc "test3 = (sin(y()+$RES/2) - sin(y()-$RES/2)) * ($RES * > $PI/180) * > > float($R)^2" > > > > I get the correct result. From the help file I understand that if any of > > the arguments is a floating number the result should be a floating > number. > > But that does not seem the case above so apparently I am missing > something > > here. Can anybody explain the logic behind the above? > > In the first case, 6371007^2 is calculated using integer arithmetic, > and the result (40589730194049) will exceed the range of an "int" > (2^31-1 = 2147483647), typically resulting in overflow (R^2 will > probably equal -2005720447). The value will be converted to "double" > for the multiplication, but by that point the error has already > occurred. > > In the second case, R is converted to "float" (single-precision > floating-point), and the result of squaring it is well within the > range of a "float", although it will be rounded to 40589731037184.0 > (using double() instead of float() will use double-precision and avoid > the rounding error). > > -- > Glynn Clements <gl...@gclements.plus.com> >
_______________________________________________ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev