Re: IEE754 and maximum correct integer value

[Eike Rathke]

Hi Winfried,

On Wednesday, 2016-09-28 16:54:17 +0200, Winfried Donkers wrote:

> > > But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 
> > > 3^31-2, 3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 
> > > as result, but 0, 0 and 3.
> > > This test is simply in Calc with e.g. =(3^31)-((3^31)-1).
> > That's due to approxSub() being used for operator '-', which lessens the
> > available precision. approxEqual() needs to be reworked to not scale in
> > such cases.
> > 
> > I'll investigate.
> > 
> OK, that explains the remaining problem with my MOD() fix
> (https://gerrit.libreoffice.org/29365) too.

Should be cured with
https://cgit.freedesktop.org/libreoffice/core/commit/?id=2135eae2a97c17d89cb47a2074830fd2d7b2226f
Please check with a cherry-pick.

  Eike


signature.asc
Description: PGP signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: IEE754 and maximum correct integer value

[Winfried Donkers]

Hi Eike,



But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2, 
3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 as result, but 
0, 0 and 3.
This test is simply in Calc with e.g. =(3^31)-((3^31)-1).

That's due to approxSub() being used for operator '-', which lessens the
available precision. approxEqual() needs to be reworked to not scale in
such cases.

I'll investigate.

OK, that explains the remaining problem with my MOD() fix 
(https://gerrit.libreoffice.org/29365) too.



Winfried
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: IEE754 and maximum correct integer value

[Eike Rathke]

Hi Winfried,

On Wednesday, 2016-09-28 13:15:53 +0200, Winfried Donkers wrote:

> I'm working on some improvements of Calc's MOD function, but I struggle with 
> some unexpected results when using large integer values.
> As I understand IEEE754, double precision means a mantissa of 53 bits, so the 
> maximum correct unsigned integer value would be 2^53.
> 
> But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2, 
> 3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 as result, 
> but 0, 0 and 3.
> This test is simply in Calc with e.g. =(3^31)-((3^31)-1).

That's due to approxSub() being used for operator '-', which lessens the
available precision. approxEqual() needs to be reworked to not scale in
such cases.

I'll investigate.

  Eike

-- 
LibreOffice Calc developer. Number formatter stricken i18n transpositionizer.
GPG key "ID" 0x65632D3A - 2265 D7F3 A7B0 95CC 3918  630B 6A6C D5B7 6563 2D3A
Better use 64-bit 0x6A6CD5B765632D3A here is why: https://evil32.com/
Care about Free Software, support the FSFE https://fsfe.org/support/?erack


signature.asc
Description: PGP signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: IEE754 and maximum correct integer value

[Luc Castermans]

Winfried,  to have an external reference I entered your input in R
(64-bit), open source
statistical SW, R confirms your math. *Calc is wrong*.  Below is from
R-commandline:


> (3^31)-((3^31)-1)[1] 1> (3^31)-((3^31)-2)[1] 2> (3^31)-((3^31)-3)[1] 3


>

Below is whtat R states about precision:

All *R* platforms are required to work with values conforming to the IEC
60559 (also known as IEEE 754) standard. This basically works with a
precision of 53 bits, and represents to that precision a range of absolute
values from about *2e-308* to *2e+308*. It also has special values NaN

(many of them), plus and minus infinity and plus and minus zero (although
*R* acts as if these are the same). There are also *denormal(ized)* (or
*subnormal*) numbers with absolute values above or below the range given
above but represented to less precision.



2016-09-28 13:15 GMT+02:00 Winfried Donkers :

> Hi,
>
> I'm working on some improvements of Calc's MOD function, but I struggle
> with some unexpected results when using large integer values.
> As I understand IEEE754, double precision means a mantissa of 53 bits, so
> the maximum correct unsigned integer value would be 2^53.
>
> But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2,
> 3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 as result,
> but 0, 0 and 3.
> This test is simply in Calc with e.g. =(3^31)-((3^31)-1).
> I use a 64bit machine and master-version of LO.
>
> What am I doing wrong here?
>
> Winfried
>
>
> ___
> LibreOffice mailing list
> LibreOffice@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/libreoffice
>



-- 
Luc Castermans
mailto:luc.casterm...@gmail.com
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

IEE754 and maximum correct integer value

[Winfried Donkers]

Hi,

I'm working on some improvements of Calc's MOD function, but I struggle with 
some unexpected results when using large integer values.
As I understand IEEE754, double precision means a mantissa of 53 bits, so the 
maximum correct unsigned integer value would be 2^53.

But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2, 
3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 as result, but 
0, 0 and 3.
This test is simply in Calc with e.g. =(3^31)-((3^31)-1).
I use a 64bit machine and master-version of LO.

What am I doing wrong here?

Winfried


___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice