Sounds to me like normal floating number behavior. I know someone is trying to document this and explain why it happens (it's a technical reason that goes beyond my skills) but this seems entirely normal behavior given the constraints of programming languages generally and hardware in particular.
Best, Joel P.S. You can see similar issues in other spreadsheet software. This seems to give some indication as to the reasons why: https://support.microsoft.com/en-us/kb/78113 (yes I know it's MSO link...just explaining the reason why floating number isn't entirely accurate all the time) On Wed, Jul 8, 2015 at 11:33 AM, Rui Pedro Caldeira <[email protected]> wrote: > Hey and thanks for your fast answer. This is the data set: > > 0.000001 0.000001 0.000001 0.000001 0.000001 0.000001 0.000001 > 0.000001 0.000001 0.000001 0.000001 0.000001 0.000001 0.000001 > 0.000002 0.000003 0.999997 0.999997 0.999997 0.999997 0.999998 > 1.000001 1.999997 > > Using the formula in the spreadsheet it gives the value: 0.328061419 (VAR) > and 0.5727664611 (STDEV) > > > And yes, i do mean CHISQ.INV :) > > Thanks, Rui. > > Cumprimentos, > Rui Pedro Caldeira > > On Wed, Jul 8, 2015 at 7:24 PM, Brian Barker <[email protected]> > wrote: > > > At 18:44 08/07/2015 +0100, Rui Pedro Caldeira wrote: > > > >> I'm having some problems with the VAR and STDEV function, when executing > >> these lines of code. The average is well calculated but the remaining > two > >> return very low, unrealistic and equal values (6.93018471335974E-310). > >> > > > > That value is 2^-1027 - which is probably the smallest floating-point > > value your system can represent without underflow. > > > > Have you compared the results produced from your data by equivalent > normal > > formulae in a spreadsheet cell? > > > > Do you mean CHISQ.INV and not CHISQINV? > > > > Brian Barker > > > > -- > > To unsubscribe e-mail to: [email protected] > > Problems? > > http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ > > Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette > > List archive: http://listarchives.libreoffice.org/global/users/ > > All messages sent to this list will be publicly archived and cannot be > > deleted > > > > > > -- > To unsubscribe e-mail to: [email protected] > Problems? > http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ > Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette > List archive: http://listarchives.libreoffice.org/global/users/ > All messages sent to this list will be publicly archived and cannot be > deleted > -- *Joel Madero* LibreOffice QA Volunteer [email protected] -- To unsubscribe e-mail to: [email protected] Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://listarchives.libreoffice.org/global/users/ All messages sent to this list will be publicly archived and cannot be deleted
