Sorry John, forget me first mail, your formula is correct ... I missed a bracket ...
I am still not sure why but in some context VARSAMP return negative number !!!, which cause a floating point error if I try to compute STDSAMP ... Regards, Alex Le 05/04/2013 09:23, Alexandre Maurel a écrit : > Hello John, > > There is a problem with the computation of VARSAMP and STDSAMP . > > In selectParser.cc file you will find, > > // sample variance is computed as > // (sum (x^2) / count(*) - (sum (x) / count(*))^2) * (count(*) / > (count(*)-1)) > > which should be corrected by > > (sum (x^2) / ( count(*) -1) - (sum (x) / count(*))^2) * (count(*) / > (count(*)-1)) > > same correction for STDSAMP > > sqrt((sum (x^2) / ( count(*) -1) - (sum (x) / count(*))^2) * (count(*) > / (count(*)-1))) > > > Regards, > > Alex _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
