-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jason Riedy wrote: > And Alois Schlögl writes: >> The only difference is that one implementation gives more often NaN >> (indicating no result) than the other implementation. Maybe its not >> wrong to return NaN, but its not what you want. You want the best >> possible estimate. > > Well, sometimes it is what *I* want. There are so many options of > what to do with NaNs that predicting what users want is tricky. > > The "best possible estimate" with missing data can be computed in > different ways! One involves an iterative process of replacing the > NaNs with an estimate based on the rest of the data. You'll find > that method in some experimental design literature, and you can > even find reference to software using signaling NaNs for that > purpose long, long ago.
Jason, Yes, I'm aware of this. You will find these techniques also under the keyword "imputation methods". Just because there are alternative methods does not make the proposed approach invalid. The proposed approach does not prevent anyone from implementing such algorithms. Actually, an efficient sumskipnan() will be beneficial for such iterative methods, because it provides an efficient way to obtain the initial value. Without such tricks, returning NaN as the > mean is a great way to identify when you should apply the iterative > method to your data set. There are alternatives, you could check for NaN's before you compute the mean. Or if the NaN-toolbox is in place, you can check this with flag_nan_occured() afterwards, and you have already the initial value for your iterative approach available. Specifically the second solution is much better because it does not waste computing the mean just to get a NaN. > > But, on the flip side, if you're using the mean to re-center the > data, then you absolutely do *not* want a NaN result. Subtracting > NaN from every entry will wipe out the entire data set and leave > you no trace of the original NaN. That's one of the cases where skipping NaN is the right thing to do. And sumskipnan() provides an efficient instrument to it. > > Baring more sophisticated handling of exceptional events and data, > the most reliable choices are to provide either a per-call optional > argument or two different routines. R takes the former route[1], > and we took the latter for max(a, b) and min(a, b) in the IEEE-754 > standard. I do not know about R, but last time I looked at IEEE-754r http://www.validlab.com/754R/drafts/archive/2006-10-04.pdf p.28, says about minNum(x,y) and maxNum(x,y) " ...[returns] the canonicalized floating point number if one operand is a floating-point number and the other a NaN". My understanding of this is, if one value is NaN, the other value is returned. In a vectorized form, all NaN's are skipped. I do not find an alternative implementation (one that propagates NaN) for min/max. The current implementation of min() and max() of Octave and Matlab ignore all NA/NaN's, too. Octave:187> min([5,NA,NaN]) ans = 5 octave:188> max([5,NA,NaN]) ans = 5 The NaN-toolbox is just applying the same principle to other statistical functions. > > A global flag that is not locally scoped and can be forgotten is > downright dangerous in this context. Such a flag will wander into > code where it was not intended and wreck havoc. Please use another > method. I see these alternatives: (1) In order to avoid wandering of the flag into code, I've added the following warning: "warning: flag_implicit_skipnan(0): You are warned!!! You have turned off skipping NaN in sumskipnan. This is not recommended. Make sure you really know what you do." Is this sufficient to address your concern ? (2) Remove the flag, here is the patch: diff sumskipnan.m sumskipnan.m.bak 82c82 < if ~isa(x,'float') || ~flag_implicit_skip_nan(), - --- > if ~isa(x,'float'), and flag_implicit_skip_nan becomes useless and can be removed. (3) there are already different functions (e.g. nanmean, nanstd, nanvar, etc) available. In order to get superior performance, You only need to modify these for using sumskipnan. In that case, you do not need the NaN-toolbox, you only need sumskipnan. But you are on your own whether to use mean or nanmean, std or nanstd, var or nanvar, etc. (4) Spend a lot of time with implementing checks of the input arguments, maintaining, and eventually changing default values, etc. I do not see a need for it. If some one else thinks this is useful, (s)he should go ahead and implement it. (5) It's also ok to leave out sumskipnan() from octave. Those who are interested have several options to install it: o) octave-nan package from Debian/Ubuntu o) the octave-forge repository o) pkg load nan (not tested by me) o) from here http://hci.tugraz.at/~schloegl/matlab/NaN/ o) its also distributed with "BioSig for Octave and Matlab" available from here: http://biosig.sourceforge.net/download.html Alois -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAknCQ3kACgkQzSlbmAlvEIgkpgCgp4I+ctYFgPhN1IYMosTTeXgb 4WwAoKA07Vl1tNMgl2fcGCdjY4Y/1Wqw =aaLM -----END PGP SIGNATURE----- ------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ Octave-dev mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/octave-dev
