Sorry, I was too quick there. Instead of >if X has a NaN in row t, then row t is deleted in both X and Y
I wanted to say if X and/or Y has a NaN in row t, then row t is deleted in both X and Y /Paul S On Monday, 18 April 2016 23:35:11 UTC+2, [email protected] wrote: > > > In particular, if X and Y contain NaNs in different places... > > I meant that if X has a NaN in row t, then row t is deleted in both X and > Y. Example: > X = [1;NaN] > Y = [10;11], > then we redefine as > Xb = [1] > Yb = [10] > and get Xb'Yb = [10] > > This is a fairly typical approach in eg. regression analysis. To > explicitly find&delete (many) such rows is time and memory intensive when > the matrices are large (eg. 200,000 rows instead of 2, with say 8,000 rows > that need to be deleted). I hoped NullableArrays would help here. > > Paul S > > > > On Monday, 18 April 2016 22:49:40 UTC+2, Milan Bouchet-Valat wrote: >> >> Le lundi 18 avril 2016 à 13:16 -0700, [email protected] a >> écrit : >> > Hi and thanks for the reply. >> > >> > However, I am not sure that I fully understand >> > >NullableArrays are not needed if you only have NaNs >> > >> > Maybe I have the wrong expectations about NullableArrays, but I hoped >> > that it would provide a quick "excise": cut out all rows where there >> > is a NaN in either X or Y and then do X'Y. Clearly, this excise can >> > be done explicitly but that costs time and memory. Am I wrong in this >> > expectation? >> I'm not sure what you mean. In particular, if X and Y contain NaNs in >> different places, removing rows/columns with NaNs may give matrices >> with incompatible dimensions. Could you provide an example? >> >> > Paul S >> > >> > >> > >> > > Le lundi 18 avril 2016 à 07:40 -0700, [email protected] a >> > > écrit : >> > > > Hi, >> > > > >> > > > I want to use NullableArrays to facilitate some multivariate >> > > > statistics (NaNs...). >> > > > >> > > > If X is a NullableArray{T,K} and Y is a NullableArray{T,L}, can I >> > > do >> > > > X'Y? (My clumsy attempts say no, but I might have missed >> > > something.) >> > > > >> > > > Thanks for the help /Paul S >> > > It looks like you need to defined zero(): >> > > Base.zero{T}(::Nullable{T}) = Nullable(zero(T)) >> > > >> > > Then it works, at least for simple cases. You should probably file >> > > an >> > > issue in GitHub against NullableArrays.jl so that we have a look at >> > > the >> > > best solution for this. This method shouldn't be defined in Julia >> > > by >> > > default (else many other methods will need a special treatment), >> > > but >> > > NullableArrays could do something about this. >> > > >> > > >> > > BTW, NullableArrays are not needed if you only have NaNs: floats >> > > handle >> > > them just fine. They are only useful when you have null/missing >> > > values >> > > other than NaN, or types other than floats. >> > > >> > > >> > > Regards >> >
