Does this computation not always return a rank-1 matrix?
On Mon, Jul 28, 2014 at 12:33 PM, John Myles White <[email protected] > wrote: > But how would you know the rank of the correlation matrix in advance? > > -- John > > On Jul 28, 2014, at 9:25 AM, Stefan Karpinski <[email protected]> > wrote: > > This is the sort of thing that just begs for a custom representation of a > rank-1 matrix, which fortunately, isn't terribly hard to implement in Julia. > > > On Mon, Jul 28, 2014 at 12:08 PM, Tim Holy <[email protected]> wrote: > >> If they're sparse along dimension 1, you can at least save time computing >> the >> dot product of the two sparse vectors. But yes, the correlation matrix >> itself >> will be dense. >> >> --Tim >> >> On Monday, July 28, 2014 11:23:31 AM Jiahao Chen wrote: >> > > I don't think sparse cor() is implemented and is falling back to the >> dense >> > > implementation. >> > Computing the correlation matrix is much like computing the outer >> > product of two sparse vectors. There will be massive fill-in and I >> > don't see how you can preserve sparsity without special knowledge >> > about the sparsity pattern. >> > Thanks, >> > >> > Jiahao Chen >> > Staff Research Scientist >> > MIT Computer Science and Artificial Intelligence Laboratory >> > >> > On Mon, Jul 28, 2014 at 11:12 AM, Stefan Karpinski < >> [email protected]> >> wrote: >> > > https://github.com/JuliaLang/julia/issues/new >> > > >> > > >> > > On Mon, Jul 28, 2014 at 10:06 AM, paul analyst <[email protected] >> > >> > > >> > > wrote: >> > >> Issue on github or on julia-dev groups? >> > >> Paul >> > >> >> > >> W dniu poniedziałek, 28 lipca 2014 12:05:27 UTC+2 użytkownik Viral >> Shah >> > >> >> > >> napisał: >> > >>> Please file an issue. I don't think sparse cor() is implemented and >> is >> > >>> falling back to the dense implementation. >> > >>> >> > >>> -viral >> > >>> >> > >>> On Monday, July 28, 2014 1:41:55 PM UTC+5:30, paul analyst wrote: >> > >>>> Correlation sparse array is very slow. Out of memory on a dense >> array >> > >>>> when we have 30,000 columns. How quickly it calculated? >> > >>>> >> > >>>> julia> I=int32((rand(10^7)*9999999).+1); >> > >>>> >> > >>>> julia> J=int32((rand(10^7)*29999).+1); >> > >>>> >> > >>>> julia> V=int8((rand(10^7)*9).+1); >> > >>>> >> > >>>> julia> D=sparse(I,J,V); >> > >>>> >> > >>>> julia> @time cor(D[:,1:30]); >> > >>>> elapsed time: 23.806328476 seconds (2458875228 bytes allocated, >> 0.14% >> > >>>> gc >> > >>>> time) >> > >>>> >> > >>>> julia> @time cor(full(D[:,1:30])); >> > >>>> elapsed time: 4.494099126 seconds (2732042496 bytes allocated, >> 5.31% gc >> > >>>> time) >> > >>>> >> > >>>> julia> >> > >>>> >> > >>>> Paul >> >> > >
