I noticed that the c-lapack svd function is very slow and memory
inefficient on my (linux) laptop. The native J version in math/misc/svd
is actually many times faster and more memory efficient, so I used
imath/misc/svd instead. Now, this flags a bug in C-lapack for certain,
as it isn't so bad when i do the same thing in R, but this also shows
the amazing power of J.
As an addendum, I noticed that R no longer uses dgesvd; they use dgesdd
instead, as I guess it is more efficient. Also, both gesvd and native
svd techniques do produce the same answer.
trn=. 250000 30 $?.1e6#0
load'math/misc/svd'
load'math/lapack'
load'math/lapack/gesvd'
ts=: 6!:2, 7!:2@]
b=.ts 'q2=:svd 10000{.trn'
a=.ts 'q=:gesvd_jlapack_ 10000{.trn'
a%b
29.1044 171.825
If you want to see what I am using SVD for, I've been fiddling with
matrix approximants, CUR decomposition in particular:
https://github.com/locklin/jCUR
CUR decomposition is a 2009 technique for efficiently approximating
matrices by selecting quasi-random pieces of the original matrix. It
has utility in dimensionality reduction in the same spirit as PCA, but
it is a more interpretable, since you have original rows and columns of
the matrix. Such things might eventually be incorporated into Jd as a
way of making sense of large amounts of data. There are other such
techniques I plan on looking at eventually, but I want to find a good
use case for CUR first (probably something in portfolio theory).
-SL
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
