I needed to compute a large correlation matrix and my own approach hit a wall after 1000 objects and 200 variables with 1GB memory, while I had larger tasks to solve. Luckily I found Brian's old post and his multivariate approach saved my day. Thanks!
PS. Is there any hope the correlation matrix computing R_multi_ could find a way into the standard 'stats' library? I am sure that others have been longing for it as well. Or does the 'stats' library already contain that in some form? Tarmo Brian Schott wrote: > When the correlations for more than 2 pairs of data > sets are being computed, some time savings can be > accomplished by multivariate statistical methods. The > following attempts to show a time comparison for Devon's > approach and a similar, but multivariate, approach. Your > mileage may vary, but I get an order of magnitude > improvement for the multivariate approach with this 3 > variable data set, Y. [Thus three correlated pairs.] > > (B=) > > > coclass 'bi' NB. bivariate approach of Devon > coinsert 'base' > > corr =: cov % *&stddev > cov =: spdev % <:@#...@] > spdev =: +/@(*~ dev) > dev =: -"_1 _ mean > mean =: +/ % # > stddev =: %:@var > var =: ssdev % <:@# > ssdev =: +/@:*:@dev > > coclass 'multi' NB. multivariate approach > > coinsert 'base' > > sum =: +/ > transpose =: |: > > ss =: sum@:*: > > ctr =: sum%# NB. centroid > mnc =: ] -"1 ctr NB. meancorrected > mp =: sum . * NB. matrix product > sscp =: transpose mp ] > SSCP =: s...@mnc > stddev =: s...@mnc %:@% <:@# > std =: mnc %"1 stddev NB. standardized > R =: s...@std % <:@# > > coclass 'base' > > NB. be careful on the next looooong line > Y =: 12 3$1 1 1 1 2 1 1 2 2 1 3 2 2 5 4 2 5 6 2 6 5 2 7 4 3 10 8 3 11 7 3 11 > 9 3 12 10 > YY =: |: Y > ]a =: corr_bi_ "1/~ |:Y > ]b =: corr_bi_ "1/~ YY > ]c =: R_multi_ Y > a -: b > a -: c > 6!:2 'corr_bi_ "1/~ |:Y' > 6!:2 'corr_bi_ "1/~ YY' > 6!:2 'R_multi_ Y' > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
