Hi,
I'm Sorry for any cross-posting. I've reviewed the archives and could
not find an exact answer to my question below.
I'm trying to generate very large sparse matrices (< 1% non-zero
entries per row). I have a sparse matrix function below which works
well until the row/col count exceeds 10,000. This is being run on a
machine with 32G memory:
sparse_matrix <- function(dims,rnd,p) {
ptm <- proc.time()
x <- round(rnorm(dims*dims),rnd)
x[((abs(x) - p) < 0)] <- 0
y <- matrix(x,nrow=dims,ncol=dims)
proc.time() - ptm
}
When trying to generate the matrix around 20,000 rows/cols on a
machine with 32G of memory, the error message I receive is:
R(335) malloc: *** vm_allocate(size=3200004096) failed (error code=3)
R(335) malloc: *** error: can't allocate region
R(335) malloc: *** set a breakpoint in szone_error to debug
R(335) malloc: *** vm_allocate(size=3200004096) failed (error code=3)
R(335) malloc: *** error: can't allocate region
R(335) malloc: *** set a breakpoint in szone_error to debug
Error: cannot allocate vector of size 3125000 Kb
Error in round(rnorm(dims * dims), rnd) : unable to find the argument
'x' in selecting a method for function 'round'
* Last error line is obvious. Question: on machine w/32G memory, why
can't it allocate a vector of size 3125000 Kb?
When trying to generate the matrix around 30,000 rows/cols, the error
message I receive is:
Error in rnorm(dims * dims) : cannot allocate vector of length 900000000
Error in round(rnorm(dims * dims), rnd) : unable to find the argument
'x' in selecting a method for function 'round'
* Last error line is obvious. Question: is this 900000000 bytes?
kilobytes? This error seems to be specific now to rnorm, but it
doesn't indicate the length metric (b/Kb/Mb) as it did for 20,000
rows/cols. Even if this Mb, why can't this be allocated on a machine
with 32G free memory?
When trying to generate the matrix with over 50,000 rows/cols, the
error message I receive is:
Error in rnorm(n, mean, sd) : invalid arguments
In addition: Warning message:
NAs introduced by coercion
Error in round(rnorm(dims * dims), rnd) : unable to find the argument
'x' in selecting a method for function 'round'
* Same.
Why would it generate different errors in each case? Code fixes? Any
simple ways to generate sparse matrices which would avoid above
problems?
Thanks in advance,
Gavin
______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
