Dirk, I've run the code a couple times with valgrind, one time calling sparse 5 times, the other calling sparse 10 times. Below is the main code and valgrind output. What do you think, could it be an Armadillo / RcppArmadillo leak?
testMatrix <- Matrix(data=0,nrow=1e7,ncol=1e7,sparse=TRUE) for(i in 1:5){ a <- sparse(testMatrix) } ==22103== 160,000,032 bytes in 2 blocks are possibly lost in loss record 4,034 of 4,035 ==22103== at 0xC658: malloc (vg_replace_malloc.c:295) ==22103== by 0x8BD503A: arma::SpMat<double>::init(unsigned long long, unsigned long long) (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so) ==22103== by 0x8BD6690: Rcpp::traits::Exporter<arma::SpMat<double> >::get() (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so) ==22103== by 0x8BD6B94: arma::SpMat<double> Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*, Rcpp::traits::r_type_generic_tag) (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so) ==22103== by 0x8BD4B37: sourceCpp_12484_sparse (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so) ==22103== by 0x86D84: do_dotcall (dotcode.c:579) ==22103== by 0xAEEF3: Rf_eval (eval.c:635) ==22103== by 0xB8E6D: Rf_applyClosure (eval.c:1003) ==22103== by 0xAEF93: Rf_eval (eval.c:654) ==22103== by 0xBB1DE: do_set (eval.c:1871) ==22103== by 0xAED66: Rf_eval (eval.c:607) ==22103== by 0xBB53F: do_begin (eval.c:1557) ==22103== ==22103== 240,000,048 bytes in 3 blocks are definitely lost in loss record 4,035 of 4,035 ==22103== at 0xC658: malloc (vg_replace_malloc.c:295) ==22103== by 0x8BD503A: arma::SpMat<double>::init(unsigned long long, unsigned long long) (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so) ==22103== by 0x8BD6690: Rcpp::traits::Exporter<arma::SpMat<double> >::get() (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so) ==22103== by 0x8BD6B94: arma::SpMat<double> Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*, Rcpp::traits::r_type_generic_tag) (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so) ==22103== by 0x8BD4B37: sourceCpp_12484_sparse (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so) ==22103== by 0x86D84: do_dotcall (dotcode.c:579) ==22103== by 0xAEEF3: Rf_eval (eval.c:635) ==22103== by 0xB8E6D: Rf_applyClosure (eval.c:1003) ==22103== by 0xAEF93: Rf_eval (eval.c:654) ==22103== by 0xBB1DE: do_set (eval.c:1871) ==22103== by 0xAED66: Rf_eval (eval.c:607) ==22103== by 0xBB53F: do_begin (eval.c:1557) ==22103== ==22103== LEAK SUMMARY: ==22103== definitely lost: 240,018,516 bytes in 14 blocks ==22103== indirectly lost: 0 bytes in 0 blocks ==22103== possibly lost: 160,010,424 bytes in 4 blocks ==22103== still reachable: 210,051,491 bytes in 52,853 blocks ==22103== suppressed: 25,144 bytes in 94 blocks ==22103== ==22103== For counts of detected and suppressed errors, rerun with: -v ==22103== ERROR SUMMARY: 8 errors from 8 contexts (suppressed: 10 from 10) testMatrix <- Matrix(data=0,nrow=1e7,ncol=1e7,sparse=TRUE) for(i in 1:10){ a <- sparse(testMatrix) } ==22148== ==22148== 320,000,064 bytes in 4 blocks are possibly lost in loss record 4,039 of 4,040 ==22148== at 0xC658: malloc (vg_replace_malloc.c:295) ==22148== by 0x8BD503A: arma::SpMat<double>::init(unsigned long long, unsigned long long) (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so) ==22148== by 0x8BD6690: Rcpp::traits::Exporter<arma::SpMat<double> >::get() (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so) ==22148== by 0x8BD6B94: arma::SpMat<double> Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*, Rcpp::traits::r_type_generic_tag) (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so) ==22148== by 0x8BD4B37: sourceCpp_93626_sparse (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so) ==22148== by 0x86D84: do_dotcall (dotcode.c:579) ==22148== by 0xAEEF3: Rf_eval (eval.c:635) ==22148== by 0xB8E6D: Rf_applyClosure (eval.c:1003) ==22148== by 0xAEF93: Rf_eval (eval.c:654) ==22148== by 0xBB1DE: do_set (eval.c:1871) ==22148== by 0xAED66: Rf_eval (eval.c:607) ==22148== by 0xBB53F: do_begin (eval.c:1557) ==22148== ==22148== 480,000,096 bytes in 6 blocks are definitely lost in loss record 4,040 of 4,040 ==22148== at 0xC658: malloc (vg_replace_malloc.c:295) ==22148== by 0x8BD503A: arma::SpMat<double>::init(unsigned long long, unsigned long long) (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so) ==22148== by 0x8BD6690: Rcpp::traits::Exporter<arma::SpMat<double> >::get() (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so) ==22148== by 0x8BD6B94: arma::SpMat<double> Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*, Rcpp::traits::r_type_generic_tag) (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so) ==22148== by 0x8BD4B37: sourceCpp_93626_sparse (in /private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so) ==22148== by 0x86D84: do_dotcall (dotcode.c:579) ==22148== by 0xAEEF3: Rf_eval (eval.c:635) ==22148== by 0xB8E6D: Rf_applyClosure (eval.c:1003) ==22148== by 0xAEF93: Rf_eval (eval.c:654) ==22148== by 0xBB1DE: do_set (eval.c:1871) ==22148== by 0xAED66: Rf_eval (eval.c:607) ==22148== by 0xBB53F: do_begin (eval.c:1557) ==22148== ==22148== LEAK SUMMARY: ==22148== definitely lost: 480,041,092 bytes in 28 blocks ==22148== indirectly lost: 0 bytes in 0 blocks ==22148== possibly lost: 320,008,408 bytes in 5 blocks ==22148== still reachable: 217,092,931 bytes in 52,888 blocks ==22148== suppressed: 25,144 bytes in 94 blocks ==22148== ==22148== For counts of detected and suppressed errors, rerun with: -v ==22148== ERROR SUMMARY: 7 errors from 7 contexts (suppressed: 10 from 10) On 18 Nov, 2013, at 15:59 , Alessandro Vitale wrote: > Thanks Dirk and Hadley. > > Dirk, asap I will try your b) and c) suggestions. While for suggestion a), I > didn't really understand how to implement it. > > Hadley, I am determining rsession process memory looking at osx activity > monitor utility, under Real Mem column. > This is what I get (I am using your own mem() function to determine r memory > usage) > > testMatrix <- Matrix(data=0,nrow=1e7,ncol=1e7,sparse=TRUE) > gc() > mem() # [1] 123.3, rsession Real Mem 203.1 MB > sparse(testMatrix) > mem() # [1] 161.5 > gc() > mem() # [1] 123.4, rsession Real Mem 507.8 MB > sparse(testMatrix) > mem() # [1] 161.5 > gc() > mem() # [1] 123.4, rsession Real Mem 585.0 MB > sparse(testMatrix) > mem() # [1] 161.5 > gc() > mem() # [1] 123.4, rsession Real Mem 661.3 MB > > and so on. > > If I call the function too many times (in fact I discovered it while using > benchmark() ), it fills up the memory and the computer gets totally > unresponsive. > > > > > > On 18 Nov, 2013, at 15:24 , Hadley Wickham wrote: > >>> After every call, the memory from gc() is stable, while rsession process >>> memory keeps growing of around 100Mb at each call and doesn't decrease after >>> the garbage collection. >> >> You don't mention how you're determining the size of the rsession >> process, but you shouldn't necessarily expect it to decrease after >> calling gc() - there may be memory fragmentation, but also the OS is >> lazy about reclaiming memory from processes, so it may only be >> returned to the OS when it actually needs it. >> >> Hadley >> >> -- >> Chief Scientist, RStudio >> http://had.co.nz/ > _______________________________________________ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel