On 20 November 2013 at 19:24, Alessandro Vitale wrote:
| I've done the new test (dense matrix with just Rcpp NumericMatrix, no
RcppArmadillo) and in this case there is no leak.
|
| This is the code I used
|
| // [[Rcpp::export]]
| NumericMatrix dense_numericMatrix(NumericMatrix A){
| A(0,0) = 1;
| A(1,0) = 2;
| return A ;
| }
|
|
| testMatrix <- matrix(data=0,nrow=1e3*5,ncol=1e3) # 40MB Dense Matrix
| for(i in 1:5){
| a <- dense_numericMatrix(testMatrix)
| }
|
| and this is valgrind output
|
| ==28325== LEAK SUMMARY:
| ==28325== definitely lost: 36 bytes in 2 blocks
| ==28325== indirectly lost: 0 bytes in 0 blocks
| ==28325== possibly lost: 0 bytes in 0 blocks
| ==28325== still reachable: 74,382,445 bytes in 20,450 blocks
| ==28325== suppressed: 21,484 bytes in 82 blocks
| ==28325==
| ==28325== For counts of detected and suppressed errors, rerun with: -v
| ==28325== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 10 from 10)
Cool. Thanks for doing that.
| Can I support in some other ways in finding what's causing the problem in
RcppArmadillo sparse matrix?
We'd have to dig through the transition from the arma::sp_mat to what is
returned to R.
You could possibly start with the posts I had on the Rcpp Gallery as they
preceded some of the other work.
Thanks for doing that -- really appreciate it.
Dirk
| Alessandro
|
|
| On 20 Nov, 2013, at 18:36 , Dirk Eddelbuettel wrote:
|
| >
| > On 18 November 2013 at 18:16, Alessandro Vitale wrote:
| > | Dirk, I've run the code a couple times with valgrind, one time calling
sparse 5 times, the other calling sparse 10 times.
| > | Below is the main code and valgrind output.
| > | What do you think, could it be an Armadillo / RcppArmadillo leak?
| >
| > Could be. For comparison, what happens when you use a regular matrix (ie
| > Rcpp::NumericMatrix) and check with valgrind?
| >
| > Dirk
| >
| >
| > | testMatrix <- Matrix(data=0,nrow=1e7,ncol=1e7,sparse=TRUE)
| > |
| > | for(i in 1:5){
| > | a <- sparse(testMatrix)
| > | }
| > |
| > | ==22103== 160,000,032 bytes in 2 blocks are possibly lost in loss record
4,034 of 4,035
| > | ==22103== at 0xC658: malloc (vg_replace_malloc.c:295)
| > | ==22103== by 0x8BD503A: arma::SpMat<double>::init(unsigned long long,
unsigned long long) (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so)
| > | ==22103== by 0x8BD6690: Rcpp::traits::Exporter<arma::SpMat<double>
>::get() (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so)
| > | ==22103== by 0x8BD6B94: arma::SpMat<double>
Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*,
Rcpp::traits::r_type_generic_tag) (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so)
| > | ==22103== by 0x8BD4B37: sourceCpp_12484_sparse (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so)
| > | ==22103== by 0x86D84: do_dotcall (dotcode.c:579)
| > | ==22103== by 0xAEEF3: Rf_eval (eval.c:635)
| > | ==22103== by 0xB8E6D: Rf_applyClosure (eval.c:1003)
| > | ==22103== by 0xAEF93: Rf_eval (eval.c:654)
| > | ==22103== by 0xBB1DE: do_set (eval.c:1871)
| > | ==22103== by 0xAED66: Rf_eval (eval.c:607)
| > | ==22103== by 0xBB53F: do_begin (eval.c:1557)
| > | ==22103==
| > | ==22103== 240,000,048 bytes in 3 blocks are definitely lost in loss
record 4,035 of 4,035
| > | ==22103== at 0xC658: malloc (vg_replace_malloc.c:295)
| > | ==22103== by 0x8BD503A: arma::SpMat<double>::init(unsigned long long,
unsigned long long) (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so)
| > | ==22103== by 0x8BD6690: Rcpp::traits::Exporter<arma::SpMat<double>
>::get() (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so)
| > | ==22103== by 0x8BD6B94: arma::SpMat<double>
Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*,
Rcpp::traits::r_type_generic_tag) (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so)
| > | ==22103== by 0x8BD4B37: sourceCpp_12484_sparse (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/Rtmp8Krm0b/sourcecpp_565778edd300/sourceCpp_27814.so)
| > | ==22103== by 0x86D84: do_dotcall (dotcode.c:579)
| > | ==22103== by 0xAEEF3: Rf_eval (eval.c:635)
| > | ==22103== by 0xB8E6D: Rf_applyClosure (eval.c:1003)
| > | ==22103== by 0xAEF93: Rf_eval (eval.c:654)
| > | ==22103== by 0xBB1DE: do_set (eval.c:1871)
| > | ==22103== by 0xAED66: Rf_eval (eval.c:607)
| > | ==22103== by 0xBB53F: do_begin (eval.c:1557)
| > | ==22103==
| > | ==22103== LEAK SUMMARY:
| > | ==22103== definitely lost: 240,018,516 bytes in 14 blocks
| > | ==22103== indirectly lost: 0 bytes in 0 blocks
| > | ==22103== possibly lost: 160,010,424 bytes in 4 blocks
| > | ==22103== still reachable: 210,051,491 bytes in 52,853 blocks
| > | ==22103== suppressed: 25,144 bytes in 94 blocks
| > | ==22103==
| > | ==22103== For counts of detected and suppressed errors, rerun with: -v
| > | ==22103== ERROR SUMMARY: 8 errors from 8 contexts (suppressed: 10 from 10)
| > |
| > |
| > |
| > | testMatrix <- Matrix(data=0,nrow=1e7,ncol=1e7,sparse=TRUE)
| > |
| > | for(i in 1:10){
| > | a <- sparse(testMatrix)
| > | }
| > |
| > | ==22148==
| > | ==22148== 320,000,064 bytes in 4 blocks are possibly lost in loss record
4,039 of 4,040
| > | ==22148== at 0xC658: malloc (vg_replace_malloc.c:295)
| > | ==22148== by 0x8BD503A: arma::SpMat<double>::init(unsigned long long,
unsigned long long) (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so)
| > | ==22148== by 0x8BD6690: Rcpp::traits::Exporter<arma::SpMat<double>
>::get() (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so)
| > | ==22148== by 0x8BD6B94: arma::SpMat<double>
Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*,
Rcpp::traits::r_type_generic_tag) (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so)
| > | ==22148== by 0x8BD4B37: sourceCpp_93626_sparse (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so)
| > | ==22148== by 0x86D84: do_dotcall (dotcode.c:579)
| > | ==22148== by 0xAEEF3: Rf_eval (eval.c:635)
| > | ==22148== by 0xB8E6D: Rf_applyClosure (eval.c:1003)
| > | ==22148== by 0xAEF93: Rf_eval (eval.c:654)
| > | ==22148== by 0xBB1DE: do_set (eval.c:1871)
| > | ==22148== by 0xAED66: Rf_eval (eval.c:607)
| > | ==22148== by 0xBB53F: do_begin (eval.c:1557)
| > | ==22148==
| > | ==22148== 480,000,096 bytes in 6 blocks are definitely lost in loss
record 4,040 of 4,040
| > | ==22148== at 0xC658: malloc (vg_replace_malloc.c:295)
| > | ==22148== by 0x8BD503A: arma::SpMat<double>::init(unsigned long long,
unsigned long long) (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so)
| > | ==22148== by 0x8BD6690: Rcpp::traits::Exporter<arma::SpMat<double>
>::get() (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so)
| > | ==22148== by 0x8BD6B94: arma::SpMat<double>
Rcpp::internal::as<arma::SpMat<double> >(SEXPREC*,
Rcpp::traits::r_type_generic_tag) (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so)
| > | ==22148== by 0x8BD4B37: sourceCpp_93626_sparse (in
/private/var/folders/2h/s9877ww97794_0fg6vrtm54c0000gn/T/RtmpLZuqjl/sourcecpp_56846fe5a651/sourceCpp_83305.so)
| > | ==22148== by 0x86D84: do_dotcall (dotcode.c:579)
| > | ==22148== by 0xAEEF3: Rf_eval (eval.c:635)
| > | ==22148== by 0xB8E6D: Rf_applyClosure (eval.c:1003)
| > | ==22148== by 0xAEF93: Rf_eval (eval.c:654)
| > | ==22148== by 0xBB1DE: do_set (eval.c:1871)
| > | ==22148== by 0xAED66: Rf_eval (eval.c:607)
| > | ==22148== by 0xBB53F: do_begin (eval.c:1557)
| > | ==22148==
| > | ==22148== LEAK SUMMARY:
| > | ==22148== definitely lost: 480,041,092 bytes in 28 blocks
| > | ==22148== indirectly lost: 0 bytes in 0 blocks
| > | ==22148== possibly lost: 320,008,408 bytes in 5 blocks
| > | ==22148== still reachable: 217,092,931 bytes in 52,888 blocks
| > | ==22148== suppressed: 25,144 bytes in 94 blocks
| > | ==22148==
| > | ==22148== For counts of detected and suppressed errors, rerun with: -v
| > | ==22148== ERROR SUMMARY: 7 errors from 7 contexts (suppressed: 10 from 10)
| > |
| > |
| > |
| > |
| > |
| > | On 18 Nov, 2013, at 15:59 , Alessandro Vitale wrote:
| > |
| > | > Thanks Dirk and Hadley.
| > | >
| > | > Dirk, asap I will try your b) and c) suggestions. While for suggestion
a), I didn't really understand how to implement it.
| > | >
| > | > Hadley, I am determining rsession process memory looking at osx
activity monitor utility, under Real Mem column.
| > | > This is what I get (I am using your own mem() function to determine r
memory usage)
| > | >
| > | > testMatrix <- Matrix(data=0,nrow=1e7,ncol=1e7,sparse=TRUE)
| > | > gc()
| > | > mem() # [1] 123.3, rsession Real Mem 203.1 MB
| > | > sparse(testMatrix)
| > | > mem() # [1] 161.5
| > | > gc()
| > | > mem() # [1] 123.4, rsession Real Mem 507.8 MB
| > | > sparse(testMatrix)
| > | > mem() # [1] 161.5
| > | > gc()
| > | > mem() # [1] 123.4, rsession Real Mem 585.0 MB
| > | > sparse(testMatrix)
| > | > mem() # [1] 161.5
| > | > gc()
| > | > mem() # [1] 123.4, rsession Real Mem 661.3 MB
| > | >
| > | > and so on.
| > | >
| > | > If I call the function too many times (in fact I discovered it while
using benchmark() ), it fills up the memory and the computer gets totally
unresponsive.
| > | >
| > | >
| > | >
| > | >
| > | >
| > | > On 18 Nov, 2013, at 15:24 , Hadley Wickham wrote:
| > | >
| > | >>> After every call, the memory from gc() is stable, while rsession
process
| > | >>> memory keeps growing of around 100Mb at each call and doesn't
decrease after
| > | >>> the garbage collection.
| > | >>
| > | >> You don't mention how you're determining the size of the rsession
| > | >> process, but you shouldn't necessarily expect it to decrease after
| > | >> calling gc() - there may be memory fragmentation, but also the OS is
| > | >> lazy about reclaiming memory from processes, so it may only be
| > | >> returned to the OS when it actually needs it.
| > | >>
| > | >> Hadley
| > | >>
| > | >> --
| > | >> Chief Scientist, RStudio
| > | >> http://had.co.nz/
| > | >
| > |
| > | _______________________________________________
| > | Rcpp-devel mailing list
| > | Rcpp-devel@lists.r-forge.r-project.org
| > | https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
| >
| > --
| > Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
|
--
Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
_______________________________________________
Rcpp-devel mailing list
Rcpp-devel@lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
