Re: [Rcpp-devel] Forcing a shallow versus deep copy
Hi,
That's great, thanks for considering this!
Following this discussion, I went to browse through my code looking for
wrap() and as() statements that could benefit from a speed-up of memory
reuse. Of course I didn't find any.
I switched to using Modules when they were introduced, the code being much
nicer to read, and these conversions only happen behind the scene.
My c++ functions thus only deal with native Armadillo / C++ objects, and I
leave it up to the modules to magically do the required conversions in and
out. It's a brilliant interface, very readable.
From what I understand, however, the resulting code can often lose a factor
2-3 in speed, compared to the now much more verbose alternative of
explicitly converting and sharing the memory with this type of code:
arma::mat A(M.begin(), M.rows(), M.cols(), false);
From this perspective, the possibility of setting copy_aux_mem to false in
as(), as used by modules, would be very welcome.
Best regards,
baptiste
On 11 July 2013 10:22, rom...@r-enthusiasts.com wrote:
Hello,
This comes up every now and then, I think we can find a syntax to initiate
an arma::mat that would allow what you want.
It is not likely it will come via attributes. The idea is to keep them
simple. The solutions I see below would eventually lead to clutter, and we
are heading in the less clutter direction.
I'll think about it and propose something.
Romain
Le 2013-07-11 14:32, Changi Han a écrit :
Hello,
I think I (superficially) understand the difference between:
// [[Rcpp::export]]
double sum1(Rcpp::NumericMatrix M) {
arma::mat A(M.begin(), M.rows(), M.cols(), false);
return sum(sum(A));
}
// [[Rcpp::export]]
double sum2(arma::mat A) {
return sum(sum(A));
}
Partly out of laziness, partly because sum2 is more elegant, and
partly to avoid namespace pollution, I was wondering if there is a way
to force a shallow copy in sum2.
If not, then may I submit a low priority feature request. An
attribute? Some thing like:
// [[Rcpp::export]]
double sum2(arma::mat A) {
// [[ Rcpp::shallow ( A ) ]]
return sum(sum(A));
}
Or (akin to C++11 generalized attributes)
// [[Rcpp::export]] { [[ Rcpp::shallow ( A ) ]] }
double sum2(arma::mat A) {
return sum(sum(A));
}
An alternative is to have an argument in sourceCpp that takes a
list/vector of objects that are to be shallow or deep copied.
For example in sum1, if M is changed within the function before
casting to the arma::mat, then might be cleaner to add M to a
list/vector of objects to be deep copied rather than cloning M within
sum1: leads to one fewer variable name.
Just a thought. I can certainly live with the additional step. As
always, thanks for all the Rcpp goodness.
Cheers,
Changi Han
__**_
Rcpp-devel mailing list
Rcpp-devel@lists.r-forge.r-**project.orgRcpp-devel@lists.r-forge.r-project.org
https://lists.r-forge.r-**project.org/cgi-bin/mailman/**
listinfo/rcpp-develhttps://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
___
Rcpp-devel mailing list
Rcpp-devel@lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Forcing a shallow versus deep copy
I am sure there are better ways to achieve the goal. I would suggest that
these two be similar if possible. I think the naive expectation is for them
to be consistent.
// [[Rcpp::export]]
stuff function(Rcpp::stuff) {
}
// [[Rcpp::export]]
stuff function(arma::stuff) {
}
Thank you again. Cheers.
On Thu, Jul 11, 2013 at 9:22 PM, rom...@r-enthusiasts.com wrote:
Hello,
This comes up every now and then, I think we can find a syntax to initiate
an arma::mat that would allow what you want.
It is not likely it will come via attributes. The idea is to keep them
simple. The solutions I see below would eventually lead to clutter, and we
are heading in the less clutter direction.
I'll think about it and propose something.
Romain
Le 2013-07-11 14:32, Changi Han a écrit :
Hello,
I think I (superficially) understand the difference between:
// [[Rcpp::export]]
double sum1(Rcpp::NumericMatrix M) {
arma::mat A(M.begin(), M.rows(), M.cols(), false);
return sum(sum(A));
}
// [[Rcpp::export]]
double sum2(arma::mat A) {
return sum(sum(A));
}
Partly out of laziness, partly because sum2 is more elegant, and
partly to avoid namespace pollution, I was wondering if there is a way
to force a shallow copy in sum2.
If not, then may I submit a low priority feature request. An
attribute? Some thing like:
// [[Rcpp::export]]
double sum2(arma::mat A) {
// [[ Rcpp::shallow ( A ) ]]
return sum(sum(A));
}
Or (akin to C++11 generalized attributes)
// [[Rcpp::export]] { [[ Rcpp::shallow ( A ) ]] }
double sum2(arma::mat A) {
return sum(sum(A));
}
An alternative is to have an argument in sourceCpp that takes a
list/vector of objects that are to be shallow or deep copied.
For example in sum1, if M is changed within the function before
casting to the arma::mat, then might be cleaner to add M to a
list/vector of objects to be deep copied rather than cloning M within
sum1: leads to one fewer variable name.
Just a thought. I can certainly live with the additional step. As
always, thanks for all the Rcpp goodness.
Cheers,
Changi Han
__**_
Rcpp-devel mailing list
Rcpp-devel@lists.r-forge.r-**project.orgRcpp-devel@lists.r-forge.r-project.org
https://lists.r-forge.r-**project.org/cgi-bin/mailman/**
listinfo/rcpp-develhttps://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
___
Rcpp-devel mailing list
Rcpp-devel@lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Forcing a shallow versus deep copy
These __are__ similar. The difference is in the classes themselves.
Rcpp classes are proxy classes so C++ copy mechanism does not apply to
them. arma classes are proper c++ classes, so C++ semantics apply.
I'm at useR right now, so I can't really work on this. I'll submit at
least ideas later.
Romain
Le 2013-07-11 15:34, Changi Han a écrit :
I am sure there are better ways to achieve the goal. I would suggest
that these two be similar if possible. I think the naive expectation
is for them to be consistent.
// [[Rcpp::export]]
stuff function(Rcpp::stuff) {
}
// [[Rcpp::export]]
stuff function(arma::stuff) {
}
Thank you again. Cheers.
On Thu, Jul 11, 2013 at 9:22 PM, rom...@r-enthusiasts.com [3]
wrote:
Hello,
This comes up every now and then, I think we can find a syntax to
initiate an arma::mat that would allow what you want.
It is not likely it will come via attributes. The idea is to keep
them simple. The solutions I see below would eventually lead to
clutter, and we are heading in the less clutter direction.
Ill think about it and propose something.
Romain
Le 2013-07-11 14:32, Changi Han a écrit :
Hello,
I think I (superficially) understand the difference between:
// [[Rcpp::export]]
double sum1(Rcpp::NumericMatrix M) {
arma::mat A(M.begin(), M.rows(), M.cols(), false);
return sum(sum(A));
}
// [[Rcpp::export]]
double sum2(arma::mat A) {
return sum(sum(A));
}
Partly out of laziness, partly because sum2 is more elegant, and
partly to avoid namespace pollution, I was wondering if there is
a way
to force a shallow copy in sum2.
If not, then may I submit a low priority feature request. An
attribute? Some thing like:
// [[Rcpp::export]]
double sum2(arma::mat A) {
// [[ Rcpp::shallow ( A ) ]]
return sum(sum(A));
}
Or (akin to C++11 generalized attributes)
// [[Rcpp::export]] { [[ Rcpp::shallow ( A ) ]] }
double sum2(arma::mat A) {
return sum(sum(A));
}
An alternative is to have an argument in sourceCpp that takes a
list/vector of objects that are to be shallow or deep copied.
For example in sum1, if M is changed within the function before
casting to the arma::mat, then might be cleaner to add M to a
list/vector of objects to be deep copied rather than cloning M
within
sum1: leads to one fewer variable name.
Just a thought. I can certainly live with the additional step. As
always, thanks for all the Rcpp goodness.
Cheers,
Changi Han
___
Rcpp-devel mailing list
Rcpp-devel@lists.r-forge.r-project.org [1]
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
[2]
Links:
--
[1] mailto:Rcpp-devel@lists.r-forge.r-project.org
[2]
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
[3] mailto:rom...@r-enthusiasts.com
___
Rcpp-devel mailing list
Rcpp-devel@lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Forcing a shallow versus deep copy
On 11 July 2013 at 10:33, baptiste auguie wrote:
| Hi,
|
| That's great, thanks for considering this!
|
| Following this discussion, I went to browse through my code looking for wrap()
| and as() statements that could benefit from a speed-up of memory reuse. Of
| course I didn't find any.
| I switched to using Modules when they were introduced, the code being much
| nicer to read, and these conversions only happen behind the scene.
| My c++ functions thus only deal with native Armadillo / C++ objects, and I
| leave it up to the modules to magically do the required conversions in and
out.
| It's a brilliant interface, very readable.
|
| From what I understand, however, the resulting code can often lose a factor
2-3
| in speed, compared to the now much more verbose alternative of explicitly
| converting and sharing the memory with this type of code:
No way.
I have seen 2 to 3 __per cent__ which is very different from a factor 2 or 3.
This whole discussion is mostly a non-issue, really, as best as I can tell
because the cost ois really not that large.
Dirk
| arma::mat A(M.begin(), M.rows(), M.cols(), false);
|
| From this perspective, the possibility of setting copy_aux_mem to false in as
| (), as used by modules, would be very welcome.
|
| Best regards,
|
| baptiste
|
|
| On 11 July 2013 10:22, rom...@r-enthusiasts.com wrote:
|
|
| Hello,
|
| This comes up every now and then, I think we can find a syntax to initiate
| an arma::mat that would allow what you want.
|
| It is not likely it will come via attributes. The idea is to keep them
| simple. The solutions I see below would eventually lead to clutter, and we
| are heading in the less clutter direction.
|
| I'll think about it and propose something.
|
| Romain
|
| Le 2013-07-11 14:32, Changi Han a écrit :
|
|
| Hello,
|
| I think I (superficially) understand the difference between:
|
| // [[Rcpp::export]]
| double sum1(Rcpp::NumericMatrix M) {
| arma::mat A(M.begin(), M.rows(), M.cols(), false);
| return sum(sum(A));
| }
|
| // [[Rcpp::export]]
| double sum2(arma::mat A) {
| return sum(sum(A));
| }
|
| Partly out of laziness, partly because sum2 is more elegant, and
| partly to avoid namespace pollution, I was wondering if there is a way
| to force a shallow copy in sum2.
|
| If not, then may I submit a low priority feature request. An
| attribute? Some thing like:
|
| // [[Rcpp::export]]
| double sum2(arma::mat A) {
| // [[ Rcpp::shallow ( A ) ]]
| return sum(sum(A));
| }
|
| Or (akin to C++11 generalized attributes)
|
| // [[Rcpp::export]] { [[ Rcpp::shallow ( A ) ]] }
| double sum2(arma::mat A) {
| return sum(sum(A));
| }
|
| An alternative is to have an argument in sourceCpp that takes a
| list/vector of objects that are to be shallow or deep copied.
|
| For example in sum1, if M is changed within the function before
| casting to the arma::mat, then might be cleaner to add M to a
| list/vector of objects to be deep copied rather than cloning M within
| sum1: leads to one fewer variable name.
|
| Just a thought. I can certainly live with the additional step. As
| always, thanks for all the Rcpp goodness.
|
| Cheers,
| Changi Han
|
|
| ___
| Rcpp-devel mailing list
| Rcpp-devel@lists.r-forge.r-project.org
| https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
|
|
|
| --
| ___
| Rcpp-devel mailing list
| Rcpp-devel@lists.r-forge.r-project.org
| https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
--
Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
___
Rcpp-devel mailing list
Rcpp-devel@lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] how to pass -fPIC instead of -fpic when building rccp package
On 11 July 2013 at 15:12, Anwar Ludin wrote: | Hello, | | When building a rcpp package is there a way of forcing -fPIC instead of -fpic during package compilation? I 'm using RStudio as my main IDE and the package is built using That is an RStudio question, and they have it all documented. Set it under (IIRC) project options, else use ~/.R/Makevars and have it work for all R compilation, inside or outside of RStudio. Dirk | R CMD INSTALL --preclean --no-multiarch package_compilation | | Thanks! | ___ | Rcpp-devel mailing list | Rcpp-devel@lists.r-forge.r-project.org | https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel -- Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Forcing a shallow versus deep copy
Everybody has this existing example in their copy of Armadillo. I am running it here from SVN rather than the installed directory, but this should not make a difference. Machine is my not-overly-powerful thinkpad used for traveling: edd@don:~/svn/rcpp/pkg/RcppArmadillo/inst/examples$ r fastLm.r Loading required package: methods Attaching package: ‘Rcpp’ The following object is masked from ‘package:inline’: registerPlugin test replications relative elapsed user.self sys.self 2 fLmTwoCasts(X, y) 50001.000 0.184 0.2040.164 1 fLmOneCast(X, y) 50001.011 0.186 0.2000.172 4 fastLmPureDotCall(X, y) 50001.141 0.210 0.2360.184 3 fastLmPure(X, y) 50002.027 0.373 0.4120.332 6 lm.fit(X, y) 50002.685 0.494 0.5280.456 5 fastLm(frm, data = trees) 5000 36.380 6.694 7.3326.028 7 lm(frm, data = trees) 5000 42.734 7.863 8.6287.068 edd@don:~/svn/rcpp/pkg/RcppArmadillo/inst/examples$ What we are talking about here is the difference between 'fLmTwoCasts' and 'fLmOneCasts'. If you use larger objects, the different with be larger. But the relative differences are tiny. It would be nice to make this more elegant, and I look forward to Romain's proposals, but methinks that we may well have bigger fish to fry. Dirk, still in Sydney -- Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] how to pass -fPIC instead of -fpic when building rccp package
Hi Dirk, Yes I agree, I'm just having issues with the way the build process works, so let's make it a Rcpp problem ;) library(Rcpp) Rcpp.package.skeleton(testpackage) R CMD INSTALL --preclean --no-multiarch testpackage icpc -I/usr/include/R -DNDEBUG -I/usr/local/include -I/usr/lib64/R/library/Rcpp/include -fpic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -c rcpp_hello_world.cpp -o rcpp_hello_world.o icpc -shared -L/usr/local/lib64 -o testpackage.so rcpp_hello_world.o -L/usr/lib64/R/library/Rcpp/lib -lRcpp -Wl,-rpath,/usr/lib64/R/library/Rcpp/lib -L/usr/lib64/R/lib -lR installing to /home/aludin/R/x86_64-redhat-linux-gnu-library/3.0/testpackage/libs Here I'm using the Intel compiler as specified in my ~/.R/Makevars file: CC=icc CXX=icpc CFLAGS=-g -O -mtune=native So basically I have no idea where all the extra flags come from, in particular -fexceptions -fstack-protector -fpic I want to get rid of the -fstack-protector -fpic flags Cheers, Anwar On Jul 12, 2013, at 12:44 AM, Dirk Eddelbuettel e...@debian.org wrote: On 11 July 2013 at 15:12, Anwar Ludin wrote: | Hello, | | When building a rcpp package is there a way of forcing -fPIC instead of -fpic during package compilation? I 'm using RStudio as my main IDE and the package is built using That is an RStudio question, and they have it all documented. Set it under (IIRC) project options, else use ~/.R/Makevars and have it work for all R compilation, inside or outside of RStudio. Dirk | R CMD INSTALL --preclean --no-multiarch package_compilation | | Thanks! | ___ | Rcpp-devel mailing list | Rcpp-devel@lists.r-forge.r-project.org | https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel -- Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] how to pass -fPIC instead of -fpic when building rccp package
Anwar, On 12 July 2013 at 01:42, Anwar Ludin wrote: | Yes I agree, I'm just having issues with the way the build process works, so let's make it a Rcpp problem ;) | | library(Rcpp) | Rcpp.package.skeleton(testpackage) | | R CMD INSTALL --preclean --no-multiarch testpackage | | icpc -I/usr/include/R -DNDEBUG -I/usr/local/include -I/usr/lib64/R/library/Rcpp/include -fpic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -c rcpp_hello_world.cpp -o rcpp_hello_world.o | icpc -shared -L/usr/local/lib64 -o testpackage.so rcpp_hello_world.o -L/usr/lib64/R/library/Rcpp/lib -lRcpp -Wl,-rpath,/usr/lib64/R/library/Rcpp/lib -L/usr/lib64/R/lib -lR | installing to /home/aludin/R/x86_64-redhat-linux-gnu-library/3.0/testpackage/libs | | Here I'm using the Intel compiler as specified in my ~/.R/Makevars file: | | CC=icc | CXX=icpc | CFLAGS=-g -O -mtune=native | | So basically I have no idea where all the extra flags come from, in particular -fexceptions -fstack-protector -fpic | | I want to get rid of the -fstack-protector -fpic flags I suspect these stem from R's own configuration, set at its compile time. Try this: edd@max:~$ R CMD config CPPFLAGS edd@max:~$ R CMD config CXXFLAGS -g -O3 -Wall -pipe -Wno-unused -pedantic edd@max:~$ R CMD config CXXPICFLAGS -fpic edd@max:~$ See 'R CMD CONFIG --help' for more, and the $RHOME/etc/Makeconf file for the settings. Dirk -- Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Forcing a shallow versus deep copy
On 11 July 2013 at 19:21, Gabor Grothendieck wrote: | 1. Just to be clear what we have been discussing here is not just how to | avoid copying but how to avoid copying while using as and wrap | or approaches that automatically generate as and wrap. I was already | aware of how to avoid copying using Armadillo how to use Armadillo types | as arguments and return values to autogen as and wrap. The problem is | not that but that these two things cannot be done at once - its either or. I must still be misunderstanding as this still reads to me as if you are suspecting that we somehow keep layers making extra copies. We're not. And I've known you long enough to know that you are not likely to suspect this either. So what is it then? As Romain said, some of the choice have to do with the representation on both the R and C++ side -- for Rcpp itself we can be lightweight and efficient via proxy classes, but this does not mean we can do this for _any arbitrary C++ class_ coming from another project. As eg Armadillo. RcppArmadilo already does pretty well, and code review may make it better. We do not know of any fat to cut, or we'd cut it ourselves. We care about a few things, but performance is clearly among them. | 2. Regarding the quesiton of performance impact there are two situations | which should be distinguished: | | i. We call C++ from R and it does some processing and then returns and | we don't call it again. In that case its likely that copying or not won't | make a big difference or at least it won't if the actual C++ computation | time is large coimpared to the time spent in copying. | | ii. We factor out the inner loop of the code and only recode that in C++ | and repeatedly call it many times. In that case the copying is multiplied | by the number of iterations and might very well have a significant impact. In case ii) I'd try to use a different design and make it more like i): You generally do not want to call down from R to object code a bazillion times as there is always some overhead, and multiplying even something rather efficient by a veryBigNumber can make small times large in the aggregate. Dirk | | On Thu, Jul 11, 2013 at 6:55 PM, Dirk Eddelbuettel e...@debian.org wrote: | | Everybody has this existing example in their copy of Armadillo. | | I am running it here from SVN rather than the installed directory, but this | should not make a difference. Machine is my not-overly-powerful thinkpad used | for traveling: | | edd@don:~/svn/rcpp/pkg/RcppArmadillo/inst/examples$ r fastLm.r | Loading required package: methods | | Attaching package: ‘Rcpp’ | | The following object is masked from ‘package:inline’: | | registerPlugin | | test replications relative elapsed user.self sys.self | 2 fLmTwoCasts(X, y) 50001.000 0.184 0.2040.164 | 1 fLmOneCast(X, y) 50001.011 0.186 0.2000.172 | 4 fastLmPureDotCall(X, y) 50001.141 0.210 0.2360.184 | 3 fastLmPure(X, y) 50002.027 0.373 0.4120.332 | 6 lm.fit(X, y) 50002.685 0.494 0.5280.456 | 5 fastLm(frm, data = trees) 5000 36.380 6.694 7.3326.028 | 7 lm(frm, data = trees) 5000 42.734 7.863 8.6287.068 | edd@don:~/svn/rcpp/pkg/RcppArmadillo/inst/examples$ | | What we are talking about here is the difference between 'fLmTwoCasts' and | 'fLmOneCasts'. If you use larger objects, the different with be larger. But | the relative differences are tiny. | | It would be nice to make this more elegant, and I look forward to Romain's | proposals, but methinks that we may well have bigger fish to fry. | | Dirk, still in Sydney | | -- | Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com | ___ | Rcpp-devel mailing list | Rcpp-devel@lists.r-forge.r-project.org | https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel | | | | -- | Statistics Software Consulting | GKX Group, GKX Associates Inc. | tel: 1-877-GKX-GROUP | email: ggrothendieck at gmail.com -- Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
