Re: [Rd] LinkingTo and C++
While linking to package shared libs is not possible in general, as Simon point out, it is possible under Windows, provided Windows knows how to find the library linked to at runtime (this requires a customized Makefile.win). One way this is done under Windows is simply to place the package/libs directories containing the package shared libs to be linked to in the Windows search path, but this may be a problem for packages released to CRAN since this would require updates to CRAN's PATH environment variable. Another possibility is to load all packages containing shared libs to be linked to before using any shared lib that is dynamically linked to them. For example, if B.dll is dynamically linked to A.dll (and both A and B are packages), and if foo() is a function in B.dll that uses functions in A.dll, then this will work: library(A) library(B) .Call('foo') But this is not a very natural or convenient solution. It requires that library() commands be used with every instance of Rscript, for example. A better solution would be some variant of LinkingTo: A that somehow has the same effect as setting Windows search path to include the libs directory containing A.dll and B.dll. Of course, most of the issues disappear when linking to static libs instead of dynamic ones, and it is not clear that the extra effort needed to support dynamic libs will yield much benefit in this case. Dominick On Thu, Feb 11, 2010 at 12:55 PM, Simon Urbanek wrote: > Romain, > > I think your'e confusing two entirely different concepts here: > > 1) LinkingTo: allows a package to provide C-level functions to other > packages (see R-ext 5.4). Let's say package A provides a function foo by > calling R_RegisterCCallable for that function. If a package B wants to use > that function, it uses LinkingTo: and calls R_GetCCallable to obtain the > function pointer. It does not actually link to package A because that is in > general not possible - it simply obtains the pointers through R. In > addition, LinkingTo: makes sure that you have access to the header files of > package A which help you to cast the functions and define any data > structures you may need. Since C++ is a superset of C you can use this > facility with C++ as long as you don't depend on anything outside of the > header files. > > 2) linking directly to another package's shared object is in general not > possible, because packages are not guaranteed to be dynamic libraries. They > are usually shared objects which may or may not be compatible with a dynamic > library on a given platform. Therefore the R-ext describes other way in > which you may provide some library independently of the package shared > object to other packages (see R-ext 5.8). The issue is that you have to > create a separate library (PKG/libs[/arch]/PKG.so won't work in general!) > and provide this to other packages. As 5.8 says, this is in general not > trivial because it is very platform dependent and the most portable way is > to offer a static library. > > To come back to your example, LinkingTo: A and B will work if you remove > Makevars from B (you don't want to link) and put your hello method into the > A.h header: > > > library (B) > Loading required package: A > > .Call("say_hello", PACKAGE = "B") > [1] "hello" > > However, your'e not really using the LinkingTo: facilities for the > functions so it's essentially just helping you to find the header file. > > Cheers, > Simon > > > > On Feb 11, 2010, at 4:08 AM, Romain Francois wrote: > > > Hello, > > > > I've been trying to make LinkingTo work when the package linked to has > c++ code. > > > > I've put dumb packages to illustrate this emails here ; > http://addictedtor.free.fr/misc/linkingto > > > > Package A defines this C++ class: > > > > class A { > > public: > > A() ; > > ~A() ; > > SEXP hello() ; > > } ; > > > > Package B has this function : > > > > SEXP say_hello(){ > > A a ; > > return a.hello() ; > > } > > > > headers of package A are copied into inst/include so that package B can > have. > > > > LinkingTo: A > > > > in its DESCRIPTION file. > > > > Also, package B has the R function ; > > > > g <- function(){ > > .Call("say_hello", PACKAGE = "B") > > } > > > > With this I can compile A and B, but then I get : > > > > $ Rscript -e "B::g()" > > Error in dyn.load(file, DLLpath = DLLpath, ...) : > > unable to load shared library '/usr/local/lib/R/library/B/libs/B.so': > > /usr/local/lib/R/library/B/libs/B.so: undefined symbol: _ZN1AD1Ev > > Calls: :: ... tryCatch -> tryCatchList -> tryCatchOne -> > > > > If I then add a Makevars in B with this : > > > > > > # find the root directory where A is installed > > ADIR=$(shell $(R_HOME)/bin/Rscript -e "cat(system.file(package='A'))" ) > > > > PKG_LIBS= $(ADIR)/libs/A$(DYLIB_EXT) > > > > > > Then it works: > > > > $ Rscript -e "B::g()" > > [1] "hello" > > > > So it appears that adding the -I flag, which is what LinkingTo does is > not enough when the package "
Re: [Rd] LinkingTo and C++
On 02/11/2010 07:40 PM, Simon Urbanek wrote: On Feb 11, 2010, at 12:24 PM, Romain Francois wrote: Thanks. On 02/11/2010 05:55 PM, Simon Urbanek wrote: Romain, I think your'e confusing two entirely different concepts here: Yes. The name "LinkingTo" probably helped my confusion. Admittedly, it's probably not the best name ;). 1) LinkingTo: allows a package to provide C-level functions to other packages (see R-ext 5.4). Let's say package A provides a function foo by calling R_RegisterCCallable for that function. If a package B wants to use that function, it uses LinkingTo: and calls R_GetCCallable to obtain the function pointer. It does not actually link to package A because that is in general not possible - it simply obtains the pointers through R. In addition, LinkingTo: makes sure that you have access to the header files of package A which help you to cast the functions and define any data structures you may need. Since C++ is a superset of C you can use this facility with C++ as long as you don't depend on anything outside of the header files. 2) linking directly to another package's shared object is in general not possible, because packages are not guaranteed to be dynamic libraries. They are usually shared objects which may or may not be compatible with a dynamic library on a given platform. Therefore the R-ext describes other way in which you may provide some library independently of the package shared object to other packages (see R-ext 5.8). The issue is that you have to create a separate library (PKG/libs[/arch]/PKG.so won't work in general!) and provide this to other packages. As 5.8 says, this is in general not trivial because it is very platform dependent and the most portable way is to offer a static library. To come back to your example, LinkingTo: A and B will work if you remove Makevars from B (you don't want to link)and put your hello method into the A.h header: Sure. but in real life I can't realistically put everything in the header files. It was just an example based on your example ;) - which was not very realistic, either. It practice it is reasonable, because it is sufficient to declare in the headers whatever you're providing so the only homework is to cast function pointers you have obtained via R_GetCCallable to the declarations from the header file. I suspect what you meant is not as much related to LinkingTo: (since the mess C++ creates at the binary level is rather hard to pass through dl pointers - but if someone has a working solution it may be worth to create a package), but rather to provide a library. That is not covered by R at this point so you're in realm of R-ext 5.8. Given how non-trivial task this is (to get it right) it may be worthwhile thinking about a portable solution and add it to R, but I don't think anyone has done that yet (mainly due to the low benefit/cost ratio I suspect). For all cases so far it was sufficient to create C or R level API for other package to use. Cheers, Simon Yes. The goal is to provide a library that other packages can just use. I thought LinkingTo would help, but now I guess not. or maybe just so that B can find headers of A. The R_RegisterCCallable/R_GetCCallable business seems to be only applicable when you are developping both A and B and you deal with plain functions, otherwise the name mangling, overloading of methods, etc would make the task hard and not fun at all. So I guess, yes the question is how to reliably and portably provide a library for package A so that package B can just use it. This would be very valuable. Romain -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |- http://tr.im/NrTG : Rcpp 0.7.5 |- http://tr.im/MPYc : RProtoBuf: protocol buffers for R `- http://tr.im/KfKn : Rcpp 0.7.2 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] LinkingTo and C++
On Feb 11, 2010, at 12:24 PM, Romain Francois wrote: > Thanks. > > On 02/11/2010 05:55 PM, Simon Urbanek wrote: >> Romain, >> >> I think your'e confusing two entirely different concepts here: > > Yes. The name "LinkingTo" probably helped my confusion. > Admittedly, it's probably not the best name ;). >> 1) LinkingTo: allows a package to provide C-level functions to other >> packages (see R-ext 5.4). Let's say package A provides a function foo by >> calling R_RegisterCCallable for that function. If a package B wants to use >> that function, it uses LinkingTo: and calls R_GetCCallable to obtain the >> function pointer. It does not actually link to package A because that is in >> general not possible - it simply obtains the pointers through R. In >> addition, LinkingTo: makes sure that you have access to the header files of >> package A which help you to cast the functions and define any data >> structures you may need. Since C++ is a superset of C you can use this >> facility with C++ as long as you don't depend on anything outside of the >> header files. >> >> 2) linking directly to another package's shared object is in general not >> possible, because packages are not guaranteed to be dynamic libraries. They >> are usually shared objects which may or may not be compatible with a dynamic >> library on a given platform. Therefore the R-ext describes other way in >> which you may provide some library independently of the package shared >> object to other packages (see R-ext 5.8). The issue is that you have to >> create a separate library (PKG/libs[/arch]/PKG.so won't work in general!) >> and provide this to other packages. As 5.8 says, this is in general not >> trivial because it is very platform dependent and the most portable way is >> to offer a static library. >> >> To come back to your example, LinkingTo: A and B will work if you remove >> Makevars from B (you don't want to link)and put your hello method into the >> A.h header: > > Sure. but in real life I can't realistically put everything in the header > files. > It was just an example based on your example ;) - which was not very realistic, either. It practice it is reasonable, because it is sufficient to declare in the headers whatever you're providing so the only homework is to cast function pointers you have obtained via R_GetCCallable to the declarations from the header file. I suspect what you meant is not as much related to LinkingTo: (since the mess C++ creates at the binary level is rather hard to pass through dl pointers - but if someone has a working solution it may be worth to create a package), but rather to provide a library. That is not covered by R at this point so you're in realm of R-ext 5.8. Given how non-trivial task this is (to get it right) it may be worthwhile thinking about a portable solution and add it to R, but I don't think anyone has done that yet (mainly due to the low benefit/cost ratio I suspect). For all cases so far it was sufficient to create C or R level API for other package to use. Cheers, Simon > > >>> library (B) >> Loading required package: A >>> .Call("say_hello", PACKAGE = "B") >> [1] "hello" >> >> However, your'e not really using the LinkingTo: facilities for the functions >> so it's essentially just helping you to find the header file. >> >> Cheers, >> Simon >> >> >> >> On Feb 11, 2010, at 4:08 AM, Romain Francois wrote: >> >>> Hello, >>> >>> I've been trying to make LinkingTo work when the package linked to has c++ >>> code. >>> >>> I've put dumb packages to illustrate this emails here ; >>> http://addictedtor.free.fr/misc/linkingto >>> >>> Package A defines this C++ class: >>> >>> class A { >>> public: >>> A() ; >>> ~A() ; >>> SEXP hello() ; >>> } ; >>> >>> Package B has this function : >>> >>> SEXP say_hello(){ >>> A a ; >>> return a.hello() ; >>> } >>> >>> headers of package A are copied into inst/include so that package B can >>> have. >>> >>> LinkingTo: A >>> >>> in its DESCRIPTION file. >>> >>> Also, package B has the R function ; >>> >>> g<- function(){ >>> .Call("say_hello", PACKAGE = "B") >>> } >>> >>> With this I can compile A and B, but then I get : >>> >>> $ Rscript -e "B::g()" >>> Error in dyn.load(file, DLLpath = DLLpath, ...) : >>> unable to load shared library '/usr/local/lib/R/library/B/libs/B.so': >>> /usr/local/lib/R/library/B/libs/B.so: undefined symbol: _ZN1AD1Ev >>> Calls: :: ... tryCatch -> tryCatchList -> tryCatchOne -> >>> >>> If I then add a Makevars in B with this : >>> >>> >>> # find the root directory where A is installed >>> ADIR=$(shell $(R_HOME)/bin/Rscript -e "cat(system.file(package='A'))" ) >>> >>> PKG_LIBS= $(ADIR)/libs/A$(DYLIB_EXT) >>> >>> >>> Then it works: >>> >>> $ Rscript -e "B::g()" >>> [1] "hello" >>> >>> So it appears that adding the -I flag, which is what LinkingTo does is not >>> enough when the package "linking from" (B) actually has to link to
Re: [Rd] LinkingTo and C++
Thanks. On 02/11/2010 05:55 PM, Simon Urbanek wrote: Romain, I think your'e confusing two entirely different concepts here: Yes. The name "LinkingTo" probably helped my confusion. 1) LinkingTo: allows a package to provide C-level functions to other packages (see R-ext 5.4). Let's say package A provides a function foo by calling R_RegisterCCallable for that function. If a package B wants to use that function, it uses LinkingTo: and calls R_GetCCallable to obtain the function pointer. It does not actually link to package A because that is in general not possible - it simply obtains the pointers through R. In addition, LinkingTo: makes sure that you have access to the header files of package A which help you to cast the functions and define any data structures you may need. Since C++ is a superset of C you can use this facility with C++ as long as you don't depend on anything outside of the header files. 2) linking directly to another package's shared object is in general not possible, because packages are not guaranteed to be dynamic libraries. They are usually shared objects which may or may not be compatible with a dynamic library on a given platform. Therefore the R-ext describes other way in which you may provide some library independently of the package shared object to other packages (see R-ext 5.8). The issue is that you have to create a separate library (PKG/libs[/arch]/PKG.so won't work in general!) and provide this to other packages. As 5.8 says, this is in general not trivial because it is very platform dependent and the most portable way is to offer a static library. To come back to your example, LinkingTo: A and B will work if you remove Makevars from B (you don't want to link)and put your hello method into the A.h header: Sure. but in real life I can't realistically put everything in the header files. Thanks again. Romain library (B) Loading required package: A .Call("say_hello", PACKAGE = "B") [1] "hello" However, your'e not really using the LinkingTo: facilities for the functions so it's essentially just helping you to find the header file. Cheers, Simon On Feb 11, 2010, at 4:08 AM, Romain Francois wrote: Hello, I've been trying to make LinkingTo work when the package linked to has c++ code. I've put dumb packages to illustrate this emails here ; http://addictedtor.free.fr/misc/linkingto Package A defines this C++ class: class A { public: A() ; ~A() ; SEXP hello() ; } ; Package B has this function : SEXP say_hello(){ A a ; return a.hello() ; } headers of package A are copied into inst/include so that package B can have. LinkingTo: A in its DESCRIPTION file. Also, package B has the R function ; g<- function(){ .Call("say_hello", PACKAGE = "B") } With this I can compile A and B, but then I get : $ Rscript -e "B::g()" Error in dyn.load(file, DLLpath = DLLpath, ...) : unable to load shared library '/usr/local/lib/R/library/B/libs/B.so': /usr/local/lib/R/library/B/libs/B.so: undefined symbol: _ZN1AD1Ev Calls: :: ... tryCatch -> tryCatchList -> tryCatchOne -> If I then add a Makevars in B with this : # find the root directory where A is installed ADIR=$(shell $(R_HOME)/bin/Rscript -e "cat(system.file(package='A'))" ) PKG_LIBS= $(ADIR)/libs/A$(DYLIB_EXT) Then it works: $ Rscript -e "B::g()" [1] "hello" So it appears that adding the -I flag, which is what LinkingTo does is not enough when the package "linking from" (B) actually has to link to the "linked to" package (A). I've been looking at http://cran.r-project.org/doc/manuals/R-exts.html#Registering-native-routines but it seems only applicable to c(++) functions and not classes ... What am I missing ? Should/can linkingto be extended in a way that accomodates c++ Romain -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |- http://tr.im/NrTG : Rcpp 0.7.5 |- http://tr.im/MPYc : RProtoBuf: protocol buffers for R `- http://tr.im/KfKn : Rcpp 0.7.2 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] LinkingTo and C++
Romain, I think your'e confusing two entirely different concepts here: 1) LinkingTo: allows a package to provide C-level functions to other packages (see R-ext 5.4). Let's say package A provides a function foo by calling R_RegisterCCallable for that function. If a package B wants to use that function, it uses LinkingTo: and calls R_GetCCallable to obtain the function pointer. It does not actually link to package A because that is in general not possible - it simply obtains the pointers through R. In addition, LinkingTo: makes sure that you have access to the header files of package A which help you to cast the functions and define any data structures you may need. Since C++ is a superset of C you can use this facility with C++ as long as you don't depend on anything outside of the header files. 2) linking directly to another package's shared object is in general not possible, because packages are not guaranteed to be dynamic libraries. They are usually shared objects which may or may not be compatible with a dynamic library on a given platform. Therefore the R-ext describes other way in which you may provide some library independently of the package shared object to other packages (see R-ext 5.8). The issue is that you have to create a separate library (PKG/libs[/arch]/PKG.so won't work in general!) and provide this to other packages. As 5.8 says, this is in general not trivial because it is very platform dependent and the most portable way is to offer a static library. To come back to your example, LinkingTo: A and B will work if you remove Makevars from B (you don't want to link) and put your hello method into the A.h header: > library (B) Loading required package: A > .Call("say_hello", PACKAGE = "B") [1] "hello" However, your'e not really using the LinkingTo: facilities for the functions so it's essentially just helping you to find the header file. Cheers, Simon On Feb 11, 2010, at 4:08 AM, Romain Francois wrote: > Hello, > > I've been trying to make LinkingTo work when the package linked to has c++ > code. > > I've put dumb packages to illustrate this emails here ; > http://addictedtor.free.fr/misc/linkingto > > Package A defines this C++ class: > > class A { > public: > A() ; > ~A() ; > SEXP hello() ; > } ; > > Package B has this function : > > SEXP say_hello(){ > A a ; > return a.hello() ; > } > > headers of package A are copied into inst/include so that package B can have. > > LinkingTo: A > > in its DESCRIPTION file. > > Also, package B has the R function ; > > g <- function(){ > .Call("say_hello", PACKAGE = "B") > } > > With this I can compile A and B, but then I get : > > $ Rscript -e "B::g()" > Error in dyn.load(file, DLLpath = DLLpath, ...) : > unable to load shared library '/usr/local/lib/R/library/B/libs/B.so': > /usr/local/lib/R/library/B/libs/B.so: undefined symbol: _ZN1AD1Ev > Calls: :: ... tryCatch -> tryCatchList -> tryCatchOne -> > > If I then add a Makevars in B with this : > > > # find the root directory where A is installed > ADIR=$(shell $(R_HOME)/bin/Rscript -e "cat(system.file(package='A'))" ) > > PKG_LIBS= $(ADIR)/libs/A$(DYLIB_EXT) > > > Then it works: > > $ Rscript -e "B::g()" > [1] "hello" > > So it appears that adding the -I flag, which is what LinkingTo does is not > enough when the package "linking from" (B) actually has to link to the > "linked to" package (A). > > I've been looking at > http://cran.r-project.org/doc/manuals/R-exts.html#Registering-native-routines > but it seems only applicable to c(++) functions and not classes ... > > What am I missing ? Should/can linkingto be extended in a way that > accomodates c++ > > Romain > > > -- > Romain Francois > Professional R Enthusiast > +33(0) 6 28 91 30 30 > http://romainfrancois.blog.free.fr > |- http://tr.im/NrTG : Rcpp 0.7.5 > |- http://tr.im/MPYc : RProtoBuf: protocol buffers for R > `- http://tr.im/KfKn : Rcpp 0.7.2 > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel > > __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] LinkingTo and C++
On 02/11/2010 10:08 AM, Romain Francois wrote: Hello, I've been trying to make LinkingTo work when the package linked to has c++ code. I've put dumb packages to illustrate this emails here ; http://addictedtor.free.fr/misc/linkingto Package A defines this C++ class: class A { public: A() ; ~A() ; SEXP hello() ; } ; Package B has this function : SEXP say_hello(){ A a ; return a.hello() ; } headers of package A are copied into inst/include so that package B can have. LinkingTo: A in its DESCRIPTION file. Also, package B has the R function ; g <- function(){ .Call("say_hello", PACKAGE = "B") } With this I can compile A and B, but then I get : $ Rscript -e "B::g()" Error in dyn.load(file, DLLpath = DLLpath, ...) : unable to load shared library '/usr/local/lib/R/library/B/libs/B.so': /usr/local/lib/R/library/B/libs/B.so: undefined symbol: _ZN1AD1Ev Calls: :: ... tryCatch -> tryCatchList -> tryCatchOne -> If I then add a Makevars in B with this : # find the root directory where A is installed ADIR=$(shell $(R_HOME)/bin/Rscript -e "cat(system.file(package='A'))" ) PKG_LIBS= $(ADIR)/libs/A$(DYLIB_EXT) Then it works: $ Rscript -e "B::g()" [1] "hello" So it appears that adding the -I flag, which is what LinkingTo does is not enough when the package "linking from" (B) actually has to link to the "linked to" package (A). I've been looking at http://cran.r-project.org/doc/manuals/R-exts.html#Registering-native-routines but it seems only applicable to c(++) functions and not classes ... What am I missing ? Should/can linkingto be extended in a way that accomodates c++ Romain One other way of course would be to have some lib support, so that for example an R library holds not only R packages but shared libraries. So for example, if as part of package A's Makevars, I copy its A.so into R.home( component = "lib" ) and rename it libA.so : RLIBDIR=$(shell $(R_HOME)/bin/Rscript -e "cat(R.home(component='lib'))" ) all: $(SHLIB) install install: cp $(SHLIB) $(RLIBDIR)/lib$(SHLIB) cp -f A.h ../inst/include Then B can just have this Makevars : PKG_LIBS=-lA as well as the "LinkingTo: A" in the description. Now I realize that R.home(component='lib') is not the right place where to host libA.so, as one might not have rights, etc ... but should there be a right place, as a per-R-library lib folder ? Romain -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |- http://tr.im/NrTG : Rcpp 0.7.5 |- http://tr.im/MPYc : RProtoBuf: protocol buffers for R `- http://tr.im/KfKn : Rcpp 0.7.2 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] LinkingTo and C++
Hello, I've been trying to make LinkingTo work when the package linked to has c++ code. I've put dumb packages to illustrate this emails here ; http://addictedtor.free.fr/misc/linkingto Package A defines this C++ class: class A { public: A() ; ~A() ; SEXP hello() ; } ; Package B has this function : SEXP say_hello(){ A a ; return a.hello() ; } headers of package A are copied into inst/include so that package B can have. LinkingTo: A in its DESCRIPTION file. Also, package B has the R function ; g <- function(){ .Call("say_hello", PACKAGE = "B") } With this I can compile A and B, but then I get : $ Rscript -e "B::g()" Error in dyn.load(file, DLLpath = DLLpath, ...) : unable to load shared library '/usr/local/lib/R/library/B/libs/B.so': /usr/local/lib/R/library/B/libs/B.so: undefined symbol: _ZN1AD1Ev Calls: :: ... tryCatch -> tryCatchList -> tryCatchOne -> If I then add a Makevars in B with this : # find the root directory where A is installed ADIR=$(shell $(R_HOME)/bin/Rscript -e "cat(system.file(package='A'))" ) PKG_LIBS= $(ADIR)/libs/A$(DYLIB_EXT) Then it works: $ Rscript -e "B::g()" [1] "hello" So it appears that adding the -I flag, which is what LinkingTo does is not enough when the package "linking from" (B) actually has to link to the "linked to" package (A). I've been looking at http://cran.r-project.org/doc/manuals/R-exts.html#Registering-native-routines but it seems only applicable to c(++) functions and not classes ... What am I missing ? Should/can linkingto be extended in a way that accomodates c++ Romain -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |- http://tr.im/NrTG : Rcpp 0.7.5 |- http://tr.im/MPYc : RProtoBuf: protocol buffers for R `- http://tr.im/KfKn : Rcpp 0.7.2 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel