Thank you so much, Jeff. The part that I do not understand is the "and make that package available in your worker environment" part. Could you please let me know how I can make the package available for each worker.
On Fri, May 14, 2021 at 1:44 PM Jeff Newmiller <jdnew...@dcn.davis.ca.us> wrote: > clusterExport works just fine if you put your Rcpp code into your own > package and make that package available in your worker environment. Given > the need for compilation in possibly a variety of computing environments > for parallel processing this is definitely recommended. > > On May 14, 2021 10:35:25 AM PDT, Naeem Khoshnevis < > khoshnevis.na...@gmail.com> wrote: > >Hi Michael, > > > > > >Thank you so much for your response. That is correct. One method for > >exporting required variables/functions is using the clusterExport > >function, > > which does not work for Rcpp-based functions. Another option is using > >clusterEvalQ (as mentioned in the shared post); however, I am not sure > >if > >CRAN likes to see the library(package name) inside the codebase. What > >are > >your thoughts? > > > >Best regards, > >Naeem > > > >On Fri, May 14, 2021 at 11:57 AM Michael Weylandt < > >michael.weyla...@gmail.com> wrote: > > > >> Hi Naeem, > >> > >> My (very quick) guess is that this isn't an Rcpp problem per se, but > >a > >> parLapply problem. You need to explicitly load your package on each > >> worker so that functions from it are available. > >> > >> See, e.g., the brief discussion here: > >> > >> > > > https://stackoverflow.com/questions/18357788/parallel-parlapply-setup#18358875 > >> > >> The "parallel" packages do not exactly replicate your environment on > >> each worker node (to avoid expensive set-up / communication costs) so > >> you need to do a bit more set-up. > >> > >> Best, > >> Michael > >> > >> On Fri, May 14, 2021 at 11:49 AM Naeem Khoshnevis > >> <khoshnevis.na...@gmail.com> wrote: > >> > > >> > Dear Rcpp developers: > >> > > >> > Thanks for developing and maintaining the Rcpp package. > >> > I wrote a function in Rcpp. It is available throughout the package > >and > >> works as expected; however, it is not available for praLapply > >workers. A > >> temporary fix is just using Rcpp::cppFunction inside the function > >that > >> parLapply workers call and copy the entire function over there. > >However, > >> this does not seem right for bigger and more complicated functions. > >> > I would be grateful if you could let me know whether there is a > >better > >> long-term solution. Here is the package and three functions that you > >might > >> want to take a look at. > >> > > >> > Original cpp function: > >> > > >> > > > https://github.com/fasrc/CausalGPS/blob/master/src/compute_closest_wgps_helper.cpp > >> > > >> > Wrapper function that calls this function + temporal fix: > >> > > >https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R > >> > > >> > The function that uses parLapply (please see line 63-89) to run the > >c++ > >> code: > >> > > >https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R > >> > > >> > Best regards, > >> > Naeem > >> > _______________________________________________ > >> > Rcpp-devel mailing list > >> > Rcpp-devel@lists.r-forge.r-project.org > >> > > >https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel > >> > > -- > Sent from my phone. Please excuse my brevity. >
_______________________________________________ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
