clusterExport works just fine if you put your Rcpp code into your own package and make that package available in your worker environment. Given the need for compilation in possibly a variety of computing environments for parallel processing this is definitely recommended.
On May 14, 2021 10:35:25 AM PDT, Naeem Khoshnevis <khoshnevis.na...@gmail.com> wrote: >Hi Michael, > > >Thank you so much for your response. That is correct. One method for >exporting required variables/functions is using the clusterExport >function, > which does not work for Rcpp-based functions. Another option is using >clusterEvalQ (as mentioned in the shared post); however, I am not sure >if >CRAN likes to see the library(package name) inside the codebase. What >are >your thoughts? > >Best regards, >Naeem > >On Fri, May 14, 2021 at 11:57 AM Michael Weylandt < >michael.weyla...@gmail.com> wrote: > >> Hi Naeem, >> >> My (very quick) guess is that this isn't an Rcpp problem per se, but >a >> parLapply problem. You need to explicitly load your package on each >> worker so that functions from it are available. >> >> See, e.g., the brief discussion here: >> >> >https://stackoverflow.com/questions/18357788/parallel-parlapply-setup#18358875 >> >> The "parallel" packages do not exactly replicate your environment on >> each worker node (to avoid expensive set-up / communication costs) so >> you need to do a bit more set-up. >> >> Best, >> Michael >> >> On Fri, May 14, 2021 at 11:49 AM Naeem Khoshnevis >> <khoshnevis.na...@gmail.com> wrote: >> > >> > Dear Rcpp developers: >> > >> > Thanks for developing and maintaining the Rcpp package. >> > I wrote a function in Rcpp. It is available throughout the package >and >> works as expected; however, it is not available for praLapply >workers. A >> temporary fix is just using Rcpp::cppFunction inside the function >that >> parLapply workers call and copy the entire function over there. >However, >> this does not seem right for bigger and more complicated functions. >> > I would be grateful if you could let me know whether there is a >better >> long-term solution. Here is the package and three functions that you >might >> want to take a look at. >> > >> > Original cpp function: >> > >> >https://github.com/fasrc/CausalGPS/blob/master/src/compute_closest_wgps_helper.cpp >> > >> > Wrapper function that calls this function + temporal fix: >> > >https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R >> > >> > The function that uses parLapply (please see line 63-89) to run the >c++ >> code: >> > >https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R >> > >> > Best regards, >> > Naeem >> > _______________________________________________ >> > Rcpp-devel mailing list >> > Rcpp-devel@lists.r-forge.r-project.org >> > >https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel >> -- Sent from my phone. Please excuse my brevity. _______________________________________________ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
