Thanks for the pointer Dirk (or in this case Rcpp::XPtr), I'll definitely take a look at this. Have to go back to the drawing board a bit as I'd currently been using a lot of built-in R functions so switching to bigmemory will be a bit non-trivial. I promise I'll make a demo if I get that far though :)
Thanks again for a great package and all the hard work! -Chuck On Wed, 13 Apr 2011 13:38:06 -0500, Dirk Eddelbuettel <e...@debian.org> wrote: > Hi Chuck, > > Thanks for posting here. > > On 13 April 2011 at 12:22, ch...@chuckcode.com wrote: > | Thanks Jay for the lightning fast insightful response! Didn't realize > that > | this limitation was intrinsic to R itself. I'll have to go look at > | bigmemory and some of the other projects. > > Yes, Rcpp uses 'proxy model' classes around the underlying SEXP objects. It > really is just R under it, and we inherit all its advantages and oddities. > Such as int32 indexing for vectors. > > The bigmemory package can be a way out, and as it interfaces R using > external > pointers, you should also be able to hook into this via the Rcpp::XPtr > class. > That would make for a great new demo or example. > > Feel like cooking one up while you work through this? ;-) > > Cheers, Dirk > > | > | Thanks, > | -Chuck > | > | On Wed, 13 Apr 2011 14:11:25 -0400, Jay Emerson <jayemer...@gmail.com> > | wrote: > | > Chuck, > | > > | > Internally, R is using 4-byte integers for indexing, and the length of > | > a vector is thus constrained to 2-billion-ish elements. Two ways > | > around this include packages ff and > | > bigmemory, for example, or relying on database-like queries. However, > | > the resulting objects can't be used with standard R functions (with > | > some special exceptions). > | > > | > Jay > | > > | > > | > On Wed, Apr 13, 2011 at 2:05 PM, <ch...@chuckcode.com> wrote: > | >> > | >> Hi All, > | >> > | >> Thanks to Dirk Eddelbuettel and the other contributors for such a > | >> wonderful package. Rcpp really transforms the way that I've been able > | to > | >> incorporate c++ code into R. Makes it possible to speed up the > critical > | >> computations while keeping all the great flexibility and features of > R. > | >> > | >> I've run into a problem lately with particularly large vectors in > Rcpp. > | I > | >> seem to be overflowing when my vectors get larger than 2^31 elements > on > | a > | >> 64 bit system. It looks from the code of both the classic (included > | >> below) > | >> and the newer versions as though this is due to using ints rather than > | >> something like size_t (http://en.wikipedia.org/wiki/Size_t) as the > type > | >> for > | >> size and indexing into the vector. It looks like RcppResultsSet's > add() > | >> functions may also be using ints. > | >> > | >> Curious to know if there is a particular reason for using ints rather > | >> than > | >> something like size_t and if the project managers would be open to > | >> changing > | >> to accomodate larger vectors. > | >> > | >> Thanks, > | >> -Chuck Sugnet > | >> > | >> template <typename T> > | >> class RcppVector { > | >> public: > | >> typedef T* iterator ; > | >> typedef const T* const_iterator ; > | >> > | >> RcppVector(SEXP vec); > | >> RcppVector(int len); > | >> int size() const; > | >> T& operator()(int i) const; > | >> T *cVector() const; > | >> std::vector<T> stlVector() const; > | >> > | >> inline const_iterator begin() const { return v ; } > | >> inline const_iterator end() const { return v + len ; } > | >> > | >> inline iterator begin(){ return v ; } > | >> inline iterator end(){ return v + len ; } > | >> > | >> private: > | >> int len; > | >> T *v; > | >> }; > | >> _______________________________________________ > | >> Rcpp-devel mailing list > | >> Rcpp-devel@lists.r-forge.r-project.org > | >> > https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel > | >> > | _______________________________________________ > | Rcpp-devel mailing list > | Rcpp-devel@lists.r-forge.r-project.org > | https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel _______________________________________________ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
