Re: [R] Accelerating binRead

Thanks Henrik, that's it. Fwiw I found this old post too, I am still surprised this doesn't seem to get used a lot(?). It's a "neat trick" for row-wise binary, without compiled code. http://cyclemumner.blogspot.com.au/2010/06/read-las-data-with-r.html?m=1 Also you should look at Paul Murrell's

Re: [R] Accelerating binRead

I second Mike's proposal - it works, e.g. https://github.com/HenrikBengtsson/affxparser/blob/5bf1a9162904c56d59c4735a8d7eb427e4f085e4/R/readCcg.R#L535-L583 Here's an outline. Say each row consists of tuple (=4-byte integer, =4-byte float, ss=2 byte integer) so that the byte-by-byte

Re: [R] Accelerating binRead

I would gladly examine your example, Mike. Cheers, Philippe > Le 18 sept. 2016 à 16:05, Michael Sumner a écrit : > > > >> On Sun, 18 Sep 2016, 19:04 Philippe de Rochambeau wrote: >> Please find below code that attempts to read ints, longs and floats from a

Re: [R] Accelerating binRead

On Sun, 18 Sep 2016, 19:04 Philippe de Rochambeau wrote: > Please find below code that attempts to read ints, longs and floats from a > binary file (which is a simplification of my original program). > Please disregard the R inefficiencies, such as using rbind, for now. > I’ve

Re: [R] Accelerating binRead

Please find below code that attempts to read ints, longs and floats from a binary file (which is a simplification of my original program). Please disregard the R inefficiencies, such as using rbind, for now. I’ve also included Java code to generate the binary file. The output shows that, at one

Re: [R] Accelerating binRead

The only difference between the below code and my program is that the former assumes that the file only contains one row of 10 ints + 10 floats , whereas my program doesn’t know in advance how many rows the file contains, unless it downloads it first and computes the potential number of rows

Re: [R] Accelerating binRead

Hi Jim, this is exactly the answer I was look for. Many thanks. I didn’t R had a pack function, as in PERL. To answer your earlier question, I am trying to update legacy code to read a binary file with unknown size, over a network, slice up it into rows each containing an integer, an integer, a

Re: [R] Accelerating binRead

Here is an example of how to do it: x <- 1:10 # integer values xf <- seq(1.0, 2, by = 0.1) # floating point setwd("d:/temp") # create file to write to output <- file('integer.bin', 'wb') writeBin(x, output) # write integer writeBin(xf, output) # write reals close(output) library(pack)

Re: [R] Accelerating binRead

I would also suggest that you take a look at the 'pack' package which can convert the binary input to the value you want. Part of your performance problems might be all the short reads that you are doing. Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me

Re: [R] Accelerating binRead

You should probably pick a forum — here or SO : http://stackoverflow.com/questions/39547398/faster-reading-of-binary-files-in-r : - vs cross-post to all of them. On Sat, Sep 17, 2016 at 11:04 AM, Ismail SEZEN wrote: > I noticed same issue but didnt care much :) > > On

Re: [R] Accelerating binRead

I noticed same issue but didnt care much :) On Sat, Sep 17, 2016, 18:01 jim holtman wrote: > Your example was not reproducible. Also how do you "break" out of the > "while" loop? > > > Jim Holtman > Data Munger Guru > > What is the problem that you are trying to solve? >

Re: [R] Accelerating binRead

Your example was not reproducible. Also how do you "break" out of the "while" loop? Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. On Sat, Sep 17, 2016 at 8:05 AM, Philippe de Rochambeau

Re: [R] Accelerating binRead

Appending to lists is only very slightly more efficient than incremental rbinding. Ideally you can figure out an upper bound for number of records, preallocate a data frame of that size, modify each element as you go in-place, and shrink the data frame once at the end as needed. If you cannot

Re: [R] Accelerating binRead

I suspect that rbind is responsible. Use list and append instead of rbind. At the end, combine elements of list by do.call(“rbind”, list). > On 17 Sep 2016, at 15:05, Philippe de Rochambeau wrote: > > Hello, > the following function, which stores numeric values extracted from a

[R] Accelerating binRead

Hello, the following function, which stores numeric values extracted from a binary file, into an R matrix, is very slow, especially when the said file is several MB in size. Should I rewrite the function in inline C or in C/C++ using Rcpp? If the latter case is true, how do you « readBin » in