Re: Read in a big file and get GC limit/outofmemory error.

2012-04-11 Thread Andy Fingerhut
On Apr 9, 2012, at 10:05 PM, Andy Wu wrote: Hi there, I'm studying algo-class.org, and one of it's programming assignment gives you a file containing contents like below: 1 2 1 7 2 100 ... There is roughly over 5 million lines, and i want to first construct a vector of vector of

Re: Read in a big file and get GC limit/outofmemory error.

2012-04-11 Thread atucker
Yet another approach that might work for you, depending on your requirements, is to use a lazy sequence to access your data. I did that for a load of Twitter data that would have been too large to hold in memory at any one time. Here's the relevant bit (I think), copied and pasted: (defn

Read in a big file and get GC limit/outofmemory error.

2012-04-10 Thread Andy Wu
Hi there, I'm studying algo-class.org, and one of it's programming assignment gives you a file containing contents like below: 1 2 1 7 2 100 ... There is roughly over 5 million lines, and i want to first construct a vector of vector of integers for further process: [[1 2][1 7][2 100]...] Below

Re: Read in a big file and get GC limit/outofmemory error.

2012-04-10 Thread Sean Corfield
On Mon, Apr 9, 2012 at 10:05 PM, Andy Wu icetorto...@gmail.com wrote: (def int-vec (with-open [rdr (clojure.java.io/reader file name)]                           (doall (map convert (line-seq rdr) This will convert all 5 million lines to a 5 million element vector of vector pairs. That's