Re: java.lang.OutOfMemoryError when slurping lots of files?

2013-04-08 Thread Adrian Muresan
Thank you very much, that worked splendidly. On Friday, April 5, 2013 5:14:30 PM UTC+2, Alex Nixon wrote: > > Java substrings prevent the original string from being garbage collected; > perhaps this also happens with regex matches? > > You can test the theory by surrounding the values in your ma

Re: java.lang.OutOfMemoryError when slurping lots of files?

2013-04-05 Thread Adrian Muresan
Alex Nixon managed to figure it out (further down) offtopic: I live in the same city as you and I'm also interested in clojure, email me if you want to have a coffee Adrian On Friday, April 5, 2013 5:08:04 PM UTC+2, Laurent PETIT wrote: > > You should show us the calling code, I guess ... > >

Re: java.lang.OutOfMemoryError when slurping lots of files?

2013-04-05 Thread Alex Nixon
Java substrings prevent the original string from being garbage collected; perhaps this also happens with regex matches? You can test the theory by surrounding the values in your map with (String. ) and seeing if the problem goes away. On 5 April 2013 15:57, Adrian Muresan wrote: > Hello everyo

Re: java.lang.OutOfMemoryError when slurping lots of files?

2013-04-05 Thread Alex Robbins
Slurp reads the entire file into memory. Maybe it is a combination of a) the program taking up more of the heap in other parts as it runs and then b) a particularly large file? Is there a reason you can't process the files as a line-seq so you don't have to load the entire thing into memory all at

Re: java.lang.OutOfMemoryError when slurping lots of files?

2013-04-05 Thread Laurent PETIT
You should show us the calling code, I guess ... 2013/4/5 Adrian Muresan : > Hello everyone, > > I'm trying to parse a large number of small reports for some data and I'm > doing this by repeatedly calling the following function (with a for) on each > of the files: > > (defn get-rep [file] > (

java.lang.OutOfMemoryError when slurping lots of files?

2013-04-05 Thread Adrian Muresan
Hello everyone, I'm trying to parse a large number of small reports for some data and I'm doing this by repeatedly calling the following function (with a for) on each of the files: (defn get-rep [file] (let [report (with-open [rdr (io/reader file)](*slurp rdr*)) reg #";\s+(\d+\.\d