Re: Loading a huge graph

2012-04-13 Thread László Török
Thanks, having a C++ background, there is definitely a lot to learn about the JVM. I'm wrote a script in Python that ran in about 10-12s and used up to 320MB of memory. I'm running a clojure repl from Emacs via jack-in. What's the best way to adjust the heap size? swank-clojure and

Re: Loading a huge graph

2012-04-13 Thread bOR_
I have this option in my project.clj file, which does the trick if you are developing from emacs+swank+clojure-jack-in, and using large networks :jvm-opts [-Xmx4000m] And yes, one of the things to do when working with the jvm is learning how to use jconsole or visualvm to see why your

Re: Loading a huge graph

2012-04-13 Thread László Török
Excellent! Thank you, that did the trick! 2012/4/13 bOR_ boris.sch...@gmail.com I have this option in my project.clj file, which does the trick if you are developing from emacs+swank+clojure-jack-in, and using large networks :jvm-opts [-Xmx4000m] And yes, one of the things to do when

Re: Loading a huge graph

2012-04-13 Thread Robert Marianski
If the jvm does have enough memory, you may want to try building up the map using a transient. And not sure if this is faster, (maybe it's slower), but you can spell the function you pass to reduce more succinctly: (fn [G [v1 v2]] (update-in G [v1] (fnil conj []) v2)) Robert On Thu, Apr 12,

Re: Loading a huge graph

2012-04-13 Thread László Török
Thanks, I know about transients, but I'm already using mutable arrays for speed :-) On Apr 13, 2012 8:01 PM, Robert Marianski r...@marianski.com wrote: If the jvm does have enough memory, you may want to try building up the map using a transient. And not sure if this is faster, (maybe it's

Loading a huge graph

2012-04-12 Thread László Török
Hi, I'm trying figure out how to load a huge file that contains some 800k pair of integers (two integers per line) which represent edges of a directed graph. So if the ith line has x and y, it means that there is an edge between x and y vertex in the graph. The goal is to load it in an array of

Re: Loading a huge graph

2012-04-12 Thread David Nolen
How much memory do Python Go consume when you do this? Are you giving the JVM enough memory? On Thu, Apr 12, 2012 at 6:17 PM, László Török ltoro...@gmail.com wrote: Hi, I'm trying figure out how to load a huge file that contains some 800k pair of integers (two integers per line) which

Re: Loading a huge graph

2012-04-12 Thread Alex Robbins
Yeah, sounds like it could definitely be a memory issue. This is one part where the JVM works a lot differently than I expected coming from a python background. Everybody may already know this, but the JVM only takes 64mb for the heap by default. You'll get an out of memory error if your program