Yeah, sounds like it could definitely be a memory issue. This is one
part where the JVM works a lot differently than I expected coming from
a python background.
Everybody may already know this, but the JVM only takes 64mb for the
heap by default. You'll get an out of memory error if your program
uses more than that. In contrast, python just takes all the memory it
needs. As your program gets closer to the JVM memory limit it'll spend
more and more time doing garbage collection, with less and less real
work getting done. You can pass an -Xmx flag to give java access to
more memory, which many (most?) programs do.
On Thu, Apr 12, 2012 at 5:22 PM, David Nolen <dnolen.li...@gmail.com> wrote:
> How much memory do Python & Go consume when you do this? Are you giving the
> JVM enough memory?
>
>
> On Thu, Apr 12, 2012 at 6:17 PM, László Török <ltoro...@gmail.com> wrote:
>>
>> Hi,
>>
>> I'm trying figure out how to load a huge file that contains some 800k pair
>> of integers (two integers per line) which represent edges of a directed
>> graph.
>>
>> So if the ith line has x and y, it means that there is an edge between x
>> and y vertex in the graph.
>>
>> The goal is to load it in an array of arrays representation, where the kth
>> array contains all the nodes, where there is a directed edge from the kth
>> node to those nodes.
>>
>> I've attempted multiple variants of with-open reader and line-seq etc. but
>> almost always ended up with OutMemoryException or sg VERY slow.
>>
>> My latest attempt that also does not work on the large input:
>>
>> (defn load-graph [input-f]
>> (with-open [rdr (io/reader input-f)]
>> (->> (line-seq rdr)
>> (map (fn [row]
>> (let [[v1str v2str] (str/split row #"\s")]
>> [ (Integer/parseInt v1str) (Integer/parseInt v2str) ]))
>> )
>> (reduce (fn [G [v1 v2]]
>> (if-let [vs (get G v1)]
>> (update-in G [v1] #(conj % v2))
>> (assoc G v1 [v2]))) { } ))))
>>
>> I'm getting a bit frustrated as there are Python, Go implementations that
>> load the graph in less the 5 seconds.
>>
>> What am I doing wrong?
>>
>> Thanks
>>
>> --
>> László Török
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
