How much memory do Python & Go consume when you do this? Are you giving the
JVM enough memory?
On Thu, Apr 12, 2012 at 6:17 PM, László Török <ltoro...@gmail.com> wrote:
> Hi,
>
> I'm trying figure out how to load a huge file that contains some 800k pair
> of integers (two integers per line) which represent edges of a directed
> graph.
>
> So if the ith line has x and y, it means that there is an edge between x
> and y vertex in the graph.
>
> The goal is to load it in an array of arrays representation, where the kth
> array contains all the nodes, where there is a directed edge from the kth
> node to those nodes.
>
> I've attempted multiple variants of with-open reader and line-seq etc. but
> almost always ended up with OutMemoryException or sg VERY slow.
>
> My latest attempt that also does not work on the large input:
>
> (defn load-graph [input-f]
> (with-open [rdr (io/reader input-f)]
> (->> (line-seq rdr)
> (map (fn [row]
> (let [[v1str v2str] (str/split row #"\s")]
> [ (Integer/parseInt v1str) (Integer/parseInt v2str) ]))
> )
> (reduce (fn [G [v1 v2]]
> (if-let [vs (get G v1)]
> (update-in G [v1] #(conj % v2))
> (assoc G v1 [v2]))) { } ))))
>
> I'm getting a bit frustrated as there are Python, Go implementations that
> load the graph in less the 5 seconds.
>
> What am I doing wrong?
>
> Thanks
>
> --
> László Török
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
