Definitely don't recursively apply concat, it will end up with a stack overflow due the way the laziness of concat is implemented.
First and foremost, are you absolutely sure you need the full list realized in memory? Your first approach with *mapcat* could be the best option if you can make sure to process the list only once, without keeping a reference to the head of that collection. If you need to process the list many times, you can try using (into [] (mapcat ...)). This will build a vector, which has less overhead per element. But if each element allocates anything more than say 300 bytes, the overhead will not be your primary problem. You'll just need a bigger heap then. On Friday, February 1, 2013 8:19:17 AM UTC+1, bruce li wrote: > > Hello, everyone. I'm experience some performance issue when using clojure. > The scenario is as follows: > > I have a huge list of xls files to process. I used the > org.clojars.boechat107/cloxls to read the files which for each file > generates a list(approximately 65,000 elements). Now I need to concatenate > all of them. I used (mapcat read-worksheet files) to get the final list, > but it soon reports out of heap space. Then I tried to use mutable > structures: > > (doseq [f files] > (swap! sheet concat (read-worksheet f))) > > where sheet is defined as (def sheet (atom []))) > > But the concat seems to slow down a lot when the list grows larger. I'm > wondering if in clojure there is some efficient idiom to handle such > situation such as efficient concatenation? > > Thanks, > Bruce Li > > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
