If you write your CSV -> XML processing as a function, you could pmap ( http://clojure.org/api#pmap) that function across the list of input files. pmap will transparently create the threads as needed, and it will probably be enough to saturate your disk.
Thanks, Stu On Mon, Mar 16, 2009 at 5:56 PM, Brian Doyle <brianpdo...@gmail.com> wrote: > I've been using Clojure for about 6 months now and really like it. I am > somewhat new to multi-threading > and using any of the parallel features in Clojure though. I have a > situation where I need to convert > 7 files from CSV to XML. Each one of these files is about 180MB apiece in > size. I have dual core machine > with 2GB of RAM and would like some advice on the best strategy for > processing these files in a way that > really utilizes both cores and my memory to really speed up the > processing. I'm sure this isn't the best > way, but I've only come up with starting up two threads at first, having > each thread open up a file, > call line-seq on that file, write out the XML for each line and then go to > the next file when it's complete. Any > advice would be great. Thanks. > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~----------~----~----~----~------~----~------~--~---
