this doesn't quite make sense: "Since Clojure is well-known for its concurrency feature, running in the same JVM should be out of question."
All the concurrency features built in to clojure are concerned with things that happen in the same process, unless you consider things like 'making it easier to use queues' to be those kinds of features that affect multi-process stuff. A concurrency focus doesn't say anything about philosophy of processes/threads or anything like that. MapReduce is dictating the process model here which goes against the grain of what java's good at doing, with the assumption that you'll be working with a *lot* of data and won't care. Something that takes 21 seconds isn't optimal use of mapreduce. Try it with something that takes a few minutes at least. However, you can help startup time in a number of ways, AOT compilation can help a bit, as well as judicious use of third-party code and keeping class dependencies low. On Fri, Apr 26, 2013 at 12:02 PM, Ji Zhang <zhangj...@gmail.com> wrote: > Hi, > > I believe I can confirm that it's the startup time issue. > > I set mapred.job.reuse.jvm.num.tasks=-1 and the overall time is > approximate to the pure java one. The best mapper task is the same time, > 3sec. Before it is 10sec. The lowest task's difference is still 12sec > (21sec - 9 sec). > > Since Clojure is well-known for its concurrency feature, running in the > same JVM should be out of question. > > Besides, my company's hadoop team seems to cut the files into too small > pieces, which result in too many mapper tasks. So the reuse is crucial for > clojure. > > > On Friday, April 26, 2013 6:05:33 PM UTC+8, Ji Zhang wrote: >> >> Hi, >> >> I'm writing map-reduce job with Clojure, yet to find that it seems to be >> much slower than a Jave job. >> >> So I write a simple test case, and upload to gist: >> https://gist.github.com/**jizhang/5466149<https://gist.github.com/jizhang/5466149> >> >> At the end of code, there is execution outputs, here are some significant >> stats: >> >> Average time taken by Map tasks: Java 7sec, Clojure 19sec >> CPU time spent (ms): Java 244,000, Clojure 1,145,440 >> >> I'm wondering what slows down the Clojure written map-reduce job. Am I >> using it wrong, or it's just an inappropriate senario. >> >> Any thoughts will be great. Thanks! >> >> Jerry >> > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
