Hank: I have looked at TC in the past, and took another look today at your suggestion. Terracotta certainly seems to have promise feature-wise, but I have to admit it's a "heavier" solution than I had been thinking of, and there are probably all sorts of gotchas (and reviewing old threads on the topic, a few are hinted at) when it comes to distributing some of Clojure's concurrency data structures in the shared-memory approach.
Kevin: I hadn't heard of Hadoop before, but at first glance it's exactly what I'm looking for. (Thanks!) The problem I am working on is a processing pipeline for massive data sets, and that seems to be the advertised use case for Hadoop. On Jan 29, 7:14 pm, Kevin Downey <redc...@gmail.com> wrote: > have you looked at the available java frameworks like hadoop? there is > also some kind of java interface to erlang > instead of reinventing the wheel again... > > > > On Thu, Jan 29, 2009 at 6:15 AM, Greg Harman <ghar...@gmail.com> wrote: > > > One of Clojure's big selling points (obviously) is the support for > > concurrent programming. However, the performance gains you get with > > this concurrency hits a scalability wall once you're fully utilizing > > all the cores available in your server. The next step, of course, is > > to add additional servers. > > > So, I've been mulling over the idea of putting together a framework > > for distributed applications. I think it should deal with issues such > > as: > > > * Locating, assessing, and registering CPUs > > * Division of large jobs into smaller sub-jobs > > * Dispatching of jobs, including optimization planning (send more jobs > > to faster CPUs) > > * Failure recovery > > * Seamless code integration (so that this can be retrofit as an > > optimization) both "horizontally" and "vertically" > > * Horizontal = take one time-consuming task and parallelize it > > * Vertical = take two sequential tasks and pipeline them across > > different servers > > > Is anybody already working on something similar? Is this already on > > the Clojure language roadmap as something to be added after 1.0? > > > Any design advice or feature requests if I were to put something like > > this together? > > > -Greg > > -- > And what is good, Phaedrus, > And what is not good— > Need we ask anyone to tell us these things? --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~----------~----~----~----~------~----~------~--~---
