I was also contemplating EC2 in regards to Hadoop. One of the issues I was thinking of was, assuming you are dynamically allocating and deallocating hosts, would you need to be careful of how fast you released hosts? Is there currently any graceful way of letting Hadoop deal with removing a host? I.E letting any mapreduce tasks finish and moving chunks to another box.
Lee On 10/11/06, Doug Cutting <[EMAIL PROTECTED]> wrote:
Trevor Strohman wrote: > Grid Engine: All the machines available to me run Sun's Grid Engine for > job submission. Grid Engine is important for us, because it makes sure > that all of the users of a cluster get their fair share of resources--as > far as I can tell, the JobTracker assumes that one user owns the > machines. Is this shared scenario you're interested in supporting? Yes. We'd like Hadoop's MapReduce to be able to live on top of such systems. Some are already experimenting with Hadoop on Condor, but I've not yet heard of anyone using Hadoop on Sun's Grid engine. http://issues.apache.org/jira/browse/HADOOP-428 http://www.cs.wisc.edu/condor/CondorWeek2006/presentations/paranjpye_yahoo_condor.ppt > Would you consider supporting job submission systems like Grid Engine or > Condor? Definitely. I'm also interested in supporting Amazon's EC2, since it removes the need of purchasing and maintaining a cluster. In particular, Amazon's prices seem, for many applications, to be considerably cheaper than operating one's own cluster. > Record I/O: [ ...] > and my TypeBuilder class generates code for all possible orderings of > this class (order by word, order by count, order by word then count, > order by count then word). Each ordering has its own hash function and > comparator. > > In addition, each ordering has its own serialization/deserialization > code. For example, if we order by count, the serialization code stores > only differences between adjacent counts to help with compression. > > Is this code you'd be interested in? Yes, this sounds very interesting. Does it build on the Record IO classes or is it completely separate? Thanks, Doug
