Trevor Strohman wrote:
Grid Engine: All the machines available to me run Sun's Grid Engine for
job submission. Grid Engine is important for us, because it makes sure
that all of the users of a cluster get their fair share of resources--as
far as I can tell, the JobTracker assumes that one user owns the
machines. Is this shared scenario you're interested in supporting?
Yes. We'd like Hadoop's MapReduce to be able to live on top of such
systems. Some are already experimenting with Hadoop on Condor, but I've
not yet heard of anyone using Hadoop on Sun's Grid engine.
http://issues.apache.org/jira/browse/HADOOP-428
http://www.cs.wisc.edu/condor/CondorWeek2006/presentations/paranjpye_yahoo_condor.ppt
Would you consider supporting job submission systems like Grid Engine or
Condor?
Definitely. I'm also interested in supporting Amazon's EC2, since it
removes the need of purchasing and maintaining a cluster. In
particular, Amazon's prices seem, for many applications, to be
considerably cheaper than operating one's own cluster.
Record I/O: [ ...]
and my TypeBuilder class generates code for all possible orderings of
this class (order by word, order by count, order by word then count,
order by count then word). Each ordering has its own hash function and
comparator.
In addition, each ordering has its own serialization/deserialization
code. For example, if we order by count, the serialization code stores
only differences between adjacent counts to help with compression.
Is this code you'd be interested in?
Yes, this sounds very interesting. Does it build on the Record IO
classes or is it completely separate?
Thanks,
Doug
