hi edward,
i fail to see why the new hadoop.enxt and yarn can do the same. i read
some of the available documentation and it looks that hadoop.next could
be used as a more general resource manager instead of eg like torque,
but that's not what HOD does.
stijn
It has been in a quasi-defunct state for a while now. It seems like
hadoop.next and yarn, helps archive a similar effect of hod. Plus it
has this new hotness factor.
On Fri, Mar 9, 2012 at 2:41 AM, Stijn De Weirdt<stijn.dewei...@ugent.be> wrote:
(my apologies for those who have received this already. i posted this mail a
few days back on the common-dev list, as this is more a development related
mail; but one of the original authors/maintainers suggested to also post
this here)
hi all,
i am a system administrator/user support person/... for the HPC team at
Ghent University (Ghent, Flanders, Belgium).
recently we have been asked to look into support for hadoop. for the moment
we are holding off on a dedicated cluster (esp dedicated hdfs setup).
but as all our systems are torque/pbs based, we looked into HOD to help out
our users.
we have started from the HOD code that was part of the hadoop 1.0.0 release
(in the contrib part).
at first it was not working, but we have been patching and cleaning up the
code for a a few weeks and now have a version that works for us (we had to
add some features besides fixing a few things).
it looks sufficient for now, although we will add some more features soon to
get the users started.
my question is the following: what is the state of HOD atm? is it still
maintained/supported? are there forks somewhere that have more up-to-date
code?
what we are now missing most is the documentation (eg
http://hadoop.apache.org/common/docs/r0.16.4/hod.html) so we can update this
with our extra features. is the source available somewhere?
i could contribute back all patches, but a few of them are identation fixes
(to use 4 space indentation throughout the code) and other cosmetic changes,
so this messes up patches a lot.
i have also shuffled a bit with the options (rename and/or move to other
sections) so no 100% backwards compatibility with the current HOD code.
current main improvements:
- works with python 2.5 and up (we have been testing with 2.7.2)
- set options through environment variables
- better default values (we can now run with empty hodrc file)
- support for mail and nodes:ppn for pbs
- no deprecation warnings from hadoop (nearly finished)
- host-mask to bind xrs addr on non-default ip (in case you have
non-standard network on the compute nodes)
- more debug statements
- gradual code cleanup (using pylint)
on the todo list:
- further tuning of hadoop parameters (i'm not a hadoop user myself, so this
will take some time)
- 0.23.X support
many thanks,
stijn
