You are my hero of the day Hemanth, these 3 things will make our life so
much simpler.
Not knowing torque at all and trying to deal with two systems that are
essentially unknown has been making the learning curve quite steep.
Hemanth Yamijala wrote:
Jason Venner wrote:
I have found that HOD writes a series of log files to directories on
the virtual cluster master, if you specify log directories.
The interesting part is figuring out which machine was the virtual
cluster master, if you have a decent sized pool of machines.
Can you explain what you mean by 'virtual cluster master' ? I guess
you could mean the node which ran the 'ringmaster' process - the
master process in a hod virtual cluster, or the hadoop 'jobtracker'.
Both this information is stored by hod for an allocated cluster. You
could retrieve it by using
hod -o list,
and
hod -o "info cluster-dir"
If the cluster is deallocated, you can get the ringmaster node still
by using torque commands:
qstat (to get torque job ids)
and
qstat -n <jobid>
This will print a list of nodes, the first one is the ringmaster.
It would be so nice if HOD could copy this back to the submission node.
I suppose we could configure syslog up but then we have to configure
the syslogd on each of the submit hosts to accept from any compute
node...
Syslog configuration has a bug which will be addressed in Hadoop 0.16.1.
Thanks
hemanth
--
Jason Venner
Attributor - Publish with Confidence <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers, contact if interested
