[HOD] Support PBS env vars in hod configuration
-----------------------------------------------
Key: HADOOP-2862
URL: https://issues.apache.org/jira/browse/HADOOP-2862
Project: Hadoop Core
Issue Type: Improvement
Components: contrib/hod
Affects Versions: 0.16.0
Environment: Torque PBS
Reporter: Craig Macdonald
In some batch environments, eg using Torque PBS, scratch spaces are provided on
cluster nodes for where jobs should put their temporary files. These are
automatically cleaned up when the job exists by an epilogue script.
For instance, in our local Torque cluster, all nodes have a /scratch partition.
For each job, the prologue script creates a scratch folder owned by the user at
/scratch/pbstmp.$PBS_JOBID - $PBS_JOBID is then the env var containing the job
id, as set by pbs_mom.
Would it be possible to use these env vars in the configuration of hod. For
instance, say I want to create an hdfs on demand using hod, but that the hdfs
space should be in /scratch/pbstmp.$PBS_JOBID, not in /tmp/hod say. This would
involve HOD supporting env vars in configuration, but knowing when to
substitute the env var with it's current value (ie not until running on the
correct node where the operation should take place).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
