[
https://issues.apache.org/jira/browse/HADOOP-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558712#action_12558712
]
eric baldeschwieler commented on HADOOP-2178:
---------------------------------------------
The user vs admin dimension is not really the same as the static vs HOD
dimension.
Even in a static JT, the job history is probably usefully part of the output
directory (for a user). I think this should just be part of the output API, no
matter how the cluster is configured. This will be much easier to document and
use.
Then we could handle static job trackers as Amareshwari describes. In the case
of HOD deployed JTs, I think we can then either set hadoop.job.history.location
to NULL or to a HOD specified output directory, probably on HDFS or another
shared FS. This would be useful if the user is running a lot jobs through a
single HOD instance. NULL is probably a fine default.
---
I agree with runping that we need to define an API for collecting central stats
from HOD deployed JTs. I think a configured URL is ok as an API, but we need
to be clear that this output will be for central collection, not user
diagnostics and as such the layout should be optimized to simplify that
(probably time sorted, not user sorted for example). Ideally this could be a
single file per JT instance.
> Job history on HDFS
> -------------------
>
> Key: HADOOP-2178
> URL: https://issues.apache.org/jira/browse/HADOOP-2178
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Amareshwari Sri Ramadasu
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.0
>
>
> This issue addresses the following items :
> 1. Check for accuracy of job tracker history logs.
> 2. After completion of the job, copy the JobHistory.log(Master index file)
> and the job history files to the DFS.
> 3. User can load the history with commands
> bin/hadoop job -history <directory>
> or
> bin/hadoop job -history <jobid>
> This will start a stand-alone jetty and load jsps
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
