I think the TOTAL_MAPS has the same meaning with the FINISHED_MAPS, which
represents the total number of map tasks successfully executed. Although there
is another metric named Launched_map_tasks, which is the sum of FINISHED_MAPS,
FAILED_MAPS and KILLED_MAPS.
And it is the same for Reduce tasks.
Cheer!
Hailong
2011-07-08
***********************************************
* Hailong Yang, PhD. Candidate
* Sino-German Joint Software Institute,
* School of Computer Science&Engineering, Beihang University
* Phone: (86-010)82315908
* Email: hailong.yang1...@gmail.com
* Address: G413, New Main Building in Beihang University,
* No.37 XueYuan Road,HaiDian District,
* Beijing,P.R.China,100191
***********************************************
发件人： sangroya
发送时间： 2011-07-07 23:50:45
收件人： hadoop-user
抄送：
主题： Re: Re: measure the time taken to complete map and reduce phase
Hi,
Thanks for the response!
I have the following queries regarding the Job History file.
I want to know if what the TOTAL_MAPS in the Job history represents.
Also, if FINISHED_MAPS represents the TOTAL_MAPS or the (TOTAL_MAPS -
FAILED_MAPS).
Does FINISHED_MAPS represents successfully executed maps.
I have the same question for REDUCE tasks.
Thanks,
Amit
On Thu, Jul 7, 2011 at 10:58 AM, Hailong [via Lucene]
<ml-node+3147566-1425016666-409...@n3.nabble.com> wrote:
> Hi sangroya,
>
> I think you may be interested in reading the following piece of code from
> JobHistory.java in Hadoop.
>
> /**
> * Generates the job history filename for a new job
> */
> private static String getNewJobHistoryFileName(JobConf jobConf, JobID
> id) {
> return JOBTRACKER_UNIQUE_STRING
> + id.toString() + "_" + getUserName(jobConf) + "_"
> + trimJobName(getJobName(jobConf));
> }
>
> /**
> * Trims the job-name if required
> */
> private static String trimJobName(String jobName) {
> if (jobName.length() > JOB_NAME_TRIM_LENGTH) {
> jobName = jobName.substring(0, JOB_NAME_TRIM_LENGTH);
> }
> return jobName;
> }
>
> Roughly speaking, the history file name is composed in the following way:
>
> hostname of JT + "_" + start time of JT + "_" + job id + "_" + user name +
> "_" + trimed job name
>
> Cheers!
>
> Hailong
>
> 2011-07-07
>
>
>
> ***********************************************
> * Hailong Yang, PhD. Candidate
> * Sino-German Joint Software Institute,
> * School of Computer Science&Engineering, Beihang University
> * Phone: (86-010)82315908
> * Email: [hidden email]
> * Address: G413, New Main Building in Beihang University,
> * No.37 XueYuan Road,HaiDian District,
> * Beijing,P.R.China,100191
> ***********************************************
>
>
>
> 发件人： sangroya
> 发送时间： 2011-07-07 15:49:58
> 收件人： hadoop-user
> 抄送：
> 主题： Re: measure the time taken to complete map and reduce phase
>
> Hi,
> Thanks!
> I am able to parse the Job History Logs(JHL). But, I need to know how
> hadoop assign a name to a file in Job History Logs(JHL).
> I can see that files are named on my local single node cluster as this:
> localhost_1309975809398_job_201107062010_0759_sangroya_word+count.
> But, I am just wondering, what is the exact pattern to name every file
> like this.
> Best Regards,
> Amit
> On Tue, Jul 5, 2011 at 6:53 AM, Hailong [via Lucene]
> <[hidden email]> wrote:
>> Hi sangroya,
>>
>> You can look at the job administration portal at port of 50030 on your
>> JobTracker such as '<a
>> href="<a href="http://localhost:50030'">http://localhost:50030'"><a
>> href="http://localhost:50030'">http://localhost:50030'. At the bottom of the
>> web page there is an item named 'Job Tracker History', click into it and
>> find you job with the job id. There goes the information you want.
>>
>>
>> Cheers!
>>
>> Hailong
>>
>> 2011-07-05
>>
>>
>>
>> ***********************************************
>> * Hailong Yang, PhD. Candidate
>> * Sino-German Joint Software Institute,
>> * School of Computer Science&Engineering, Beihang University
>> * Phone: (86-010)82315908
>> * Email: [hidden email]
>> * Address: G413, New Main Building in Beihang University,
>> * No.37 XueYuan Road,HaiDian District,
>> * Beijing,P.R.China,100191
>> ***********************************************
>>
>>
>>
>> 发件人： sangroya
>> 发送时间： 2011-07-05 10:56:38
>> 收件人： hadoop-user
>> 抄送：
>> 主题： measure the time taken to complete map and reduce phase
>>
>> Hi,
>> I am trying to monitor the time to complete a map phase and reduce
>> phase in hadoop. Is there any way to measure the time taken to
>> complete map and reduce phase in a cluster.
>> Thanks,
>> Amit
>> --
>> View this message in context:
>>
>> http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
>> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3139665.html
>> To unsubscribe from measure the time taken to complete map and reduce
>> phase,
>> click here.
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3147426.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3147566.html
> To unsubscribe from measure the time taken to complete map and reduce phase,
> click here.
--
View this message in context:
http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3148620.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
