Sorry, but what's the answer from Aaron? It's not in this mail.
And, I think that the cleanup task has nothing to do with HDFS. As far
as I can see, the hadoop contains a setup and a clean up task for map
and reduce tasks. I don't understand why, but it has. Here's the part
of the code that I'm talking about:
[code]
// cleanup map tip. This map doesn't use any splits. Just assign an empty
// split.
JobClient.RawSplit emptySplit = new JobClient.RawSplit();
cleanup[0] = new TaskInProgress(jobId, jobFile, emptySplit,
jobtracker, conf, this, numMapTasks);
cleanup[0].setJobCleanupTask();
// cleanup reduce tip.
cleanup[1] = new TaskInProgress(jobId, jobFile, numMapTasks,
numReduceTasks, jobtracker, conf, this);
cleanup[1].setJobCleanupTask();
// create two setup tips, one map and one reduce.
setup = new TaskInProgress[2];
// setup map tip. This map doesn't use any split. Just assign an empty
// split.
setup[0] = new TaskInProgress(jobId, jobFile, emptySplit,
jobtracker, conf, this, numMapTasks + 1 );
setup[0].setJobSetupTask();
// setup reduce tip.
setup[1] = new TaskInProgress(jobId, jobFile, numMapTasks,
numReduceTasks + 1, jobtracker, conf, this);
setup[1].setJobSetupTask();
[/code]
If anyone could explain also what's the purpose of these tasks I would
be very appreciated?
Thanks,
On Thu, Jun 9, 2011 at 9:05 AM, Laurent Hatier <laurent.hat...@gmail.com> wrote:
> Oh i don't see that it was in the HDFS. Aaron has answered i think
>
> 2011/6/9 Laurent Hatier <laurent.hat...@gmail.com>
>>
>> Have you try to restart your hadoop node ? (or all hadoop node). When you
>> go to restart, the namenode go to format the directory that you have choosen
>>
>> 2011/6/8 Pedro Costa <psdc1...@gmail.com>
>>>
>>> Hi,
>>>
>>> After I run the command "bin/hadoop job -history /temp/history/", I've
>>> got these 2 task summary. In one of them, it run a cleanup task and in
>>> the other hasn't run the cleanup task. This means that a cleanup task
>>> doesn't run always. So, when a cleanup task should run?
>>>
>>> Task Summary
>>> ============================
>>> Kind Total Successful Failed Killed StartTime
>>> FinishTime
>>>
>>> Setup 1 1 0 0 8-Jun-2011 15:54:50
>>> 8-Jun-2011 15:54:51 (0sec)
>>> Map 600 600 0 0 8-Jun-2011 15:54:54
>>> 8-Jun-2011 15:57:05 (2mins, 11sec)
>>> Reduce 252 250 0 2 8-Jun-2011 15:55:06
>>> 8-Jun-2011 15:57:48 (2mins, 42sec)
>>> Cleanup 0 0 0 0
>>> ============================
>>>
>>>
>>>
>>> Task Summary
>>> ============================
>>> Kind Total Successful Failed Killed StartTime
>>> FinishTime
>>>
>>> Setup 1 1 0 0 8-Jun-2011 16:04:19
>>> 8-Jun-2011 16:04:20 (1sec)
>>> Map 500 383 0 117 8-Jun-2011
>>> 16:04:40
>>> Reduce 0 0 0 0
>>> Cleanup 1 1 0 0 8-Jun-2011 16:04:37
>>> 8-Jun-2011 16:04:38 (1sec)
>>> ============================
>>>
>>> Thanks,
>>
>>
>>
>> --
>> Laurent HATIER
>> Étudiant en 2e année du Cycle Ingénieur à l'EISTI
>
>
>
> --
> Laurent HATIER
> Étudiant en 2e année du Cycle Ingénieur à l'EISTI
>
