[
https://issues.apache.org/jira/browse/MAPREDUCE-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated MAPREDUCE-1607:
-----------------------------------------------
Attachment: patch-1607-ydist.txt
Illustrative patch for Yahoo! distribution not for commit here.
Patch does the following :
1. It changes the directory structure of the task log directory from
$hadoop.log.dir/userlogs/jobid/taskid to
$hadoop.log.dir/userlogs/jobid/taskid[.cleanup]
2. Changes the index file name from log.index[.cleanup] to log.index
Did this change because log.index file need not have a suffix now, because it
is going into separate directory.
3. Changes the contents of index file for LOG_DIR from
LOG_DIR:<attemptid> to LOG_DIR:<actual log location>
This change is needed for the indexing into proper directory.
We can just add LOG_DIR:<attemptid>[.cleanup], but caller has to pass jobid
also to find the actual directory. I
think that adding the actual log path will avoid any other directory structure
change (like adding user) to make
changes in all the places where log location is used. One more use case for
doing this is distributing logs in all the disks, similar to "mapred.local.dir".
4. Patch adds a regression test
TestTaskTrackerLocalization.testCleanupTaskLocalization() which fails without
the patch and passes with the patch.
5. Patch adds a Junit test to test tasklogs and indices for all the tasks of
job with jvm-reuse. The existing test
TestTaskFail already tests logs of task attempts and cleanup attempts.
> Task controller may not set permissions for a task cleanup attempt's log
> directory
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1607
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1607
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task-controller
> Reporter: Hemanth Yamijala
> Assignee: Amareshwari Sriramadasu
> Attachments: patch-1607-ydist.txt
>
>
> Task controller uses the INITIALIZE_TASK command to initialize task attempt
> and task log directories. For cleanup tasks, task attempt directories are
> named as task-attempt-id.cleanup. But log directories do not have the
> .cleanup suffix. The task controller is not aware of this distinction and
> tries to set permissions for log directories named task-attempt-id.cleanup.
> This is a NO-OP. Typically the task cleanup runs on the same node that ran
> the original task attempt as well. So, the task log directories are already
> properly initialized. However, the task cleanup can run on a node that has
> not run the original task attempt. In that case, the initialization would not
> happen and this could result in the cleanup task failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
