[
https://issues.apache.org/jira/browse/MAPREDUCE-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13708269#comment-13708269
]
Hua xu commented on MAPREDUCE-5388:
-----------------------------------
We fix that:
{code}
private static class PathCleanupThread extends Thread {
// cleanup queue which deletes files/directories of the paths queued up.
private LinkedBlockingQueue<PathDeletionContext> queue =
new LinkedBlockingQueue<PathDeletionContext>();
public PathCleanupThread() {
setName("Directory/File cleanup thread");
setDaemon(true);
start();
}
void addToQueue(PathDeletionContext[] contexts) {
+ boolean interrupted = false;
for (PathDeletionContext context : contexts) {
try {
queue.put(context);
} catch(InterruptedException ie) {
+ interrupted = true;
}
}
+ if(interrupted){
+ Thread.currentThread().interrupt();
+ }
}
public void run() {
if (LOG.isDebugEnabled()) {
LOG.debug(getName() + " started.");
}
PathDeletionContext context = null;
while (true) {
try {
context = queue.take();
// delete the path.
if (!deletePath(context)) {
LOG.warn("CleanupThread:Unable to delete path " + context.fullPath);
}
else if (LOG.isDebugEnabled()) {
LOG.debug("DELETED " + context.fullPath);
}
} catch (InterruptedException t) {
LOG.warn("Interrupted deletion of " + context.fullPath);
return;
} catch (Exception e) {
LOG.warn("Error deleting path " + context.fullPath + ": " + e);
}
}
}
}
{code}
> There are two TaskLaunchers for map after the TaskTracker reinit
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-5388
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5388
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 0.21.0, 0.22.0, 1.0.3
> Reporter: Hua xu
>
> The TaskTracker was asked to reinit as the full gc,then the process contained
> of two TaskLauncher instance for map:
> {code}
> 2013-06-22 08:32:59,457 INFO org.apache.hadoop.mapred.TaskTracker: Process
> Thread Dump: lost task
> 36 active threads
> ......
> Thread 43 (TaskLauncher for REDUCE tasks):
> State: WAITING
> Blocked count: 73824
> Waited count: 72848
> Waiting on java.util.LinkedList@98a9557
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
>
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2157)
> Thread 42 (TaskLauncher for MAP tasks):
> State: WAITING
> Blocked count: 358814
> Waited count: 356105
> Waiting on org.apache.hadoop.ipc.Client$Call@6f764bc1
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.ipc.Client.call(Client.java:891)
>
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
> $Proxy5.getFileInfo(Unknown Source)
> sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> $Proxy5.getFileInfo(Unknown Source)
> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:850)
>
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:620)
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:167)
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1543)
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1524)
>
> org.apache.hadoop.mapred.TaskTracker.localizeJobTokenFile(TaskTracker.java:3995)
>
> org.apache.hadoop.mapred.TaskTracker.localizeJobFiles(TaskTracker.java:1036)
> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:977)
> ......
> Thread 1 (main):
> State: RUNNABLE
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
>
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:157)
>
> org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:211)
>
> org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks(TaskTracker.java:1792)
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1468)
> org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2329)
> org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3461)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
