[
https://issues.apache.org/jira/browse/HDFS-9612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15090835#comment-15090835
]
Zoran Dimitrijevic commented on HDFS-9612:
------------------------------------------
168:
try {
work = inputQueue.take();
} catch (InterruptedException e) {
LOG.debug("Interrupted while waiting for request from inputQueue.");
// if interrupt is triggered by shutdown(), terminate the thread
// otherwise, attempt to take again
Thread.currentThread().interrupt();
return;
}
I don't see how you distinguish between shutdown and other interrupts? How I
read this code, you actually stop the thread using return every time thread
gets any InterruptedException. Whether it is possible to get this exception for
any reason except calling shutdown, I am not 100% certain - that is why I
protected this thread (but apparently failed to realize shutdown is not able to
complete). What you want is to let thread execute shutdown() but continue
working if it is interrupted while waiting to take() from input queue.
Give me a day to find time to check how to do this, or explain me how this
return does not get out of run() method and finishes the thread. Or maybe it
does work if you simply remove "return" from your patch?
> DistCp worker threads are not terminated after jobs are done.
> -------------------------------------------------------------
>
> Key: HDFS-9612
> URL: https://issues.apache.org/jira/browse/HDFS-9612
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
> Affects Versions: 2.8.0
> Reporter: Wei-Chiu Chuang
> Assignee: Wei-Chiu Chuang
> Attachments: HDFS-9612.001.patch, HDFS-9612.002.patch,
> HDFS-9612.003.patch, HDFS-9612.004.patch, HDFS-9612.005.patch,
> HDFS-9612.006.patch
>
>
> In HADOOP-11827, a producer-consumer style thread pool was introduced to
> parallelize the task of listing files/directories.
> We have a use case where a distcp job is run during the commit phase of a MR2
> job. However, it was found distcp does not terminate ProducerConsumer thread
> pools properly. Because threads are not terminated, those MR2 jobs never
> finish.
> In a more typical use case where distcp is run as a standalone job, those
> threads are terminated forcefully when the java process is terminated. So
> these leaked threads did not become a problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
