[
https://issues.apache.org/jira/browse/HADOOP-4664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679077#action_12679077
]
Jothi Padmanabhan commented on HADOOP-4664:
-------------------------------------------
True. There are dfs calls in JobHistory*. However, I do not foresee a deadlock,
these calls just might make it sequential in what otherwise could have been
achieved in parallel, no? What this patch does is not insulate from slow data
nodes, but just mitigates the effect.
On the different thread-safely items:
# JobHistory.openJobs - Agreed. This needs to be made into a concurrentHashMap
# dnsToSwitchMapping. The method in question is resolve. From what I can see,
this method is already thread safe in all its current implementations.
# JobEndNotifier.queue - This is already a delayedqueue.
# storeCompledJob() - CompletedJobStatusStore.store seems to be thread safe
already.
> Parallelize job initialization
> ------------------------------
>
> Key: HADOOP-4664
> URL: https://issues.apache.org/jira/browse/HADOOP-4664
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Matei Zaharia
> Assignee: Jothi Padmanabhan
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hadoop-4664-v1.patch, parallel-job-init-v1.patch
>
>
> The job init thread currently initializes one job at a time. However, this is
> a lengthy and partly IO-bound process because all of the job's block
> locations need to be resolved through the namenode and a map of them needs to
> be built. It can take tens of seconds. As a result, the cluster sometimes
> initializes jobs too slowly for full utilization to be achieved, if there are
> many small jobs queued up. It would be better to have a pool of threads that
> initialize multiple jobs in parallel. One thing to be careful of, however, is
> not causing deadlocks or holding locks for too long in these threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
