[
https://issues.apache.org/jira/browse/MAPREDUCE-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-6473:
----------------------------------
Target Version/s: 2.8.0
Status: Open (was: Patch Available)
Thanks for the patch, Kuhu!
There's a race condition in the initialization of the provider list. A thread
can try to init while another thread is busy building the provider list. In
that case the second thread may try to walk the ArrayList while it is being
constructed since providerList.isEmpty() will be false yet the list isn't fully
constructed. This could cause a client to fail to find a provider that should
have been found and violates the concurrency safety of ArrayList since we're
reading it while it's being written. We could use a volatile that's
initialized to null when it hasn't been set. That way either the list isn't
there and we'll synchronize the threads that are trying to create it within the
null check block or the list will be there in its entirety and not modified
beyond that.
Nit: initProviderList is probably a better name since set implies the ability
to provide the value to set it to
> Job submission can take a long time during Cluster initialization
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-6473
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6473
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
> Attachments: 99%ile.png, MAPREDUCE-6473-PerfTest.txt,
> MAPREDUCE-6473-v1.patch, MAPREDUCE-6473-v2.patch, MAPREDUCE-6473-v3.patch,
> MAPREDUCE-6473-v4.patch, avgtime.png
>
>
> During initialization in Cluster.java, the framework provider classes are
> loaded inside a sync block which can considerably increase job submission
> time when the number of submissions are high. The motive is to reduce time
> spent in this sync block safely to improve performance.
> {noformat}
> synchronized (frameworkLoader) {
> for (ClientProtocolProvider provider : frameworkLoader) {
> LOG.debug("Trying ClientProtocolProvider : "
> + provider.getClass().getName());
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
