zhihai xu created YARN-4095:
-------------------------------
Summary: Avoid sharing AllocatorPerContext object in
LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService.
Key: YARN-4095
URL: https://issues.apache.org/jira/browse/YARN-4095
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Reporter: zhihai xu
Assignee: zhihai xu
Currently {{ShuffleHandler}} and {{LocalDirsHandlerService}} share
{{AllocatorPerContext}} object in {{LocalDirAllocator}} for configuration
{{NM_LOCAL_DIRS}} because {{AllocatorPerContext}}s are stored in a static
TreeMap with configuration name as key
{code}
private static Map <String, AllocatorPerContext> contexts =
new TreeMap<String, AllocatorPerContext>();
{code}
{{LocalDirsHandlerService}} and {{ShuffleHandler}} both create a
{{LocalDirAllocator}} using {{NM_LOCAL_DIRS}}. Even they don't use the same
{{Configuration}} object, but they will use the same {{AllocatorPerContext}}
object. Also {{LocalDirsHandlerService}} may change {{NM_LOCAL_DIRS}} value in
its {{Configuration}} object to exclude full and bad local dirs,
{{ShuffleHandler}} always uses the original {{NM_LOCAL_DIRS}} value in its
{{Configuration}} object. So every time {{AllocatorPerContext#confChanged}} is
called by {{ShuffleHandler}} after {{LocalDirsHandlerService}},
{{AllocatorPerContext}} need be reinitialized because {{NM_LOCAL_DIRS}} value
is changed. This will cause some overhead.
{code}
String newLocalDirs = conf.get(contextCfgItemName);
if (!newLocalDirs.equals(savedLocalDirs)) {
{code}
So it will be a good improvement to not share the same {{AllocatorPerContext}}
instance between {{ShuffleHandler}} and {{LocalDirsHandlerService}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
