[
https://issues.apache.org/jira/browse/HDFS-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John George updated HDFS-2081:
------------------------------
Description:
While trying to run yarn with 1024 block size, it looks as though the
dfsclients are hung waiting to write to DN. The reason is because yarn holds
the configuration file lock and tries to write to hdfs which in turn tries to
read from the configuration which requires the same lock.
The temporary fix is to make sure that dfsclient threads dont read the config
in its write path. There might be better long term solutions like read/write
locks and this needs to be investigated in another JIRA.
was:
While trying to run yarn with 1024 block size, it looks as though the
dfsclients are hung waiting to write to DN. The reason is because yarn golds
the configuration file lock and tries to write to hdfs which in turn tries to
read from the configuration which requires the same lock.
The temporary fix is to make sure that dfsclient threads dont read the config
in its write path. There might be better long term solutions like read/write
locks and this needs to be investigated in another JIRA.
Affects Version/s: 0.23.0
Fix Version/s: 0.23.0
> trying to read configuration in hdfs write path might result in deadlock
> ------------------------------------------------------------------------
>
> Key: HDFS-2081
> URL: https://issues.apache.org/jira/browse/HDFS-2081
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.23.0
> Environment: MR-279 branch. This is specific to yarn since yarn tries
> to write the configuration file to hdfs while holding the lock of the
> configuration file.
> Reporter: John George
> Assignee: John George
> Fix For: 0.23.0
>
> Attachments: HDFS-2081.patch
>
>
> While trying to run yarn with 1024 block size, it looks as though the
> dfsclients are hung waiting to write to DN. The reason is because yarn holds
> the configuration file lock and tries to write to hdfs which in turn tries to
> read from the configuration which requires the same lock.
> The temporary fix is to make sure that dfsclient threads dont read the config
> in its write path. There might be better long term solutions like read/write
> locks and this needs to be investigated in another JIRA.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
