Is this related to JIRA HDFS-378?
On Wed, Feb 26, 2014 at 11:54 AM, Jack Levin <[email protected]> wrote: > Submitted JIRA patch: https://issues.apache.org/jira/browse/HDFS-6022 > (with test) > > On Mon, Feb 24, 2014 at 12:16 PM, Jack Levin <[email protected]> wrote: >> I will do that. >> >> -Jack >> >> On Mon, Feb 24, 2014 at 6:23 AM, Steve Loughran <[email protected]> >> wrote: >>> that's a very old version of cloudera's branch you are working with there; >>> patching that is not a good way to go, as you are on the slippery slope of >>> having your own private branch and all the costs of it. >>> >>> It looks like dead node logic has -> DFSInputStream, where it is still >>> instance-specific: >>> >>> /* XXX Use of CocurrentHashMap is temp fix. Need to fix >>> * parallel accesses to DFSInputStream (through ptreads) properly */ >>> private final ConcurrentHashMap<DatanodeInfo, DatanodeInfo> deadNodes = >>> new ConcurrentHashMap<DatanodeInfo, DatanodeInfo>(); >>> >>> This implies the problem still exists -and the opportunity to fix it -but >>> you will need to modify your patch to apply to hadoop trunk, ideally think >>> of a test, then submit a patch to the HDFS project on JIRA. >>> >>> >>> On 19 February 2014 04:48, Stack <[email protected]> wrote: >>> >>>> On Sat, Feb 15, 2014 at 8:01 PM, Jack Levin <[email protected]> wrote: >>>> >>>> > Looks like I patched it in DFSClient.java, here is the patch: >>>> > https://gist.github.com/anonymous/9028934 >>>> > >>>> > .... >>>> >>>> >>>> > I moved 'deadNodes' list outside as global field that is accessible by >>>> > all running threads, so at any point datanode does go down, each >>>> > thread is basically informed that the datanode _is_ down. >>>> > >>>> >>>> We need to add something like this to current versions of DFSClient, a >>>> global status, so each stream does not have to discover bad DNs for itself. >>>> St.Ack >>>> >>> >>> -- >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity to >>> which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You.
