[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HDFS-3703:
-------------------------
Attachment: 3703-hadoop-1.0.txt
Attaching first patch for hadoop 1.0
I don't see how heart beat can be temporarily disabled for DataNode, so I
commented out related calls.
I currently got:
{code}
Testcase: testReadSelectNonStaleDatanode took 0.879 sec
Caused an ERROR
Directory /home/hduser/1-hadoop/build/test/data/dfs/name1 is in an inconsistent
state: storage directory does not exist or is not accessible.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory
/home/hduser/1-hadoop/build/test/data/dfs/name1 is in an inconsistent state:
storage directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:304)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:104)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:410)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:378)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:277)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:529)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1403)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:278)
{code}
> Decrease the datanode failure detection time
> --------------------------------------------
>
> Key: HDFS-3703
> URL: https://issues.apache.org/jira/browse/HDFS-3703
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, name-node
> Affects Versions: 1.0.3, 2.0.0-alpha
> Reporter: nkeywal
> Assignee: Jing Zhao
> Fix For: 1.0.4
>
> Attachments: 3703-hadoop-1.0.txt, HDFS-3703-branch2.patch,
> HDFS-3703.patch, HDFS-3703-trunk-read-only.patch,
> HDFS-3703-trunk-read-only.patch, HDFS-3703-trunk-read-only.patch,
> HDFS-3703-trunk-read-only.patch, HDFS-3703-trunk-read-only.patch,
> HDFS-3703-trunk-read-only.patch, HDFS-3703-trunk-with-write.patch
>
>
> By default, if a box dies, the datanode will be marked as dead by the
> namenode after 10:30 minutes. In the meantime, this datanode will still be
> proposed by the nanenode to write blocks or to read replicas. It happens as
> well if the datanode crashes: there is no shutdown hooks to tell the nanemode
> we're not there anymore.
> It especially an issue with HBase. HBase regionserver timeout for production
> is often 30s. So with these configs, when a box dies HBase starts to recover
> after 30s and, while 10 minutes, the namenode will consider the blocks on the
> same box as available. Beyond the write errors, this will trigger a lot of
> missed reads:
> - during the recovery, HBase needs to read the blocks used on the dead box
> (the ones in the 'HBase Write-Ahead-Log')
> - after the recovery, reading these data blocks (the 'HBase region') will
> fail 33% of the time with the default number of replica, slowering the data
> access, especially when the errors are socket timeout (i.e. around 60s most
> of the time).
> Globally, it would be ideal if HDFS settings could be under HBase settings.
> As a side note, HBase relies on ZooKeeper to detect regionservers issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
