[
https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554456#comment-13554456
]
Todd Lipcon commented on HDFS-4350:
-----------------------------------
Hey Andrew. I took another look at this and think it can be cleaned up a bit
more... having two separate booleans (and respective getters) for
isAvoidingStaleDataNodesForWrite vs shouldAvoidStaleDatanodesForWrite is pretty
messy.
What if we did the following?
- kill off the boolean for {{isAvoidingStaleDatanodesForWrite}}
- change heartbeatCheck to always check stale nodes. It seems like there is
practically zero cost to this. In the usual case where the heartbeat check does
not find a dead node, we already iterate over the whole list of nodes. Checking
isStale() is just a comparison, so essentially free.
- then DatanodeManager only needs to expose the current state:
{{shouldAvoidStaleDataNodesForWrite}}, which would be implemented something
like":
{code}
return avoidStaleDatanodesForWrite && numStaleNodes < datanodeMap.size() *
ratioUseStaleDatanodesForWrite
{code}
(or still keep a cached boolean inside DatanodeManager, but set inside
setNumStaleNodes)
That also allows you to move the ratio into DatanodeManager, which makes more
sense in my opinion. I think all these changes together could allow you to
remove the reference backward from HeartbeatManager into DatanodeManager - the
circular reference between the classes is the thing that smells bad to me.
> Make enabling of stale marking on read and write paths independent
> ------------------------------------------------------------------
>
> Key: HDFS-4350
> URL: https://issues.apache.org/jira/browse/HDFS-4350
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.0.0
> Reporter: Andrew Wang
> Assignee: Andrew Wang
> Attachments: hdfs-4350-1.patch, hdfs-4350-2.patch, hdfs-4350-3.patch,
> hdfs-4350-4.patch
>
>
> Marking of datanodes as stale for the read and write path was introduced in
> HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys,
> {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and
> {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently
> exists a dependency, since you cannot enable write marking without also
> enabling read marking, since the first key enables both checking of staleness
> and read marking.
> I propose renaming the first key to
> {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled
> if either of the keys are set. This will allow read and write marking to be
> enabled independently.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
