[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961864#comment-16961864
]
Istvan Fajth commented on HDFS-14882:
-------------------------------------
Yes, KeyInputStream is something else I was working on, and forgot there, thank
you for cleaning up.
The shuffle is there by default at the current code, the patch preserves that
behaviour when the readConsiderLoad is false, that is there to give some random
distribution to the DN read load already.
Patch 009 looks good to me, thank you for incorporating the suggestion and
provide a combined patch. I would suggest two more things to do, we might
deprecate the old sorter methods, as we most likely won't need them on the long
run, as their use effectively overrides the new setting, and an update would be
nice to the APIDoc of these methods.
Other than that if CI does not bring up any relevant issues, I support the
change, but the only thing I can give is a non-binding +1 :)
> Consider DataNode load when #getBlockLocation
> ---------------------------------------------
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Xiaoqiao He
> Assignee: Xiaoqiao He
> Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch,
> HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch,
> HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch,
> HDFS-14882.009.patch, HDFS-14882.suggestion
>
>
> Currently, we consider load of datanode when #chooseTarget for writer,
> however not consider it for reader. Thus, the process slot of datanode could
> be occupied by #BlockSender for reader, and disk/network will be busy
> workload, then meet some slow node exception. IIRC same case is reported
> times. Based on the fact, I propose to consider load for reader same as it
> did #chooseTarget for writer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
