[
https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205337#comment-15205337
]
stack commented on HDFS-3702:
-----------------------------
bq. How would DFSClient know which nodes are disfavored nodes? How could it
enforce disfavored nodes?
You postulated an application that wanted to '....distribute its files
uniformly in a cluster.' I was just trying to suggest that users would prefer
that HDFS would just do it for them. HDFS would know how to do it better being
the arbiter of what is happening in the cluster. An application will do a poor
job compared. 'distribute its files uniformly...' sounds like a good feature to
implement with a block placement policy.
bq. Since we already have favoredNodes, adding disfavoredNodes seems more
natural than adding a flag.
As noted above at 'stack added a comment - 12/Mar/16 15:20', favoredNodes is an
unexercised feature that has actually been disavowed by the originators of the
idea, FB, because it proved broken in practice. I'd suggest we not build more
atop a feature-under-review as adding disfavoredNodes would (or at least until
we hear of successful use of favoredNodes -- apparently our Y! are trying it).
bq. In addition, the new FileSystem CreateFlag does not look clean to me since
it is too specific to HDFS. How would other FileSystems such as LocalFileSystem
implement it?
The flag added by the attached patch is qualified throughout as a 'hint'. When
set against LFS, it'll just be ignored. No harm done. The 'hint' didn't take.
If we went your suggested route and added a disfavoredNodes route, things get a
bit interesting when hbase, say, passes localhost. What'll happen? Does the
user now have to check the FS implementation type before they select DFSClient
method to call?
I don't think you are objecting to the passing of flags on create, given this
seems pretty standard fare in FSs.
> Add an option for NOT writing the blocks locally if there is a datanode on
> the same box as the client
> -----------------------------------------------------------------------------------------------------
>
> Key: HDFS-3702
> URL: https://issues.apache.org/jira/browse/HDFS-3702
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Affects Versions: 2.5.1
> Reporter: Nicolas Liochon
> Assignee: Lei (Eddy) Xu
> Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HDFS-3702.000.patch, HDFS-3702.001.patch,
> HDFS-3702.002.patch, HDFS-3702.003.patch, HDFS-3702.004.patch,
> HDFS-3702.005.patch, HDFS-3702.006.patch, HDFS-3702.007.patch,
> HDFS-3702.008.patch, HDFS-3702_Design.pdf
>
>
> This is useful for Write-Ahead-Logs: these files are writen for recovery
> only, and are not read when there are no failures.
> Taking HBase as an example, these files will be read only if the process that
> wrote them (the 'HBase regionserver') dies. This will likely come from a
> hardware failure, hence the corresponding datanode will be dead as well. So
> we're writing 3 replicas, but in reality only 2 of them are really useful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
