[jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client

[Colin Patrick McCabe (JIRA)]

    [ 
https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235651#comment-15235651
 ] 

Colin Patrick McCabe commented on HDFS-3702:
--------------------------------------------

So the proposal is to add a new API to {{DistributedFileSystem}} that would 
support NO_LOCAL_WRITE.  Since we would want to continue to support all the 
existing options to create, it would look something like this:

{code}
  public FSDataOutputStream create(Path f,
      FsPermission permission,
      EnumSet<CreateFlag> flags,
      int bufferSize,
      short replication,
      long blockSize,
      Progressable progress,
      ChecksumOpt checksumOpt,
      boolean noLocalWrite) throws IOException {
{code}

That's 9 different parameters, which seems like a definite code smell.

Is it really worth adding this ugly API to avoid introducing a new 
{{CreateFlag}}?  Adding a new {{CreateFlag}} seems harmless to me.  If 
{{CreateFlag.NO_LOCAL_WRITE}} doesn't work out, we will stick a 
{{\@deprecated}} next to it and start ignoring it.  It's still better than 
adding a 9-argument function which requires a typecast to use.  We already have 
a lot of {{CreateFlags}} that are HDFS-specific such as {{LAZY_PERSIST}}, 
{{SYNC_BLOCK}}, and {{APPEND_NEWBLOCK}}.

Also keep in mind that once we add this ugly 9-argument API to 
{{DistributedFileSystem}}, we won't be able to remove it.  In order for HBase 
to use it, it has to be public and part of an official Apache release of 
Hadoop.  By our own API policies, it becomes permanent at that point.  Why is 
HDFS so unkind to our downstream projects?

> Add an option for NOT writing the blocks locally if there is a datanode on 
> the same box as the client
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3702
>                 URL: https://issues.apache.org/jira/browse/HDFS-3702
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 2.5.1
>            Reporter: Nicolas Liochon
>            Assignee: Lei (Eddy) Xu
>            Priority: Minor
>              Labels: BB2015-05-TBR
>         Attachments: HDFS-3702.000.patch, HDFS-3702.001.patch, 
> HDFS-3702.002.patch, HDFS-3702.003.patch, HDFS-3702.004.patch, 
> HDFS-3702.005.patch, HDFS-3702.006.patch, HDFS-3702.007.patch, 
> HDFS-3702.008.patch, HDFS-3702.009.patch, HDFS-3702.010.patch, 
> HDFS-3702.011.patch, HDFS-3702_Design.pdf
>
>
> This is useful for Write-Ahead-Logs: these files are writen for recovery 
> only, and are not read when there are no failures.
> Taking HBase as an example, these files will be read only if the process that 
> wrote them (the 'HBase regionserver') dies. This will likely come from a 
> hardware failure, hence the corresponding datanode will be dead as well. So 
> we're writing 3 replicas, but in reality only 2 of them are really useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)