[
https://issues.apache.org/jira/browse/HDFS-15022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17296707#comment-17296707
]
Yang Yun commented on HDFS-15022:
---------------------------------
Thanks [~ayushtkn] for the review.
Yes, when he datanode goes down, and it was using a bucket and now I want that
bucket to be used by another datanode, I only copy the path and bucket
infomation to target datanode and the target datanode trigger Block Report.In
this process, there is no real data transmission, so the speed is very fast.
But this function was very dependent on the implementation of the underlying
storage, so I add a external shell script. In our setup, we use fuse to mount
remote storage. The shell script just add file info in target node linked to
the remote data (It's similar to hard link of Linux).
The new API can be used for Mover/Balancer if the underlying storage support
this function.
> Add new RPC to transfer data block with external shell script across Datanode
> -----------------------------------------------------------------------------
>
> Key: HDFS-15022
> URL: https://issues.apache.org/jira/browse/HDFS-15022
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: Yang Yun
> Assignee: Yang Yun
> Priority: Minor
> Attachments: HDFS-15022.patch, HDFS-15022.patch,
> link_block_across_datanode.pdf
>
>
> Replicating data block is expensive when some Datanodes are down, especially
> for slow storage. Add a new RPC to replicate block with external shell script
> across datanode. User can choose more effective way to copy block files.
> In our setup, Archive volume are configured to remote reliable storage. we
> just add a new link file in new datanode to the remote file when do
> replication.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
