[
https://issues.apache.org/jira/browse/HDFS-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006657#comment-17006657
]
Jinglun edited comment on HDFS-15087 at 1/2/20 1:36 PM:
--------------------------------------------------------
Hi [~elgoiri], thanks your nice comments !
{quote}Would it be possible to leverage HDFS snapshots instead of blocking
writes and having the new tree related calls? Intuitively, I would expect for
snapshots to cover 90% of the features described in the doc. I would try to
improve snapshots to cover 100%.
{quote}
-I'm not familiar with the snapshot. In my rough thought as long as the
snapshot meta could be transferred and rebuilt the HFR could support it.- -I'll
try to write a demo to transfer and rebuild the snapshot across NameNodes-.
I have a quick look of snapshot and I'm not sure how to use it. Do you mean to
use the diff of snapshots so we can do the balance in an incremental way ?
{quote}The approach described in the doc requires hard linking. I think this is
a good idea for the start but I would push to make it pluggable/abstract so in
the future we can have other implementations.
{quote}
Good idea. The design of HFR has considered it. The HFR is a combination of
many tasks. Each task is plugable. For example if we want to use copy instead
of hardlink, we can switch the HardLink task to a CopyReplica task.
{quote}Is hard linking available in Windows?
{quote}
After HADOOP-11483 we use jdk Files.createLink() to do the hardlinks. I test
Files.createLink() on windows and it works.
See java doc [https://docs.oracle.com/javase/tutorial/essential/io/links.html]
was (Author: lijinglun):
Hi [~elgoiri], thanks your nice comments !
{quote}Would it be possible to leverage HDFS snapshots instead of blocking
writes and having the new tree related calls? Intuitively, I would expect for
snapshots to cover 90% of the features described in the doc. I would try to
improve snapshots to cover 100%.
{quote}
I'm not familiar with the snapshot. In my rough thought as long as the snapshot
meta could be transferred and rebuilt the HFR could support it. I'll try to
write a demo to transfer and rebuild the snapshot across NameNodes.
{quote}The approach described in the doc requires hard linking. I think this is
a good idea for the start but I would push to make it pluggable/abstract so in
the future we can have other implementations.
{quote}
Good idea. The design of HFR has considered it. The HFR is a combination of
many tasks. Each task is plugable. For example if we want to use copy instead
of hardlink, we can switch the HardLink task to a CopyReplica task.
{quote}Is hard linking available in Windows?
{quote}
After HADOOP-11483 we use jdk Files.createLink() to do the hardlinks. I test
Files.createLink() on windows and it works.
See java doc [https://docs.oracle.com/javase/tutorial/essential/io/links.html]
> RBF: Balance/Rename across federation namespaces
> ------------------------------------------------
>
> Key: HDFS-15087
> URL: https://issues.apache.org/jira/browse/HDFS-15087
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jinglun
> Priority: Major
> Attachments: HFR_Rename Across Federation Namespaces.pdf
>
>
> The Xiaomi storage team has developed a new feature called HFR(HDFS
> Federation Rename) that enables us to do balance/rename across federation
> namespaces. The idea is to first move the meta to the dst NameNode and then
> link all the replicas. It has been working in our largest production cluster
> for 2 months. We use it to balance the namespaces. It turns out HFR is fast
> and flexible. The detail could be found in the design doc.
> Looking forward to a lively discussion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
