[
https://issues.apache.org/jira/browse/HDFS-16782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom McCormick updated HDFS-16782:
---------------------------------
Description:
At LinkedIn, we run very large HDFS volumes and many of them. We have a growing
need to balance data across our hdfs volumes without downtime. (More info on
our
[setup|https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr]
The goal is to migrate data from one hdfs volume to another, with zero read
downtime and zero (or near zero) write downtime.
This is a new mount point that will opportunistically write new files to the
new volume with the goal being to reduce the amount of new data created on the
original volume.
Similar to existing RBF subcluster mount points, it knows how to reconcile
files and directories between hdfs volumes.
-----
*Related tickets*
https://issues.apache.org/jira/browse/HDFS-15294
* Currently this solution has consistency issues
* assumes some downtime
was:
The goal is to migrate data from one hdfs volume to another, with zero read
downtime and zero (or near zero) write downtime.
This is a new mount point that will opportunistically write new files to the
new volume with the goal being to reduce the amount of new data created on the
original volume.
Similar to existing RBF subcluster mount points, it knows how to reconcile
files and directories between hdfs volumes.
> RBF Migrate Sub cluster mount point
> -----------------------------------
>
> Key: HDFS-16782
> URL: https://issues.apache.org/jira/browse/HDFS-16782
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
> Reporter: Tom McCormick
> Priority: Major
>
> At LinkedIn, we run very large HDFS volumes and many of them. We have a
> growing need to balance data across our hdfs volumes without downtime. (More
> info on our
> [setup|https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr]
>
> The goal is to migrate data from one hdfs volume to another, with zero read
> downtime and zero (or near zero) write downtime.
> This is a new mount point that will opportunistically write new files to the
> new volume with the goal being to reduce the amount of new data created on
> the original volume.
> Similar to existing RBF subcluster mount points, it knows how to reconcile
> files and directories between hdfs volumes.
>
>
> -----
> *Related tickets*
> https://issues.apache.org/jira/browse/HDFS-15294
> * Currently this solution has consistency issues
> * assumes some downtime
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
