[
https://issues.apache.org/jira/browse/HDFS-16782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom McCormick updated HDFS-16782:
---------------------------------
Description:
At LinkedIn, we run very large HDFS volumes and many of them. We have a growing
need to balance data across our hdfs volumes without downtime. (More info on
our
[setup|https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr]
)
The *goal* is to migrate data from one hdfs volume to another, with zero read
downtime and zero (or near zero) write downtime.
The *proposal* is a new mount point that will opportunistically write new files
to the new volume with the goal being to reduce the amount of new data created
on the original volume.
Similar to existing RBF subcluster mount points, it knows how to reconcile
files and directories between hdfs volumes.
This solution will have some assumptions / dependencies on move tooling based
to ensure the strategy implemented by the subcluster mount point and the data
movement tooling to ensure consistency and gracefully handing error cases.
----
*Related tickets*
https://issues.apache.org/jira/browse/HDFS-15294
* Currently this solution has consistency issues
* assumes some downtime
was:
At LinkedIn, we run very large HDFS volumes and many of them. We have a growing
need to balance data across our hdfs volumes without downtime. (More info on
our
[setup|https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr]
The goal is to migrate data from one hdfs volume to another, with zero read
downtime and zero (or near zero) write downtime.
This is a new mount point that will opportunistically write new files to the
new volume with the goal being to reduce the amount of new data created on the
original volume.
Similar to existing RBF subcluster mount points, it knows how to reconcile
files and directories between hdfs volumes.
-----
*Related tickets*
https://issues.apache.org/jira/browse/HDFS-15294
* Currently this solution has consistency issues
* assumes some downtime
> RBF Migrate Sub cluster mount point
> -----------------------------------
>
> Key: HDFS-16782
> URL: https://issues.apache.org/jira/browse/HDFS-16782
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
> Reporter: Tom McCormick
> Priority: Major
>
> At LinkedIn, we run very large HDFS volumes and many of them. We have a
> growing need to balance data across our hdfs volumes without downtime. (More
> info on our
> [setup|https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr]
> )
> The *goal* is to migrate data from one hdfs volume to another, with zero read
> downtime and zero (or near zero) write downtime.
> The *proposal* is a new mount point that will opportunistically write new
> files to the new volume with the goal being to reduce the amount of new data
> created on the original volume.
> Similar to existing RBF subcluster mount points, it knows how to reconcile
> files and directories between hdfs volumes.
> This solution will have some assumptions / dependencies on move tooling based
> to ensure the strategy implemented by the subcluster mount point and the data
> movement tooling to ensure consistency and gracefully handing error cases.
>
>
> ----
> *Related tickets*
> https://issues.apache.org/jira/browse/HDFS-15294
> * Currently this solution has consistency issues
> * assumes some downtime
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
