Leon Gao created HDFS-15547:
-------------------------------
Summary: Dynamic disk-level tiering
Key: HDFS-15547
URL: https://issues.apache.org/jira/browse/HDFS-15547
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode
Reporter: Leon Gao
Assignee: Leon Gao
Attachments: Proposal - Dynamic disk-level tiering.pdf
This is a proposal for a new use case based on archival storage, to allow
configuring DISK and ARCHIVE storage types on the same device (filesystem) to
balance disk IO for disks with different density.
The proposal is to mainly solve two problems:
1) The disk IO of ARCHIVE disks is underutilized. This is normal in many use
cases where the data hotness is highly skewed.
2) Over the years, as better/cheaper hard drives showing on the market, a large
production environment can have mixed disk densities. For example, in our prod
environment, we have 2TB, 4TB, 8TB, and 16TB disks. When putting all different
HDDs into the cluster, we should be able to utilize disk capacity and disk IO
efficiently for all of them.
When moving blocks from DISK to ARCHIVE, we can prefer the same disk and simply
rename the files instead of copying.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
