[
https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357754#comment-14357754
]
Zhe Zhang commented on HDFS-7285:
---------------------------------
We have been discussing how to fit EC with other storage policies since the
first [meetup |
https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14192480&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14192480]
and haven't reached a clear conclusion. This design is now blocking several
ongoing JIRAs: HDFS-7068, HDFS-7349, HDFS-7839, HDFS-7866. I'd like to propose
the following potential solution based on the ideas we have exchanged:
To reiterate the challenge: Multiple dimensions of storage policies could be
applied to the same file. Across these dimensions we could have a large number
of combinations -- easily over 50, could be over 100. Fitting them in a single
dimension policy space is inefficient for the system to manage and inconvenient
for admins to set / get.
* Storage-type preference: HOT / WARM / COLD
* Erasure coding schema: ReedSolomon-6-3 / XOR-2-1 (targeting 5~10)
* Block layout: Striping / contiguous
* Other potential policies, e.g. compression
We can setup a family of storage policy XAttrs, where each dimension can be
independently set / get:
* {{system.hdfs.storagePolicy.type}}
* {{system.hdfs.storagePolicy.erasurecoding}}
* {{system.hdfs.storagePolicy.layout}}
Each dimension has a default value. So if an admin only wants to change the EC
schema, the following command can be used. The {{getStoragePolicy}} should
return policies on all dimensions unless an optional argument like
{{-erasureCoding}} is used.
{code}
setStoragePolicy -erasureCoding RS63 /home/zhezhang/foo
getStoragePolicy /home/zhezhang/foo
{code}
Like the current storage policy semantics, the initial policy of a file or dir
is inherited from its parent. Nested policy setting is allowed (/home is not
ECed but /home/zhezhang is). A single file can have a storage policy without
being in a zone.
Any feedbacks are very welcome. [~jingzhao], [~szetszwo], I think we should
have another meetup to sync on this (and several other issues)?
> Erasure Coding Support inside HDFS
> ----------------------------------
>
> Key: HDFS-7285
> URL: https://issues.apache.org/jira/browse/HDFS-7285
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Weihua Jiang
> Assignee: Zhe Zhang
> Attachments: ECAnalyzer.py, ECParser.py, HDFS-7285-initial-PoC.patch,
> HDFSErasureCodingDesign-20141028.pdf, HDFSErasureCodingDesign-20141217.pdf,
> HDFSErasureCodingDesign-20150204.pdf, HDFSErasureCodingDesign-20150206.pdf,
> fsimage-analysis-20150105.pdf
>
>
> Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice
> of data reliability, comparing to the existing HDFS 3-replica approach. For
> example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks,
> with storage overhead only being 40%. This makes EC a quite attractive
> alternative for big data storage, particularly for cold data.
> Facebook had a related open source project called HDFS-RAID. It used to be
> one of the contribute packages in HDFS but had been removed since Hadoop 2.0
> for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends
> on MapReduce to do encoding and decoding tasks; 2) it can only be used for
> cold files that are intended not to be appended anymore; 3) the pure Java EC
> coding implementation is extremely slow in practical use. Due to these, it
> might not be a good idea to just bring HDFS-RAID back.
> We (Intel and Cloudera) are working on a design to build EC into HDFS that
> gets rid of any external dependencies, makes it self-contained and
> independently maintained. This design lays the EC feature on the storage type
> support and considers compatible with existing HDFS features like caching,
> snapshot, encryption, high availability and etc. This design will also
> support different EC coding schemes, implementations and policies for
> different deployment scenarios. By utilizing advanced libraries (e.g. Intel
> ISA-L library), an implementation can greatly improve the performance of EC
> encoding/decoding and makes the EC solution even more attractive. We will
> post the design document soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
