[
https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391207#comment-14391207
]
Zhe Zhang commented on HDFS-7285:
---------------------------------
Yesterday we had another offline meetup. I think the discussion was very
productive. Below please find the summary:
*Attendees*: Nicholas, Jing, Zhe
*Project phasing*
We went over the list of subtasks under this JIRA and separated them into 3
categories:
# Basic EC functionalities under the striping layout. Those subtasks were kept
under this umbrella JIRA. The goal is for the HDFS-7285 branch to be ready for
merging into trunk upon their completion.
# Follow-on tasks for EC+striping (including code and performance optimization,
as well as support for advanced HDFS features). Those subtasks were moved under
HDFS-8031. Following the common practice, those follow-on tasks are targeted
for trunk, after HDFS-7285 is merged.
# EC with non-striping / contiguous block layout. Those subtasks were moved to
HDFS-8030, which represents the 2nd phase of the erasure coding project.
Extending from the initial [PoC prototype |
https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14339006&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14339006],
the following _basic EC functionalities_ will be finished under this JIRA
([~szetszwo] please let me know if I missed anything from your list):
* A striped block group is distributed evenly on racks
* NN handles striped block groups in existing block management logics:
** Missing and corrupted blocks
** To-invalidate blocks
** Lease recovery
** DN decommissioning
* NN periodically distributes tasks to DN to reconstruct missing striped blocks
* DN executes the reconstruction task by pulling data from peer DNs
* Client can read a striped block group even if some blocks are missing,
through decoding
* Client should handle DN failures during writing
* Basic command for directory-level EC configuration (similar to a zone)
* Correctly handle striped block groups in file system statistics and metrics
* Documentation
* More comprehensive testing
* _Optional_: instead of hard-coding, incorporate the {{ECSchema}} class with
1~2 schemas
*Key remaining tasks*
We think the following remaining tasks are _key_ in terms of complexity and
amount of work:
# Client writing: the basic striped writing logic is close to complete (patch
available under HDFS-7889), but it's challenging to handle failures during
writing in an elegant way.
# Client reading: the logic isn't too complex but amount of work is non-trivial
# DN reconstruction: logic is clean but work has not been started yet
*Client design*
We also dived into more details of the design of client reading/writing paths,
and are synced on the overall approach. A few points were raised and will be
addressed:
# Cell size in striping currently has default value of 1M. We should study its
impact more carefully. Intuitively, a smaller value (like 128K) might be more
suitable.
# Pread in striping format should always try to fetch data in parallel, when
the requested range spans multiple striping cells.
# Stateful read in striping format should maintain multiple block readers to
minimize overhead of creating new readers.
> Erasure Coding Support inside HDFS
> ----------------------------------
>
> Key: HDFS-7285
> URL: https://issues.apache.org/jira/browse/HDFS-7285
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Weihua Jiang
> Assignee: Zhe Zhang
> Attachments: ECAnalyzer.py, ECParser.py, HDFS-7285-initial-PoC.patch,
> HDFSErasureCodingDesign-20141028.pdf, HDFSErasureCodingDesign-20141217.pdf,
> HDFSErasureCodingDesign-20150204.pdf, HDFSErasureCodingDesign-20150206.pdf,
> fsimage-analysis-20150105.pdf
>
>
> Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice
> of data reliability, comparing to the existing HDFS 3-replica approach. For
> example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks,
> with storage overhead only being 40%. This makes EC a quite attractive
> alternative for big data storage, particularly for cold data.
> Facebook had a related open source project called HDFS-RAID. It used to be
> one of the contribute packages in HDFS but had been removed since Hadoop 2.0
> for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends
> on MapReduce to do encoding and decoding tasks; 2) it can only be used for
> cold files that are intended not to be appended anymore; 3) the pure Java EC
> coding implementation is extremely slow in practical use. Due to these, it
> might not be a good idea to just bring HDFS-RAID back.
> We (Intel and Cloudera) are working on a design to build EC into HDFS that
> gets rid of any external dependencies, makes it self-contained and
> independently maintained. This design lays the EC feature on the storage type
> support and considers compatible with existing HDFS features like caching,
> snapshot, encryption, high availability and etc. This design will also
> support different EC coding schemes, implementations and policies for
> different deployment scenarios. By utilizing advanced libraries (e.g. Intel
> ISA-L library), an implementation can greatly improve the performance of EC
> encoding/decoding and makes the EC solution even more attractive. We will
> post the design document soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
