HDFS-10910. HDFS Erasure Coding doc should state its currently supported erasure coding policies. Contributed by Yiqun Lin.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ee33a022 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ee33a022 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ee33a022 Branch: refs/heads/YARN-2915 Commit: ee33a02234511ac69c1e491fd38490a141ec907e Parents: 2ab1ef1 Author: Wei-Chiu Chuang <weic...@apache.org> Authored: Fri Sep 30 12:48:11 2016 -0700 Committer: Wei-Chiu Chuang <weic...@apache.org> Committed: Fri Sep 30 12:48:11 2016 -0700 ---------------------------------------------------------------------- .../hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/ee33a022/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md index 9066a15..76c1b3a 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md @@ -59,9 +59,9 @@ Architecture 1. _Read the data from source nodes:_ Input data is read in parallel from source nodes using a dedicated thread pool. Based on the EC policy, it schedules the read requests to all source targets and reads only the minimum number of input blocks for reconstruction. - 1. _Decode the data and generate the output data:_ New data and parity blocks are decoded from the input data. All missing data and parity blocks are decoded together. + 2. _Decode the data and generate the output data:_ New data and parity blocks are decoded from the input data. All missing data and parity blocks are decoded together. - 1. _Transfer the generated data blocks to target nodes:_ Once decoding is finished, the recovered blocks are transferred to target DataNodes. + 3. _Transfer the generated data blocks to target nodes:_ Once decoding is finished, the recovered blocks are transferred to target DataNodes. * **ErasureCoding policy** To accommodate heterogeneous workloads, we allow files and directories in an HDFS cluster to have different replication and EC policies. @@ -69,10 +69,9 @@ Architecture 1. _The ECSchema:_ This includes the numbers of data and parity blocks in an EC group (e.g., 6+3), as well as the codec algorithm (e.g., Reed-Solomon). - 1. _The size of a striping cell._ This determines the granularity of striped reads and writes, including buffer sizes and encoding work. + 2. _The size of a striping cell._ This determines the granularity of striped reads and writes, including buffer sizes and encoding work. - Currently, HDFS supports the Reed-Solomon and XOR erasure coding algorithms. Additional algorithms are planned as future work. - The system default scheme is Reed-Solomon (6, 3) with a cell size of 64KB. + There are three policies currently being supported: RS-DEFAULT-3-2-64k, RS-DEFAULT-6-3-64k and RS-LEGACY-6-3-64k. All with default cell size of 64KB. The system default policy is RS-DEFAULT-6-3-64k which use the default schema RS_6_3_SCHEMA with a cell size of 64KB. Deployment --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
