JyotinderSingh commented on a change in pull request #3006: URL: https://github.com/apache/ozone/pull/3006#discussion_r789418341
########## File path: hadoop-hdds/docs/content/feature/ErasureCoding.md ########## @@ -0,0 +1,215 @@ +--- +title: "Ozone Erasure Coding" +weight: 1 +menu: + main: + parent: Features +summary: Erasure Coding Support for Ozone. +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +## Background + +Distributed systems basic expectation is to provide the data durability. +To provide the higher data durability, many popular storage systems use replication +approach which is expensive. The Apache Ozone supports `RATIS/THREE` replication scheme. +The Ozone default replication scheme `RATIS/THREE` has 200% overhead in storage +space and other resources (e.g., network bandwidth). +However, for warm and cold datasets with relatively low I/O activities, additional +block replicas rarely accessed during normal operations, but still consume the same +amount of resources as the first replica. + +Therefore, a natural improvement is to use Erasure Coding (EC) in place of replication, +which provides the same level of fault-tolerance with much less storage space. +In typical Erasure Coding (EC) setups, the storage overhead is no more than 50%. +The replication factor of an EC file is meaningless. Instead of replication factor, +we introduced ReplicationConfig interface to specify the required type of replication, +either `RATIS/THREE` or `EC`. + +Integrating EC with Ozone can improve storage efficiency while still providing similar +data durability as traditional replication-based Ozone deployments. +As an example, a 3x replicated file with 6 blocks will consume 6*3 = `18` blocks of disk space. +But with EC (6 data, 3 parity) deployment, it will only consume `9` blocks of disk space. + +## Architecture + +The storage data layout is a key factor in the implementation of EC. After deep analysis +and several technical consideration, the most fitting data layout is striping model. +The data striping layout is not new. The striping model already adapted by several other +file systems(Ex: Quantcast File System, Hadoop Distributed File System etc) successfully before. + +For example, with the EC (6 data, 3 parity) scheme, the data chunks will be distributed to first 6 data nodes in order +and then client generates the 3 parity chunks and transfer to remaining 3 nodes in order. +These 9 chunks together we call as "Stripe". Next 6 chunks will be distributed to the same first 6 data nodes again +and the parity to remaining 3 nodes. These 9 data nodes stored blocks together called as "BlockGroup". + +If the application is continuing to write beyond teh size of `6 * BLOCK_SIZE`, then client will request new block group from Ozone Manager. + +### Erasure Coding Write + +The core logic of erasure coding writes are placed at ozone client. +When client creates the file, ozone manager allocates the block group(`d + p`) +number of nodes from the pipeline provider and return the same to client. +As data is coming in from the application, client will write first d number of chunks +to d number of data nodes in block group. It will also cache the d number chunks +to generate the parity chunks. Once parity chunks generated, it will transfer the +same to the remaining p nodes in order. Once all blocks reached their configured sizes, +client will request the new block group nodes. + +Below diagram depicts the block allocation in containers as logical groups. +For interest of space, we assumed EC(3, 2) replication config for the diagram. + + Review comment: Could you use change the Markdown Image tag to the Hugo shortcode defined in `hadoop-hdds/docs/themes/ozonedoc/layouts/shortcodes/image.html` This would ensure that the images don't overflow content boundaries on the website. You can change it to the following: ``` {{< image src="EC-Write-Block-Allocation-in-Containers.png">}} ``` Once you do this, the image won't be visible on the intellij markdown editor, but will be available on the website after the Hugo build process. You can preview the website as it will appear on the web by running the following ([reference](https://github.com/apache/ozone/tree/master/hadoop-hdds/docs)): ``` hugo serve ``` note: you will need to have hugo available on your machine to run the serve command `brew install hugo` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
