smengcl commented on code in PR #463: URL: https://github.com/apache/ozone-site/pull/463#discussion_r3555033237
########## docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/04-hdds-3816-erasure-coding.md: ########## @@ -0,0 +1,91 @@ +# HDDS-3816: Erasure Coding Phase 1 + +Distributed systems basic expectation is to provide the data durability. +To provide the higher data durability, many popular storage systems use replication +approach which is expensive. The Apache Ozone supports \`RATIS/THREE\` replication scheme. +The Ozone default replication scheme \`RATIS/THREE\` has 200% overhead in storage +space and other resources (e.g., network bandwidth). +However, for warm and cold datasets with relatively low I/O activities, additional +block replicas rarely accessed during normal operations, but still consume the same +amount of resources as the first replica. + +Therefore, a natural improvement is to use Erasure Coding (EC) in place of replication, +which provides the same level of fault-tolerance with much less storage space. +In typical EC setups, the storage overhead is no more than 50%. The replication factor of an EC file is meaningless. +Instead of replication factor, we introduced ReplicationConfig interface to specify the required type of replication, +either \`RATIS/THREE\` or \`EC\`. + +Integrating EC with Ozone can improve storage efficiency while still providing similar +data durability as traditional replication-based Ozone deployments. +As an example, a 3x replicated file with 6 blocks will consume 6\*3 = \`18\` blocks of disk space. +But with EC (6 data, 3 parity) deployment, it will only consume \`9\` blocks of disk space. + +## Git Branch + +## 1. builds/intermittent test failures Review Comment: Link to branch is lost somehow: https://cwiki.apache.org/confluence/spaces/OZONE/pages/199534150/Ozone+EC+Branch+HDDS-3816-ec+Phase-1+Merge+Checklist ```suggestion ## Git Branch https://github.com/apache/ozone/tree/HDDS-3816-ec ## 1. builds/intermittent test failures ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
