[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066024#comment-15066024
]
Kai Zheng commented on HDFS-7866:
---------------------------------
Thanks Rui for the great work and moving this on!
bq. Add a new system EC policy RS-3-2 and set it as the default policy. The 3-2
schema is nothing specific and just for testing. And we can revert to our
current default policy (or make it configurable) when the patch is ready.
It would be good to have the new built-in policy RS-3-2, considering it will
help a lot in some testing, experimental trying of EC on a small cluster (5
nodes). [~zhz] how do you think of this? However, please don't set it as
default policy or change the default policy at all.
bq. EC policy can be retrieved either by name or by ID. The ID is stored in
HEADER::replication in InodeFile, as suggested by the TODO comment.
This sounds good. Would you share the details about making use of the remain
bits in the replicaiton field? How many policies will it support in the way?
How many polices would we have if we could estimate assuming 4+ codecs (XOR,
RS, LRC, HitchHicker) and their derivations, 10+ cell sizes (64KB, 128KB,
256KB, ..., 32MB and so on), and the EC form (striping or contiguous).
Consuming the whole remain bits for this purpose (not sure how you did) may be
not a good idea, considering there might be some other usages.
bq. Lots of modifications to the tests to make them work with multiple
policies. This may better fit as a follow-on task. But to verify the patch I
have to make the tests pass here.
I thought fixing these tests is essential for this work and glad you have made
it work. I don't have yet looked into the large patch yet, do we add/have tests
to test multiple policies can work at the same time in a cluster?
> Erasure coding: NameNode manages EC schemas
> -------------------------------------------
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Kai Zheng
> Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
