[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2016-01-17 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104051#comment-15104051
 ] 

Kai Zheng commented on HDFS-7866:
-

Hi [~lirui], the patch is very large and I noticed much of the changes are not 
very essential to the multiple policy support. Can you refactor and extract 
such changes into separate patches to prepare for this? Thanks!

> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch, 
> HDFS-7866.7.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in 
> authorized and controlled approach. The provided facilities will be used to 
> implement DFSAdmin commands so admin can list available EC schemas, then 
> could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2016-01-04 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15081093#comment-15081093
 ] 

Kai Zheng commented on HDFS-7866:
-

Hi [~walter.k.su], thanks for your reminding. Yes I was aware the discussion 
and planed to read the related discussions in related issues and sort it out 
how we would proceed here and HDFS-7337.

> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch, 
> HDFS-7866.7.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in 
> authorized and controlled approach. The provided facilities will be used to 
> implement DFSAdmin commands so admin can list available EC schemas, then 
> could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-12-30 Thread Rui Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075676#comment-15075676
 ] 

Rui Li commented on HDFS-7866:
--

Thanks [~drankye] for the review!
1. Sure, will do.
2. Since the system policies are static members, they'll have the same ID after 
NN restarts. But I agree it's better not to let {{ErasureCodingPolicy}} 
generate the ID itself. How about pass the ID to it as a constructor parameter, 
and have the ID generator in {{ErasureCodingPolicyManager}}?
3. Currently striped files all have 0 as the replication factor. E.g. in the 
INodeFile part of the patch:
{code}
-  // Replication factor for striped files is zero
+  // For erasure coded files, replication is used to store ec policy id
+  if (isStriped) {
+Preconditions.checkArgument(replication >= 0);
+  }
+  h = REPLICATION.BITS.combine(replication, h);
   if (isStriped) {
-h = REPLICATION.BITS.combine(0L, h);
 h = IS_STRIPED.BITS.combine(1L, h);
   } else {
-h = REPLICATION.BITS.combine(replication, h);
 h = IS_STRIPED.BITS.combine(0L, h);
   }
{code}
With the patch, striped file's replication factor is actually the policy ID. 
The modification you mentioned in {{HdfsFileStatus}} is to avoid confusing 
users by displaying policy ID as replication factor.
4. {{QUEUE_VERY_UNDER_REPLICATED}} means the current redundancy is 1/3 of 
desired redundancy. So I changed the comment to be consistent with the code. 
You can refer to discussions in HDFS-8461.
5. Which map do you think we should keep?

> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch, 
> HDFS-7866.7.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in 
> authorized and controlled approach. The provided facilities will be used to 
> implement DFSAdmin commands so admin can list available EC schemas, then 
> could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-12-30 Thread Walter Su (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075037#comment-15075037
 ] 

Walter Su commented on HDFS-7866:
-

To [~drankye],
[There|https://issues.apache.org/jira/browse/HDFS-8833?focusedCommentId=14658978=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14658978]
 is a discussion about whether to hard-code policy or not.
I think hard-coding will do. But there is {{ECSchema.extraOptions}}. I don't 
know if your original idea is to make it configurable?

> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch, 
> HDFS-7866.7.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in 
> authorized and controlled approach. The provided facilities will be used to 
> implement DFSAdmin commands so admin can list available EC schemas, then 
> could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-12-30 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074997#comment-15074997
 ] 

Kai Zheng commented on HDFS-7866:
-

Some comments so far. Will continue the review tomorrow.

1. Ref. below codes:
{code}
+  public ErasureCodingPolicy(ECSchema schema, int cellSize) {
+this.name = composePolicyName(schema, cellSize);
+this.schema = schema;
+this.cellSize = cellSize;
+id = getNextID();
+  }
{code}
How about this to reuse the constructor.
{code}
public ErasureCodingPolicy(ECSchema schema, int cellSize) {
  this(composePolicyName(schema, cellSize), cellSize);
}
{code}

2. See the codes. What will happen when NN restarts? 
{code}
+  private static final AtomicInteger ID_GEN = new AtomicInteger(0);
+
{code}

{code}
+  private static short getNextID() {
+return (short) ID_GEN.getAndIncrement();
+  }
{code}

3. In below codes, not sure better or not use 0 or 1 for the block_replication 
value in striping case. Would you check related codes?
{code}
-this.block_replication = (short)block_replication;
+this.block_replication = ecPolicy == null ? (short) block_replication : 0;
{code}

4. A little confused.
{code}
-  // can only afford one replica loss
+  // there is less than a third as many blocks as requested;
{code}

5. I guess better to use only one map for the most often case to avoid out of 
sync in two maps. For lookup using the other key, simple iteration may do the 
work.
{code}
-this.activePolicies = new TreeMap<>();
-for (ErasureCodingPolicy policy : SYS_POLICY) {
-  activePolicies.put(policy.getName(), policy);
+this.activePoliciesByName = new TreeMap<>();
+this.activePoliciesByID = new TreeMap<>();
+for (ErasureCodingPolicy policy : SYS_POLICIES) {
+  activePoliciesByName.put(policy.getName(), policy);
+  activePoliciesByID.put(policy.getId(), policy);
 }
{code}

> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch, 
> HDFS-7866.7.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in 
> authorized and controlled approach. The provided facilities will be used to 
> implement DFSAdmin commands so admin can list available EC schemas, then 
> could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-12-22 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068012#comment-15068012
 ] 

Hadoop QA commented on HDFS-7866:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 21 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s 
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 10s 
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 29s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 37s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 21s 
{color} | {color:red} Patch generated 1 new checkstyle issues in 
hadoop-hdfs-project (total was 121, now 118). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s 
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK 
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 51s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s 
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK 
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 25s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
26s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 136m 44s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| JDK v1.7.0_91 Failed junit tests |

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-12-20 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066024#comment-15066024
 ] 

Kai Zheng commented on HDFS-7866:
-

Thanks Rui for the great work and moving this on!

bq. Add a new system EC policy RS-3-2 and set it as the default policy. The 3-2 
schema is nothing specific and just for testing. And we can revert to our 
current default policy (or make it configurable) when the patch is ready.
It would be good to have the new built-in policy RS-3-2, considering it will 
help a lot in some testing, experimental trying of EC on a small cluster (5 
nodes). [~zhz] how do you think of this? However, please don't set it as 
default policy or change the default policy at all. 

bq. EC policy can be retrieved either by name or by ID. The ID is stored in 
HEADER::replication in InodeFile, as suggested by the TODO comment.
This sounds good. Would you share the details about making use of the remain 
bits in the replicaiton field? How many policies will it support in the way? 
How many polices would we have if we could estimate assuming 4+ codecs (XOR, 
RS, LRC, HitchHicker) and their derivations, 10+ cell sizes (64KB, 128KB, 
256KB, ..., 32MB and so on), and the EC form (striping or contiguous). 
Consuming the whole remain bits for this purpose (not sure how you did) may be 
not a good idea, considering there might be some other usages.

bq. Lots of modifications to the tests to make them work with multiple 
policies. This may better fit as a follow-on task. But to verify the patch I 
have to make the tests pass here.
I thought fixing these tests is essential for this work and glad you have made 
it work. I don't have yet looked into the large patch yet, do we add/have tests 
to test multiple policies can work at the same time in a cluster? 

> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in 
> authorized and controlled approach. The provided facilities will be used to 
> implement DFSAdmin commands so admin can list available EC schemas, then 
> could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-12-20 Thread Rui Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066031#comment-15066031
 ] 

Rui Li commented on HDFS-7866:
--

Thanks Kai for your comments.
bp. However, please don't set it as default policy or change the default policy 
at all.
Yeah I didn't mean to change the default policy. The change in the patch is 
temporary to make the tests pick the new policy.
bp. Would you share the details about making use of the remain bits in the 
replicaiton field?
Sure. The HeaderFormat is a 64-bit long and its layout is {{[4-bit 
storagePolicyID][1-bit isStriped][11-bit replication][48-bit 
preferredBlockSize]}}. The patch doesn't change the layout -- it just uses the 
11-bit replication to store the policy ID. Therefore it can support up to 
{{2^11-1=2047}} different policies. I think that'll be enough for all the 
combinations of codecs, cell sizes and layouts.
bq. do we add/have tests to test multiple policies can work at the same time in 
a cluster
Good point. I'll add a test to cover this.

> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in 
> authorized and controlled approach. The provided facilities will be used to 
> implement DFSAdmin commands so admin can list available EC schemas, then 
> could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-12-18 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063739#comment-15063739
 ] 

Hadoop QA commented on HDFS-7866:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 20 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s 
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 23s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 11s 
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 27s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 33s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s 
{color} | {color:red} Patch generated 1 new checkstyle issues in 
hadoop-hdfs-project (total was 129, now 127). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 0s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s 
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK 
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 22s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s 
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK 
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 7s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
25s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 19s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | 
hadoop.hdfs.server.datanode.TestFsDatasetCache |
|   |

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-12-18 Thread Rui Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063885#comment-15063885
 ] 

Rui Li commented on HDFS-7866:
--

The latest failures cannot be reproduced locally.

> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in 
> authorized and controlled approach. The provided facilities will be used to 
> implement DFSAdmin commands so admin can list available EC schemas, then 
> could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-12-17 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062185#comment-15062185
 ] 

Hadoop QA commented on HDFS-7866:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 18 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
7s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s 
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 31s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 18s 
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 39s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 48s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s 
{color} | {color:red} Patch generated 2 new checkstyle issues in 
hadoop-hdfs-project (total was 96, now 95). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 36s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 45s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 18s 
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK 
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 53s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s 
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK 
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 41s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 27s 
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 47s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | 
hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
|   |

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-12-09 Thread Rui Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048347#comment-15048347
 ] 

Rui Li commented on HDFS-7866:
--

I tried to add another system policy (with different schema) and hit the 
following problem.

In {{FSDirErasureCodingOp::getErasureCodingPolicyForPath}}, if the inode is a 
file, we retrieve the policy by a numeric ID: 
{{INodeFile::getErasureCodingPolicyID}}.
For directories however, the policy is retrieved by name:
{code}
final XAttrFeature xaf = inode.getXAttrFeature();
if (xaf != null) {
  XAttr xattr = xaf.getXAttr(XATTR_ERASURECODING_POLICY);
  if (xattr != null) {
ByteArrayInputStream bIn = new 
ByteArrayInputStream(xattr.getValue());
DataInputStream dIn = new DataInputStream(bIn);
String ecPolicyName = WritableUtils.readString(dIn);
return fsd.getFSNamesystem().getErasureCodingPolicyManager().
getPolicy(ecPolicyName);
  }
}
{code}
Therefore the newly added policy works for dirs but not files. I think we need 
a unified way to store/get a policy. Any thoughts?

> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
> HDFS-7866-v3.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in 
> authorized and controlled approach. The provided facilities will be used to 
> implement DFSAdmin commands so admin can list available EC schemas, then 
> could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-12-08 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046665#comment-15046665
 ] 

Kai Zheng commented on HDFS-7866:
-

Thanks [~lirui] for the taking! Just assigned this to you.

> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
> HDFS-7866-v3.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in 
> authorized and controlled approach. The provided facilities will be used to 
> implement DFSAdmin commands so admin can list available EC schemas, then 
> could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-12-08 Thread Rui Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046636#comment-15046636
 ] 

Rui Li commented on HDFS-7866:
--

Hi [~drankye], it seems HDFS-9345 depends on this JIRA. I can help with it if 
you're busy with other tasks. Thanks.

> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
> HDFS-7866-v3.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in 
> authorized and controlled approach. The provided facilities will be used to 
> implement DFSAdmin commands so admin can list available EC schemas, then 
> could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-15 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497386#comment-14497386
 ] 

Kai Zheng commented on HDFS-7866:
-

I agree this moves out, and will resolve some TODOs left for this in HDFS-8156. 
Thanks.

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
 HDFS-7866-v3.patch


 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-14 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493725#comment-14493725
 ] 

Kai Zheng commented on HDFS-7866:
-

I thought it's good to have this in the patch and HDFS-7859 can remove them 
when adding the codes. Sounds good?

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch


 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-14 Thread Vinayakumar B (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493755#comment-14493755
 ] 

Vinayakumar B commented on HDFS-7866:
-


Hi [~xinwei], Thanks for working on this.
Patch looks good.
Some nits.

1. namesystem is unused now. Let it be added whenever actually used.
{code}
+  /**
+   * The FSNamesystem that contains this ECSchemaManager.
+   */
+  private final FSNamesystem namesystem;{code}

2. can remove ): in the log message. I think you wanted to put sad smiley ( :( 
) , but logger wont understand your feelings ;)
{code}
+  LOG.warn(A schema {} is updated but will be ignored as not  +
+  supported yet):, schema.getSchemaName());{code}

3. Can also update TODO in the {{FSNameSystem#getECSchemas()}} to return loaded 
schemas.

4. Rename {{TestSchemaManager}} to {{TestECSchemaManager}}

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch


 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-14 Thread Xinwei Qin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493727#comment-14493727
 ] 

Xinwei Qin  commented on HDFS-7866:
---

OK, that sounds good.

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch


 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-14 Thread Xinwei Qin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493720#comment-14493720
 ] 

Xinwei Qin  commented on HDFS-7866:
---

Hi, [~drankye]
{code}
+/**
+ * TODO: HDFS-7859 persist into NameNode
+ * load persistent schemas from image and editlog, which is done only once
+ * during NameNode startup. This can be done here or in a separate method.
+ */ 
{code}
These annotation can be removed. Now loading persistent schemas from fsimage 
and editlog is done in {{loadECSchemas()}} or {{loadState()}} method and these 
methods are called during NameNode startup.

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch


 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-14 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493863#comment-14493863
 ] 

Kai Zheng commented on HDFS-7866:
-

Hi [~vinayrpet], thanks for your good comments. Patch updated. Would you review 
one more time? Thanks.

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
 HDFS-7866-v3.patch


 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-14 Thread Vinayakumar B (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493922#comment-14493922
 ] 

Vinayakumar B commented on HDFS-7866:
-

Some more nits,
1. Following changes not required. Why to have same value in two places, one in 
variable and one in map. In fact you need to remove from the map while 
assigning to variable.
{code}+if (options == null) {
+  options = new HashMap();
+}
+options.put(CODEC_NAME_KEY, codecName);
+options.put(NUM_DATA_UNITS_KEY, String.valueOf(numDataUnits));
+options.put(NUM_PARITY_UNITS_KEY, String.valueOf(numParityUnits));
+this.options = Collections.unmodifiableMap(options);{code}

2. {{ECSchemaManager#reloadPredefined()}} should be called on initialization. 
May be in constructor.

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
 HDFS-7866-v3.patch


 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-14 Thread Tsz Wo Nicholas Sze (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494953#comment-14494953
 ] 

Tsz Wo Nicholas Sze commented on HDFS-7866:
---

[~drankye], [~vinayrpet] I think we need a design for EC schema to specify what 
operations are supported; see [this 
comment|https://issues.apache.org/jira/browse/HDFS-7859?focusedCommentId=14494933page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14494933]
 in HDFS-7859.  Also, we only support one schema (6,3)-Reed-Solomon in the 
first phase HDFS-7285. I think we should focus on finishing a complete, working 
basic EC feature and get HDFS-7285 merged to trunk. How about deferring this 
work to HDFS-8031? Sorry for commenting on this late and thanks for all the 
good works.

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
 HDFS-7866-v3.patch


 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-14 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495044#comment-14495044
 ] 

Kai Zheng commented on HDFS-7866:
-

Hi [~szetszwo], thanks for taking care of this.
The overall design and discussion are in HDFS-7337. Let's discuss further 
there. I will sort out latest discussions and update the design doc there. 
Thanks. 

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
 HDFS-7866-v3.patch


 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-14 Thread Tsz Wo Nicholas Sze (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495056#comment-14495056
 ] 

Tsz Wo Nicholas Sze commented on HDFS-7866:
---

Since HDFS-7337 is under HDFS-8031, let's move all the related works to 
HDFS-8031.

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
 HDFS-7866-v3.patch


 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-14 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494050#comment-14494050
 ] 

Kai Zheng commented on HDFS-7866:
-

Thanks [~vinayrpet] for the further thoughts.
bq.Following changes not required. 
I agree, better to remove the duplication.
bq.reloadPredefined() should be called on initialization.
I have an offline discussion with Vinay about this. Let me summarize the 
discussion below. Please correct me if anything incorrect.

Originally {{ec-schemas.xml}} was thought of as an input method for admin to 
define their schemas because it's not easy to specify schema parameters well in 
command line if we allow that by providing a command. Therefore it's expected 
to trigger a reload command after manually editing the schema file. To avoid 
bad situation when have an immature schema file, the automatic loading of the 
file isn't expected. 

Vinay has different thoughts, and suggests that we have system defined schemas 
as defined in the current ec-schema.xml, which are fixed and not allowed to 
change. Additionally, we also allow site specific schemas, which can be defined 
in site specific xml file. The system defined xml file should be loaded 
automatically. The site specific xml file needs admin to trigger the loading 
once it's changed.

I agree with the thoughts above. To simplify a little bit, we can have the 
system schemas defined just in Java codes, as storage policies do, not using 
any xml file; and rename the existing ec-schema.xml file to 
ec-schema-template.xml, so a deployment can have its own site specific xml file 
like {{ec-schema-site.xml}}, for admin to define their own schemas. Loading 
this file will need admin to trigger a reload command, not automatically.

Would anyone have more thoughts? Thanks!

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, 
 HDFS-7866-v3.patch


 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-09 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14487002#comment-14487002
 ] 

Kai Zheng commented on HDFS-7866:
-

HDFS-8112 was opened to handle the authorization protection aspect.

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng

 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas

2015-04-07 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483193#comment-14483193
 ] 

Kai Zheng commented on HDFS-7866:
-

I'm still working on the codes. Before we can fully get this done, to resolve 
some dependencies, HDFS-8074 was opened to have a system default EC schema 
first but in good place.

 Erasure coding: NameNode manages EC schemas
 ---

 Key: HDFS-7866
 URL: https://issues.apache.org/jira/browse/HDFS-7866
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng

 This is to extend NameNode to load, list and sync predefine EC schemas in 
 authorized and controlled approach. The provided facilities will be used to 
 implement DFSAdmin commands so admin can list available EC schemas, then 
 could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)