[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14583047#comment-14583047
]
Zhe Zhang commented on HDFS-8499:
-
Regarding the [comparison |
https://issues.apache.org/jira/browse/HDFS-8499?focusedCommentId=14578064page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14578064]
of the 2 inheritance structures, yes the main motivation for choosing option
#1 in 04~06 patches is to minimize changes to trunk.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14583002#comment-14583002
]
Yi Liu commented on HDFS-8499:
--
Thanks Walter for the comments. Your comment #1 looks good, but I think it's
OK to put these static methods in {{BlockInfoContiguous}}. Zhe, you can
address Walter's #1 comment if you think it's good too, since both looks good
to me.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14583016#comment-14583016
]
Vinayakumar B commented on HDFS-8499:
-
{quote}
It's not better than the approach done in 04/05 patch. I just think the
inheritance is more clear. In 04/05 patch, BlockInfoContiguous,
BlockInfoStriped, BlockInfoUC those three extends BlockInfo, so they are all
first-class citizen. But BlockInfoUC is different with the others. I think UC
is more like a feature but not entity, like color to apple.{quote}
[~walter.k.su], Looks like your approach is one of approaches discussed in
earlier
[comment|https://issues.apache.org/jira/browse/HDFS-8499?focusedCommentId=14578064page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14578064],
but not used just to minimize the changes.
bq. It's inappropriate for me to comment on this critical jira. I don't have
much coding experience. Please ignore this if what I said is wrong.
Dont worry, every opinion counts. And whatever you said was right from the
other angle.
Some nits in 05 patch:
1. unused
{code}public static final BlockInfoContiguous[] EMPTY_ARRAY = {};{code}
2. Use Preconditions instead of asserts in all files. As these needs to be
checked always, not only in tests.
{code}+assert getBlockUCState() !=
HdfsServerConstants.BlockUCState.COMPLETE :
+BlockInfoUnderConstructionContiguous cannot be in COMPLETE
state;{code}
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582993#comment-14582993
]
Yi Liu commented on HDFS-8499:
--
Thanks Zhe, +1 for {{.05}} patch, will commit it later.
For striped, let's handle it in the the EC branch.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14583210#comment-14583210
]
Hadoop QA commented on HDFS-8499:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 50s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 10 new or modified test files. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 40s | The applied patch generated 3
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 15s | The applied patch generated 7
new checkstyle issues (total was 1137, now 1121). |
| {color:green}+1{color} | whitespace | 0m 5s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 14s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 161m 33s | Tests passed in hadoop-hdfs.
|
| | | 207m 58s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12739177/HDFS-8499.06.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 83e8110 |
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/11317/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11317/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11317/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11317/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11317/console |
This message was automatically generated.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14583424#comment-14583424
]
Yi Liu commented on HDFS-8499:
--
Thanks Zhe for the patch. Sorry there is a confusion for the comments:
Seems you modified the patch based on {{04}}, so missed my another comment
again (We should include what you did in {{05}} about The update patch also
moves the implementation of 3 methods to BIUCContiguous: setExpectedLocations,
getTruncateBlock and setTruncateBlock).
So I mean we modify the patch based on {{05}}, just keep the static methods in
{{ContiguousBlockStorageOp}}.
+1 after addressing.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14583810#comment-14583810
]
Hadoop QA commented on HDFS-8499:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 26s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 10 new or modified test files. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 34s | The applied patch generated 3
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 58s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 5s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 16s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 12s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 164m 18s | Tests passed in hadoop-hdfs.
|
| | | 206m 57s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12739245/HDFS-8499.07.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e4489d9 |
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/11323/artifact/patchprocess/diffJavadocWarnings.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11323/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11323/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11323/console |
This message was automatically generated.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14583830#comment-14583830
]
Andrew Wang commented on HDFS-8499:
---
I think the 07 patch addresses Yi's comment. I'll commit shortly based on the
very thorough reviews from Vinay, Walter, Yi.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581893#comment-14581893
]
Yi Liu commented on HDFS-8499:
--
The approach {{#1}}(04 patch) looks good. Some comments:
We don't need {{BlockInfoContiguousFeature}} (and {{BlockInfoStripedFeature}}),
just write the static methods in {{BlockInfoContiguous}} and
{{BlockInfoStriped}}, and we can call these static methods in
{{BlockInfoUnderConstructionContiguous}} and
{{BlockInfoUnderConstructionStriped}}.
{quote}
In BlockInfoUC, there are some methods should have different implementations
for Continuous and Striped, for example, setExpectedLocations, currently the
default in BlockInfoUC is for continuous, of course we can override it in
striped. Maybe they should be abstract, and implemented in both continuous and
striped, then it's more clear? So I suggest if the implementation of some
method is different for continuous and striped, then we make them abstract.
{quote}
You missed this?
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582620#comment-14582620
]
Hadoop QA commented on HDFS-8499:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 45s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 10 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 37s | The applied patch generated 3
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 13s | The applied patch generated 7
new checkstyle issues (total was 1137, now 1121). |
| {color:green}+1{color} | whitespace | 0m 8s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 14s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 64m 20s | Tests failed in hadoop-hdfs. |
| | | 110m 34s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.fs.contract.hdfs.TestHDFSContractOpen |
| Timed out tests | org.apache.hadoop.fs.contract.hdfs.TestHDFSContractCreate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12739123/HDFS-8499.05.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b258b34 |
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/11313/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11313/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11313/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11313/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11313/console |
This message was automatically generated.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582625#comment-14582625
]
Zhe Zhang commented on HDFS-8499:
-
All 3 reported failures / timeout pass locally.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582975#comment-14582975
]
Walter Su commented on HDFS-8499:
-
Commonts on 05 patch and Yi's previous comment:
BIContigous and BIUCContiguous has no inheritance relationship, technically. Is
it weird the second one calls the first one's static method? I think 04 patch
is more natural.
You could try either:
1. Use 04 patch. Since the common code is storage operation. Maybe rename
{{BlockInfoContiguousFeature}} to {{ContiguousStorageOp}}.
2. Or Use 05 patch. But put all static method in
{code}
public class BlockInfoContiguous extends BlockInfo {
public static class ContiguousStorageOp {
//here
{code}
3. BlockInfoStripedUC has all the features BlockInfoStriped has. You are just
trying mimic multiple inheritance. I didn't mean it's wrong. I think it's
something unavoidable. We could mimic multiple inheritance based on EC branch.
We just put all common code of two UC classes in {{BlockInfoUCOp}}. So
{{BlockInfoUC}} is more like a class.
{code}
BlockInfo (abstract)
/ \
BlockInfoStriped BlockInfoContiguous
||
| BlockInfoUCOp |
| (static methods) |
| BlockInfoUC |
| (interface) |
| / \ |
BlockInfoStripedUC BlockInfoContiguousUC
{code}
It's not better than the approach done in 04/05 patch. I just think the
inheritance is more clear. In 04/05 patch, BlockInfoContiguous,
BlockInfoStriped, BlockInfoUC those three extends BlockInfo, so they are all
first-class citizen. But BlockInfoUC is different with the others. I think UC
is more like a feature but not entity, like color to apple.
-
It's inappropriate for me to comment on this critical jira. I don't have much
coding experience. Please ignore this if what I said is wrong.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582972#comment-14582972
]
Zhe Zhang commented on HDFS-8499:
-
Thanks for the comment Yi. We certainly can have static methods in
{{BlockInfoStriped}} just as 05 patch. It's just a little ugly, like shown
below.
{code}
static boolean removeStorage(BlockInfo b, DatanodeStorageInfo storage) {
assert b instanceof BlockInfoStriped || b instanceof
BlockInfoUnderConstructionStriped;
byte[] indices;
if (b instanceof BlockInfoStriped) {
indices = ((BlockInfoStriped)b).indices;
} else {
indices = ((BlockInfoUnderConstructionStriped)b).indices;
}
...
}
{code}
So in option #2 we can just add some interfaces, like {{isStriped}},
{{getStripedFeature}}, to make the above code less ugly. It's just an
optimization, not really an option. Sorry for the confusion.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582750#comment-14582750
]
Yi Liu commented on HDFS-8499:
--
Zhe, I don't quite get your point. For Striped, we will have separate static
methods in {{BlockInfoStriped}}, just as we do for {{BlockInfoContiguous}}, so
how we need these 2 choices?
{quote}
Then we face 2 choices in structuring BIStriped and BIUCStriped:
# We can handle them similarly as the 2 contiguous classes. The static methods
will verify the BlockInfo from the argument is instanceof either BIStriped or
BIUCStriped, and so some type casting. This is because the static methods need
to operate on both the triplets and indices.
# We can also explicitly create an interface for BIStriped and BIUCStriped.
I will file a separate JIRA for the 2 striped classes and start writing a patch.
{quote}
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.05.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14580079#comment-14580079
]
Zhe Zhang commented on HDFS-8499:
-
The reported failures and timeout all pass fine locally (they also passed with
03 patch, and 04 patch only adds cosmetic changes).
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14580086#comment-14580086
]
Yi Liu commented on HDFS-8499:
--
Thanks Zhe for the patch, I will take a look at it later today.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578458#comment-14578458
]
Hadoop QA commented on HDFS-8499:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 10s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 10 new or modified test files. |
| {color:green}+1{color} | javac | 9m 11s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 11m 37s | The applied patch generated 3
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 28s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 55s | The applied patch generated 7
new checkstyle issues (total was 1137, now 1121). |
| {color:green}+1{color} | whitespace | 0m 7s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 52s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 43s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 17s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 4m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 87m 57s | Tests failed in hadoop-hdfs. |
| | | 143m 38s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.server.namenode.ha.TestQuotasWithHA |
| Timed out tests | org.apache.hadoop.hdfs.web.TestWebHDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738496/HDFS-8499.04.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fc2ed4a |
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/11284/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11284/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11284/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11284/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11284/console |
This message was automatically generated.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578100#comment-14578100
]
Hadoop QA commented on HDFS-8499:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 27s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 10 new or modified test files. |
| {color:red}-1{color} | javac | 1m 31s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738465/HDFS-8499.UCFeature.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 025a3a8 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11281/console |
This message was automatically generated.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578110#comment-14578110
]
Hadoop QA commented on HDFS-8499:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 14s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 10 new or modified test files. |
| {color:green}+1{color} | javac | 7m 40s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 51s | The applied patch generated 3
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 20s | The applied patch generated
30 new checkstyle issues (total was 1136, now 1143). |
| {color:green}+1{color} | whitespace | 0m 5s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 17s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 19s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 208m 4s | Tests failed in hadoop-hdfs. |
| | | 255m 31s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.TestReadWhileWriting |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation |
| | hadoop.hdfs.server.namenode.ha.TestQuotasWithHA |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestBlockReaderLocalLegacy |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.TestAppendDifferentChecksum |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.TestFileAppend3 |
| |
hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
|
| | hadoop.hdfs.server.namenode.TestSnapshotPathINodes |
| | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
| | hadoop.fs.permission.TestStickyBit |
| | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate |
| | hadoop.hdfs.server.namenode.TestFSImageWithSnapshot |
| | hadoop.hdfs.TestDataTransferProtocol |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestPipelines |
| | hadoop.fs.TestHDFSFileContextMainOperations |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.cli.TestHDFSCLI |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.server.namenode.TestAddBlock |
| | hadoop.hdfs.server.namenode.ha.TestDNFencing |
| | hadoop.hdfs.TestLeaseRecovery |
| Timed out tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
| | org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738413/HDFS-8499.02.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0e80d51 |
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/11279/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11279/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11279/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11279/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578307#comment-14578307
]
Hadoop QA commented on HDFS-8499:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 16s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 10 new or modified test files. |
| {color:green}+1{color} | javac | 7m 46s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 54s | The applied patch generated 3
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 14s | The applied patch generated
30 new checkstyle issues (total was 1136, now 1143). |
| {color:green}+1{color} | whitespace | 0m 6s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 19s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 3s | Tests passed in hadoop-hdfs.
|
| | | 209m 28s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738480/HDFS-8499.03.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fc2ed4a |
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/11282/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11282/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11282/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11282/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11282/console |
This message was automatically generated.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574551#comment-14574551
]
Yi Liu commented on HDFS-8499:
--
Thanks Zhe for updating the patch.
I just noticed one thing:
{code}
boolean addStorage(DatanodeStorageInfo storage) {
+return convertToCompleteBlock().addStorage(storage);
+ }
{code}
We actually have not done anything. Since {{convertToCompleteBlock}} will
create a new instance of {{BlockInfoContinuous}} (use continuous as example),
and {{triplets}} is different, so the modification of the created
{{BlockInfoContinous}} will not affect original instance (BlockInfoUC).
We should find another way...
{quote}
assert getBlockUCState() != HdfsServerConstants.BlockUCState.COMPLETE :
+BlockInfoContiguousUnderConstruction cannot be in COMPLETE state;
{quote}
We'd better to modify the string in the message too. Besides using IDE to
refactor, we'd better to search the String.
In BlockInfoUC, there are some methods should have different implementations
for Continuous and Striped, for example, {{setExpectedLocations}}, currently
the default in BlockInfoUC is for continuous, of course we can override it in
striped. Maybe they should be abstract, and implemented in both continuous and
striped, then it's more clear? So I suggest if the implementation of some
method is different for continuous and striped, then we make them abstract.
{quote}
In BlockInfo, convertToBlockUnderConstruction should be abstract, and
continuous/striped block implement it.
Agreed. Will address in the next rev.
{quote}
Seems you missed this?
Let's also wait for Jing if he has some suggestions. Besides, I want to see
the Jenkins to make sure we didn't miss anything.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574833#comment-14574833
]
Zhe Zhang commented on HDFS-8499:
-
[~hitliuyi] great points! I didn't notice that the copy constructor of
{{BlockInfo}} creates a new empty {{triplets}}. Will look deeper into that.
Somehow Jenkins doesn't like this JIRA.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573187#comment-14573187
]
Zhe Zhang commented on HDFS-8499:
-
Many thanks for Yi and Vinay for the reviews!
bq. In BlockInfoUnderConstruction, we can implement
addStorage/removeStorage/numNodes/replaceBlock, no need to push them to
subclass.
Good point, subclass only need to implement {{convertToCompleteBlock}}
separately.
bq. In BlockInfo, convertToBlockUnderConstruction should be abstract, and
continuous/striped block implement it.
Agreed. Will address in the next rev.
bq. I think these methods you can directly move to BlockInfo itself as non of
these methods are having any difference in subclasses, at least as of now.
Actually {{BlockInfoStriped}} will implement
addStorage/removeStorage/numNodes/replaceBlock differently than
{{BlockInfoContiguous}}. So as shown below, {{BlockInfoStriped}},
{{BlockInfoContiguous}}, and {{BlockInfoUC}} each has a different
{{addStorage}} logic. In particular, {{BlockInfoUC}} just converts itself to a
complete block -- this conversion has different implementations in
{{BlockInfoUCStriped}} and {{BlockInfoUCContiguous}} -- and calls the
corresponding {{addStorage}}.
The {{convertToCompleteBlock()}} logic is necessary because
{{BlockInfoUCStriped}} no longer subclasses {{BlockStriped}} and we needs an
explicit conversion to the right type of complete block.
{code}
BlockInfo (abstract)
/ | \
BlockInfoStriped |BlockInfoContiguous
#addStorage_s|#addStorage_c
|
BlockInfoUC(abstract)
#addStorage_uc =
convertToCompleteBlock().addStorage
/\
BlockInfoStripedUC BlockInfoContiguousUC
#convertToCompleteBlock_s #convertToCompleteBlock_c
{code}
Right now the striped classes are not in, so yes, we can put these methods in
the superclass and move them later. But I guess it's easier to put them in the
right subclasses so we can catch incoming changes during the merge process?
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572262#comment-14572262
]
Vinayakumar B commented on HDFS-8499:
-
Thanks [~zhz] for the work. I hope this will resolve lot of merge conflicts in
HDFS-7285 branch :)
Overall hierarchy looks good to me as well.
bq. In BlockInfoUnderConstruction, we can implement
addStorage/removeStorage/numNodes/replaceBlock, no need to push them to
subclass.
I think these methods you can directly move to BlockInfo itself as non of these
methods are having any difference in subclasses, at least as of now. And there
is no need to call {{convertToCompleteBlock()}} as well. So I feel it can be
moved to {{BlockInfo}} itself.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572001#comment-14572001
]
Yi Liu commented on HDFS-8499:
--
Thanks Zhe for the work, the interfaces/classes hierarchy looks good overall.
In {{BlockInfoUnderConstruction}}, we can implement
{{addStorage/removeStorage/numNodes/replaceBlock}}, no need to push them to
subclass.
In {{BlockInfo}}, {{convertToBlockUnderConstruction}} should be abstract, and
continuous/striped block implement it.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571457#comment-14571457
]
Zhe Zhang commented on HDFS-8499:
-
Casting from {{BI}} to {{BIUC}} isn't actually an issue because the
{{instanceof}} check is always used before casting.
In the next rev I plan to convert some {{BIUC}} methods as abstract and move
the implementation to {{BIUCCongtiguous}}, based on whether the logic differs
in contiguous and striped subclasses. [~jingzhao] I wonder if we should unify
{{replicas}} in both subclasses to either list or array?
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568668#comment-14568668
]
Zhe Zhang commented on HDFS-8499:
-
In HDFS-7285 branch {{BlockInfoContiguousUnderConstruction}} and
{{BlockInfoStripedUnderConstruction}} share much of their internal code. There
is also some code dup in using these 2 classes. For example,
{{BlockManager#createLocatedBlock}}:
{code}
if (blk.isStriped()) {
final BlockInfoStripedUnderConstruction uc =
(BlockInfoStripedUnderConstruction) blk;
final DatanodeStorageInfo[] storages = uc.getExpectedStorageLocations();
final ExtendedBlock eb = new ExtendedBlock(namesystem.getBlockPoolId(),
blk);
return newLocatedStripedBlock(eb, storages, uc.getBlockIndices(), pos,
false);
} else {
assert blk instanceof BlockInfoContiguousUnderConstruction;
final BlockInfoContiguousUnderConstruction uc =
(BlockInfoContiguousUnderConstruction) blk;
final DatanodeStorageInfo[] storages = uc.getExpectedStorageLocations();
final ExtendedBlock eb = new ExtendedBlock(namesystem.getBlockPoolId(),
blk);
return newLocatedBlock(eb, storages, pos, false);
}
{code}
Maybe we can take this chance and optimize the inheritance structure (the idea
was discussed in a meetup with [~jingzhao] and [~drankye]). My current plan is
to explore the following structure:
{code}
BlockInfo (abstract)
/ | \
BlockInfoStriped |BlockInfoContiguous
|
BlockInfoUC(abstract)
/ \
BlockInfoStripedUC BlockInfoContiguousUC
{code}
To support this structure, we can provide static methods in the uppermost
{{BlockInfo}} class to explicitly convert between {{BlockInfoStriped}} and
{{BlockInfoStripedUC}} (similarly for contiguous), instead of relying on
downcasting.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Zhe Zhang
Assignee: Zhe Zhang
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
