[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[
https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8728:
Attachment: HDFS-8728-HDFS-7285.02.patch
Uploading branch-based {{HDFS-8728-HDFS-7285.02.patch}} to address some of
Andrew's comments:
# Since the patch is already large, will leave the ideas of {{getOp}} and
saving reference in {{StripedBlockStorageOp}} as follow-ons.
# I filed HDFS-8784 to rename {{numNodes}}.
This patch basically makes necessary changes to merge trunk's {{BlockInfo}}
hierarchy back to HDFS-7285 branch (as well as adding the striped
counterparts). If we agree upon this direction I will create another patch to
replace all unnecessary usages of {{BIC}}, {{BIS}}, {{BIUCC}}, {{BIUCS}} with
{{BlockInfo}} and {{BlockInfoUC}}.
After reaching a conclusion here, I plan to update {{Merge-1}} to {{Merge-14}}
patches accordingly, and then rebase the HDFS-7285 branch to catch up with
trunk.
[~jingzhao] Could you share some advice here? Thanks!
Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
---
Key: HDFS-8728
URL: https://issues.apache.org/jira/browse/HDFS-8728
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8728-HDFS-7285.00.patch,
HDFS-8728-HDFS-7285.01.patch, HDFS-8728-HDFS-7285.02.patch,
HDFS-8728.00.patch, HDFS-8728.01.patch, HDFS-8728.02.patch,
Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch,
Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch,
Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch,
Merge-8-inodeFile.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8728: Status: Open (was: Patch Available) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile --- Key: HDFS-8728 URL: https://issues.apache.org/jira/browse/HDFS-8728 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: HDFS-8728-HDFS-7285.00.patch, HDFS-8728-HDFS-7285.01.patch, HDFS-8728-HDFS-7285.02.patch, HDFS-8728.00.patch, HDFS-8728.01.patch, HDFS-8728.02.patch, Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[
https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8728:
Attachment: HDFS-8728-HDFS-7285.03.patch
Thanks Jing for reviewing HDFS-8787 patch!
Uploading patch 03 (which requires HDFS-8787). It is still large but a major
portion is code refactoring around {{BlockInfo}} classes.
Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
---
Key: HDFS-8728
URL: https://issues.apache.org/jira/browse/HDFS-8728
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8728-HDFS-7285.00.patch,
HDFS-8728-HDFS-7285.01.patch, HDFS-8728-HDFS-7285.02.patch,
HDFS-8728-HDFS-7285.03.patch, HDFS-8728.00.patch, HDFS-8728.01.patch,
HDFS-8728.02.patch, Merge-1-codec.patch, Merge-2-ecZones.patch,
Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch,
Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch,
Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8728: Attachment: HDFS-8728-HDFS-7285.01.patch Forgot that the .00 patch requires HDFS-8058 patch. Uploading .01 patch which includes HDFS-8058 patch. Erasure coding: revisit and simplify BlockInfoStriped and INodeFile --- Key: HDFS-8728 URL: https://issues.apache.org/jira/browse/HDFS-8728 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: HDFS-8728-HDFS-7285.00.patch, HDFS-8728-HDFS-7285.01.patch, HDFS-8728.00.patch, HDFS-8728.01.patch, HDFS-8728.02.patch, Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[
https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8728:
Attachment: HDFS-8728-HDFS-7285.00.patch
Updating {{HDFS-8728-HDFS-7285.00.patch}} which changes branch code to be
compatible with the trunk version of {{BlockInfo}} hierarchy.
Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
---
Key: HDFS-8728
URL: https://issues.apache.org/jira/browse/HDFS-8728
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8728-HDFS-7285.00.patch, HDFS-8728.00.patch,
HDFS-8728.01.patch, HDFS-8728.02.patch, Merge-1-codec.patch,
Merge-2-ecZones.patch, Merge-3-blockInfo.patch,
Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch,
Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch,
Merge-8-inodeFile.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8728: Summary: Erasure coding: revisit and simplify BlockInfoStriped and INodeFile (was: Erasure coding: revisit BlockInfoStriped based on code hierarchy from HDFS-8499) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile --- Key: HDFS-8728 URL: https://issues.apache.org/jira/browse/HDFS-8728 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[
https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8728:
Attachment: Merge-8-inodeFile.patch
Merge-7-replicationMonitor.patch
Merge-6-locatedStripedBlock.patch
Merge-5-blockPlacementPolicies.patch
Merge-4-blockmanagement.patch
Merge-3-blockInfo.patch
Merge-2-ecZones.patch
Merge-1-codec.patch
Attaching sub-patches 1~8 from the consolidated HDFS-7285 patch as background
of this proposed change. Note that they are trunk-based.
The root of this proposed change is the new {{BlockInfo}} hierarchy introduced
by HDFS-8499. Based on that, we are able to simplify striped blocks handling in
several perspectives:
# {{BlockUCContiguous}} and {{BlockInfoUCStriped}} can now share the UC logic.
This can be seen in {{Merge-3-blockInfo.patch}}. That sub-patch also separates
out a new {{StripedBlockStorageOp}} class to handle striping-specific logic.
# By keeping {{BlockInfoUC}} as a subclass of {{BlockInfo}} (instead of a
separate interface as in the branch), we can avoid changing trunk code that
relies on this relationship. This can be seen in
{{Merge-4-blockmanagement.patch}} (Allocate and manage striped blocks in
NameNode blockmanagement module.). With this change and other pre-merge
efforts (HDFS-8487, HDFS-8608, HDFS-8623), the [rebased consolidated patch |
https://issues.apache.org/jira/secure/attachment/12744101/Consolidated-20150707.patch]
has much fewer intrusive changes to {{blockmanagement}} and {{namenode}}
modules than the current HDFS-7285 consolidated patch:
{code}
Current HDFS-7285:
2532 insertions(+), 1156 deletions(-) in blockmanagement
1826 insertions(+), 444 deletions(-) in namenode
Rebased (including this proposed simplification):
1251 insertions(+), 201 deletions(-) in blockmanagement
1324 insertions(+), 168 deletions(-) in namenode
{code}
# As we [discussed |
https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14600362page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14600362]
under HDFS-7285, this new {{BlockInfo}} hierarchy also requires us to unify
contiguous and striped blocks handling along the direction of HDFS-8058. I
think we should also take this chance to reach a conclusion on this issue. My
current thought is that we can take the HDFS-8058 approach to avoid duplicating
code, and thoroughly address type safety as a follow-on (some thoughts can be
found on HDFS-8655).
The attached sub-patches 1, 2, 5, 6, 7 are mostly the same as the branch; 3, 4,
8 contain the proposed simplification and should be more thoroughly reviewed
([~andrew.wang]: we went over these ideas in offline discussions; could you
take a look at the patches?). Per Jing's suggestion I'm also working on
generating a patch against the current HDFS-7285 branch.
Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
---
Key: HDFS-8728
URL: https://issues.apache.org/jira/browse/HDFS-8728
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: Merge-1-codec.patch, Merge-2-ecZones.patch,
Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch,
Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch,
Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8728: Status: Patch Available (was: Open) Submitting consolidated patches 1~8 to trigger Jenkins. Erasure coding: revisit and simplify BlockInfoStriped and INodeFile --- Key: HDFS-8728 URL: https://issues.apache.org/jira/browse/HDFS-8728 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8728: Attachment: HDFS-8728.00.patch Erasure coding: revisit and simplify BlockInfoStriped and INodeFile --- Key: HDFS-8728 URL: https://issues.apache.org/jira/browse/HDFS-8728 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: HDFS-8728.00.patch, Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8728: Attachment: HDFS-8728.01.patch Rebased patch. Erasure coding: revisit and simplify BlockInfoStriped and INodeFile --- Key: HDFS-8728 URL: https://issues.apache.org/jira/browse/HDFS-8728 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: HDFS-8728.00.patch, HDFS-8728.01.patch, Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8728: Attachment: (was: HDFS-8728.02.patch) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile --- Key: HDFS-8728 URL: https://issues.apache.org/jira/browse/HDFS-8728 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: HDFS-8728.00.patch, HDFS-8728.01.patch, Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[
https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8728:
Attachment: HDFS-8728.02.patch
Updating patch to fix test failures. Most failures are due to patch splitting:
# The {{updateBlockForPipeline}} to {{bumpBlockGenerationStamp}} change should
be moved to client support patch.
# {{TestDFSUpgradeFromImage}} will pass in the final consolidated patch and I
haven't figured out which change fixes it.
Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
---
Key: HDFS-8728
URL: https://issues.apache.org/jira/browse/HDFS-8728
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8728.00.patch, HDFS-8728.01.patch,
HDFS-8728.02.patch, Merge-1-codec.patch, Merge-2-ecZones.patch,
Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch,
Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch,
Merge-7-replicationMonitor.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8728: Attachment: Merge-8-inodeFile.patch Erasure coding: revisit and simplify BlockInfoStriped and INodeFile --- Key: HDFS-8728 URL: https://issues.apache.org/jira/browse/HDFS-8728 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: HDFS-8728.00.patch, HDFS-8728.01.patch, HDFS-8728.02.patch, Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8728: Attachment: (was: Merge-8-inodeFile.patch) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile --- Key: HDFS-8728 URL: https://issues.apache.org/jira/browse/HDFS-8728 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: HDFS-8728.00.patch, HDFS-8728.01.patch, Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8728: Attachment: HDFS-8728.02.patch Erasure coding: revisit and simplify BlockInfoStriped and INodeFile --- Key: HDFS-8728 URL: https://issues.apache.org/jira/browse/HDFS-8728 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: HDFS-8728.00.patch, HDFS-8728.01.patch, HDFS-8728.02.patch, Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8728: Status: Open (was: Patch Available) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile --- Key: HDFS-8728 URL: https://issues.apache.org/jira/browse/HDFS-8728 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: HDFS-8728.00.patch, HDFS-8728.01.patch, Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
[ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8728: Status: Patch Available (was: Open) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile --- Key: HDFS-8728 URL: https://issues.apache.org/jira/browse/HDFS-8728 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: HDFS-8728.00.patch, HDFS-8728.01.patch, Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch, Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch, Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
