[jira] [Updated] (HDFS-7369) Erasure coding: distribute block recovery work to DataNode
[
https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7369:
Attachment: (was: HDFS-7396-000.patch)
Erasure coding: distribute block recovery work to DataNode
--
Key: HDFS-7369
URL: https://issues.apache.org/jira/browse/HDFS-7369
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-7369-000-part1.patch, HDFS-7369-000-part2.patch
This JIRA updates NameNode to handle background / offline recovery of erasure
coded blocks. It includes 2 parts:
# Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to
appropriate priority levels.
# Update {{ReplicationMonitor}} to distinguish block codec tasks and send a
new DataNode command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7369) Erasure coding: distribute block recovery work to DataNode
[
https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7369:
Attachment: HDFS-7369-002.patch
Erasure coding: distribute block recovery work to DataNode
--
Key: HDFS-7369
URL: https://issues.apache.org/jira/browse/HDFS-7369
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-7369-000-part1.patch, HDFS-7369-000-part2.patch,
HDFS-7369-001.patch, HDFS-7369-002.patch
This JIRA updates NameNode to handle background / offline recovery of erasure
coded blocks. It includes 2 parts:
# Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to
appropriate priority levels.
# Update {{ReplicationMonitor}} to distinguish block codec tasks and send a
new DataNode command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7369) Erasure coding: distribute block recovery work to DataNode
[
https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7369:
Attachment: HDFS-7369-000-part1.patch
Erasure coding: distribute block recovery work to DataNode
--
Key: HDFS-7369
URL: https://issues.apache.org/jira/browse/HDFS-7369
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-7369-000-part1.patch, HDFS-7369-000-part2.patch
This JIRA updates NameNode to handle background / offline recovery of erasure
coded blocks. It includes 2 parts:
# Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to
appropriate priority levels.
# Update {{ReplicationMonitor}} to distinguish block codec tasks and send a
new DataNode command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7369) Erasure coding: distribute block recovery work to DataNode
[
https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7369:
Attachment: HDFS-7369-001.patch
The main change in this patch is to extend existing {{ReplicationWork}} class
in {{BlockManager}}
# Create a new parent class {{BlockRecoveryWork}} to abstract the common
properties of replication- and EC-based recovery task
# Create a subclass {{ErasureCodingWork}} with EC-specific logic
A re-replication task works in a pushing mode, with 1 source (pusher) and
multiple targets. A EC-based recovery task works in a pull mode, with multiple
sources and 1 target (puller). Therefore the {{BlockRecoveryWork}} has both
sources and targets as arrays. {{ReplicationWork}} and {{ErasureCodingWork}}
picks the first element from the corresponding array as their source/target. An
{{ErasureCodingWork}} should also record the index of the missing block, which
is a TODO to be added in the next rev or another JIRA.
Erasure coding: distribute block recovery work to DataNode
--
Key: HDFS-7369
URL: https://issues.apache.org/jira/browse/HDFS-7369
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-7369-000-part1.patch, HDFS-7369-000-part2.patch,
HDFS-7369-001.patch
This JIRA updates NameNode to handle background / offline recovery of erasure
coded blocks. It includes 2 parts:
# Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to
appropriate priority levels.
# Update {{ReplicationMonitor}} to distinguish block codec tasks and send a
new DataNode command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7369) Erasure coding: distribute block recovery work to DataNode
[
https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7369:
Description:
This JIRA updates NameNode to handle background / offline recovery of erasure
coded blocks. It includes 2 parts:
# Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to
appropriate priority levels.
# Update {{ReplicationMonitor}} to distinguish block codec tasks and send a new
DataNode command.
was:{{ErasureCodingBlocks}} on NameNode should insert parity blocks into
{{UnderReplicatedBlocks}}. Then {{ReplicationMonitor}} should be extended to
distinguish block encoding tasks and send a new DataNode command.
Erasure coding: distribute block recovery work to DataNode
--
Key: HDFS-7369
URL: https://issues.apache.org/jira/browse/HDFS-7369
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
This JIRA updates NameNode to handle background / offline recovery of erasure
coded blocks. It includes 2 parts:
# Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to
appropriate priority levels.
# Update {{ReplicationMonitor}} to distinguish block codec tasks and send a
new DataNode command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7369) Erasure coding: distribute block recovery work to DataNode
[
https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7369:
Attachment: HDFS-7396-000.patch
A very simple patch for step 1 in the JIRA description. Ultimately we want to
accurately assess the _risk level_ of a block group when some of its EC blocks
are lost. This patch is the first order approximation:
# A block group with _m_ data blocks and _k_ parity blocks is equivalent to a
contiguous block with replication factor _k+1_, because both can tolerate _k_
failures
# If _n_ blocks are healthy among the _m+k_ blocks, it's equivalent to having
_n-m+1_ healthy replicas, calculated from _(k+1) - ((m+k) - n)_, or
_replicationFactor - numLostReplicas_.
Erasure coding: distribute block recovery work to DataNode
--
Key: HDFS-7369
URL: https://issues.apache.org/jira/browse/HDFS-7369
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-7396-000.patch
This JIRA updates NameNode to handle background / offline recovery of erasure
coded blocks. It includes 2 parts:
# Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to
appropriate priority levels.
# Update {{ReplicationMonitor}} to distinguish block codec tasks and send a
new DataNode command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7369) Erasure coding: distribute block recovery work to DataNode
[
https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7369:
Attachment: HDFS-7369-000-part2.patch
This is a very rough patch for step #2 in the JIRA description.
Erasure coding: distribute block recovery work to DataNode
--
Key: HDFS-7369
URL: https://issues.apache.org/jira/browse/HDFS-7369
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-7369-000-part2.patch, HDFS-7396-000.patch
This JIRA updates NameNode to handle background / offline recovery of erasure
coded blocks. It includes 2 parts:
# Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to
appropriate priority levels.
# Update {{ReplicationMonitor}} to distinguish block codec tasks and send a
new DataNode command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7369) Erasure coding: distribute block recovery work to DataNode
[
https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7369:
Summary: Erasure coding: distribute block recovery work to DataNode (was:
Distribute parity block encoding work to DataNode via heartbeat)
Erasure coding: distribute block recovery work to DataNode
--
Key: HDFS-7369
URL: https://issues.apache.org/jira/browse/HDFS-7369
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
{{ErasureCodingBlocks}} on NameNode should insert parity blocks into
{{UnderReplicatedBlocks}}. Then {{ReplicationMonitor}} should be extended to
distinguish block encoding tasks and send a new DataNode command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
