[
https://issues.apache.org/jira/browse/HDFS-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-10999:
--------------------------------------
Attachment: HDFS-10999.01.patch
Based on previous discussions, attaching v01 patch to address the following.
* {{ClientProtocol}} extended to support {{ReplicatedBlocksStats
getReplicatedBlocksStats()}} and {{ECBlockGroupsStats getECBlockGroupsStats()}}
apart from the current {{long[] getStats()}}
* Introduced new MBeans {{ECBlockGroupsStatsMBean}} and
{{ReplicatedBlocksStatsMBean}} for the consumers like {{DFSAdmin}}, WebUI.
* {{FSNamesystemMBean}} will continue to carry aggregated stats combining both
replicates and ec block stats. Since these are aggregated stats now, deprecated
the methods to use proper naming.
* {{FSNamesystem}} now implements {{ECBlockGroupsStatsMBean}} and
{{ReplicatedBlocksStatsMBean}} apart from the already implemented
{{FSNamesystemMBean}}
* {{BlockMAnager}} changes to expose the stats specific to Replicated and EC
Blocks
* {{LowRedundancyBlocks}}, {{CorruptReplicasMap}} and {{InvalidateBlocks}}
updated to track Replicated and EC Blocks separately using LongAccumulators.
Already existing aggregate blocks tracking size() methods are not altered for
backward compatibility.
* {{PBHelperClient}}, {{ClientNamenodeProtocolTranslatorPB}},
{{ClientNamenodeProtocolServerSideTranslatorPB}} are updated to plumb in the
new ClientProtocol services.
* {{ClientNameNodeProtocol.proto}} updated to define the new ClientProtocol
services proto buf messages.
* {{TestNameNodeMetrics}}, {{TestUnderReplicatedBlocks}} are updated to verify
the new stats. Several other tests are updated to verify the needed block
counts.
* PS: {{DfsAdmin -report}} and WebUI are not updated to make use of the newer
infrastructure. Probably after we finalize on this infra, I can take up the
consumers separately in a new jira.
[~andrew.wang], [~tasanuma0829], [~jojochuang], can you please take a look at
the attached patch ?
> Introduce separate stats for Replicated and Erasure Coded Blocks apart from
> the current Aggregated stats
> --------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10999
> URL: https://issues.apache.org/jira/browse/HDFS-10999
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
> Affects Versions: 3.0.0-alpha1
> Reporter: Wei-Chiu Chuang
> Assignee: Manoj Govindassamy
> Labels: hdfs-ec-3.0-nice-to-have, supportability
> Attachments: HDFS-10999.01.patch
>
>
> Per HDFS-9857, it seems in the Hadoop 3 world, people prefer the more generic
> term "low redundancy" to the old-fashioned "under replicated". But this term
> is still being used in messages in several places, such as web ui, dfsadmin
> and fsck. We should probably change them to avoid confusion.
> File this jira to discuss it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
