[
https://issues.apache.org/jira/browse/HDFS-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-10999:
--------------------------------------
Attachment: HDFS-10999.04.patch
Thanks [~tasanuma0829] and [~andrew.wang] for the detailed review comments.
Attaching patch v04 to address the following. Can you please take a look?
Takanobu:
bq. For unit tests: I think it would be good if we add more unit tests for
these changes in this jira or follow-on jiras. Add more validations for new
metrics in TestComputeInvalidateWork, TestCorruptReplicaInfo and
TestLowRedundancyBlockQueues.
Done. Added new unit tests to verify stats added in {{CorruptRepicasMap}},
{{InvalidateBlocks}}, {{LowRedundancyBlocks}}
bq. TestUnderReplicatedBlocks covers only replicated files. If we use
DFSTestUtil#verifyClientStats in TestReconstructStripedBlocks, we may be able
to cover the EC case.
Done.
Andrew:
bq. Regarding "CorruptReplicaOne" and similar, the full name is "corrupt
replication factor one". Abbreviating it as "CorruptReplicaOne" is a bit
ambiguous since "replica" has its own meaning. I'd prefer we name all these as
"CorruptReplOne" or "CorruptReplicationOne" instead.
Done.
bq. BlocksStats#toString and ECBlockGroupsStats, optional, but should the names
be expanded to match the field names? In terms of code formatting, it'd also be
clearer if the key and value were on the same line.
Done.
bq. Need javadoc on methods in ECBlockGroupsStatsMBean and
ReplicatedBlocksStatsMBean. Javadoc on aggregate methods in FSNamesystemMBean
should make it clear that they are aggregates too.
Done.
bq. Looks like some of my testing comments from last time were not addressed.
Do you prefer to handle this in a follow-on JIRA? If so, appreciate if you
could file that and link it here. Still a bit concerned about the TODO in
TestNameNodeMetrics.
Fixed the tests in TestNameNodeMetrics and the TODOs are removed now. Also,
added bunch of new unit tests under other Tests to cover all the newly added
stats. If at all more need to be added, I can take it up in new jiras.
> Introduce separate stats for Replicated and Erasure Coded Blocks apart from
> the current Aggregated stats
> --------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10999
> URL: https://issues.apache.org/jira/browse/HDFS-10999
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
> Affects Versions: 3.0.0-alpha1
> Reporter: Wei-Chiu Chuang
> Assignee: Manoj Govindassamy
> Labels: hdfs-ec-3.0-nice-to-have, supportability
> Attachments: HDFS-10999.01.patch, HDFS-10999.02.patch,
> HDFS-10999.03.patch, HDFS-10999.04.patch
>
>
> Per HDFS-9857, it seems in the Hadoop 3 world, people prefer the more generic
> term "low redundancy" to the old-fashioned "under replicated". But this term
> is still being used in messages in several places, such as web ui, dfsadmin
> and fsck. We should probably change them to avoid confusion.
> File this jira to discuss it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
