Andrew Wang commented on HDFS-10999:

bq. Are we exposing how many blocks are EC blocks and how many blocks are 
normally replicated blocks?

I don't think so, and you're right that we should in some fashion. Sounds like 
we want these counts both for the whole filesystem as well as recovery-related 

As a first-cut, I think these counts can ignore the EC policy. I think most 
clusters will only use a single EC policy since it heavily depends on the # of 
racks. We can expand this to per-policy metrics if we find it necessary.

bq. Most of the advanced admins I know do it several times a year, either 
because the NN was too stupid to fix it's own replication problems and/or 
because it was simply faster for us to do it rather than wait for the normal 
block replication process.

I choose to interpret this as HDFS needing better knobs for emergency 
replication :)

This has been great info, I'll file some JIRAs to track these work items. 
Sounds like:

* I/O based pending replication metrics / throttles
* EC block counts
* dynamically configurable replication throttles

> Use more generic "low redundancy" blocks instead of "under replicated" blocks
> -----------------------------------------------------------------------------
>                 Key: HDFS-10999
>                 URL: https://issues.apache.org/jira/browse/HDFS-10999
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: erasure-coding
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Wei-Chiu Chuang
>            Assignee: Yuanbo Liu
>              Labels: supportability
> Per HDFS-9857, it seems in the Hadoop 3 world, people prefer the more generic 
> term "low redundancy" to the old-fashioned "under replicated". But this term 
> is still being used in messages in several places, such as web ui, dfsadmin 
> and fsck. We should probably change them to avoid confusion.
> File this jira to discuss it.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to