[
https://issues.apache.org/jira/browse/HDFS-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15232824#comment-15232824
]
Ming Ma commented on HDFS-10175:
--------------------------------
Interesting work.
* Seems like a useful feature to provide per-job counters for this. Note that
NN also has the data, except nntop or audit log don't track them on per
DFSClient client name basis. For the HDFS-9579 scenario, besides the per-job
counters functionality, another motivation is the extra work required to track
and aggregate on DN side (something we still want to experiment to track hot
block, or remote read or write for all HDFS apps, not just MR).
* Question about using inline long or map object. The original patch in
HDFS-9579 used map as I tried to make it more dynamic in two aspects; no need
to hard code network distance or memory allocation for FileSystems that don't
generate such data. But the small extra memory for static longs and mostly
static network distance definition make map approach less interesting.
* Growth of the metrics. It seems each time we add a RPC method, we need to add
another entry. Also if someone wants to track cross-DC mkdir and intra-DC
mkdir, then you added another dimension to it.
* Making it optional is a good idea.
> add per-operation stats to FileSystem.Statistics
> ------------------------------------------------
>
> Key: HDFS-10175
> URL: https://issues.apache.org/jira/browse/HDFS-10175
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Reporter: Ram Venkatesh
> Assignee: Mingliang Liu
> Attachments: HDFS-10175.000.patch, HDFS-10175.001.patch,
> HDFS-10175.002.patch, HDFS-10175.003.patch, TestStatisticsOverhead.java
>
>
> Currently FileSystem.Statistics exposes the following statistics:
> BytesRead
> BytesWritten
> ReadOps
> LargeReadOps
> WriteOps
> These are in-turn exposed as job counters by MapReduce and other frameworks.
> There is logic within DfsClient to map operations to these counters that can
> be confusing, for instance, mkdirs counts as a writeOp.
> Proposed enhancement:
> Add a statistic for each DfsClient operation including create, append,
> createSymlink, delete, exists, mkdirs, rename and expose them as new
> properties on the Statistics object. The operation-specific counters can be
> used for analyzing the load imposed by a particular job on HDFS.
> For example, we can use them to identify jobs that end up creating a large
> number of files.
> Once this information is available in the Statistics object, the app
> frameworks like MapReduce can expose them as additional counters to be
> aggregated and recorded as part of job summary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
