[
https://issues.apache.org/jira/browse/HDFS-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259851#comment-15259851
]
Steve Loughran commented on HDFS-10175:
---------------------------------------
bq. One thing that is a bit concerning about metrics2 is that I think people
feel that this interface should be stable (i.e. don't remove or alter things
once they're in), which would be a big constraint on us.
Ops teams don't like metrics they rely on being taken away; they also view
published metrics as the API. As the compatibility docs say "Metrics should
preserve compatibility within the major release."
bq. Perhaps we could document that per-fs stats were @Public @Evolving rather
than stable
+1 to that, though it'll be important not to break binary compatibility with
external filesystems.
FWIW, issues I have with metrics2, apart from "steve doesn't understand the
design fully" is its preference for singletons registered with JMX. This makes
sense in deployed services, not for tests.
bq. Do we have any ideas about how Spark will consume these metrics in the
longer term?
Spark ia a Coda Hale instrumented codebase, as are many other apps these days.
Integration between hadoop metrics of any form and the Coda Hale libraries
would be something to address, but not here.
> add per-operation stats to FileSystem.Statistics
> ------------------------------------------------
>
> Key: HDFS-10175
> URL: https://issues.apache.org/jira/browse/HDFS-10175
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Reporter: Ram Venkatesh
> Assignee: Mingliang Liu
> Attachments: HDFS-10175.000.patch, HDFS-10175.001.patch,
> HDFS-10175.002.patch, HDFS-10175.003.patch, HDFS-10175.004.patch,
> HDFS-10175.005.patch, HDFS-10175.006.patch, TestStatisticsOverhead.java
>
>
> Currently FileSystem.Statistics exposes the following statistics:
> BytesRead
> BytesWritten
> ReadOps
> LargeReadOps
> WriteOps
> These are in-turn exposed as job counters by MapReduce and other frameworks.
> There is logic within DfsClient to map operations to these counters that can
> be confusing, for instance, mkdirs counts as a writeOp.
> Proposed enhancement:
> Add a statistic for each DfsClient operation including create, append,
> createSymlink, delete, exists, mkdirs, rename and expose them as new
> properties on the Statistics object. The operation-specific counters can be
> used for analyzing the load imposed by a particular job on HDFS.
> For example, we can use them to identify jobs that end up creating a large
> number of files.
> Once this information is available in the Statistics object, the app
> frameworks like MapReduce can expose them as additional counters to be
> aggregated and recorded as part of job summary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
