[
https://issues.apache.org/jira/browse/HDFS-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258726#comment-15258726
]
Colin Patrick McCabe commented on HDFS-10175:
---------------------------------------------
bq. I see that, but stream-level counters are essential at least for the tests
which verify forward and lazy seeks. Which means that yes, they do have to go
into the 2.8.0 release. What I can do is set up the scope so that they are
package private, then, in the test code, implement the assertions about
metric-derived state into that package.
I guess my hope here is that whatever mechanism we come up with is something
that could easily be integrated into the upcoming 2.8 release. Since we have
talked about requiring our new metrics to not modify existing stable public
interfaces, that seems very reasonable.
One thing that is a bit concerning about metrics2 is that I think people feel
that this interface should be stable (i.e. don't remove or alter things once
they're in), which would be a big constraint on us. Perhaps we could document
that per-fs stats were \@Public \@Evolving rather than stable?
bq. Regarding the metrics2 instrumentation in HADOOP-13028, I'm aggregating the
stream statistics back into the metrics 2 data. That's something which isn't
needed for the hadoop tests, but which I'm logging in spark test runs, such as
(formatted for readability):
Do we have any ideas about how Spark will consume these metrics in the longer
term? Do they prefer to go through metrics2, for example? I definitely don't
object to putting this kind of stuff in metrics2, but if we go that route, we
have to accept that we'll just get global (or at best per-fs-type) statistics,
rather than per-fs-instance statistics. Is that acceptable? So far, nobody
has spoken up strongly in favor of per-fs-instance statistics.
> add per-operation stats to FileSystem.Statistics
> ------------------------------------------------
>
> Key: HDFS-10175
> URL: https://issues.apache.org/jira/browse/HDFS-10175
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Reporter: Ram Venkatesh
> Assignee: Mingliang Liu
> Attachments: HDFS-10175.000.patch, HDFS-10175.001.patch,
> HDFS-10175.002.patch, HDFS-10175.003.patch, HDFS-10175.004.patch,
> HDFS-10175.005.patch, HDFS-10175.006.patch, TestStatisticsOverhead.java
>
>
> Currently FileSystem.Statistics exposes the following statistics:
> BytesRead
> BytesWritten
> ReadOps
> LargeReadOps
> WriteOps
> These are in-turn exposed as job counters by MapReduce and other frameworks.
> There is logic within DfsClient to map operations to these counters that can
> be confusing, for instance, mkdirs counts as a writeOp.
> Proposed enhancement:
> Add a statistic for each DfsClient operation including create, append,
> createSymlink, delete, exists, mkdirs, rename and expose them as new
> properties on the Statistics object. The operation-specific counters can be
> used for analyzing the load imposed by a particular job on HDFS.
> For example, we can use them to identify jobs that end up creating a large
> number of files.
> Once this information is available in the Statistics object, the app
> frameworks like MapReduce can expose them as additional counters to be
> aggregated and recorded as part of job summary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
