Steve Loughran created HADOOP-18617:
---------------------------------------
Summary: Make IOStatisticsStore and binding APIs public for use
beyond our code
Key: HADOOP-18617
URL: https://issues.apache.org/jira/browse/HADOOP-18617
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 3.3.5
Reporter: Steve Loughran
it's really useful to be able to collect iostats in things other than the FS
classes -we do it in the S3A and manifest committers.
But external code -such as the spark committers can't use the methods in
{{org.apache.hadoop.fs.statistics.impl))
Proposed
Make some classes/interfaces public
* IOStatisticsBinding
* IOStatisticsStore
* IOStatisticsStoreBuilder
Ideally we should actually move the IOStatisticsStore interface into
org.apache.hadoop.fs.statistics and the builder to match -but we can't do that
without causing trauma elsewhere (google gcs).
Strategy there: Add a new interface IOStatisticsCollector in .impl which is
then implemented by IOStatisticsStore, and a new builder API which forwards to
IOStatisticsStoreBuilder.
Side issue: we don't make any use of the "clever, elegant functional" bit of
DynamicIOStatisticsBuilder/DynamicIOStatistics, where every counter is mapped
to a function which is then invoked to get at the atomic longs. It's used in
IOStatisticsStoreImpl, but only with AtomicLong and MeanStatistic instances. If
we just move to simple maps we will save on lambda-expressions and on lookup
overhead. The original intent was something like coda hale metrics where we
could add dynamic lookup to other bits of instrumentation; in practise we
measure durations and build counts/min/max.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
