[
https://issues.apache.org/jira/browse/HADOOP-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684625#comment-17684625
]
Steve Loughran commented on HADOOP-18617:
-----------------------------------------
actually, got my plan wrong
* new interface in o.a.h.fs.statistics
* leave all existing stuff in the same place as before
* but implement the new interface
* new builder builds the right thing. which is always a bit tricky in builder
apis; we would have to wrap the impl builder
> Make IOStatisticsStore and binding APIs public for use beyond our code
> ----------------------------------------------------------------------
>
> Key: HADOOP-18617
> URL: https://issues.apache.org/jira/browse/HADOOP-18617
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
> Affects Versions: 3.3.5
> Reporter: Steve Loughran
> Priority: Major
>
> it's really useful to be able to collect iostats in things other than the FS
> classes -we do it in the S3A and manifest committers.
> But external code -such as the spark committers can't use the methods in
> {{org.apache.hadoop.fs.statistics.impl))
> Proposed
> Make some classes/interfaces public
> * IOStatisticsBinding
> * IOStatisticsStore
> * IOStatisticsStoreBuilder
> Ideally we should actually move the IOStatisticsStore interface into
> org.apache.hadoop.fs.statistics and the builder to match -but we can't do
> that without causing trauma elsewhere (google gcs).
> Strategy there: Add a new interface IOStatisticsCollector in .impl which is
> then implemented by IOStatisticsStore, and a new builder API which forwards
> to IOStatisticsStoreBuilder.
> Side issue: we don't make any use of the "clever, elegant functional" bit of
> DynamicIOStatisticsBuilder/DynamicIOStatistics, where every counter is mapped
> to a function which is then invoked to get at the atomic longs. It's used in
> IOStatisticsStoreImpl, but only with AtomicLong and MeanStatistic instances.
> If we just move to simple maps we will save on lambda-expressions and on
> lookup overhead. The original intent was something like coda hale metrics
> where we could add dynamic lookup to other bits of instrumentation; in
> practise we measure durations and build counts/min/max.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
