steveloughran opened a new pull request #1982: URL: https://github.com/apache/hadoop/pull/1982
Contributed by Steve Loughran. This patch adds to hadoop-common an API For querying IO classes (Especially input and output streams) for statistics. It includes a big rework of the S3A Statistics including * implementation of the IOStatistics APIs * and contract tests for those and any other streams which implement the same interfaces and the same bytes read/written counters. * A split of the existing S3AInstrumentation classes into interface/implementations. * Troubled attempt to wire up the AWSSDK metrics The AWS metric binding is breaking some of the S3 region handling code, so is disabled, we're still using the old "create client then set endpoint" logic rather than the builder API for constructing the S3 client. Doing the public interface hand-in-hand with that implementation helps evolve the interface, but it makes for a bigger patch. There are contract tests for those and any other streams which implement the same interfaces and the same bytes read/written counters. Proposed: once the reviewers are happy with the design we can split the two up into the hadoop-common changes (which can be used in ABFS) and the S3A FS changes. Writing up the package info class makes me think it is a bit overcomplicated right now and that maybe we should go for "dynamic always" statistics. If you want a snapshot, it can be wrapped with IOStatisticsSupport.takeSnapshot(). This will simplify the client code and remove ambiguity in implementations as to what they should be doing. We either provide callbacks to evaluate values or references to AtomicLongs/AtomicIntegers which are probed on demand. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
