steveloughran commented on issue #1820: HADOOP-16830. Add public IOStatistics API + S3A implementation URL: https://github.com/apache/hadoop/pull/1820#issuecomment-579385297 HADOOP-16830. Evolution of S3AStatisticsContext This is still at a -is this the right design?- stage. New class ActiveOperationContext. This is meant to be superclass of all operation context's, supplanting the S3AOpContext. I've also made WriteOperationHelper a subclass of it -which is one thing I'm not so sure of. Better as a constructor param. Almost all production references to the S3AInstrumentation class now refer to a much more minimal S3AStatisticsContext. The goal there is to have one which increments op-specific counters as well as the FS counters/metrics; for now there's a relay to S3AInstrumentation, and an empty one for testing/no-ops. Sticking this up while I do other things briefly, its here to show my direction of change. Regarding the IOStats API, I want to move off always returning something to making it Optional<>; that way callers can see its not there, but don't need to play is !null calls before acting. As usual, I have mixed feelings there. Next Steps * IOStats again * Make WriteOperationHelper something we can construct with the stats, so all ops which go through it pass the stats back to the caller. * Pass in to the AbstractStoreOperation constructor * and have S3AFileSystem.OperationCallbacksImpl do the same. These will both be instantiated for each operation, so can keep the stats context around without having to force the *Operation classes to add it everywhere We inevitably will need to pass it around as an argument to *every single innerXYX and other internal operation in S3AFS*. Oops. It is critical, therefore, that ActiveOperationContext is something we don't find we need to replace in future; something we can just extend with things like * trace span info * custom UA header * custom AWS auth chain A WriteOperationContext will be a subclass with * enum about parent dir state to enable us to skip empty dir marking when we knew it wasn't needed * any actions to do in finishedWrite ... ## Testing. All the intermittently flaky tests failed for me. I'm going to do an aggregate "fix the flaky's" patch
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
