[
https://issues.apache.org/jira/browse/HADOOP-14973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222135#comment-16222135
]
Steve Loughran commented on HADOOP-14973:
-----------------------------------------
s3a should not be be doing anything special w.r.t logging stats given we have
metrics2 for that. It adds non standard metrics & documentation, doesn't hook
up with other systems and won't be directly manageable through tools.
By the time you are looking at logs, you are into debugging. What you want is
alerts based on published data (e.g. throttle rate), which again says "metrics2"
Why not automate the publishing of all fs client storage stats by adding some
heartbeat publishing of metrics, and then users get to set up metrics2 to log
if that is how they want the stats published.
+ [~aw] for his insight into metrics.
> [s3a] Log StorageStatistics
> ---------------------------
>
> Key: HADOOP-14973
> URL: https://issues.apache.org/jira/browse/HADOOP-14973
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0-beta1, 2.8.1
> Reporter: Sean Mackrory
> Assignee: Sean Mackrory
>
> S3A is currently storing much more detailed metrics via StorageStatistics
> than are logged in a MapReduce job. Eventually, it would be nice to get
> Spark, MapReduce and other workloads to retrieve and store these metrics, but
> it may be some time before they all do that. I'd like to consider having S3A
> publish the metrics itself in some form. This is tricky, as S3A has no daemon
> but lives inside various other processes.
> Perhaps writing to a log file at some configurable interval and on close()
> would be the best we could do. Other ideas would be welcome.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
