Steve Loughran commented on SPARK-9004:

HADOOP-13605 added a whole new set of counters for HDFS, S3 and hopefully soon 
Azure; there's an API call on the FS {{getStorageStatistics()}} to query these.

One problem though: this isn't shipping in Hadoop branch-2 yet, so you can't 
write code that uses it, not unless there's some introspection/plugin 

All the stats are just {{name: String -> value: Long}}, so a something to 
collect a {{Map[String, Long]}} would work. 

> Add s3 bytes read/written metrics
> ---------------------------------
>                 Key: SPARK-9004
>                 URL: https://issues.apache.org/jira/browse/SPARK-9004
>             Project: Spark
>          Issue Type: Improvement
>          Components: Input/Output
>            Reporter: Abhishek Modi
>            Priority: Minor
> s3 read/write metrics can be pretty useful in finding the total aggregate 
> data processed

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to