[
https://issues.apache.org/jira/browse/SPARK-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432462#comment-15432462
]
Steve Loughran commented on SPARK-9004:
---------------------------------------
If you know the filesystem, you can get summary stats from
{{FileSystem.getStatistics()}}; they'd have to be collected across all the
executors
These counters are per-JVM, not isolated into individual jobs
> Add s3 bytes read/written metrics
> ---------------------------------
>
> Key: SPARK-9004
> URL: https://issues.apache.org/jira/browse/SPARK-9004
> Project: Spark
> Issue Type: Improvement
> Components: Input/Output
> Reporter: Abhishek Modi
> Priority: Minor
>
> s3 read/write metrics can be pretty useful in finding the total aggregate
> data processed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
