[ https://issues.apache.org/jira/browse/SPARK-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432462#comment-15432462 ]
Steve Loughran commented on SPARK-9004: --------------------------------------- If you know the filesystem, you can get summary stats from {{FileSystem.getStatistics()}}; they'd have to be collected across all the executors These counters are per-JVM, not isolated into individual jobs > Add s3 bytes read/written metrics > --------------------------------- > > Key: SPARK-9004 > URL: https://issues.apache.org/jira/browse/SPARK-9004 > Project: Spark > Issue Type: Improvement > Components: Input/Output > Reporter: Abhishek Modi > Priority: Minor > > s3 read/write metrics can be pretty useful in finding the total aggregate > data processed -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org