Re: How to measure IO time in Spark over S3
s. Needs a committer which doesn't do renames. The 1.6 DirectOutputCommitter did, but it couldn't handle failure & retry. Future ones will. -Steve From: Gili Nachum Sent: 13 February 2017 06:55 To: user@spark.apache.org Subject: How to measure IO ti
How to measure IO time in Spark over S3
Hi! How can I tell IO duration for a Spark application doing R/W from S3 (using S3 as a filesystem sc.textFile("s3a://...")? I would like to know the % of time doing IO of the overall app execution time. Gili.