Kostas Sakellis created SPARK-5645: -------------------------------------- Summary: Track local bytes read for shuffles - update UI Key: SPARK-5645 URL: https://issues.apache.org/jira/browse/SPARK-5645 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Kostas Sakellis
Currently we do not track the local bytes read for a shuffle read. The UI only shows the remote bytes read. This is pretty confusing to the user because: 1) In local mode all shuffle reads are local 2) the shuffle bytes written from the previous stage might not add up if there are some bytes that are read locally on the shuffle read side 3) With https://github.com/apache/spark/pull/4067 we display the total number of records so that won't line up with only showing the remote bytes read. I propose we track the remote and local bytes read separately. In the UI show the total bytes read and in brackets show the remote bytes read for a shuffle. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org