GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/23207
[SPARK-26193][SQL] Implement shuffle write metrics in SQL
## What changes were proposed in this pull request?
1. Implement `SQLShuffleWriteMetricsReporter` on the SQL side as the
customized `ShuffleWriteMetricsReporter`.
2. Add shuffle write metrics to `ShuffleExchangeExec`, and use these
metrics to create corresponding `SQLShuffleWriteMetricsReporter` in shuffle
dependency.
3. Expand current `ShuffleWriteMetrics` in context as a proxy, register the
shuffle write metrics reporter to it during ShuffleMapTask is created on
executor.
## How was this patch tested?
Add UT in SQLMetricsSuite.
Manually test locally, update screen shot to document attached in JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuanyuanking/spark SPARK-26193
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23207.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23207
----
commit 6b26c629439973045da77f7bcd4b852afe8ebd8b
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-12-02T12:19:55Z
Commit for fist time success
commit a8a1225837419c99a3d9941046a2ca6b501f6dc8
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-12-03T12:06:34Z
Simplify implement by add metrics in ShuffleExchangeExec
commit 7c8e5161904f1fd0fa4d99e6c497ef1be3542bdb
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-12-03T12:40:41Z
code clean and comments
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
