GitHub user 10110346 opened a pull request:
https://github.com/apache/spark/pull/22065
[SPARK-23992][CORE] ShuffleDependency does not need to be deserialized
every time
In the same stage, 'ShuffleDependency' is not necessary to be deserialized
each time.
I have tested 3 times in my production environment , it has a bit of
performance improvement( about 0.7%):
Before this PRï¼
duration: 20189(s)
App count: 4736
the sum of time for all Apps: 530470(s)
Afterthis PRï¼
duration: 20035(s)
the count of Apps : 4736
the sum of time for all Apps: 525220(s)
How was this patch tested?
Existing unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/10110346/spark notdeserializedep22
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22065.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22065
----
commit a99769dd1aac779e972ed2e23aa7598e6d7c7105
Author: liuxian <liu.xian3@...>
Date: 2018-08-10T04:20:54Z
fix
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
