GitHub user wujianping10043419 opened a pull request:
https://github.com/apache/spark/pull/20526
[SPARK-23349][SQL]ShuffleExchangeExec: Duplicate and redundant type
determination for ShuffleManager Object
## What changes were proposed in this pull request?
org.apache.spark.sql.execution.exchange.ShuffleExchangeExec:
There is a nested "if or else" branch within the
"needtocopyobjectsbeforguffle()" function.
The <sortBasedShufffleOn> condition in the first layer "if" has the same
value as the <bypassIsSupported> condition in the second layer "if", that is,
<bypassIsSupported> must be true when <sortBasedShufffleOn> is true.
In addition, the <byPassMergeThreshold> condition will be used in the
second layer "if" and should not be calculated until needed.
It's to remove the <bypassIsSupported> condition in the second layer " if"
and to move the <bypassmergethreshold> calculation backward
## How was this patch tested?
Existing tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wujianping10043419/spark-1 wujp-0107
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20526.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20526
----
commit 1c8b73ad9a662ca44f0bc2553b8c43130ab632d4
Author: wujianping10043419 <wu.jianping@...>
Date: 2018-02-07T08:04:07Z
It's to remove the bypassmergethreshold condition in the second layer " if"
and to move the <bypassmergethreshold> calculation backward
Signed-off-by: wujianping10043419 <[email protected]>
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
