GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/12600
[SPARK-14791] [SQL] fix risk condition between broadcast and subquery
## What changes were proposed in this pull request?
SparkPlan.prepare() could be called in different threads (BroadcastExchange
will call it in a thread pool), it only make sure that doPrepare() will only be
called once, the second call to prepare() may return earlier before all the
children had finished prepare(). Then some operator may call doProduce() before
prepareSubqueries(), `null` will be used as the result of subquery, which is
wrong. This cause TPCDS Q23B returns wrong answer sometimes.
This PR added synchronization for prepare(), make sure all the children had
finished prepare() before return. Also call prepare() in produce() (similar to
execute()).
Added checking for ScalarSubquery to make sure that the subquery has
finished before using the result.
## How was this patch tested?
Manually tested with Q23B, no wrong answer anymore.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark fix_risk
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12600.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12600
----
commit e5bd221eccee7d0194c691519d65c3a88c168578
Author: Davies Liu <[email protected]>
Date: 2016-04-22T04:52:36Z
fix risk condition between broadcast and subquery
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
