[
https://issues.apache.org/jira/browse/SPARK-26746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-26746:
---------------------------------
Description:
When we turn on the {{spark.sql.adaptive.enabled}} switch, the following
actions trigger the shuffle calculation, but not when the switch is off:
{code}
sql("select a, sum(a) from test group by a").rdd
{code}
The reason is _'_ExchangeCoordinator' submitMapStage too early, the code is
like this:
{code}
while (j < submittedStageFutures.length) {
// This call is a blocking call. If the stage has not finished, we will wait
at here.
mapOutputStatistics(j) = submittedStageFutures(j).get()
j += 1
}
{code}
was:
When we turn on the spark. sql. adaptive. enabled switch, the following actions
trigger the shuffle calculation, but not when the switch is off:
sql("select a, sum(a) from test group by a").rdd
The reason is _'_ExchangeCoordinator' submitMapStage too early, the code is
like this:
while (j < submittedStageFutures.length) {
// This call is a blocking call. If the stage has not finished, we will wait
at here.
mapOutputStatistics(j) = submittedStageFutures(j).get()
j += 1
}
> Adaptive causes non-action operations to trigger computation
> ------------------------------------------------------------
>
> Key: SPARK-26746
> URL: https://issues.apache.org/jira/browse/SPARK-26746
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.1, 2.3.2, 2.4.0
> Reporter: eaton
> Priority: Major
>
> When we turn on the {{spark.sql.adaptive.enabled}} switch, the following
> actions trigger the shuffle calculation, but not when the switch is off:
> {code}
> sql("select a, sum(a) from test group by a").rdd
> {code}
> The reason is _'_ExchangeCoordinator' submitMapStage too early, the code is
> like this:
> {code}
> while (j < submittedStageFutures.length) {
> // This call is a blocking call. If the stage has not finished, we will wait
> at here.
> mapOutputStatistics(j) = submittedStageFutures(j).get()
> j += 1
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
