chenliang created SPARK-26542:
---------------------------------
Summary: Support the coordinator to demerminte post-shuffle
partitions more reasonably
Key: SPARK-26542
URL: https://issues.apache.org/jira/browse/SPARK-26542
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.3.2, 2.3.1, 2.3.0, 2.2.2, 2.2.1, 2.2.0
Reporter: chenliang
Fix For: 2.3.0
For SparkSQL ,when we open AE by 'set spark.sql.adapative.enable=true'，the
ExchangeCoordinator will introduced to determine the number of post-shuffle
partitions. But in some certain conditions,the coordinator performed not very
well, there are always some tasks retained and they worked with Shuffle Read
Size / Records 0.0B/0 ,We could increase the
spark.sql.adaptive.shuffle.targetPostShuffleInputSize to solve this,but this
action is unreasonable as targetPostShuffleInputSize Should not be set too
large.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
